“You can’t cross-examine an algorithm.” —Michele Gilman
What Is the Automated State?
In replacing human workers with machines, the automated state seeks to handle rote and routine work more efficiently. Automation is also aimed at improving the accuracy, fairness, or consistency of decision-making. By removing such decision-making from human discretion, the automated state seeks to depoliticize it (or at least appear to).
But these motivations are anything but neutral: they raise urgent concerns about implementation, oversight, and accountability. What does automation look like in practice? Who and what gets overlooked? What does interacting with systems, interfaces, and datasets require of people, including skills and resources they may not have? Does automation leave room for individual voices? What are the implications for participatory democracy, and for people’s willingness and ability to trust systems of all kinds?