Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor

The Allegheny County Office of Children, Youth and Families (CYF) in Pennsylvania uses a judgment protocol called the Key Information and Demographics System (KIDS) to process social services requests. The aim of the KIDS system is to identify which indicators for kids in vulnerable situations are connected to child abuse. However, “Many struggles common among poor families are officially defined as child maltreatment, including not having enough food, having inadequate or unsafe housing, lacking medical care, or leaving a child alone while you work” (130).

In the late 1990s, when Marc Cherna took over CYF, “Seventy percent of children in the foster care system were Black, though African Americans made up only 11 percent of the population of Allegheny County” (134). There was huge racial bias inherent in the system. Cherna’s first priority was a comprehensive data warehouse, meant to “increase agency communication and accountability, provide wraparound services for clients, and cut costs” (135). Again, as with Governor Daniels in Indiana, the aim was to funnel public money to private business: In 2012, the state offered a million-dollar contract to build an automated triage data system. The winning proposal by New Zealand economist Rhema Vaithianathan was “a predictive model using 132 variables” which gave results with only “fair, approaching good” accuracy (137). Despite the fact that academic researchers in New Zealand warned that the model (also in use in that country) “was wrong about nearly 70 percent of the children it identified as at highest risk of harm in the historical data” (138) and neglected to account for inherent bias against Maori families and the poor, Pennsylvania awarded the contract anyway.

The new predictive risk tool was called the Allegheny Family Screen Tool (AFST). It was supposed to partner with human call screeners, supporting “human decision-making in the call center. And yet, in practice, the algorithm seems to be training the intake workers” (142). Human decision-making elements are visible in three components of the AFST: outcome variables, predictive variables, and validation data.

Outcome variables measure the phenomenon they are trying to predict. In this case, these variables were the number of children initially screened out but later screened again, and the number of children initially screened in and later placed in foster care. Predictive variables are data that correlate with the outcome variables. Finally, validation data is universally used to see how well a model performs. AFST turned out to only be 76% accurate, “about the same accuracy as a yearly mammogram” (145), raising questions about its data sources. The reason that data was biased and skewed was immediately apparent: “Allegheny County has an extraordinary amount of information about the use of public programs stored in its data warehouse. But the county has no access to data about people who do not use public services” (147). By never comparing the poor to the non-poor, dataset makers were unable to teach their algorithm which behaviors were normal throughout the population, thus perpetuating disparities. As a result, as of 2016, “African American children are more than two and a half times as likely to end up in foster care than they should be, given their proportion of the population” (153). A study of Allegheny County CYF found that the racial disparity exposed by the data was actually from a misapplication of data restraints, and not a true indication of racial reality.

“Poverty profiling” is the assessment of poor people’s morals according to their spending habits and data about “TANF, Supplemental Security Income, SNAP, and county medical assistance” (156). By contrast, information from private sources used by the non-poor, like nannies, therapists, and rehabilitation centers, could also potentially predict child abuse in the population above the poverty line, but this data is protected by privacy laws. Eubanks concludes that the poor have no privacy, and their automatic monitoring is akin to being in prison. A source familiar with the CYF system seconds that opinion, saying that the expungement process is more complex than for a criminal record.