CPD Using Opaque Algorithm to Compile List of Citizens by Risk of Violence
Posted on June 15, 2017
In a recent story by The New York Times, reporters attempt to probe into the operation of an opaque algorithm which the Chicago Police Department uses to direct law enforcement resources by creating and maintaining a list of citizens at risk for perpetrating or falling victim to violent crime. Although CPD has released a sample dataset from the list, known as the “Strategic Subject List” or SSL, with identifying information removed to protect the privacy of listed individuals, the department has declined to reveal the source code for public auditing, prompting Chicago-area journalists to file a Freedom of Information Act request to obtain the source code.
CPD has made public assurances that the algorithm does not utilize or process demographic information which could create discriminatory bias, such as gender identity, race, or location. However, without an independent analysis of publicly available source code for SSL, these claims of precaution against discriminatory practices cannot be verified. Moreover, no matter what criteria or data points are fed into the algorithm to produce the list–CPD states that it uses criminal records and other crime data on how many incidents of violent crime an individual has been involved in, as perpetrator or victim–any evaluative technique or program which places citizens on a list which subjects them to more concentrated law enforcement action, and which is unavailable for public review, is inherently troubling.
While a full assessment of the algorithm is not possible absent its source code, the reporters on the New York Times article compared the statistics submitted in CPD’s disclosed SSL sample data against the program’s risk scores to reverse engineer how data points are weighted. This reverse engineering process illuminated flaws in the algorithm, such as the egregiously minuscule impact that gang membership has on risk scores, that call its accuracy into question.
A common misconception is that software whose developers openly publish the source code, commonly known as “open-source software,” is more susceptible to breach or circumvention. On the contrary, by making source code available to the public, the number of professional developers and security auditors who are able to review the program increases dramatically. Another myth is that if an evaluative program like SSL is publicly disclosed for review, people will be empowered to “game” the system, but if an algorithm is written well enough–which, again, is made easier by opening it to more potential reviews–no amount of knowledge of its inner workings should leave room to circumvent its assessments. Case in point, some of the most secure encryption programs, such as (coincidentally) the openSSL encryption suite, are open-source. If CPD officers and resources are to be deployed in accordance with the evaluations the SSL algorithm assigns to people, it is imperative that the public has access to its source code so that the program can be properly subject to review through the democratic process.
You can read the full piece from The New York Times here.
Jonathan Terrasi has been a Research Assistant with the Chicago Committee to Defend the Bill of Rights since January 2017. His interests include computer security, encryption, history, and philosophy. In his writing, he regularly covers topics on current affairs and political developments, as well as technical analyses and guides on security issues, published on his blog, Cymatic Scanning, and Linux Insider.
