Reframing AI Explainability

The opacity and inaccessibility of algorithmic systems are fundamental challenges in contemporary society, explored in a range of in-depth sociological analyses over recent years. In response, ‘explainable AI’ is widely proposed, providing technical explanations of data and algorithms to enable critical scrutiny, debate and intervention. However, our starting point in this paper is that technical explanation itself is contingent, dependent on what is deemed to be in need of explanation (and what isn’t), the decisions made in assembling technical explanation, and a wider ecology of social explanations for algorithmic decision-making that are in circulation in any particular instance. In this paper, we draw on new empirical research to explore how explainability is crafted and operates in two different organizations, a credit agency (in Germany) and an insurance company (in the UK). Based on ethnographic research, we argue that a) developers inscribe certain ideas of explainability into the models to mediate interactions with a variety of stakeholders, anticipating needs and managing expectations of what AI is and how it works; b) that explainability is dynamic and multiple, reproduced as it comes into dialogue with other practices of explanation in everyday work; and c) that ethnographic research on explainability may also become an actor in the space it seeks to describe. Reframing explainable AI in this way underscores the importance of sociological voices in shaping as well as critiquing emerging AI practice.