I’m just back from Beatrice Fazi’s seminar on ‘Deep Learning, Explainability and Representation.’ This was a fascinating account of opacity in deep learning processes, grounded in the philosophy of science but also ranging further afield.
Beatrice brought great clarity to a topic which — being implicated with the limits of human intelligibility — is by its nature pretty tough-going. The seminar talk represented work-in-progress building on her recently published book, Contingent Computation: Abstraction, Experience, and Indeterminacy in Computational Aesthetics, exploring the nature of thought and representation.
I won’t try to summarise the shape of the talk, but I’ll briefly pick up on two of the major themes (as advertised by the title), and then go off on my own quick tangent.
First, representation. Or more specifically, abstraction (from, I learned, the Latin abstrahere, ‘to draw away’). Beatrice persuasively distinguished between human and deep learning modes of abstraction. Models abstracted by deep learning, organised solely according to predictive accuracy, may be completely uninterested in representation and explanation. Such models are not exactly simplifications, since they may end up as big and detailed as the problems they account for. Such machinic abstraction is quasi-autonomous, in the sense that it produces representational concepts independent of the phenomenology of programmers and users, and without any shared nomenclature. In fact, even terms like ‘representational concept’ or ‘nomenclature’ deserve to be challenged.
This brought to my mind the question: so how do we delimit abstraction? What do machines do that is not abstraction? If we observe a machine interacting with some entity in a way which involves receiving and manipulating data, what would we need to know to decide whether it is an abstractive operation? If there is a deep learning network absorbing some inputs, is whatever occurs in the first few layers necessarily ‘abstraction,’ or might we want to tag on some other conditions before calling it that? And is there non-representational abstraction? There could perhaps be both descriptive and normative approaches to these questions, as well as fairly domain-specific answers.
Incidentally, the distinction between machine and human abstraction also made me wonder if pattern-recognition actually belongs with terms such as generalization, simplification, reduction, and (perhaps!) conceptualization, and (perhaps even!) modelling, terms which pertain only in awkward and perhaps sort of metaphorical ways to machine abstraction. It also made me wonder how applicable other metaphors might be: rationalizing, performing, adapting, mocking up? Tidying? — like a machinic Marie Kondo, discarding data points that fail to spark joy?
The second theme was explanation. Beatrice explored the incommensurability between the abstractive operations of human and (some) machine cognition from a number of angles, including Jenna Burrell’s critical data studies work, ongoing experiments by DARPA, and the broader philosophical context of scientific explainability, such as Kuhn and Feyerabend’s influential clashes with conceptual conservativism. She offered translation as a broad paradigm for how human phenomenology might interface with zones of machinic opacity. However, to further specify appropriate translation techniques, and/or ways of teaching machine learning to speak a second language, we need to clarify what we want from explanation.
For example, we might want ways to better understand the impact of emerging machine learning applications on existing policy, ways to integrate machine abstractions into policy analysis and formation, to clarify lines of accountability which extend through deep learning processes, and to create legibility for (and of) human agents capable of bearing legal and ethical responsibility. These are all areas of obvious relevance to the Science Policy Research Unit, which hosted today’s seminar. But Beatrice Fazi’s project is at the same time fundamentally concerned with the ontologies and epistemologies which underlie translation, whether it is oriented to these desires or to others. One corollary of such an approach is that it will not reject in advance the possibility that (to repurpose Langdon Winner’s phrase) the black box of deep learning could be empty: it could contain nothing translateable at all.
For me, Beatrice’s account also sparked questions about how explanation could enable human agency, but could curtail human agency as well. Having something explained to you can be the beginning of something, but it can also be the end. How do we cope with this?
Might we want to mobilise various modes of posthumanism and critical humanism, to open up the black box of ‘the human’ as well? Might we want to think about who explanation is for, where in our own socio-economic layers explanation could insert itself, and what agencies it could exert from there? Think about how making automated processes transparent might sometimes place them beyond dispute, in ways which their opaque predecessors were not? Think about how to design institutions which — by mediating, distributing, and structuring it — make machinic abstraction more hospitable for human being, in ways relatively independent of its transparency or opacity to individual humans? Think about how to encourage a plurality of legitimate explanations, and to cultivate an agonistic politics in their interplay and rivalry?
Might we want to think about distinguishing explainable AI from splainable AI? The word mansplain has been around for about ten years. Rebecca Solnit’s ‘Men Explain Things To Me‘ (2008), an essay that actually intersects with many of Rebecca Solnit’s interests and which she is probably recommended at parties, doesn’t use the word, but it does seem to have inspired it.
Splain has splayed a little, and nowadays a watered-down version might apply to any kind of pompous or unwelcome speech, gendered or not. However, just for now, one way to specify splaining might be: overconfident, one-way communication which relies on and enacts privilege, which does not invite the listener as co-narrator, nor even monitor via backchannels the listener’s ongoing consent. Obviously splained content is also often inaccurate, condescending, dull, draining, ominously interminable, and even dangerous, but I think these are implications of violating consent, rather than essential features of splaining: in principle someone could tell you something (a) that is true, (b) that you didn’t already know, (c) that you actually care about, (d) that doesn’t sicken or weary you, (e) that doesn’t impose on your time, and wraps up about when you predict, (f) that is harmless … and you could still sense that you’ve been splained, because there is no way this bloke could have known (a), (b), (c), (d), (e), and/or (f).
“Overconfident” could maybe be glossed a bit more: it’s not so much a state of mind as a rejection of the listener’s capacity to evaluate; a practiced splainer can even splain their own confusion, doubt, and forgetfulness, so long as they are acrobatically incurious about the listener’s independent perspective. So overconfidence makes possible the minimalist splain (“That’s a Renoir,” “You press this button”), but it also goes hand-in-hand with the impervious, juggernaut quality of longform splaining.
Splainable AI, by analogy, would be translatable into human phenomenology, without human phenomenology being translatable into it. AI which splains itself might well root us to the spot, encourage us to doubt ourselves, and insist we sift through vast swathes of noise for scraps of signal, and at a systemic level, devalue our experience, our expertise, and our credibility in bearing witness. I’m not really sure how it would do this or what form it would take: perhaps, by analogy with big data, big abductive reasoning? I.e. you can follow every step perfectly, there are just so many steps? Splainable AI might also give rise to new tactics of subversion, resistance, solidarity. Also, although I say ‘we’ and ‘us,’ there is every reason to suppose that splainable AI would exacerbate gendered and other injustices.
It is interesting, for example, that DARPA mention “trust” among the reasons they are researching explainable artificial intelligence. There is a little link here with another SHL-related project, Automation Anxiety. When AIs work within teams of humans, okay, the AI might be volatile and difficult to explain, evaluate, debug, veto, learn from, steer to alternative strategies, etc. … but the same is true of the humans. The same is particularly true of the humans if they have divergent and erratic expectations about their automated team-mates. In other words, rendering machine learning explainable is not only useful for co-ordinating the interactions of humans with machines, but also useful for co-ordinating the interactions of humans with humans in proximity to machines. Uh-oh. For that purpose, there only needs to be a consistent and credible, or perhaps even incontrovertible, channel of information about what the AI is doing. It does not need to be true. And in fact, a cheap way to accomplish such incontrovertibility is to make such a channel one-way, to reject its human collaborators as co-narrators. Maybe, after all, getting splAIned will do the trick.
JLW
Earlier: Messy Notes from the Messy Edge.
Sussex Humanities Lab Blog 