The algorithm itself is neither ethical, nor unethical. It is the human who created it, who made more or less ethical decisions. So he’s the one responsible. Maciej Chojnowski talks with Przemysław Biecek, professor in Human-Centered Artificial Intelligence at Warsaw University of Technology
Maciej Chojnowski: During your lectures, you often recall Cathy O’Neil’s book “Weapons of Math Destruction.” Why does an expert in artificial intelligence focus on the dangers of artificial intelligence rather than the benefits?
Przemysław Biecek*: It is not true that artificial intelligence researchers are automatically uncritical advocates of AI. What motivates me – and I think many others – is the opportunity to build accurate, trusted and fair models. For models to be accurate in a long run, they need to undergo a detailed critical analysis.
Cathy O’Neil’s book shows a series of failures resulting from the deployment of untested machine learning models. For me, it was a very interesting reading, which made me realize that when building machine learning models, for years we didn’t notice certain threats, risks and dangers. This is a very valuable lesson: we should be more creative in predicting and anticipating the potential negative consequences of what we create.
In the book, there are many examples and unfortunate cases when algorithms are not working the way they should and someone got hurt as a result. What is the key issue described there?
For those who develop models, it is obvious that by building a model we make a compromise. The compromise between how many decisions we want to automatize or speed up, and the number of mistakes we agree on. those who are not working in the field may find the consent for mistakes surprising. Users of artificial intelligence systems often assume that the machine will be reliable or at least less unreliable than a human. But mistakes do happen. In the process of building a model, we estimate number of these errors, and manage the level of accuracy. O’Neil describes cases where model decisions were taken uncritically and the errors were very severe.
When can a mistake be accepted in favor of doing something much faster?
For example when we use a predictive model to recommend online ads. If mistakes do not occur systematically, then even a large number of them will be acceptable. Poorly matched ads generate a certain loss for the advertiser, but usually not a significant one. And it is compensated by a fraction of effective ads.
This is different in the case of health, life or, for example, employment. Here, the social costs of wrong decisions are higher. Knowing how many mistakes can be expected, we can (but we do not have to) accept certain risks.
It’s not true that artificial intelligence researchers are automatically uncritical advocates of AI
The problem arises when someone assumes that the system must be flawless. This creates a gap between users’ expectations that the system will always be right and the awareness of researchers who design tools and know their imperfections.
What is the source of exorbitant expectations for artificial intelligence?
I think it’s down to people and their laziness. A lot of people would like someone else to make a decision for them, and that’s still the best for them. As long as these decisions are favourable, no one questions them. But when they are not, complaining starts: I thought it would work differently.
Besides, it is easy to get overwhelmed by every new technology, choke on it. We used to fantasize about intergalactic space flight, now we fantasize about artificial intelligence. We would like to believe that a breakthrough is around the corner.
Some companies are taking advantage of our faith in a better future, promising that soon everything will be done automatically, fast, cheaply, and efficiently. There is giant progress, of course, in the field of process automation , but where there’s a lot of money, there’s a temptation to exaggerate that progress to win new customers. I have the impression that promises are now growing faster than the actual pace of AI development.
You work on the so-called “explainable AI.” How does it work?
The research I carry out is designed to determine when and how we can explain the reasons why a particular decision was made by a statistical or machine learning model.
There are many reasons why we need such explanations. As a model developer, I need to understand the reasons for their decisions so that I can correct them when they are wrong. For example, I have a model that predicts the price of an apartment and I see that the model does not work well for specific apartments, e.g. it undervalues their price. So I need to understand the factors that make it work so that I can calibrate it later.
We should be more creative in anticipating the potentially negative consequences of what we create.
An enduser of a model has a different perspective. An apartment seller wants to know what can change to increase the price of the apartment. Does it pay to swap the old windows for new ones? Does it pay to refurbish the apartment?
The structure of some models are easy to analyze, they are called interpretable or transparent models. But complex models, which are becoming more and more popular, are also much more difficult to analyse. In the MI2 research team, we create the DrWhy.AI toolkit, which helps to analyse even the most complex cases.
Are the solutions you work on universal?
The tools that my team is developing are most often used in financial and medical applications. Every area of application has different needs regarding the explainability. You cannot use a single tool that opens all the black boxes. It is impossible, because these boxes are complex, each one in its own way. A tool that is great for X-rays images, might not be so great in explaining credit decisions.
But although the individual tools for explanation are not universal, they are governed by similar laws. This is why we want to understand better these laws. We need a language in which a human can effectively communicate with statistical models. For us, visual language is the first choice. How to visualize a model, how to show its strong and weak points, the most important factors influencing its decisions, how to keep up interaction with the model?
On the basis of these data visualization techniques, we create new techniques for visualizing statistical models. But the language for communicating with the models does not have to be visual, it can be based on other forms of communication.
How is the language that mentioned supposed to work?
We won’t know in detail how the deep neural network works, because there are millions of parameters to be taken into account and each of them can influence the final decision. But even for such complex models, we can get some answers to questions like: “How would the model prediction change, if there was a different input data?” or “What would have to happen for the model prediction to change to the opposite one.”
When we reduce the explainability of the models to a certain number of elementary questions and for each question we will be able to construct explanations, then we will be able to explore any model: a neural network, millions of decision trees, or anything else.
Let’s imagine a complex AI evaluating a credit application. You cannot do it in the banks at the moment, but let’s suppose it happens after all. A black box model, for whatever reason, gives someone a credit – or not. To understand how the model works, I can ask, what do I need to change in my behaviour to change the model decision. If I decrease the number of credit cards, then my chance to get credit increases, or decreases? And what if I had life insurance?
I don’t need to fully understand all the components of a particular black box decision. I just need to understand how I can change my situation. And that’s much easier to explain.
How would it look like in practice?
There are many possible implementations of model explanations. Personally, I believe most in three approaches.
One, developed by Michał Kuźba, a student at the Faculty of Mathematics, Informatics and Mechanics, University of Warsaw, is a xai-bot. Xai-bot is a dialogue system for explaining the model decisions. The created prototype answers questions related to hypothetical chances of surviving the sinking of Titanic. For example, someone says he is a man of a certain age and what chances would he have to survive the disaster, what would happen if he was someone older, or who had bought a ticket in a different harbor, whether he traveled in a different class. In order to determine the survival rate, a so-called “random forest” model is used which has many parameters. But even watching them all, it is impossible to guess how the model is going to behave. By interacting with it, we can build some intuition about this behaviour.
In software development We used to pay a lot of attention to communication between human and computer. What we did for interaction with computers, we now have to do for interaction with artificial intelligence models.
A bit like when playing with a dog: we see it brings a ball, so we assume, that it likes it and that is why it does it. We do not know the processes taking place in the dog’s brain, but based on the interaction we build an intuition about its behaviour. The same principle applies to machine learning. If after another interaction everything happens as we expected, our confidence in the system will increase.
There are other approaches apart from a chatbot.
Yes, our team has developed many prototype tools that explain some specific aspect of the AI. For example, in the case of credit application rejection, the tool may suggest that the deciding factor was the number of credit cards. Or in a medical application, that the refusal of a medical treatment was determined by the fact that someone is diabetic.
After all, it turns out that the most efficient tools are the ones that show the model from various perspectives. Just as with the visualization of data. One chart says a lot, but if we juxtapose it with others, then we can look at a particular problem from various perspectives and we will better understand the whole.
An example?
An ideal example is the modelStudio, a tool developed by Hubert Baniecki, a student at the Faculty of Mathematics and Information Science, Warsaw University of Technology. It allows for a multi-aspect analysis of decisions of the predictive model. The prototype explains the predictions of footballers worth. Using the data on thousands of players, the model learned what characteristics (e.g., controlling the ball, shooting efficiency, speed) have the strongest influence on the players worth. The tool allows for multiple perspectives, thanks to which we understand the decisions of the system better, but we can also be more critical of them.
This approach assumes that better understanding is when the problem is exposed from many perspectives. Today, when someone thinks about artificial intelligence deployed in a doctor’s office, they can imagine one model which recommends something to a doctor. Would it not be safer to have several models competing with each other that need to present their arguments in order to convince the doctor? In particularly difficult cases, a decision should result from such a discussion. A singular point of view will not make it happen.
And the third approach?
It is about showing the same content at the same time in different channels, e.g. in the form of both a graph and a text. In our team, we often work with physicians on medical data. For them, the visual language, meaning charts, is not natural and even causes apprehension: “Did I notice everything that was important on the chart?” “I was looking to the right, and maybe I also had to look to the left?.” The textual explanation has an advantage: we know where it begins and where it ends. It can be analyzed word-by-word, and assessed if everything is understandable. For some audiences such a message can be safer and more effective than the visual language.
There are many possibilities of experimenting at the interface between human and model communication. Our ambitions are big, because we want to have effective tools for everybody. For model developers who create predictive models, i.e. highly skilled in understanding the mechanisms of machine learning models, and for people who know nothing about AI but are interested in the final decision.
Who will operate the tools for explainable AI? A specialist in a given field, or a client?
Everyone should be able to do so. In her book, O’Neil points out that at the level of model-development, the whole layer of explanation of model decisions to the human operator has been overlooked. And the human aspect should be considered at every step of the construction of AI system.
Explanation, meaning communication?
Yes. The interface between a model and a human is important as well. When I graduated in software engineering at Wroclaw University of Technology several years ago, we paid a lot of attention to communication between human and computer. What we once did for computers, we now have to do for artificial intelligence models. And it is not only about interface. Unit tests and debuggers are tools that allow programmers to verify and fix their programs. Now we need similar tools that will help model developers to verify and fix machine learning models.
It is about the creation of an additional communication layer which would allow for the explanation of model operation to various users, the ones more and less advanced?
Yes, because we could not talk with models before. Today, if someone went to court and said that some model is unfair, it wouldn’t even be apparent, how this justice was to be measured. This is why we must create a language in which it can talk also about fairness of predictive models.
I have an impression that the promises are growing faster than the actual pace of AI development
The co-author of the most popular AI manual, Peter Norvig, believes that the explainability will work like a bit based on the rule of rationalization: “Black box, please explain, why did you do it this way?” – and the black box is going to work and will blurt out a calming explanation, although not necessarily compatible with the factual process,. Is it right?
I agree with this, and I also disagree at the same time. In some areas, it can be the case. However I do not believe that the main aim of the system for explaining black boxes is calming the human operator and creating an illusion that he understands something. Paradoxically, if an aim was given to the algorithms of machine learning, for them to increase the trust to its decisions, it may be easy to create a confidence-inspiring system at declaration level, but not necessarily working well.
However, if the system is going to support critical thinking and nitpicking, then such explanations can be really useful. Whether the system shows us a single solution and artificially imposes trust for the solution, or several solutions, and says: “Think all the time, you have to choose,” depends solely on its creators.
Explainable AI must then rely on reverse rationalization, because it does not need to show a single message. The explanation can involve two contradictory messages “for and against” which will require an active decision of a human. The level of required human attention can be minimized for typical cases. But we are also able to create a system which will force us to make a choice in extreme scenarios.
Explainable artificial intelligence can be seen as part of responsible AI. It covers not only ethics, but also legal regulations, codes of conduct, and the discourse pertaining to artificial intelligence. Is your work to be viewed in that context?
Definitely, yes. In the case of artificial intelligence, it is not like it builds itself – for now, anyway, and it probably is going to be the case for a long time. The people who develop it must be responsible for their work and must have high ethical standards and be responsible for their creations. We do not say that a hammer is ethical, unethical; a person that used it had made more or less ethical decisions. AI is a tool, when it comes to the question of ethics, I would not take away the responsibility from a human who created this tool, or uses it.
*Przemysław Biecek, professor in Human Centered Artificial Intelligence at the Warsaw University of Technology. His adventure in data analysis began with the software engineering studies and mathematical statistics at the Wrocław University of Science and Technology. He uses years of experience in data visualization and statistical modeling to build tools for explaining predictive models under the name DrWhy.AI. He established a research group MI2DataLab, spread between Faculty of Mathematics and Information Science, Warsaw University of Technology and Faculty of Mathematics, Informatics and Mechanics, University of Warsaw. Author of numerous science articles, as well as monographs on data analysis, data visualization and programming. In his free time, an enthusiast of the so-called data-literacy. An author of popular science books: Beta and Bit series. He enjoys travelling, board games and audiobooks.
Przeczytaj polską wersję tego tekstu TUTAJ