How Posos is committed to explainable AI

XAI tools and frameworks

• LIME: An Explainability Tool for AI

Among the various tools and frameworks developed to make complex models more understandable, one of the most known is LIME (Local Interpretable Model-agnostic Explanations). LIME, introduced in the paper "Why Should I Trust You?": Explaining the Predictions of Any Classifier”, has gained popularity as a versatile tool for explaining AI decisions across different types of models.

LIME is designed to provide explanations for the predictions of any machine learning model, regardless of its complexity, being particularly useful in cases where models act as 'black boxes'. It is model-agnostic, meaning it can be applied to a wide range of models, from simple linear classifiers to complex deep learning networks. This versatility makes it an invaluable tool in fields where understanding the reasoning behind AI decisions is crucial, such as finance, healthcare, and law. For example, in healthcare, LIME can be used to explain why a predictive model might diagnose a patient with a particular disease. This transparency is essential for doctors who need to understand the model's reasoning to make informed treatment decisions.

                                                   
In the finance sector, LIME can help explain credit scoring models, providing insights into why a loan application was approved or denied. This helps in building trust with customers and ensuring compliance with regulations.

LIME operates by generating local approximations of the model's decision boundary around a particular instance. Here’s a simplified explanation of the process:

1. Perturbation: LIME starts by creating a new dataset consisting of perturbed versions of the instance being explained. These perturbations involve altering some features while keeping others constant.
2. Prediction: The model makes predictions on this new dataset.
3. Weighting: LIME assigns weights to these new instances based on their similarity to the original instance.
4. Training an Interpretable Model: Using the weighted dataset, LIME trains a simple, interpretable model (such as a linear model or decision tree) that approximates the complex model’s behavior locally around the instance of interest.
5. Generating Explanations: The interpretable model is then used to provide an explanation of the complex model’s prediction, highlighting which features were most influential.

For example, consider a neural network used to classify images of animals, as shown in the image below. If the model incorrectly classifies a cat as a dog, LIME might show that certain features, like the shape of the ears or the texture of the fur, were key factors in the decision, helping users understand the source of the error.

LIME has had a significant impact on increasing the transparency of AI models. By providing clear, understandable explanations for individual predictions, LIME helps demystify complex models, making them more accessible to non-experts. This is particularly important in regulated industries, where users need to understand the decision-making process to ensure compliance and fairness. Moreover, LIME can be used to identify and mitigate biases in AI systems. For example, if a model consistently rates loan applications differently based on gender or race, LIME can highlight the problematic features, allowing developers to address these issues. This capability is crucial for building fair and ethical AI systems.

In summary, LIME is a powerful tool in the XAI toolkit, offering a practical and versatile solution for enhancing the transparency of AI models. By providing local, interpretable explanations, LIME helps understand, trust, and improve AI systems across a wide range of applications.

• Microsoft Responsible AI toolbox

Another popular XAI framework is Azure Machine Learning, presented by Microsoft. It supports tools for enabling developers and data scientists to implement algorithms which are safe, reliable, transparent, and fair. Similar to LIME, it generates human-understandable descriptions of model predictions and introduces, among others, the Error Analysis Dashboard. It helps by providing insights into how model failures are distributed across different data subsets. Error patterns in models are often complex, involving multiple features, making it difficult for developers to explore all possible combinations to identify critical failures. Errors are assessed through error heatmaps and binary trees. The latter automatically partitions benchmark data into interpretable subgroups with unusually high or low error rates, using input features to distinguish model errors from successes. Users can investigate each node, examining the error rate (intensity of red color), error coverage (fill rate of the node), and data representation (thickness of the incoming edge and total instances in the node).

• DICE

While the tools cited above focus on local and global explanations of different sets of features, DiCE, on the contrary, uses counterfactual analysis to determine what changes to a specific data point would alter the model's decision. These minimal changes are crucial because they highlight the most influential features and the smallest possible modifications needed to alter the outcome, making the explanation actionable and understandable. For example, the figure below shows minimal changes needed to alter the prediction of class 0 (low income) to class 1 (high income).

• Fairlearn

Fairlearn is another XAI tool that offers metrics and visualizations to help detect biases in model predictions. These metrics can reveal disparities in performance or outcomes across different demographic groups, such as race, gender, age, etc. Most importantly, after identifying biases, it offers solutions that can be integrated into machine learning pipelines. Fairlearn uses parity constraints to ensure that predictor behaviors are comparable across groups defined by sensitive features like race or gender. It offers two primary types of unfairness mitigation algorithms:

• Reduction algorithms reweight training datasets to generate retrained models that address disparities, allowing users to choose a model balancing accuracy and fairness based on their needs.
• Post-processing algorithms adjust predictions of existing classifiers without retraining. For instance, threshold optimization modifies decision thresholds to achieve fairness, providing a simple and flexible solution.

‍

Explainable AI in Healthcare: Posos’ approach

The aforementioned tools can enable users to get a better grip on why a deep learning model is making some predictions. But at Posos, we believe, like many, that this is not the only approach for explainable AI-based decision support tools. Namely, explainability can be provided through the way AI models are included in such tools, and overconfidence in wrong predictions can be addressed with auxiliary models that compute confidence scores.

‍

An incremental instead of an end-to-end approach

One of Posos’ central features is that a healthcare professional can scan a prescription with our app which will automatically detect all the prescribed drugs with their dosage. Then, they can provide some undesirable effects that the patient endures, and Posos will find which drug is more likely to provoke this side effect and suggest alternatives to it.

• Avoiding the 'Black Box' Problem with an Incremental Approach

We could have thrown the problem directly at an LLM like GPT-4 in an end-to-end manner, by providing the scan of the prescription and the endured side-effect, and then waited for an answer. But explaining the decision of such a model suffers greatly from the ‘Black box’ issue. While it is possible to ask an LLM for an explanation, there is no guarantee that this is the right explanation and that the model is not simply hallucinating.

Our incremental approach gives back control to the users. The first way to do that is to include them in several steps of the decision. For instance, Posos identifies the drug most likely to cause a particular side effect, but the physician retains the ability to choose a different option based on their expertise and knowledge. Additionally, this approach allows for real-time adjustments, such as when a patient reveals that they have not been taking certain prescribed medications.

• Breaking Down the Task into Smaller Models for Enhanced Explainability

More importantly, Posos uses deep learning models for solving smaller lower-stakes tasks that serve to the end goal of providing a safe alternative to a potentially harmful drug. Instead of making one single high-stake prediction which raises a huge ‘Black Box’ problem, we cut the black box into many smaller ones. In the case of scanning a prescription to find an alternative drug, here are three key examples of models that come into play:

• An Optical Character Recognition (OCR) model converts the scan into text.
• A model based on a language model (LM) determines whether each line of text is medical information or not, discarding the prescriber's or patient's information.
• A named entity recognition model (NER) extracts important entities like drugs, doses, drug forms, quantities, or frequency of intake.

The prediction of those models can be easily checked and corrected by the user if needed. While this is not strictly an XAI approach, breaking down the task into smaller, verifiable subtasks enhances the explainability and security of the final prediction.

‍

A reassurance use case: posology structuration confidence score

Another important and often overlooked way to improve interpretability is to provide a confidence score. While most deep learning models already return a probability measure of their prediction, this likelihood is known to be unreliable. Due to their training, deep learning models often produce probabilities that are very close to discrete values. This can give the impression that the model is highly confident in its predictions, even when those predictions are incorrect.

• Addressing Overconfidence with Auxiliary Models

To address this issue of ‘overconfidence’, one solution is to use an auxiliary model to compute the confidence score. Using an auxiliary model has the advantage that its prediction can be relatively independent from the original model. It can take as input the input and output of the evaluated model, but also other features such as information about the model, like the model weights, activation values, etc. Auxiliary models are not only used to provide a confidence score. For instance, in ChatGPT’s RLHF training, an auxiliary model is used to compute a reward that the main model must maximize.

At Posos, auxiliary models provide an additional benefit: they help address the challenge of delivering a confidence score for the overall prediction when deep learning is used for smaller, intermediary tasks. By utilizing these auxiliary models, we can offer a comprehensive confidence score in an end-to-end manner. As described in our previous blog article, we use a random forest to predict the confidence score for the conversion of the prescription scan into a structured object containing the prescribed drugs and their dosage instructions.

• Practical Application: Confidence Levels in Prescription Structuring

To effectively integrate this confidence score into our prescription scanning workflow, we focus on highlighting cases where our algorithms are most confident that the structuration we made requires manual review. This ensures that alerts are meaningful and prevents them from being systematically overridden. Hence, we assigned confidence levels—High, Medium, and Low—to each structured dosage. These categories help users understand the confidence we have in the accuracy of the structuring. Additionally, each confidence level serves a distinct purpose: “High” maintains the initial workflow without changes, “Medium” indicates potential issues that require user review but can be bypassed without extra clicks, and “Low” requires mandatory user review before the output can be exported. This approach is integrated seamlessly into our interface, where users are guided to verify dosages based on these confidence levels, ensuring both transparency and trust in the system’s outputs.This integration process is often put aside when thinking about explainable AI, yet the design and presentation of AI are crucial for identifying potential mistakes and enhancing overall safety in AI applications.

Posos’ strategy of improving the interpretability of our models relies on those two mainstays: giving back control to the user by splitting a high-stake prediction into smaller ones and reassuring the user using a carefully crafted confidence score. It shows that explainability does not only require working on new XAI models and frameworks but that there is also a need for carefully thinking about how AI models can be safely included in a product design.

‍

Authors: Laura ZANELLA and Natalia Bobkova

‍

In the article, Laura Zanella covered the theoretical aspects of XAI, including the ‘black box’ problem, dimensions of explainability, and advanced techniques like probing and counterfactual explanations. She also described the XAI tool, LIME, and discussed future challenges and opportunities in XAI. Natalia Bobkova focused on describing the XAI tools: Microsoft Responsible AI toolbox, DICE, and Fairlearn.

‍