Recent years have seen a growing concern among policymakers and the public about the explainability of artificial intelligence systems. As AI becomes more advanced and is applied to various domains such as healthcare, hiring, and criminal justice, there is a call for these systems to be more transparent and interpretable. The fear is that the black box nature of modern machine learning models makes them unaccountable and potentially dangerous. However, the importance of AI explainability is often overstated, as a lack of explainability does not necessarily make an AI system unreliable or unsafe.
While creators of state-of-the-art deep learning models cannot fully articulate how these models transform inputs into outputs due to their complexity, the same could be said of many other technologies we use daily. The focus should be on validating the performance of high-stakes AI systems and ensuring they behave as intended, rather than solely on explainability. Techniques in the emerging field of AI interpretability aim to open up the black box of deep learning to some extent, allowing for a clearer understanding of how these models process data to arrive at outputs.
It is important to note that AI systems may never be totally explainable like a simple equation or decision tree might be as the most powerful models will likely always entail some level of complexity. There should not be an emphasis on explainability to the detriment of other priorities, as there can be trade-offs between performance and explainability. Ultimately, AI systems should be evaluated based on their real-world impact, with a focus on their reliability and effectiveness in achieving their objectives.
Developers should take appropriate precautions, test AI systems extensively, validate their real-world performance, and align them with human values before deploying them into the broader world. It is essential to strive to make AI systems interpretable where possible but not at the cost of the benefits they deliver. Even a black box model can be a powerful tool for good as long as the output is successful in achieving its intended purpose. The focus should remain on the output of the AI system rather than solely on the process that delivered it.
