‍
CertifAI is an Artificial Intelligence (AI) testing and certification company. We serve the requirements of the EU AI Act and applicable sectoral laws and regulations for external assessments and testing support. But let’s put standards or compliance aside for now. Our goal is broader. Real-world applications of AI are still rare compared to its potential. We firmly believe that a critical piece has to be solved first to usher in the next era of AI-driven applications.

AI in its infancy

Since AlexNet handily won the ImageNet competition in 2012, AI has seen a rise in popularity and industry adoption. Google’s Brain team is said to be one of the first teams to implement modern AI into Google’s Maps and later Ads products, as Andrew Ng likes to recall.

As the popularity of AI grew, our CTO Jan wrote an article in 2020 predicting that AI-based applications will become ubiquitous in the future. The story is too compelling and logical. Recording and storing data will become cheaper, thus we are going to produce more data in the future. This will be the fuel to power a new wave of AI-based applications.

Yet, more than a decade after AlexNet jumpstarted an entire industry around the sexiest job of the 21st century, AI adoption is not equally distributed. We see tremendous potential for growth.

Lagging AI adoption

While ChatGPT spurred a Generative AI investment flurry of 18bn in 2023, according to the State of AI report, AI adoption remains below its potential. Stanford estimates that since the AlexNet inception in 2012, private investments in AI exceeded $400bn. IBM surveyed in 2022 that only 35% of more than 7,500 institutions have deployed AI in their business. AI adoption grew about 4% from the year prior. In their State of AI 2023 report, McKinsey estimates that only 6% of surveyed companies generate 20% or more of their EBIT with AI.

We also observe that digital-first industries have had fewer challenges operationalizing AI-based applications. However, more traditional industries haven’t been as successful. Many reasons exist. They include siloed data, a lack of talent, or missing infrastructure to deploy AI-based applications. However, the key challenge is another one.

The missing ingredient to increased AI adoption

Responsible AI. Ethical AI. Safe AI. Trustworthy AI. Reliable AI. These words are used interchangeably at times, but they are all describe parts of what we simply call good AI engineering.

Kent Beck is attributed with inventing test-driven development (TDD) as part of the extreme programming development practice. Over two decades ago, software applications often lacked quality. They failed to scale with increasing complexity. Developers often frowned upon testing and didn’t test their applications well enough. After the dotcom bubble crashed, the tech community slowly but surely understood the need for testing. They adopted TDD and other testing techniques. Today, you’ll be hard-pressed to find successful software applications that aren’t well-tested.

We also observe that digital-first industries have had fewer challenges operationalizing AI-based applications. However, more traditional industries haven’t been as successful. Many reasons exist. They include siloed data, a lack of talent, or missing infrastructure to deploy AI-based applications. However, the key challenge is another one.

Testing your AI-based application

So, how do you test your AI-based application? Utilizing MLOps is a great first step, but not the full answer. AI works non-deterministically and is mostly based on data. This makes it hard to understand its inner workings, especially for neural networks. It requires a different testing approach than traditional software testing.

In essence, we advocate for combining a structured methodology with targeted testing. Start with a risk-based evaluation to systematically find vulnerabilities of your application and implement the corresponding tests. This is a very common practice across industries. This approach is also proposed by the Fraunhofer IAIS AI Audit Catalogue and others. Tests for your AI-based component differ depending on the use-case, AI model, or data used. However, AI tests are abundantly available. Based on the EU High-Level Expert Group’s recommendation on trustworthy AI, you can evaluate the risk of your AI along six dimensions. Here, AI-based applications don’t differ from traditional software applications. Every product manager of every product, digital or analog, should always manage the risks of their application. This is the foundation of good engineering cultures which produce qualitative products.

After analyzing the risks, deploy tests to increase the reliability or security of your application. This approach works for any AI-based application. It could be an LLM-, a CV-, or a Logistic Regression-based application. A follow-up article will guide you through an example of applying this approach practically.

If you evaluate the risks of your AI-based application and systematically test against them, you’ve established a good foundation for building qualitative products and great customer experiences. This will help you make the AI do more of what it’s supposed to do, thus driving the pace of AI adoption.

Good AI engineering enables regulatory compliance

As mentioned in the introduction, CertifAI is an AI testing and certification company. We serve our clients’ needs to follow EU law. The (close to final draft) AI Act classifies AI-based applications based on its risk level. High-risk applications will face more scrutiny before gaining access to the European market. While the EU AI Act is the first EU wide horizontal AI regulation, governments around the world work on additional AI regulation.

Certification and regulatory compliance follows good AI engineering practices. Even if you don’t work on high-risk classified applications, you’ll vastly benefit from a good AI engineering approach. It will improve the quality of your AI-based applications. That is what we set out to do. CertifAI helps you implement good AI engineering practices and complying with EU AI Act requirements. We’re convinced that good AI engineering helps building reliable AI applications. This will increase the pace of AI adoption. Let’s work together to implement good AI engineering practices to build the AI-driven world we want to live in now.