AI has been around for more than 50 years. Artificial intelligence is not new, but the recent developments of deep learning and convolutional neural networks combined with increased compute power and extended storage capacity have made it possible to create an AI that can solve complex problems. These new AIs are replacing humans in several areas, but is everything better with AI?
This blog post will cover some of the pros of using AI test automation for software tests and some of the cons. To learn more about how testing can be improved with AI, read on!
Artificial Intelligence is the simulation of human intelligence processes by machines, especially computer systems. AI is a broad and complex topic, but we can break it down into two major categories: artificial narrow intelligence and artificial general intelligence.
Artificial narrow intelligence is when a machine mimics a human’s intellectual ability. It can solve a very specific problem and has a limited range of tasks.
Artificial general intelligence is when a machine mimics a human’s intellectual ability and can do any intellectual task that a human can.
AI test automation for software tests is an example of artificial narrow intelligence that is capable of performing software tests.
For software testing you can use AI in two different scenarios.
As we have already established AI test automation for software tests is an example of artificial narrow intelligence. In practical terms that means it cannot distinguish if something is right or wrong on its own. For this, the AI needs to have an understanding of the intended use of an application. Something only a general intelligence will be able to do.
Today, AI needs training and cannot deviate new knowledge from training data. Let me give you a practical example: You train your AI to always click the “Cancel” button if the “Do you accept cookies” popup occurs. One day the popup is redesigned to ask “Do you accept all cookies or do you want to set up your preference”. The AI will still press the Cancel button because that is what you’ve trained it to do. While pressing cancel did not accept the cookies till today it will now accept all. While you don’t retrain your AI it will continue to work but give wrong results.
For decision scenarios, AI can click around in your software and see if it crashes, as that is clearly not the intent of your application. But that is it.
In summary, do not use AI for decision-making tasks in software testing today.
If you have a big amount of data and need to find similarities, that is one of the scenarios in which AI can shine. Assume you have 100 failed test cases and you have stack traces for all of them. Over time, your AI can learn correlations between stack traces and actual root causes a lot better than every human.
The same is true if you need to identify a lot of similar elements for Visual Testing. Train your AI with a few and it will be able to find similar elements in your application.
That is what test.ai did in the past: They’ve trained their model with thousands of shopping cart symbols to allow you to write “click on shopping cart”. The AI would be able to identify the shopping cart button in your app while it has never seen it before.
Brilliant use of AI to help you to save time.
While there are valid scenarios, never forget that Artificial Intelligence is still a buzz word and you’ll hear a lot of things that AI magically fixes, that cannot be true.
A good example is flakiness, i.e. a test case randomly fails because of the automation and not because of the subject under test (SUT). AI is only a band-aid that brings you from x% flakiness to x-1% flakiness.
Next time someone tells you about their great AI for software testing, ask them how they have tested it (the AI) — Please let me know the answers 😉
In Software Testing, AI is mostly applied to automated test execution, therefore the usual test automation pros apply:
What are the pros of using AI in software testing? I want to highlight two, that are usually not mentioned:
What are the cons of using AI in software testing?
Overall, I think that AI will not revolutionize the software testing industry. But it will improve continuous testing, which will help catch bugs before they are released to the public in DevOps environments.
Today, AI in software testing is used for tasks like improving OCR results, identifying fuzzy images, classifying elements, comparing layouts by perception, etc.
As time goes on, I think that AI will be able to do more complex things, like testing for security risks or performance issues that a human could not even think of. But it will not deliver on the promise like the one of zero flakiness, for example. That’s okay, as there is no need for AI to do so as flakiness is a problem introduced by how the tests are executed. Take TestResults.io as an example for a test platform that uses a fundamentally different approach and therefore removes scenarios that usually lead to flakiness from the equation. Check it out.
However, this is IT and who can predict the next 5–7 years? Once we are able to develop artificial general intelligence which understands the intended use of our applications, we don’t need to test anymore. On the other hand, at that point, we do not need to develop applications anymore, at all.
The software will finally be able to understand our intent. No more manuals, support hotlines, IT experts, … It will really be like “Siri, work through all my emails today and do what I would have done and, btw, do this every day from now on”.
In summary, AI is nothing new, but we did move the limits in which it operates.
.png)
.png)