Day 19: Experiment with AI for test prioritisation and evaluate the benefits and risks

Day 19 already! I hope you can all appreciate how much we have covered already - well done!

Today we want to turn our attention to whether AI can help us make decisions about test selection and prioritisation and evaluate some of the risks and benefits of this as an approach.

Using data to make decisions about what to test and how much has been around for a long time (most testers are familiar with the idea of Risk Based Testing) and it’s natural to think about automating these decisions to accelerate. The technical evolution for this process it to delegate to an AI model that learns from data in your context about the testing performed and the observable impact of the testing.

The critical question is…Should we?

Task Steps

You have two options for today’s task (you can do both if you want):

  • Option 1 - If your company already uses an AI powered tool for test prioritisation and selection, then write a short case study and share it with the community by responding to this post. Consider sharing:

    • The tool you are using
    • How does the tool select/prioritise tests? Is this understandable to you?
    • How does your team use the tool? For example, only for automated checks or only for Regression?
    • Has the performance of the tool improved over time?
    • What are the key benefits your team gains by using this tool?
    • Have there been any notable instances where the tool was wrong?

  • Option 2 - Consider and evaluate the idea of using AI to select and prioritise your testing.

    • Find and read a short article that discusses the use of AI in test prioritisation and selection.
      • Tip: if you are short on time, why not ask your favourite chatbot or copilot to summarise the current approaches and benefits of using AI in test prioritisation and selection?
    • Consider how you or your team currently perform this task. Some thinking prompts are:
      • To what extent do you need to select/prioritise tests in your context?
      • What factors do you use when selecting/prioritising tests? Are they qualitative or quantitative?
      • How do you make decisions when there is a lack of data?
      • What is the implication if you get this wrong?
    • In your context, would delegating this task to an AI be valuable? If so, how would your team benefit?
    • What are the risks of delegating test prioritisation and selection to an AI model? Some thinking prompts are:
      • How might test prioritisation and selection fail, and what would the impact be?
      • Do you need to understand and explain the decision made by the AI?
      • “How did test/qa miss this?” is an unjust but common complaint - how does this change if an AI is making the decisions about what to test?
      • How could you mitigate these?
      • If we mitigate risks using a Human in the loop, how does this impact the benefits of using AI?
    • How could you fairly evaluate the performance of an AI tool in this task?
    • Share your key insights by replying to this post. Consider sharing:
      • A brief overview of your context (e.g. what industry you work in or the type of applications you test).
      • Share your key insights about the benefits and risks of adopting AI for test prioritisation and selection.

Why Take Part

  • Understanding where AI can help: There is excitement/hype about using AI to improve and accelerate testing. For teams managing large numbers tests, complex systems or time-consuming tests being more data driven about selecting and prioritising tests might provide real benefits. By taking part in today’s task, you are critically evaluating whether it works for your context, you learn about specific risks of delegating responsibility to AI and are better prepared to make a considered decision about AI based Testing Selection and Prioritisaiton.

https://club.ministryoftesting.com/t/day-19-experiment-with-ai-for-test-prioritisation-and-evaluate-the-benefits-and-risks/75216

My Day 19 Task

Based on my current work status, for Day 19’s task, I chose Option 2

Regarding Reading an Article Discussing the Use of AI in Test Prioritization and Selection

The article I read is AI-Driven Test Prioritization

The main content of the article is as follows:

AI-driven test prioritization enhances efficiency and reduces costs in software testing.

  • 📄 Understanding Test Prioritization: Determining test case execution order for maximum coverage and defect detection.
  • ⭐️ Significance of AI-Driven Test Prioritization: Enhanced efficiency, faster feedback, optimal resource utilization, improved test coverage, and reduced costs.
  • 🔨 Implementing AI-Driven Test Prioritization: Data collection, feature selection, machine learning model, prediction, and integration with testing tools.
  • 🏢 Real-Life Examples and Implementations: Google and Microsoft successfully use AI-driven test prioritization.

AI-driven test prioritization revolutionizes software testing by utilizing machine learning to rank test cases based on their likelihood to fail, enhancing efficiency and coverage while optimizing resources. This method relies on analyzing vast data, including historical test results and code changes. Implementations by giants like Google and Microsoft illustrate its value in reducing testing times and accelerating development. By integrating AI prioritization with testing tools, organizations can significantly improve their software quality and reduce costs, underscoring the transformative impact of AI in software development.

Regarding Summarizing the Methods and Benefits of Using AI Tools in Test Prioritization and Selection

I chose ChatGPT-4, and its response was:

AI in test prioritisation and selection, particularly within the context of software development and quality assurance, is a rapidly evolving area that leverages machine learning (ML), natural language processing (NLP), and other AI methodologies to enhance the efficiency and effectiveness of testing processes. Here's a summary of the current approaches and benefits:

### Approaches

1. **Machine Learning Models**: AI systems are trained on historical data, including past test results, code changes, and defect rates, to predict which tests are most likely to detect faults in new or modified code. Techniques such as regression, classification, and even deep learning models are employed.

2. **Natural Language Processing (NLP)**: Used to analyze requirements and change logs to identify relevant tests. This approach helps in understanding the context and semantics of changes, thereby selecting tests that are more likely to be affected by those changes.

3. **Test Case Embeddings**: Similar to word embeddings in NLP, test cases are represented in a high-dimensional space, where their similarities to code changes or features can be measured. This helps in identifying which tests are most relevant to the current changes.

4. **Prioritization Algorithms**: These algorithms rank tests based on various factors, such as historical failure rates, test execution time, code coverage, and impact of recent code changes. The aim is to run the most critical tests early in the cycle.

5. **Feedback Loops**: Incorporating continuous feedback from test outcomes to refine and adjust the models and algorithms used for test selection and prioritization, ensuring they become more accurate over time.

### Benefits

1. **Reduced Test Times**: By prioritizing and selecting the most relevant tests, AI can significantly reduce the overall time required for testing, enabling faster release cycles.

2. **Improved Test Coverage**: AI techniques can uncover gaps in test coverage by analyzing code paths and identifying untested or under-tested areas, leading to better software quality.

3. **Cost Efficiency**: Reducing the amount of testing without compromising on quality leads to lower computing resources usage and human effort, thereby cutting costs.

4. **Early Defect Detection**: Prioritizing tests that are more likely to fail can lead to early detection of defects, which is generally less costly to fix and leads to more stable releases.

5. **Adaptability**: AI models can adapt to changes in the software and testing environments, ensuring that the test prioritization and selection processes remain efficient over time.

6. **Risk Management**: By focusing on high-risk areas based on code changes and historical defect data, AI helps in better managing the risks associated with software releases.

These AI-driven approaches represent a significant shift from manual and static methods to dynamic, efficient, and intelligent test management strategies, aiming at improving software quality, reducing costs, and accelerating development cycles.

Regarding Reflecting on How You or Your Team Currently Execute This Task

Background and Available Information of My Current Project

  • The current project has tight delivery timelines, with significant pressure on task delivery and a risk of scope creep.
  • Existing project SOPs require compatibility with different browsers, resolutions, and devices.
  • The project is an e-commerce online shopping website, primarily for the web platform.

The Necessity of Test Priority Sorting

It is necessary. From the beginning of the project, based on the background and SOP, an initial testing strategy was developed. It included test priorities: business functionality testing first, followed by compatibility testing, then performance and network testing, and finally usability and ease-of-use testing.

Factors Relied Upon for Choosing/Prioritizing Tests

Mostly quantitative, with some qualitative aspects

  • Project team background
  • Project delivery pressure
  • Project SOPs
  • Team personnel configuration, more about the ratio of developers to testers
  • Results of negotiations and communications with the team

Decision-Making in the Absence of Data

Refer to useful information from past projects, negotiate and confirm with the team, and then make a decision.

Here, I must mention that the testing strategy and priorities are always iterative and updated, not set in stone. They can be adjusted based on the project situation and more information obtained.

Regarding The Value of Test Priority Selection to AI Models

There would definitely be value. AI could provide more reasonable and lower-risk outcomes based on known qualitative and quantitative historical data within the model.

Regarding The Risks of Test Priority Selection to AI Models

Risks are inevitable, mainly because of my concerns about data privacy and security with AI model tools. I would not transmit 100% of the project’s context and known information to AI. Therefore, the results generated by AI without full information might significantly differ from the team’s expectations. If the project is executed based on the outcomes provided by the AI model, it might not be possible to complete the project’s delivery on time.

About Event

The “30 Days of AI in Testing Challenge” is an initiative by the Ministry of Testing community. The last time I came across this community was during their “30 Days of Agile Testing” event.

Community Website: https://www.ministryoftesting.com

Event Link: https://www.ministryoftesting.com/events/30-days-of-ai-in-testing

Challenges: