Context

The performance of large systems is crucial for business efficiency, as unresponsive systems can harm user experience and software quality. Companies, such as Mozilla, invest in performance testing and monitoring, but these are typically conducted outside the development context, often during pre-release QA which causes inefficiencies leading to resources/time loss.

Vision

• The goal is to enhance Mozilla's performance workflow by trying different strategies including empirical analysis of the performance workflow, both from a system perspective and from manual process perspective, in order to explore the landscape. Afterwards, based on the findings, we want to propose possible changes to the workflow and we want to report the changes' effects on the performance engineering workflow if the changes are implemented.

Challenges

In order to make the vision concrete, the project is divided to multiple challeneges

• Challenge 0: Extracting performance related data from Mozilla systems As Mozilla's systems are open to the general public, the data collection should take place. That can be followed by an analysis of the data to identify insights.

• Challenge 1: Proposing changes to Mozilla's workflow Based on the insigths from the previous challenge, we proceed with identifying areas of change in the performance engineering workflow. The current idea is to try different techniques for predicting change points in Mozilla's performance measuremnt time series to enhance precision and accuracy of performance anomaly detection.

• Challenge 2: Implementing the changes into Mozilla systems Based on the results fro mthe previous challenge, we implement changes in the Mozilla systems.

• Challenge 3: Evaluating the effect of the changes Once the changes are implemented and has been around for a while, we assess the impact of these changes on the long term on the performance engineering workflow at Mozilla.

Work done

Currently, the work done has revolved around extracting data from Mozilla system and try different change point detection techniques to detect anomalies in Mozilla data. The work that has been done exists in the folders data, data_extraction_transformation, prediction_generation, and prediction_postprocessing_scripts.

data_extraction_transformation overview

The data_extraction_transformation folder contains scripts and notebooks to extract performance-related data from Treeherder API, tranform them, and do preliminary analysis on them. Performance-related alerts are extracted from one year ago (from beginning of May 2023 to the beginning of May 2024).

prediction_generation overview

The prediction_generation folder contains scripts to detect the change points in the data using TCPDBench. The YCPDBench project is extended and altered to fit the use case of this project.

prediction_postprocessing_scripts overview

The prediction_postprocessing_scripts folder contains scripts to further tranform and analyze the data generated by TCPDBench by combining the results of several CPD methods for example.

Data artifacts challenge

The folders to consider for the artifacts challenge are data which contains the data and data_extraction_transformation which contains the necessary code used to extract the data.

Contact information

This work is a collaboration between Mozilla and REALISE Lab in Concordia University. Lab members participating in the project are :

• Prof. Diego Elias Costa (Supervising Professor)
• Mohamed Bilel Besbes (Research assistant)