Automatic Tests Generation

Updates:

• Implemented function generate_column_tests() to generate similar tests for many columns automatically

Dataset Null-related tests

• Implemented TestNumberOfNulls to replace TestNumberOfNANs and TestNumberOfNullValues
• Implemented TestShareOfNulls
• Implemented TestShareOfColumnsWithNulls
• Implemented TestShareOfRowsWithNulls
• Implemented TestNumberOfDifferentNulls

Column Null-related tests

• Implemented TestColumnNumberOfNulls to replace TestColumnNumberOfNullValues
• Implemented TestColumnShareOfNulls to replace TestColumnNANShare

Fixes:

• Fixed metric duplication to reduce an amount of calculations while building TestSuites (basically, now same metrics from the one test suite are not recalculated multiple times)
• Implemented NAN filtering for all dashboards in way, that each column is filtered separately

Binary Classification: TPR, TNR, FPR, FNR

Updates:

• added TPR, TNR, FPR, FNR Tests for Binary Classification Model Performance
• Renamed status "No Group" to "Dataset-level tests" in TestSuites filtering menu

Fixes:

• #207
• #265
• #256
• fixed unit tests for different versions of python and pandas

Tests Grouping in the UI

Updates:

1. Updated the UI to let users group tests by the following properties:

• All tests
• By Status
• By feature
• By test type
• By test group

2. New Tests:

• Added tests for binary probabilistic classification models
• Added tests for multiclass classification models
• Added tests for multiclass probabilistic classification models
  The full list of tests will be available in the docs.

3. New Tests Presets:

• Regression
• MulticlassClassification
• BinaryClassificationTopK
• BinaryClassification

Data Quality and Integrity Tests

• added default configurations for Data Quality Tests
• added default configurations for Data Integrity Tests
• added visualisation for Data Quality Tests
• added visualisation for Data Integrity Tests
• Test descriptions are updated (column names are highlighted)

Test Suites with Individual Tests and Test Presets

Implemented new interfaces to test data and models in a batch: Test Suite.

Implemented the following Individual tests:

• TestNumberOfColumns()
• TestNumberOfRows()
• TestColumnNANShare()
• TestShareOfOutRangeValues()
• TestNumberOfOutListValues()
• TestMeanInNSigmas()
• TestMostCommonValueShare()
• TestNumberOfConstantColumns()
• TestNumberOfDuplicatedColumns()
• TestNumberOfDuplicatedRows()
• TestHighlyCorrelatedFeatures()
• TestTargetFeaturesCorrelations()
• TestShareOfDriftedFeatures()
• TestValueDrfit()
• TestColumnsType()

Implemented the following test presets:

• Data Quality. This preset is focused on the data quality issues like duplicate rows or null values.  
• Data Stability. This preset identifies the changes in the data or differences between the batches.
• Data Drift. This one compares feature distributions using statistical tests and distance metrics.  
• NoTargetPerformance. This preset combines several checks to run when there are model predictions, there are no actuals or ground truth labels. This includes checking for prediction drift and some of the data quality and stability checks.

v0.1.51.dev0

Updates:

• Updated DataDriftTab: added target and prediction rows in DataDrift Table widget
• Updated CatTargetDriftTab: added additional widgets for probabilistic cases in both binary and multiclasss probabilistic classification, particularly widget for label drift and class probability distributions.

Fixes:

• #233
• fixed previes in DataDrift Table widget. Now histogram previews for refernce and current data share an x-axis. This means that bins order in refernce and current histograms is the same, it makes visual distribution comparion esier.

Automatic Stattest Selection

Release scope:

1. Stat test auto selection algorithm update: https://docs.evidentlyai.com/reports/data-drift#how-it-works

For small data with <= 1000 observations in the reference dataset:

• For numerical features (n_unique > 5): two-sample Kolmogorov-Smirnov test.
• For categorical features or numerical features with n_unique <= 5: chi-squared test.
• For binary categorical features (n_unique <= 2), we use the proportion difference test for independent samples based on Z-score.
  All tests use a 0.95 confidence level by default.

For larger data with > 1000 observations in the reference dataset:

• For numerical features (n_unique > 5): Wasserstein Distance.
• For categorical features or numerical with n_unique <= 5): Jensen–Shannon divergence.
  All tests use a threshold = 0.1 by default.

2. Added options for setting custom statistical test for Categorical and Numerical Target Drift Dashboard/Profile:
   cat_target_stattest_func: Defines a custom statistical test to detect target drift in CatTargetDrift.
num_target_stattest_func: Defines a custom statistical test to detect target drift in NumTargetDrift.

3. Added options for setting custom threshold for drift detection for Categorical and Numerical Target Drift Dashboard/Profile:
   cat_target_threshold: Optional[float] = None
num_target_threshold: Optional[float] = None
   These thresholds highly depends on selected stattest, generally it is either threshold for p_value or threshold for a distance.

Fixes:
#207

Statistical Tests for Categorical Features

StatTests
The following statistical tests now can be used for both numerical and categorical features:

• 'jensenshannon'
• 'kl_div'
• 'psi

Grafana monitoring example

• Updated the example to be used with several ML models
• Added monitors for NumTargetDrift, CatTargetDrift

Dashboard Customisation: Colour Scheme and Statistical Test

Colour Scheme
Support for custom colours in the Dashboards:

• primary_color
• secondary_color
• current_data_color
• reference_data_color
• color_sequence
• fill_color
• zero_line_color
• non_visible_color
• underestimation_color
• overestimation_color
• majority_color

Statistical Test:
Support for user implemented statistical tests
Support for custom statistical tests in Dashboards and Profiles
Available tests:

• 'ks'
• 'z'
• 'chisquare'
• 'jensenshannon'
• 'kl_div'
• 'psi'
• 'wasserstein'
  more info: docs

Fixes:
#193

Custom Text Comments in Dashboards

Custom Text Comments in Dashboards

• Added type type="text” for BaseWidgetInfo (for text widgets implementation)
• Markdown syntax is supported

see the example: https://github.com/evidentlyai/evidently/blob/main/examples/how_to_questions/text_widget_usage_iris.ipynb