For kaldi based experiments, like the 2020 VoicePrivacy Challenge, the code snippet demo_voiceprivacy_challenge.py provides an overview on:
We assume team submissions are in the folder
expwith subfoldersTEAM-ID/SYSTEM/results-TAG. Results are shown for the primary and contrastive baseline systems.
The evaluation plan of the 2020 VoicePrivacy challenge covers automatic speaker verification (ASV) and automatic speech recognition (ASR). The former quantifies the biometric privacy; the latter quantifies the non-biometric utility. Additionally, speech naturalness and intelligibility are addressed in the challenge.
Results are exported to:
-
Comparison: multiple systems at one condition (e.g., libri-a-a-f)
-
Comparison: one system at multiple conditions
To sort legend entries, ZEBRA profiles need to be memorized first. For sorting by DECE values, list of lists strcutures are used:
from copy import deepcopy
from os import sep
dece_values = []
title_strings = []
zebra_objects = []
filename_strings = []
# memorize ZEBRA profiles
for single_assessment, title in zip(assessment_categories, category_titles):
zebra_plot = PriorLogOddsPlots()
# handles for later sorting
dece_values.append([])
dece_handle = dece_values[-1]
zebra_objects.append([])
zebra_handle = zebra_objects[-1]
# and depending descriptions
title_strings.append(title)
filename_strings.append('voiceprivacy-challenge-2020' + sep + title)
for scr, key, label in zip(score_paths, key_paths, profile_labels) # conditions or systems to look into for each single assessment
zebra_framework(zebra_plot, scr, key, label, color_min=None) # color_min=None deactivates profile plotting; we just want to compute profiles
# memorizing ZEBRA profiles
dece_handle.append(zebra_plot.get_delta_ECE())
zebra_handle.append(deepcopy(zebra_plot))
Then, the following function sorts DECE values, creates plots with sorted legends, and exports them:
zebra_plots_sorted_legend(dece_values, zebra_objects, title_strings, filename_strings)
Grouped by databases and gender.
