You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, when we run an experiment, the output is stored in a folder that is defined by the name attribute in Experiment. If we run the same experiment again with the same output folder, the folder is partially overwritten, which can lead to confusing situations.
Example: I ran an experiment for sensitive attribute A with four categories: 1, 2, 3 and 4. So, 4 different folders were created for each of those categories. Then, I ran the same experiment with sensitive attribute B with two categories, 1 and 2. Since the output folder name was the same for both experiments, the second experiment overwrote the folders for categories 1 and 2 but retained the categories 3 and 4 from the last experiment. This led to incorrect plots when plotting the results.
Ideally, we should have an attribute overwrite that's False by default. If it's False, the experiment should throw an error if the given folder name already exists. If it's True, we should delete the contents of the output folder before running the experiment to prevent data from two experiments being mixed up.
The text was updated successfully, but these errors were encountered:
Currently, when we run an experiment, the output is stored in a folder that is defined by the
name
attribute inExperiment
. If we run the same experiment again with the same output folder, the folder is partially overwritten, which can lead to confusing situations.Example: I ran an experiment for sensitive attribute
A
with four categories:1
,2
,3
and4
. So, 4 different folders were created for each of those categories. Then, I ran the same experiment with sensitive attributeB
with two categories,1
and2
. Since the output folder name was the same for both experiments, the second experiment overwrote the folders for categories1
and2
but retained the categories3
and4
from the last experiment. This led to incorrect plots when plotting the results.Ideally, we should have an attribute
overwrite
that'sFalse
by default. If it'sFalse
, the experiment should throw an error if the given folder name already exists. If it'sTrue
, we should delete the contents of the output folder before running the experiment to prevent data from two experiments being mixed up.The text was updated successfully, but these errors were encountered: