Implemented Privacy Metrics

.github/workflows/tests.yml

@@ -21,7 +21,7 @@ jobs:
         python-version: ${{ matrix.python-version }}
     - if: matrix.os == 'windows-latest'
       name: Install dependencies - Windows
-      run: pip install 'torch>=1,<2' -f https://download.pytorch.org/whl/torch_stable.html
+      run: pip install 'torch>=1,<1.8' -f https://download.pytorch.org/whl/torch_stable.html
     - name: Install package
       run: pip install invoke .[dev]
     - name: invoke lint
@@ -58,7 +58,7 @@ jobs:
         python-version: ${{ matrix.python-version }}
     - if: matrix.os == 'windows-latest'
       name: Install dependencies - Windows
-      run: pip install 'torch>=1,<2' -f https://download.pytorch.org/whl/torch_stable.html
+      run: pip install 'torch>=1,<1.8' -f https://download.pytorch.org/whl/torch_stable.html
     - name: Install package and dependencies
       run: pip install invoke .[test]
     - name: invoke pytest
@@ -105,7 +105,7 @@ jobs:
     - if: matrix.os == 'windows-latest'
       name: Install dependencies - Windows
       run: |
-        pip install 'torch>=1,<2' -f https://download.pytorch.org/whl/torch_stable.html
+        pip install 'torch>=1,<1.8' -f https://download.pytorch.org/whl/torch_stable.html
         choco install graphviz
     - name: Install package and dependencies
       run: pip install invoke jupyter .[ctgan]

conda/meta.yaml

@@ -25,8 +25,8 @@ requirements:
     - pomegranate >=0.13.4,<0.14.2
     - pytorch >=1.4,<2
     - sktime >=0.4,<0.6
+    - copulas>=0.5.0,<0.6
     - rdt >=0.4.0,<0.5
-
   run:
     - python >=3.6,<3.9
     - scikit-learn >=0.23,<1
@@ -36,6 +36,7 @@ requirements:
     - pomegranate >=0.13.4,<0.14.2
     - pytorch >=1.4,<2
     - sktime >=0.4,<0.6
+    - copulas>=0.5.0,<0.6
     - rdt >=0.4.0,<0.5
 
 about:

sdmetrics/single_table/README.md

@@ -44,6 +44,30 @@ Implemented metrics:
     * `MLEfficacy`: Generic ML Efficacy metric that detects the type of ML Problem associated
       with the dataset by analyzing the target column type and then applies all the metrics
       that are compatible with it.
+* Privacy Metrics: Metrics that fit an adversial attacker model on the synthetic data and
+    then evaluate its accuracy (or probability of making the correct attack) on the real data.
+    * `CategoricalCAP`: Privacy Metric for categorical columns, based
+    on the Correct Attribution Probability method.
+    * `CategoricalZeroCAP`: Privacy Metric for categorical columns, based
+    on the Correct Attribution Probability method.
+    * `CategoricalGeneralizedCAP`: Privacy Metric for categorical columns, based
+    on the Correct Attribution Probability method.
+    * `NumericalMLP`: Privacy Metric for numerical columns, based
+    on MLPRegressor from scikit-learn.
+    * `NumericalLR`: Privacy Metric for numerical columns, based
+    on LinearRegression from scikit-learn.
+    * `NumericalSVR`: Privacy Metric for numerical columns, based
+    on SVR from scikit-learn.
+    * `CategoricalKNN`: Privacy Metric for categorical columns, based
+    on KNeighborsClassifier from scikit-learn.
+    * `CategoricalNB`: Privacy Metric for categorical columns, based
+    on CategoricalNB from scikit-learn.
+    * `CategoricalRF`: Privacy Metric for categorical columns, based
+    on RandomForestClassifier from scikit-learn.
+    * `CategoricalEnsemble`: Privacy Metric for categorical columns, based
+    on an 'ensemble' of other categorical Privacy Metrics.
+    * `NumericalRadiusNearestNeighbor`: Privacy Metric for numerical columns, based
+    on an implementation of the Radius Nearest Neighbor method.
 * MultiSingleColumn Metrics: Metrics that apply a Single Column metric on each column from
   the table that is compatible with it and then compute the average across all the columns.
     * `CSTest`: MultiSingleColumn metric based on applying the Single Column CSTest on all
@@ -86,7 +110,18 @@ Out[2]:
  'KSTest': sdmetrics.single_table.multi_single_column.KSTest,
  'KSTestExtended': sdmetrics.single_table.multi_single_column.KSTestExtended,
  'ContinuousKLDivergence': sdmetrics.single_table.multi_column_pairs.ContinuousKLDivergence,
- 'DiscreteKLDivergence': sdmetrics.single_table.multi_column_pairs.DiscreteKLDivergence}
+ 'DiscreteKLDivergence': sdmetrics.single_table.multi_column_pairs.DiscreteKLDivergence,
+ 'CategoricalCAP': sdmetrics.single_table.privacy.cap,
+ 'CategoricalGeneralizedCAP': sdmetrics.single_table.privacy.cap,
+ 'CategoricalZeroCAP': sdmetrics.single_table.privacy.cap,
+ 'CategoricalKNN': sdmetrics.single_table.privacy.cap,
+ 'CategoricalNB': sdmetrics.single_table.privacy.cap,
+ 'CategoricalRF': sdmetrics.single_table.privacy.cap,
+ 'CategoricalEnsemble': sdmetrics.single_table.privacy.ensemble,
+ 'NumericalLR': sdmetrics.single_table.privacy.numerical_sklearn,
+ 'NumericalMLP': sdmetrics.single_table.privacy.numerical_sklearn,
+ 'NumericalSVR': sdmetrics.single_table.privacy.numerical_sklearn,
+ 'NumericalRadiusNearestNeighbor': sdmetrics.single_table.privacy.radius_nearest_neighbor}
 ```
 
 ## Single Table Inputs and Outputs

sdmetrics/single_table/privacy/__init__.py

@@ -0,0 +1,25 @@
+from sdmetrics.single_table.privacy.base import CategoricalPrivacyMetric, NumericalPrivacyMetric
+from sdmetrics.single_table.privacy.cap import (
+    CategoricalCAP, CategoricalGeneralizedCAP, CategoricalZeroCAP)
+from sdmetrics.single_table.privacy.categorical_sklearn import (
+    CategoricalKNN, CategoricalNB, CategoricalRF)
+from sdmetrics.single_table.privacy.ensemble import CategoricalEnsemble
+from sdmetrics.single_table.privacy.numerical_sklearn import (
+    NumericalLR, NumericalMLP, NumericalSVR)
+from sdmetrics.single_table.privacy.radius_nearest_neighbor import NumericalRadiusNearestNeighbor
+
+__all__ = [
+    'CategoricalCAP',
+    'CategoricalZeroCAP',
+    'CategoricalGeneralizedCAP',
+    'NumericalMLP',
+    'NumericalLR',
+    'NumericalSVR',
+    'CategoricalKNN',
+    'CategoricalNB',
+    'CategoricalRF',
+    'CategoricalPrivacyMetric',
+    'NumericalPrivacyMetric',
+    'CategoricalEnsemble',
+    'NumericalRadiusNearestNeighbor'
+]