Anomaly Detection for Imbalanced Data

[Noazao]

HI everyone,
I have a dataset for network traffic. The dataset contains 100k from the normal class and 1k from the anomaly class. I applied the Halfspace tree anomaly detection algorithm, but I had a worse performance ( ROC : 53% ). So I tried different techniques to work with data imbalance, such as (Undersampling, Oversampling, SMOTE). I also tried to follow the steps in this link :
https://riverml.xyz/latest/examples/imbalanced-learning/
but it turns out that the RandomOverSampler doesn't accept the Halfspace tree, and this is the error that I got " AttributeError: 'RandomOverSampler' object has no attribute 'score_one'"

Can anyone guide me on working with imbalanced data in online anomaly detection?

[MaxHalford]

Can anyone guide me on working with imbalanced data in online anomaly detection?

You seem to figuring things out by yourself. And we don't have enough information to help you out in terms of performance (no code, no data, etc.).

but it turns out that the RandomOverSampler doesn't accept the Halfspace tree, and this is the error that I got " AttributeError: 'RandomOverSampler' object has no attribute 'score_one'"

That's expected. RandomOverSampler is meant to be used with a classifier, which HalfSpaceTrees isn't. I guess we could extend RandomOverSampler it to support score_one. Although my first feeling is that it's weird to oversample the minority class when you're doing anomaly detection. But why not.

At this point I would recommend you try out scikit-learn on your data and check if it works better. It might just be that your problem is too difficult, even for a batch model.

[Noazao]

Thanks, @MaxHalford , for your quick response. Please have a look at the snapshot of the dataset that I used, and it is available in this repo (https://github.com/dad-repository/dad):

https://www.photobox.co.uk/my/photo/full?photo_id=504672320392

This dataset is collected from the data centre's thermal sensors using TCP and MQTT. It contains 87096 normal data and 1088 anomalous data. The snapshot includes all features after cleaning and extracting the most relevant features. I followed your instructions, using 6 different supervised/unsupervised learning algorithms presented in RiverML and here his the results:

https://www.photobox.co.uk/my/photo/full?photo_id=504672319984
https://www.photobox.co.uk/my/photo/full?photo_id=504672321210

I am trying to use unsupervised learning such as HST or one class SVM to detect the anomaly in the dataset. Some of the algorithms above are supervised ML. The best results (ROCAUC/Recall/F1) were obtained by Adaptive Random forest with Adwin drift detector and leveraging bagging model with Hoeffding tree. I believe this is because of the nature of the drift concept and ensemble of LB. However, in the real world, we will not have the label. Therefore, the only options I have here are HalfSpave Tree and One Class SVM. Both of them didn't work well in this dataset (although I used different hyperparameters ), so I assume it will not work in real implementation in a smart home environment where the data is exchanged over the TCP with a similar setting used to collect the data in the above dataset. This is my code:

def preprocessing():

  categorical_features = compose.Pipeline(
      compose.SelectType(str),preprocessing.OneHotEncoder())
  numerical_features = compose.Pipeline(
      compose.SelectType(numbers.Number),preprocessing.MinMaxScaler())
  
  
  preModel =  categorical_features + numerical_features 
  return preModel


def evaluate_hst(X_y, WINDOW_SIZE):
  model = compose.Pipeline ( preprocessing() ,anomaly.QuantileThresholder(
          anomaly.HalfSpaceTrees(window_size=WINDOW_SIZE,seed=42),
         q=0.95
      ))
  
  report = metrics.ClassificationReport() 
  auc = metrics.ROCAUC()
  
  for x, y in X_y:
     score = model.score_one(x)
     model = model.learn_one(x)
     report = report.update(y, score) 
     auc = auc.update(y,score)
  print(auc)
  print(report)


X = main_df.drop(['label_fl'],axis='columns')
y = main_df['label_fl']
X_y = stream.iter_pandas(X, y)

evaluate_hst(X_y, 500)

Here are my questions:

1. Is my approach correct for anomaly detection? If yes, how would you interpret the result above?
2. In the code, I used the label provided by the dataset to measure the performance of HST. In the real world scenario, how I would measure the performance of HST, one class SVM without the label?
3. Is it possible to use ADWIN drift with HST or One class SVM without the label?

I am new to this, so I greatly appreciate your help.

[MaxHalford]

Hey there! Sorry, I was on holiday.

Is my approach correct for anomaly detection? If yes, how would you interpret the result above?

It looks good to me.

In the code, I used the label provided by the dataset to measure the performance of HST. In the real world scenario, how I would measure the performance of HST, one class SVM without the label?

As far as I know, you wouldn't. You would need some human intervention to validate the outputs. In general you need labels to measure a model's performance. No free lunch!

Is it possible to use ADWIN drift with HST or One class SVM without the label?

ADWIN is useful when you want to detect drift for a single variable. Also, drift detection is different to anomaly detection (one-class SVM, HST). In drift detection, the distribution changes and might change forever. In anomaly detection, you're assuming that the underlying distribution doesn't change, but that some observations are very different from the distribution. Anomaly detection and drift detection are therefore different yet complementary techniques.