Accuracy too good to be true?

[EarlTheCurl]

Hi.

I've been recording two different R/C boats while driving them around. So thats two classes there and one more that is just silence. I've removed parts of the original recordings of the boats that were basically silent and chopped them up into 1-second segments and made 4 augmentations of each file.

So each class consists of about 3000 1s clip (that includes the augmentations) and when training with batch size of 20 and 2000 epochs, I get 99.3% accuracy with 14 mistakes out of 2160 attempts.

That doesn't seem right to me and would love to get feedback from anyone with some experience since I'm totally new to this.

Thank you, Scott, for creating panotti.

[drscotthawley]

Hi @EarlTheCurl ; I've had a busy summer and am just now getting to Issues.

Hmm. Is that accuracy for the Validation set, or just the Training set? Or is it an extra Testing set?

If you're talking about the Training set, then sure, you're likely to get a high accuracy if you train for a while, but this is likely "overfitting", which means that when you apply the trained model to a new set of files, it won't do so well.

[EarlTheCurl]

Hi, and thank you for the reply. I appreciate it.

That accuracy is on an extra testing set. Most of my recordings is from an indoor pool, with little to no background noise, while some of the files in the Testing set is recorded in the ocean with lots of different sounds.

Even when testing on just the files from the ocean, I'll get some really good results. So I've come to the conclusion that it's Panotti that is well made.