Sensitive to noise

[bruunand]

Hi

I've found that Lingua generally works well with clean text. However, sometimes it can really be thrown off by words that are not in any language vocabulary (to my knowledge). For example, the text hello this is a test SIARxKnru3t gives the following classifications.

1. eu - 1.0
2. ca - 0.87
3. sk - 0.86
4. nn - 0.86
5. cs - 0.86

English is not even in the top 5 here. If I remove SIARxKnru3t it has 100% confidence in en. I've seen similar behavior with URLs and hashtags, but those can be filtered out. Is there anything I can do to reduce this noise sensitivity?

[pemistahl]

Hi Anders, thanks for trying my library and for reaching out to me.

I'm afraid, there is nothing you can do about this. The language detector is sensitive to noise especially for very short texts because there are not enough distinct ngrams available for calculating a reliable language estimate. If the sentence was longer with the word SIARxKnru3t being the only noisy one, the detector would surely output English. You should try to filter out the noise before trying to detect the language.