Language detector always loads the small ngrams set

[Toflar]

Hey, thanks for the cool library! I'm using it in Loupe.

I noticed a performance issue:

<?php

use Nitotm\Eld\LanguageDetector;

include __DIR__ . '/vendor/autoload.php';

$languageDetector = new LanguageDetector();
$languageDetector->langSubset(['de']);

var_dump($languageDetector->detect('Guten Tag.'));

On the first call, this will write a subset for just de into the vendor (/subsets) directory. This is perfect!
However, the LanguageDetector will always load the small.php ngrams, even though I only need the subset for German.

Also it is loading the dataset already in the __construct() which means that you cannot instantiate the LanguageDetector object without causing the data to be loaded into memory. This means, it's always loaded even if nobody calls ->detect() on the object later. This should ideally be converted to a lazy evaluation. So only load the data once it's used for the first time 😊

[Toflar]

I just saw that according to the docs, one should pass the subset via constructor. That doesn't make sense to me, why can't LanguageDetector do that itself?
Something like

$languageDetector = new LanguageDetector(); // This does not do anything at all, so no data is loaded
$languageDetector->langSubset(['de']); // This sets the internal subset choice, still no data is loaded

var_dump($languageDetector->detect('Guten Tag.')); // Now the data is loaded, either the subset or `small.php`

Also, small.php is 3 MB on disk. But it's 64 MB when PHP compiles it into its memory structures. Maybe the storage format is not the best? What's your opinion on this? 😊

[nitotm]

Regarding your initial concern and issue title, as you have already seen, you don’t need to load a database you don’t need. In your specific case, you would do:

$languageDetector = new LanguageDetector(‘small_1_em’);
$languageDetector->detect('Guten Tag.')

The ‘em’ from small_1_em, it’s just internal encoding, I agree is not the most intuitive, it’s just the file that ->langSubset(['de']) returns. I could change this behavior, and maintain internal file names, internal.
Also point out that subsets have “size” too, so each database has two characteristics: size and languages.

---

As for the database size on memory, I imagine you have seen https://github.com/nitotm/efficient-language-detector?tab=readme-ov-file#databases where I added detailed information. For example small.php when cached should use 21+4 MB from the OPcache memory.

Still, yes, it’s a concern, especially for the larger databases.
I would say it is not a problem of the array structure, I believe it’s good, but that PHP arrays are high level.
I tried SplFixedArray, and some other solutions, with bad results.
I am open to suggestions and try new things, but I prefer not to use extensions.

---

Regarding loading data on first detect, I don't have a thoughtful answer right now.
I was probably optimizing for the most common/normal usage.
I will think about it.

You can leave this issue open.

[Toflar]

The ‘em’ from small_1_em, it’s just internal encoding, I agree is not the most intuitive, it’s just the file that ->langSubset(['de']) returns. I could change this behavior, and maintain internal file names, internal.
Also point out that subsets have “size” too, so each database has two characteristics: size and languages.

Yes, I understood but this means that every user of your library needs to store the result of ->langSubset() elsewhere in order to then pass it to the constructor which seems not very intuitive 😊

As for the database size on memory, I imagine you have seen https://github.com/nitotm/efficient-language-detector?tab=readme-ov-file#databases where I added detailed information. For example small.php when cached should use 21+4 MB from the OPcache memory.

Yes, I did read that. OPcache doesn't help on CLI for worker processes because it's mostly disabled there.

I am open to suggestions and try new things, but I prefer not to use extensions.

I haven't looked into the format at all but depending on what exactly needs to be looked up it might be interesting to see how sqlite performs. It's widely available. It would likely be slower but depending on how many % of the entire dataset you actually need to load in order to detect, it might be more efficient to load the partial data vs. loading the entire dataset into memory. Would need to be investigated but that's one suggestion 😊

Regarding loading data on first detect, I don't have a thoughtful answer right now.
I was probably optimizing for the most common/normal usage.
I will think about it.

Sure, thanks!

[Toflar]

Do you want me to create a PR for this? 😊