The List is growing a bit too much

[Rudxain]

Intro

As of now, uad_lists is ~1.4MB.

#608 (reply in thread)

Some text editors may have performance problems when loading the file (but let's be honest, those editors are poorly optimized).

Potential solutions

Format

On #516, I proposed migrating from JSON to some other standard format. But maybe we could benefit from a specially-designed format. I could write a parser that not only validates syntax, but also the values of the fields (similar to #536)

I managed to reduce the List to ~0.9MB by:

• removing unnecessary quotes (like YAML does)
• using single-letter keys
• removing all indentation

For reference, compressing the "non-minified" file with

xz -9e -T1 uad_lists.json

yields a ~0.19MB file. While the minified one compresses to ~0.18MB

For simplicity, we could also choose CSV

Split

Aside from modifying the file, we could also break it in multiple files, for better organization.

I would propose 1 file for each major OEM, as that would allow UAD-ng to avoid loading all lists, iff 1-or-more devices from the same OEM are connected. This can also serve as a network optimization, as only 1 OEM list needs to be updated at a time.

#608 (reply in thread)

See also

• meta(pkg-lists): Move uad_lists to a separate repository #612 / Move `uad_lists` to a separate repository #646

[Rudxain]

I've noticed we use spaces for indentation 💀. We really should use tabs, while we decide what to do

[AnonymousWP]

Sounds good for a major future release (e.g. 2.0.0).

[Rudxain]

Yeah. ~1MB isn't really "too much". But if it ever gets to ~4MB we may get into problems, because GH refuses to render "large files"

[samolego]

1 file per OEM seems quite reasonable.

Question about carriers: don't they also sometimes put stuff on the phones (that's not the case for my country, but idk how it is in US / elsewhere). In that case, perhaps also 1 file per carrier, then you choose the phone+carrier combo.

[Rudxain]

Nice idea! I forgot about carriers, lol.

In that case, instead of:

• 1 list per OEM
• An extra list for everything else

We would have:

• 1 per OEM
• 1 per carrier
• Only 1 Google + AOSP. Because sometimes it's unclear which is which, and they can switch between versions.
• 1 for everything else: chipset manufacturers, unknown/unlisted stuff, 3rd-party partners (such as Meta and Amazon), etc...

[AnonymousWP]

How do we want to migrate that though? Nobody's going to do this manually, and not all package names lead to an OEM. Based on descriptions may help, but still perfect.

This is why I prefer 1 file per OEM. Downside is that if we add a new OEM to the list, we need to create a new file and also build-in support for that within UAD-ng.

[Rudxain]

not all package names lead to an OEM.

Good point!

build-in support for that within UAD-ng.

I assume the GH API supports listing directories. So we could put all lists in the same folder, then code UADNG to download all files from that dir

[samolego]

Also, how about making dependencies, labels and neededBy optional? Would save some space by those apps which pnly have empty arrays there.

[AnonymousWP]

Yes, I think we agreed (and 0x192 also wanted to do this) upon removing these fields, as they're not displayed in UAD-ng anyway as far as I know.

[Rudxain]

I was considering that if a package "A" is a dependency of "B", then removing "A" should auto-rm "B". This behavior would turn UAD-NG into a package manager (it already is, in some way)

[lavafroth]

Ok, here's a more exotic idea. What if these files also double as a knowledge-base? We create a separate repo with markdown files for each category. uad pulls these markdown files, parses them for use. That way, even a passer by will be able to see descriptions of the apps.

Also see: dnscrypt-proxy2 which uses markdown files for public resolvers.

[Rudxain]

That would help a lot with longer descriptions! The MD files can also contain "testimonials" from users (with links to the original comments), so that other users can get more confidence about removing packs.

Example: If another user has the same device as me, they can be confident that if I had any issues (or no issues), they would have them too.

Currently, this info is "lost" to the list, but can be retrieved by searching Issues & PRs (here, or the legacy repo). If we had a KB, it would be easier to find all those details

[AnonymousWP]

I do prefer Markdown, but maybe it's more prone to error? If you make a small typo (perhaps syntax), then it can't be parsed. Is there a Rust lib that can parse Markdown stuff then?