Add nicer syntax for file sizes

[JakobDev]

It would be nice to have Filesizes in TOML.

test = 1MB # Results in 1.000 * 1.000 = 1.000.000
test2 = 1MiB # Results in 1.024 * 1.024 = 1.048.576

Things like max Filesize etc. are used in many configurations and it should be easy to implement for the different parsers.

[eksortso]

We would not introduce a special new type for file sizes, when integers suffice.

That said, we did talk about adding many different multiplier suffixes for integers and floats back in #427. We considered Mi instead of MiB, for instance. Ultimately, the proposal was closed four months ago. It was ambitious, but ambiguities appeared, and it was deemed too confusing in the context of, as it turns out, file sizes. Perhaps if it was limited in scope, it would pass muster, no offense intended to @JeppeKlitgaard.

Personally, I recoil at the thought of using a comment in place of a well-established suffix with a clear meaning. With all due respect to @pradyunsg, there are real problems with doing this, and comments can be deceptive.

size1a = 10_300_000  # 10.3 MB, idk
size1b = 10_800_333  # 10.3 MB, idk
size1c = 10_547_000  # 10.3 MB, idk
size1z = 20e+06      # 10.3 MB, idk+idc

I'd be open to hear a new proposal if it could garner a lot more support than the old proposal did. Make a suggestion or a PR, and rally your colleagues to show their support and spread the word. But maximize utility, expressiveness, and simplicity to make your case.

[marzer]

My main issue with this idea is the distinction between MB and MiB. Personally I find the MiB syntax detestable and always treat filesizes as the sensible-in-technical-domains pow2 form. I know I'm not alone here.

About the only time the (completely insane) base-10 form is acceptable when talking about file sizes is in the (unfortunately) already-established practice of hardware manufacturers trying to make their shit seem more powerful/bigger than it is (which I'm pretty sure is the root cause of the problem to begin with).

Plus, what if someone wanted to write mb, mB et cetera? I've seen all these in the wild, and they're all going to seem perfectly valid to lots of people.

Note that it's only file sizes specifically that suffer from these dumb problems. The more general suffixes proposed in #427 would have avoided all that baggage.

[eksortso]

@marzer wrote:

Plus, what if someone wanted to write mb, mB et cetera? I've seen all these in the wild, and they're all going to seem perfectly valid to lots of people.

ABNF uses case-insensitive matching strings by default. We'd pick up all those variants immediately just by specifying "MB".

[marzer]

Ah, that's true. Guess that eliminates half of my qualms above.

[eksortso]

@JakobDev Could you give this issue a follow-up? It's been five months since your suggestion, and it prompted some discussion on that day. But without some additional interest, continued discussion, or a PR with concrete changes proposed, this issue may be closed like the one that came before it.

[JakobDev]

I'm still interested in this, but I can't contribute much to the discussion here.

[tintin10q]

I would like to add that if you would deserialize file sizes in programming languages then what type should it be converted to? Most programming languages do not have a type for file sizes. This brings in a lot of ambiguity because there are many valid interpretations. Do you parse 4kb to a number of bytes (4000 or 4096)? Do you parse to a string '4kb'? Maybe the programming language does have a native file size type so you parse to that? This leads to differences between parsers from languages which is bad and not simple or obvious.

[JakobDev]

It should be parsed to the number of bytes. The functions to get the file size in programming languages are usually return the number of bytes.

e.g if you have a website that allows uploading files. you can set the max allowed file size:

[Upload]
max-size = 10MB

Pseudo python code:

if os.path.getsize(path) > config["Upload"]["max-size"]:
    show_error("Your file is too large")

if len(file_bytes) > config["Upload"]["max-size"]:
    show_error("Your file is too large")

If you want to calculate if a file is bigger/smaller than the given size, which will be the most common use case, it needs to be a number.

It should be a long to support large sizes such as 1TB, if someone needs that.

Do you parse to a string '4kb'?

If this is wanted, you can just use

[Upload]
max-size = "4kb"

Maybe the programming language does have a native file size type so you parse to that?

I don't know if there any language out there that has and I don't know why such a type is needed, but in this case, use the native filesize type of the language.

[eksortso]

File sizes are integers. There are no fractional bytes out there. Not touching kubits or anything!

The strangulation point is whether kilobytes are 1000 bytes or 1024 bytes. Like it or not, it's that ambiguity in storage quantities that ended #427. We haven't allowed 4k or 4Ki because nobody knows what a kibibyte is, outside of our rarified circle, or which one to use in which cases. And if we knew, we could just write 4_000 or 4_096 and be done with it.

The reason we don't allow something like 4kb is that it is not obvious. And as long as we can write 10_300_000_000 to represent 10.3 billion, then we can stay minimal and not introduce unnecessary features. I know they'd be easy for parser writers to implement. But frankly, a user writing something like max-size-kb = 4 when they know what they're dealing with makes more sense in the end.

It really pains me that we can't have SI suffixes for integers, but it's those stupid ambiguous file sizes that make our lives difficult!

[rmunn]

New idea for solving the file-size ambiguity: K means 1000, KB means 1024. M means 1000000 (one million), MB means 1048576 (1024*1024). And so on.

Rationale: if #427 was ultimately rejected because, as @eksortso says, "nobody knows what a kibibyte is, outside of our rarified circle", then let's change the syntax. A suffix without a trailing B is a kilo/mega/etc as used in scientific notation (powers of 10), whereas a suffix with a trailing B means "kilobyte/megabyte" with the traditional powers-of-2 meaning thereof (1024, 1048576, and so on).

Suffix	Meaning	Suffix	Meaning
K	1000	KB/KiB	1024
M	1000²	MB/MiB	1024²
G	1000³	GB/GiB	1024³
T	1000⁴	TB/TiB	1024⁴
P	1000⁵	PB/PiB	1024⁵
E	1000⁶	EB/EiB	1024⁶
Z	1000⁷	ZB/ZiB	1024⁷
Y	1000⁸	YB/YiB	1024⁸
R	1000⁹	RB/RiB	1024⁹
Q	1000¹⁰	QB/QiB	1024¹⁰

My reasoning is that when people are working with bytes, powers of 1024 are what they expect. Even in technical circles, nobody actually says the word "kibibyte" out loud: we all talk about kilobytes even though we really do mean 1024 bytes, and nobody thinks this is ambiguous. In writing, we might write "kibibytes", but only in technical contexts where precision matters more than clear communication; most of the time if you see someone write "kilobytes" in an article, you expect that it means 1024 bytes unless the author specifically clarifies that it means 1000 bytes.

So why not allow KB, MB, etc. to default to the power-of-2 meaning that everyone expects? The K, M, etc. suffixes will remain powers of 10, to allow watts = 1.21G. But KB is 1024, MB is 1048576, and so on, because in a configuration file, that's what nearly everyone will expect.

Further parts of my proposal:

• KiB, MiB, etc., will be exactly synonymous with KB, MB
• Suffixes are case-insentive (kb and Kb and KB mean the same thing in TOML)
• Floating-point values will be allowed to have KB, etc., suffixes; discussion below

Floating-point values like 2.5MB could be treated in one of five ways:

• Disallow them entirely.
• Parse as floats, rounding left up to consuming application
• Parse as ints, with truncation rounding (1023.5 -> 1023, -1023.5 -> -1023)
• Parse as ints, with ceiling rounding (1023.5 -> 1024, -1023.5 -> -1024)
• Parse as ints only when unambigous (2.5MB, 8.25KB, etc), disallow values like 10.3MB that aren't an exact int value.

I'm coming around to what @eksortso suggested here, which is to parse such values as floats, because it's impossible to tell which of ceiling or truncation rounding would be right for any given application. Also, one other way to handle values like 2.5MB would be to parse them as ints if unambiguous (2.5MB), but leave as floats if not a whole int (10.3MB). I believe this would be a very BAD idea, so I didn't include it in the list above, but it's worth at least mentioning if only to immediately reject the idea out of hand.

[rmunn]

One more consideration: the E ("exa", 10⁶) potentially conflicts with the E used in scientific notation. Adding this would make parsers more difficult to write, as encountering an E at the end of a number now has two possibilities:

1. It's followed by a digit or a + or -, so it's part of a floating-point value written in scientific notation
2. It's the end of the number, or it's followed by B or iB, so it's part of a suffix

Both can be parsed unambiguously, but it makes parsers slightly harder to write. We would also need to add a rule that you cannot use scientific notation and numeric suffixes together. No writing 1.0e3K to represent a thousand thousand (1 million), as that's just unnecessary cruelty to parser writers.

And another consideration: once you get up into the exabyte range, you're approaching the limits of 64-bit integers. TOML says that the values -2^63...2^63-1 (the natural range of signed 64-bit ints) must be accepted. But 8EB is 2^63, so any value of 8EB or greater would be too large to fit into a 64-bit int. My suggestion for handling this issue is the following:

1. Values strictly less than 8EB are treated as ints or floats as mentioned above (ints if no decimal point present, floats if decimal point is present).
2. Integer values of 8EB or higher should be represented as platform-dependent "big integer" types (called BigInt / BigInteger in most languages) if such types are available.
3. On platforms that don't have a BigInt / BigInteger data type, integer values of 8EB or higher should be represented as floats. Any loss of precision involved in representing a large int as a float is unlikely, because the most common scenario is going to be a small mantissa and a large exponent (e.g., 3ZB would be 3×1024⁷ = 3×2⁷⁰, which can be precisely represented as a float).

[tintin10q]

I just don't think this should be added. It is not obvious.

As was said before just do something like max-size-kb = 4.

And another big problem with this is automatically writing a config file from something like a python dict. How would the writer know that it has to write a number back to something ending with kb. It wouldn't so then you would get the number of bytes. So this would only add something for reading filed but then as soon as you write to the file its gone.

Just don't make it more complicated.