Should tartufo insist on valid base64 encodings?

[rbailey-godaddy]

Historically, we did a spotty job of this. The original get_strings_of_set() allowed = to appear anywhere in a "base64" string, although technically it is only permitted as padding at the end. There's a PR in review that replaces this function with a regex-based find_string_encodings() that does a better (but still imperfect) job of matching only legal base64 encodings.

However, the reason I bring this up is that we have a request to add support for base64url encodings. By way of quick recap, base64url is just like base64 except it replaces + and / in the encoding alphabet with - and _. I am considering speed-vs-accuracy tradeoffs in possible implementations. I see three options:

1. (Conservative and slow) Treat base64url as entirely new and distinct, and scan for these encodings like we do for base64 and hex. This results in 3 full-text entropy scans vs the existing 2 full-text entropy scans now, i.e. a 50% increase in effort.
2. (Sloppy and fast) Recognizing that base64 and base64url are nearly the same, we could modify the base64 regex to include the two new characters. The good news is that this would recognize both base64 and base64url encodings with negligible additional effort. The bad news is this would also recognize things that are neither base64 nor base64url encodings
3. (Middle of the road) We could take the "sloppy" approach and then re-test any matches to ensure we return only valid encodings. The retesting seems likely to be expensive, mitigated only by the thought that it could be restricted to strings we thought likely to be returned and thus hopefully less expensive than option 1.

You do not see "option 4 - make the user tell us with a command line flag" because there is nothing that says a repository -- or even a single file -- might not have both base64 and base64url encodings in it.

Given that we are looking for entropy and not sanity-checking validity, I am very drawn to option 2 due to its efficiency and simplicity, but I am looking for feedback on the real-world consequences of this strategy. There are two corner cases I have thought of:

• Tartufo may find a string that contains (for example) both + (from base64) and _ (from base64url). Do we really care if this string is examined for high entropy, even if it is not actually a valid known encoding? History suggests that we do not. (Note previous mishandling of = and lack of valid-length check.)
• Tartufo may find a string that consists of (for example) onething-anotherthing. Previously this would be considered as two separate base64 strings (onething and anotherthing), but now we would see one onething-anotherthing string. Previous findings that might be excluded by signature would no longer match, because the detected string had changed. A special case of this would be something like _onething which previously would return onething and now would return _onething.

As you consider the second point, think about the likelihood of an otherwise-valid base64 encoding appearing in text immediately adjacent to either a - or _ (the two new characters). This seems to me to be unlikely -- but if I thought it was totally safe, I wouldn't be asking.