Bi-Directional Transcoding of Invalid Identifiers

[miguelh-nvidia]

Description of Proposal

TfMakeValidIdentifier, was used in OpenUSD to convert any identifier into a valid identifier. However, it creates a
non-bidirectional relationship, for example, something like カーテンウォール would be transformed into ________________.

The objective of this proposal is to provide an alternative to TfMakeValidIdentifier that can take any identifier
(potentially with invalid characters) and transform it into a OpenUSD valid identifier.

Contributing

• I agree to and accept the Supplemental Terms.

[nvmkuruc]

The proposed implementation should touch on whether or not invalid UTF-8 code points are allowed to be encoded or decoded, specifically code points that are greater than the max code point, the replacement code point, and the surrogate ranges.

[miguelh-nvidia]

The proposed implementation should touch on whether or not invalid UTF-8 code points are allowed to be encoded or decoded, specifically code points that are greater than the max code point, the replacement code point, and the surrogate ranges.

Agree, it is not clear in the proposed API. I changed to reflect on that.

• input should be valid UTF-8
• output with invalid UTF-8
• reference to TfUtf8CodePoint for the other aspects (i.e. max code point, replacement code point and surrogate ranges).

[nvmkuruc]

At one point we had discussed that the right behavior is to attempt to decode and then reencode. Motivation-- Let's say I have "tn__MünchenGermany_rEi5, an identifier previously encoded with SdfBoostringEncodeIdentifier, and I want to ensure it's ASCII encode with SdfBoostringEncodeAsciiIdentiifer. It also can function as a validator. Run SdfBootrstringEncodeIdentifier as a way of ensuring that an identifier is properly encoded.

[nvmkuruc]

Or actually vice versa-- I'm leaving a domain that required Ascii identifiers and now I want to "upgrade" to Utf8.

[miguelh-nvidia]

I think the input for encode should by any UTF-8 string and we do not really have any interpretation of what that string is. The output of encoding should be:

• empty string: in case of invalid UTF-8 string
• the same string: if the UTF-8 string is already in the domain of valid characters
• encoded string (i.e. with tn__ prefix): if the UTF-8 string is not in the domain of valid characters.

I think the 3 proposed methods pose the minimum set of operations to fix the problem mentioned above:

std::optional<std::string> SdfBootstringReencodeIdentifier(const std::string& identifier) {
   std::string originalIdentifier = SdfBootstringDecodeIdentifier(identifier).value_or(identifier);
   return SdfBootstringEncodeIdentifier(originalIdentifier);
}

That can be added into the set of methods to the API, and I think the intent is clear: it will attempt to check the passed identifier is a valid encoded identifier and it will encode it again.