[plsql,tsql] Fix CPD being case sensitive in PLSQL and TSQL

[oowekyala]

Describe the PR

Since these are two different lexer implementations (Javacc and Antlr), there are two different solutions which use the same idea, which is to project the image of most tokens into uppercase. I didn't use a language property to disable this behavior (like done in Apex) although that could be done.

Related issues

• Fixes [core] CPD is always case sensitive #4396

Ready?

• Added unit tests for fixed bug/feature
• Passing all unit tests
• Complete build ./mvnw clean verify passes (checked automatically by github actions)
• Added (in-code) documentation (if needed)

[pmd-test]

	1 Message
📖	Compared to master: This changeset changes 0 violations, introduces 0 new violations, 0 new errors and 0 new configuration errors, removes 0 violations, 0 errors and 0 configuration errors. Download full report as build artifact
✅	Compared to master: This changeset changes 0 violations, introduces 0 new violations, 0 new errors and 0 new configuration errors, removes 0 violations, 0 errors and 0 configuration errors. Download full report as build artifact
✅	Compared to master: This changeset changes 0 violations, introduces 0 new violations, 0 new errors and 0 new configuration errors, removes 0 violations, 0 errors and 0 configuration errors. Download full report as build artifact

Generated by 🚫 Danger

[adangel]

Thanks!

I didn't use a language property to disable this behavior (like done in Apex) although that could be done.

That's ok. We already removed the property in Apex, as it usually doesn't make sense to have this configurable. It's defined by the language and the language is either case-insensitive or not.

Maybe we could mention this (the fact that a language could be case-insensitive) in https://docs.pmd-code.org/latest/pmd_devdocs_major_adding_new_cpd_language.html and the like?

[jsotuyod]

Up to now these things would have been done in the token filter, which is supported by both Javacc and antlr.

I'm a little wary of adding yet another mechanism that can overlap. The only reason to get something into the lexer itself (the way Javacc is implemented) is because the behavior should apply both to cpd and pmd, as it's inherent to the language. The way this is hooked into antlr languages it's only applicable to cpd.

Yes, the tokenfilter was a catch all that made for some complex implementations and would probably use some cleanup, but I'm unsure this is is.

I feel we need to have a better defined boundary / limit extension points. Once this is published we will have to support it. I'd love to see this behavior pushed into the lexer rather the token / token manager as is the case for Javacc.

[oowekyala]

I'd love to see this behavior pushed into the lexer rather the token / token manager as is the case for Javacc.

Do you mean having this behavior pushed into the Antlr lexer, so that the Antlr parser also sees the normalized image? In that case I agree, it would be nicer. I can look into it.

I think overall using the token filter to do that is an abuse when the language itself applies specific normalizations. That's why I believe our Javacc extension point is justified, and we need a mirror extension point for Antlr.

Edit: I see now that on the line you commented on, the reference to CPD_ANONYMIZE_LITERALS should be removed, as this part should be exclusive to the CpdLexer

[jsotuyod]

Yes, you got my intent right.

Try and push this down to the antlr lexer, the same way the Javacc one works. It will seamlessly apply to pmd and cpd, make it non optional (it's language behavior), and exclude it as an extension point.

Other cpd specific behaviors such as anonymize literals should be done in cpd specific code. Right now that is the token filter, but as we both agree, at some point that api will need to evolve.