Ok, let's try to describe, what the behaviour should be, then compare it to the current behavior. I assume, when you say "PMDException" you actually mean "processing error", right?

According to the documentation on https://pmd.github.io/latest/pmd_userdocs_cli_reference.html, here are the combinations:

• Analysis result: (no) violations | (no) processing errors | (no) exceptions
• CLI Flag --failOnViolation: true (default) | false
• Exit Code: 0 | 1 | 4

Explanations / Definitions:

• violations: one or more rules have been executed and found problematic code. Violations are part of the (e.g. xml) report.
• processing errors: one or more rules failed or parsing failed. This might lead to false negatives since not everything is analyzed. Processing errors are part of the (e.g. xml) report as well.
• exceptions - "PMDException": PMD didn't run at all, e.g. configuration error (invalid CLI arg, not existing rule referenced, ruleset file not found, couldn't write report, ...)

It boils down to the question: How should be handle processing errors?

If we consider processing errors as potential false negatives, then we should probably return "4". 4 basically means: PMD executed, but something is wrong with a rule or part of the code couldn't be parsed and has been skipped. --> processing errors == violations

Or we could return "1". But "1" usually means, that PMD complete crashed and the analysis didn't run at all, so there are no results, not even partial results, that can be used. If we consider processing errors as exceptions, then we should return "1" regardless of "--failOnViolation". --> processing errors == exceptions

I think currently processing errors are simply ignored completely.

Technically, processing errors are potential false negatives i.e., they could be no violations (when the problem is resolved) or there could be violations. The exit code then could be 0 or 4, based on boolean flag failOnViolations.
Let's add a new exit code 2 (in between) if you wish to treat them as potential false negatives.
If you wish to treat them as exceptions, then exit code 1 will do nicely but the new meaning will have to be made apparent.
If I'm not mistaken, 0 and 4 are 'success' codes; 4 is returned to fail the build process. So the real question is, do you wish the build to 'succeed' on failed processing? I don't think so.

PS: The tabling of the scenarios helped clarify the thinking. Thanks!

Do you need a definitive answer to the above?

I like the levels set out for exit codes in AmigaOS.
We can treat exit code 2 as a warning code.

A pity that completely reworking the exit codes to the levels set out is out of scope for this issue.

https://en.m.wikipedia.org/wiki/Exit_status#AmigaOS

I think the exit code for processing errors should be 0. Processing errors should be caused only by bugs in PMD, which can only be fixed after a release cycle at the earliest. Failing builds because of them would make a user's build fail without a possible fix on their part, apart from excluding the problematic files from analysis. Excluding the files would cause the same false-negatives as the processing errors, only they wouldn't be reported anywhere.

So, I think processing errors should be ignored in the exit code, but logged on stderr / added to the report, ie, the current behavior is correct.

I think the exit code for processing errors should be 0. Processing errors should be caused only by bugs in PMD, which can only be fixed after a release cycle at the earliest. Failing builds because of them would make a user's build fail without a possible fix on their part, apart from excluding the problematic files from analysis. Excluding the files would cause the same false-negatives as the processing errors, only they wouldn't be reported anywhere.

So, I think processing errors should be ignored in the exit code, but logged on stderr / added to the report, ie, the current behavior is correct.

Hmm... Makes sense...
Since you expect PMD to not process files rarely...

To close or not to close, @oowekyala ?

Or do we need another flag, failOnProcessingErrors? What would that do?
Fail the build...

Then what does developing team do?
Change it to false or exclude files...
Hopefully, also log issue with PMD...

Or revert back to the previous working PMD version...

I am getting an exit code of 0 when encountering lexical errors. This is on PMD 7.0.0, Windows 2019 Server and PowerShell, e.g.

PS C:> pmd.bat cpd --minimum-tokens 500 --dir src --language cpp --format xml > cpd-output.xml
[ERROR] Error while tokenizing: Lexical error in file 'src\example.cpp' at line 16, column 25: <EOF> after : "" (in lexical state PREPROCESSOR_OUTPUT)
[ERROR] Exception while running CPD: Errors were detected while lexing source, exiting because --skip-lexical-errors is unset.
PS C:> $LastExitCode
0

If I fix the above error THEN I get $LastExitCode of 4 so violations are found, The CI job is passing because of the 0 exit code, so we don't know that CPD is hitting errors and it's also not failing on violations.

Not setting an exit code > 0 for lexical errors is causing violations to be missed.

I did some testing - and both PMD and CPD behave the same in this regard: ParseExceptions or LexExceptions are logged but don't affect the exit code and therfore don't fail the build.

This causes the issue, that such problems are not being noticed at all. I personally only check the logs, if the build failed. If the build was successful, then I assume, that everything worked as expected. I don't scroll up to look at potential log output.

I think the exit code for processing errors should be 0. Processing errors should be caused only by bugs in PMD, which can only be fixed after a release cycle at the earliest. Failing builds because of them would make a user's build fail without a possible fix on their part, apart from excluding the problematic files from analysis. Excluding the files would cause the same false-negatives as the processing errors, only they wouldn't be reported anywhere.

So, I think processing errors should be ignored in the exit code, but logged on stderr / added to the report, ie, the current behavior is correct.

I think, we should at least have an option to change the behavior to fail the build in such cases. Processing errors might be caused by a bug in PMD or might be caused by a bug in the application being analyzed (probably not a Java application, because we expect already compileable code for Java).

See also the similar case for the maven-pmd-plugin here: https://issues.apache.org/jira/browse/MPMD-383

Reading this case, the default behavior should indeed be to fail the build if there are any processing errors or lex exceptions.