Weak Hashing findings vanished from 1.1.11 ruleset?

[davewichers]

I maintain the OWASP Benchmark project:

• https://owasp.org/www-project-benchmark/
• https://github.com/OWASP-Benchmark/BenchmarkJava

Which is a test suite for testing the effectiveness of software security analysis tools. I've had codeQL scripts for scanning OWASP Benchmark for a while.

They can be found in the scripts folder here: https://github.com/OWASP-Benchmark/BenchmarkJava/tree/master/scripts

• translateCodeQL.sh
• runCodeQL.sh

You have to follow the codeQL install instructions listed in the translateCodeQL.sh script, then run translateCodeQL.sh and then runCodeQL.sh.

After completing, it puts the codeQL SARIF results file into the results/ folder and you can then score the tool against Benchmark by running createScoreards.sh.

I noticed that when using the latest version of CodeQL 2.20.1, with the 1.1.9 ruleset, it properly detected 69% of the Weak Hashing test cases in Benchmark (and had zero false positives). But when I upgraded to ruleset 1.1.11, it now detects none of them.

Is this on purpose? Or was a bug introduced. or mistake made, to cause those rules to go away in 1.1.11?

[redsun82]

Hey @davewichers, thanks a lot for reaching out with this! I'll circle this back to our internal team responsible for Java analysis 👍

[redsun82]

Hey @davewichers, as reported in this change note, we recently removed reporting of MD5 and SHA1 hashing from the java/weak-cryptographic-algorithm to the less precise java/potentially-weak-cryptographic-algorithm, as the former was alerting on too many cases of legitimate non cryptographic usages of those hashes. Maybe you can switch to using that query instead in your benchmarking?

[davewichers]

@redsun82 - the codeQL documentation related to finding out which rulepacks exist and how to use them is super confusing. Where is the list of ALL the published codeql rulepacks? I only found: codeql\java-queries and codeql\java-all, but when I try to use java-all, it says 'this is a library and does not contain queries to run'. I want to use ALL the codeQL java rules in my run. How do I do that?

[redsun82]

Hi @davewichers, the concept of a set of queries is captured by "query suites":

• docs for running with the CLI (see in particular the explanation of the <packs,queries> positional argument). What I also found indeed a bit confusing here, is that we don't stress enough that this documentation relates to full codeQL bundles published as releases of codeql-action. Please find more instructions below.
• docs for codescanning
• files defining query suites for java here)

I heard back from the team, and they advise to use the java-security-extended suite for the benchmark. To do so, you should provide java-security-extended or java-security-extended.qls as second positional argument to codeql database analyze. If you download the latest release for the full codeql bundle, that will be included together with precompiled queries which will make the command run faster. You can however also run the queries from a checkout of codeql, provided you give the right --search-path.

Where is the list of ALL the published codeql rulepacks?

Probably the best way to get that list is to run

codeql resolve queries

This will list all query suites with their .qls extension. The query suite argument to codeql database analyze can be provided either with or without that extension.

I'm sorry if you found the documentation lacking, we will try to make it better! 🙌

[davewichers]

Well, These clarifications didn't help either. I'm doing these steps: codeql pack download codeql/java-queries (works fine) codeql database create owasp-benchmark --language=java --overwrite --command="mvn clean package" (works fine) codeql database analyze owasp-benchmark codeql/java-queries --format-sarifv2.1.0 --output=results/Benchmark-v1.2_CodeQL.sarif (works fine) So trying to use a different 'stronger' java rule pack I tried this: codeql pack download codeql/java-all (works fine) codeql database create owasp-benchmark --language=java --overwrite --command="mvn clean package" (works fine) codeql database analyze owasp-benchmark codeql/java-all --format-sarifv2.1.0 --output=results/Benchmark-v1.2_CodeQL.sarif ERROR: CodeQL pack 'codeql/java-all' is a library and does not contain queries. So, I try to use the java-security-extended rulepack you suggest with: codeql pack download codeql/java-security-extended ERROR: blah blah Does this CodeQL pack exist on the public GitHub Container registry blah blah And this doesn't work either: codeql database analyze owasp-benchmark codeql/java-security-extended --format-sarifv2.1.0 --output=results/Benchmark-v1.2_CodeQL.sarif ERROR: blah blah that rulepack doesn't exist You state: "If you download the latest release <https://github.com/github/codeql-action/releases/latest> for the full codeql bundle," I finally figured out how to do that, but there is no java-security-extended query pack in that bundle. There are only java-queries and java-examples So, 1) How do you download the rule pack that contains java-security-extended, and then 2) How do you invoke it properly with: codeql database analyze owasp-benchmark WHATRULEPACKNAMEHERE?

…

On Mon, Jan 20, 2025 at 3:49 AM Paolo Tranquilli ***@***.***> wrote: Hi @davewichers <https://github.com/davewichers>, the concept of a set of queries is captured by "query suites": - docs for running with the CLI <https://docs.github.com/en/code-security/codeql-cli/getting-started-with-the-codeql-cli/analyzing-your-code-with-codeql-queries> (see in particular the explanation of the <packs,queries> positional argument). What I also found indeed a bit confusing here, is that we don't stress enough that this documentation relates to full codeQL bundles published as releases of codeql-action <https://github.com/github/codeql-action>. Please find more instructions below. - docs for codescanning <https://docs.github.com/en/code-security/code-scanning/managing-your-code-scanning-configuration/codeql-query-suites> - files defining query suites for java here <https://github.com/github/codeql/tree/main/java/ql/src/codeql-suites>) I heard back from the team, and they advise to use the java-security-extended suite for the benchmark. To do so, you should provide java-security-extended or java-security-extended.qls as second positional argument to codeql database analyze. If you download the latest release <https://github.com/github/codeql-action/releases/latest> for the full codeql bundle, that will be included together with precompiled queries which will make the command run faster. You can however also run the queries from a checkout of codeql, provided you give the right --search-path. Where is the list of ALL the published codeql rulepacks? Probably the best way to get that list is to run codeql resolve queries This will list all query suites with their .qls extension. The query suite argument to codeql database analyze can be provided either with or without that extension. I'm sorry if you found the documentation lacking, we will try to make it better! 🙌 — Reply to this email directly, view it on GitHub <#18518 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGFWBN6WC4RC4CHUH5LOST2LS2DDAVCNFSM6AAAAABVKJBOTOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMBRG44DEOBUGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[redsun82]

Hi @davewichers

I'm very sorry for answering so late to this, and that you encountered these problems!

I think the source of confusion here is that java-security-extended is not a query pack, but rather a subset thereof (which we call query suites). They are included in the codeql/java-all pack you already download, and then they can be run by providing them as is to codeql database analyze, i.e. without the codeql/ prefix. See my comment above:

To do so, you should provide java-security-extended or java-security-extended.qls as second positional argument to codeql database analyze

So in your case the correct analyze command should be

codeql database analyze owasp-benchmark java-security-extended --format=sarifv2.1.0 --output=results/Benchmark-v1.2_CodeQL.sarif

Before running that you can double check the query suite is included by running

codeql resolve queries

This should contain the lines

  java-code-scanning.qls - Standard Code Scanning queries for Java
  java-lgtm-full.qls - Standard LGTM queries for Java, including ones not displayed by default
  java-lgtm.qls - Standard LGTM queries for Java
  java-security-and-quality.qls - Security-and-quality queries for Java
  java-security-experimental.qls - Extended and experimental security queries for Java and Kotlin
  java-security-extended.qls - Security-extended queries for Java

Those are all possible positional arguments to codeql database analyze, with the .qls extension being optional, no further prefixes required.

I will try out a PR to clarify the documentation to the <packs,queries> argument to database analyze.

[davewichers]

This does not work either. When I run: codeql pack download codeql/java-all It returns: Package install location: /home/MYUSERID/.codeql/packages Already installed ***@***.*** (library) And when I run: codeql database create PROJECTNAME --language=java --overwrite --command"mvn clean package" that appears to run successfully, and it ends with: Successfully created database at CURRENTDIR/PROJECTNAME. But when I run: codeql resolve queries, it returns: No queries were specified Also, no discoverable query suites were found. Perhaps you need to download some QL packs or configure a --search-path so your packs can be found? And when I try: PATHTO/codeql database analyze owasp-benchmark java-security-extended --format=sarifv2.1.0 --output=results/Benchmark-v1.2_CodeQL.sarif It returns: A fatal error occurred: java-security-extended cannot be found. Using codeql/java-security-extended fails the same way. Using java-security-extended.qls returns: java-security-extended.qls is not a .ql file, .qls file, a directory, or a query pack specification. Note that codeql is NOT on my path. I'm using a hardcoded path to reach the codeql executable in all of these instances. Could that be contributing to this problem?

…

On Tue, Feb 4, 2025 at 3:02 AM Paolo Tranquilli ***@***.***> wrote: Hi @davewichers <https://github.com/davewichers> I'm very sorry for answering so late to this, and that you encountered these problems! I think the source of confusion here is that java-security-extended is not a query pack, but rather a subset thereof (which we call query suites). They are included in the codeql/java-all pack you already download, and then they can be run by providing them *as is* to codeql database analyze, i.e. without the codeql/ prefix. See my comment above: To do so, you should provide java-security-extended or java-security-extended.qls as second positional argument to codeql database analyze So in your case the correct analyze command should be codeql database analyze owasp-benchmark java-security-extended --format=sarifv2.1.0 --output=results/Benchmark-v1.2_CodeQL.sarif Before running that you can double check the query suite is included by running codeql resolve queries This should contain the lines java-code-scanning.qls - Standard Code Scanning queries for Java java-lgtm-full.qls - Standard LGTM queries for Java, including ones not displayed by default java-lgtm.qls - Standard LGTM queries for Java java-security-and-quality.qls - Security-and-quality queries for Java java-security-experimental.qls - Extended and experimental security queries for Java and Kotlin java-security-extended.qls - Security-extended queries for Java Those are all possible positional arguments to codeql database analyze, with the .qls extension being optional, no further prefixes required. I will try out a PR to clarify the documentation to the <packs,queries> argument to database analyze. — Reply to this email directly, view it on GitHub <#18518 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABGFWBM3H6BLASFJUANCPDL2OBXYPAVCNFSM6AAAAABVKJBOTOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMZTGE2TGNRSGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[davewichers]

I added the codeql home directory to my path so I don't have to specify the path anymore, and that did not resolve these problems. So that's not it.

…

On Wed, Feb 19, 2025 at 12:33 PM Dave Wichers ***@***.***> wrote: This does not work either. When I run: codeql pack download codeql/java-all It returns: Package install location: /home/MYUSERID/.codeql/packages Already installed ***@***.*** (library) And when I run: codeql database create PROJECTNAME --language=java --overwrite --command"mvn clean package" that appears to run successfully, and it ends with: Successfully created database at CURRENTDIR/PROJECTNAME. But when I run: codeql resolve queries, it returns: No queries were specified Also, no discoverable query suites were found. Perhaps you need to download some QL packs or configure a --search-path so your packs can be found? And when I try: PATHTO/codeql database analyze owasp-benchmark java-security-extended --format=sarifv2.1.0 --output=results/Benchmark-v1.2_CodeQL.sarif It returns: A fatal error occurred: java-security-extended cannot be found. Using codeql/java-security-extended fails the same way. Using java-security-extended.qls returns: java-security-extended.qls is not a .ql file, .qls file, a directory, or a query pack specification. Note that codeql is NOT on my path. I'm using a hardcoded path to reach the codeql executable in all of these instances. Could that be contributing to this problem? On Tue, Feb 4, 2025 at 3:02 AM Paolo Tranquilli ***@***.***> wrote: > Hi @davewichers <https://github.com/davewichers> > > I'm very sorry for answering so late to this, and that you encountered > these problems! > > I think the source of confusion here is that java-security-extended is > not a query pack, but rather a subset thereof (which we call query suites). > They are included in the codeql/java-all pack you already download, and > then they can be run by providing them *as is* to codeql database analyze, > i.e. without the codeql/ prefix. See my comment above: > > To do so, you should provide java-security-extended or > java-security-extended.qls as second positional argument to codeql > database analyze > > So in your case the correct analyze command should be > > codeql database analyze owasp-benchmark java-security-extended --format=sarifv2.1.0 --output=results/Benchmark-v1.2_CodeQL.sarif > > Before running that you can double check the query suite is included by > running > > codeql resolve queries > > This should contain the lines > > java-code-scanning.qls - Standard Code Scanning queries for Java > java-lgtm-full.qls - Standard LGTM queries for Java, including ones not displayed by default > java-lgtm.qls - Standard LGTM queries for Java > java-security-and-quality.qls - Security-and-quality queries for Java > java-security-experimental.qls - Extended and experimental security queries for Java and Kotlin > java-security-extended.qls - Security-extended queries for Java > > Those are all possible positional arguments to codeql database analyze, > with the .qls extension being optional, no further prefixes required. > > I will try out a PR to clarify the documentation to the <packs,queries> > argument to database analyze. > > — > Reply to this email directly, view it on GitHub > <#18518 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ABGFWBM3H6BLASFJUANCPDL2OBXYPAVCNFSM6AAAAABVKJBOTOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMZTGE2TGNRSGU> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >