Quick Eval on Mac for Standard QL Libraries Not Functioning Correctly

[knewbury01]

Describe the bug
Quick eval in out of the box QL libraries (on Mac) does not succeed in any circumstance

Version
The CodeQL and VSCode extension version in which the bug occurs.

CodeQL CLI 2.15.5 and extension 1.13.0

(but other CLIs too have been observed to have same issue - 2.14.6, 2.16.4, multiple pairings of standard library versions)

To reproduce
When developing a custom query, for which "Install Pack Dependencies" has been used to get the libraries, and a codeql-pack.lock.yml is successfully generated, then any QL class is used in the custom query, a jump to definition successfully navigates to the expected .qll definition in the packages location for the lib version listed in the lock file. However any quick evals on any predicates in any library fail. Clearly the extension/CLI can resolve the library because the custom query using the library succeeds (full eval completes, any quick evals complete, no error messages etc). However it is less ideal that quick evals cannot be used for quick checks.

Expected behavior
Click quick eval on any class name/predicate etc in a library file and get a result/complete evaluation

Additional context

error message:

Failed to run query: Could not resolve library path for /Users/<user>/.codeql/packages/codeql/java-all/0.8.12
Error: Failed to run query: Could not resolve library path for /Users/<user>/.codeql/packages/codeql/java-all/0.8.12
    at redactableError (/Users/<user>/.vscode/extensions/github.vscode-codeql-1.13.0/out/extension.js:95019:12)
    at LocalQueryRun.getCompletedQueryInfo (/Users/<user>/.vscode/extensions/github.vscode-codeql-1.13.0/out/extension.js:113369:57)
    at LocalQueryRun.complete (/Users/<user>/.vscode/extensions/github.vscode-codeql-1.13.0/out/extension.js:113303:30)
    at LocalQueries.compileAndRunQueryInternal (/Users/<user>/.vscode/extensions/github.vscode-codeql-1.13.0/out/extension.js:113716:9)
    at LocalQueries.compileAndRunQuery (/Users/<user>/.vscode/extensions/github.vscode-codeql-1.13.0/out/extension.js:113654:5)
    at withProgress.title (/Users/<user>/.vscode/extensions/github.vscode-codeql-1.13.0/out/extension.js:113528:9)

[dbartol]

@knewbury01 Quick question: How exactly are you invoking Quick Eval: Context menu in the editor pane, key binding, command palette? And what's the exact text of the menu command you're invoking (we have a couple similar commands depending on whether you've got the Canary feature flag enabled or not).

[knewbury01]

I can confirm that the command palette "CodeQL: Quick Evaluation" gives the same error , and also I believe this UI element is the "Context menu in the editor pane"?

[aeisenberg]

@NlightNFotis will triage this.

[NlightNFotis]

Hello,

I have spent some time on this and I have been unable to reproduce this (more precisely: I'm seeing the expected behaviour) following the instructions.

My extension version is 1.13.0 and the CLI version as reported by the extension is:

Searching for latest release including codeql-osx64.zip.
CodeQL CLI already up to date.
Checking CodeQL version using CodeQL CLI: /Users/nlightnfotis/Library/Application Support/Code/User/globalStorage/github.vscode-codeql/distribution8/codeql/codeql version -v --log-to-stderr --format=json...
[2024-05-30 10:47:01] This is codeql version -v --log-to-stderr --format=json
[2024-05-30 10:47:01] Terminating normally.

CLI command succeeded.
Found compatible version of CodeQL CLI (version 2.17.3)

I have attached a small screen recording that shows me following the steps, just to make sure I'm not missing anything obvious. The only step missing is the "Install Pack Dependencies", which I had performed prior to recording the video.

Screen.Recording.2024-05-30.at.10.43.55.mov

Will spend a bit of time to see if I can downgrade the version used and see if I can reproduce in this manner, but I hope it's a positive signal that it's not reproducible in the latest version - modulo me being wrong in the repro method I'm using.

[NlightNFotis]

Follow up: Downloaded a version of the bundle (v2.16.4) and pointed the extension to its local path. I have also checked out the tag codeql-cli/v2.16.4 of the github/codeql repository and I remain similarly unable to observe the erroneous behaviour (conversely, I observe the expected behaviour):

[knewbury01]

@NlightNFotis the feature works as expected for myself as well with bundles (or checkouts of the codeql repo for the libs), the problem has only been observed when using the "Install Pack Dependencies" functionality

[NlightNFotis]

Hm, when I did a first run of the triage, I downloaded the codeql-starter-pack, and then the apache/kafka Python database, and then I selected the Python example in the starter pack. At that point I run the Install Pack Dependencies option through the command palette (edit: installed the python-all dependency) which I verified by ensuring that the codeql-pack.lock.yml is present in the same directory. Then I followed the same process I have captured in the video above.

I'm wondering if there's a different way I should execute the Install Pack Dependencies that might have triggered the issue. Which way are you using to perform this action? (I'm assuming it exists in a context menu or somewhere outside the command palette - otherwise I'm not sure how best to trigger this).

[NlightNFotis]

Ahh, wait, re-reading your comment I think I understand what you mean. I will try again from a repo where the ql libraries aren't installed and will report back.

This will probably happen towards the end of the day, as I'm currently working on something else.

[NlightNFotis]

@knewbury01 I've tried again in a new repository, copying the example Python query, and adding the example qlpack.yml in the repo.

The files added look like this:

Python_example.ql

/**
 * @name Empty scope
 * @kind problem
 * @problem.severity warning
 * @id python/example/empty-scope
 */

 import python
 
 from Scope s
 where count(s.getAStmt()) = 0
 select s, "This is an empty scope."

qlpack.yml

# Change 'getting-started' to a user name or organization that you have write access to
name: getting-started/codeql-extra-queries-python
version: 0.0.0
dependencies:
  # This uses the latest version of the codeql/python-all library.
  # You may want to change to a more precise semver string.
  codeql/python-all: "*"

In the VSCode extension settings I'm referencing a CLI versioned 2.16.4.

I proceed to install the pack dependencies referenced in the qlpack.yml (there's only one option, getting-started/codeql-extra-queries-python), but I remain similarly unable to observe the erroneous behaviour as stated - seeing behaviour similar to the one in the video recorded above.

If I navigate to the Scope.qll file, I see it's in a path close to the binary, so I'm assuming these are bundled together. If I take the CLI binary and put it in a different path, then reference it from the extension settings as the Executable Path, then I get Unhandled error: No distribution found Error: No distribution found.

So the error as far as I understand is predicated on the CodeQL CLI not having a query pack present and installing it through the Install Pack Dependencies option - but I'm not sure how to get to that state.

Would it be possible to record a short video with the MacOS Screenshot tool that shows the conditions under which the bug gets triggered? If that's not possible, some more information on the environment configuration would also be helpful.

[aeisenberg]

@NlightNFotis are you using the codeql bundle (downloaded from github/codeql-action)? This version includes the dependencies with it. Instead, you can download the CLI from https://github.com/github/codeql-cli-binaries. Even better you can use the ghcodeql CLI extension to help you manage different versions of the CLI.

$ gh extension install github/gh-codeql
$ gh codeql download 2.16.4
$ gh codeql version   # use this to get the path to the installation folder and set the value in the `codeQL.cli.executablePath` setting

[NlightNFotis]

Thanks Andrew, yeah this is where I had sourced the codeql bundle from. I wasn't aware that I can download the CLI alone, so I was basically trying to extract the binary from the bundle in a number of ways and wasn't successful doing that.

I will give this another try later today following what you suggested and report back here.

[aeisenberg]

Let me know if you need any help. I'm around. Or we can go through this together at the offsite next week.

[knewbury01]

@aeisenberg @NlightNFotis ah yep, sorry maybe hadnt been explicit enough on that part, I observe this problem when the standalone CLI distribution is paired with the Install Pack Dependencies functionality
(was implying that with bundles or a repo checkout, installing a pack would not be needed, but again my apologies for not making that explicit! 😅 )

[NlightNFotis]

Hello, update:

I've followed what Andrew suggested, and I can confirm that I'm now seeing the issue described in the OP:

Failed to run query: Could not resolve library path for /Users/<user>/.codeql/packages/codeql/python-all/0.12.1
Error: Failed to run query: Could not resolve library path for /Users/<user>/.codeql/packages/codeql/python-all/0.12.1
    at redactableError (/Users/<user>/.vscode/extensions/github.vscode-codeql-1.13.1/out/extension.js:94418:12)
    at LocalQueryRun.getCompletedQueryInfo (/Users/<user>/.vscode/extensions/github.vscode-codeql-1.13.1/out/extension.js:112843:57)
    at LocalQueryRun.complete (/Users/<user>/.vscode/extensions/github.vscode-codeql-1.13.1/out/extension.js:112771:30)
    at LocalQueries.compileAndRunQueryInternal (/Users/<user>/.vscode/extensions/github.vscode-codeql-1.13.1/out/extension.js:113190:9)
    at LocalQueries.compileAndRunQuery (/Users/<user>/.vscode/extensions/github.vscode-codeql-1.13.1/out/extension.js:113128:5)
    at withProgress.title (/Users/<user>/.vscode/extensions/github.vscode-codeql-1.13.1/out/extension.js:113002:9)

I've followed this traceback, and it appears to be coming from the CLI (as in, the error seems to be coming from the CodeQL server). I'll follow up with Andrew as to the next best step here.