discovery failed: bad grpc status code: The operation was cancelled

[dBianchii]

Verify canary release

• I verified that the issue exists in the latest Turborepo canary release.

Link to code that reproduces this issue

https://github.com/dBianchii/kodix-turbo

What package manager are you using / does the bug impact?

pnpm

What operating system are you using?

Linux

Which canary version will you have in your reproduction?

turbo 2.0.3

Describe the Bug

When trying to run turbo watch dev -F @kdx/kdx... for my project, I get this error:

gabriel@gabriel-ubuntu:~/Documents/Github/kodix-turbo$ turbo watch dev -F @kdx/kdx...
• Packages in scope: @kdx/api, @kdx/auth, @kdx/date-fns, @kdx/dayjs, @kdx/db, @kdx/eslint-config, @kdx/kdx, @kdx/locales, @kdx/prettier-config, @kdx/react-email, @kdx/shared, @kdx/tailwind-config, @kdx/tsconfig, @kdx/ui, @kdx/upstash-dev, @kdx/validators
• Running dev in 16 packages
• Remote caching disabled
  × failed to connect to daemon
  ╰─▶ server is unavailable: channel closed

Expected Behavior

Should run the project in watch mode without any problems

To Reproduce

Clone https://github.com/dBianchii/kodix-turbo and run pnpm dev:kdx after running pnpm i

Additional context

I tested it on 2.0.4-canary.4

[dBianchii]

I also get this error sometimes:

× failed to connect to daemon
  ╰─▶ server is unavailable: channel closed

But after runing turbo daemon clean the error goes back to "bad grpc status code"

[aaronadamsCA]

Same version, same issue:

$ pnpm exec turbo watch typecheck lint:fix
• Packages in scope: //, ...
• Running typecheck, lint:fix in 27 packages
• Remote caching enabled
  × discovery failed: bad grpc status code: The operation was cancelled

Apparently it's not reliably reproducible, as I'm currently running watch mode without issue. I have no idea what changed.

[NicholasLYang]

Interesting. Could you run watch mode with debug verbosity, i.e. add -vv to the invocation? And also if you could send over your daemon logs (run turbo daemon logs), that'd be helpful too!

[ericanderson]

I see the same thing in my repo. Some notes:

• turbo daemon status was fine.
• The file was readable. The PID in it was still running.
• turbo daemon status reported no issues.
• I had a hunch after seeing that turbo daemon status -vv was fine and cloned my repo onto the same drive as /var and it works. Might be a red herring. Also possibly useful: the dev drive is case sensitive. MacOS.

Hope this helps. Thanks for your hard work

[ericanderson]

Also with many -v's:

2024-06-18T10:19:58.448-0400 [DEBUG] turborepo_lib::daemon::connector: looking for pid in lockfile: AbsoluteSystemPathBuf("/var/folders/qx/[redact]/T/turbod/[redact]/turbod.pid")
                                             2024-06-18T10:19:58.453-0400 [TRACE] log: deregistering event source from poller
                                                                                                                               × failed to connect to daemon
  ╰─▶ server is unavailable: channel closed

[ericanderson]

I tried with the Kodix repo from the OP and it worked on the same drive as /var and failed on my dev drive. Thats your repro I think.

[gegenschall]

I have the very same issue with watch. I'm running turbo inside a Docker container. Logs I can see:

2024-06-21T10:53:34.000+0000 [DEBUG] turborepo_lib::daemon::connector: looking for pid in lockfile: AbsoluteSystemPathBuf("/tmp/turbod/f53b52ad6d21cceb/turbod.pid")
2024-06-21T10:53:34.000+0000 [DEBUG] turborepo_lib::daemon::connector: found pid: 55
2024-06-21T10:53:34.000+0000 [DEBUG] turborepo_lib::daemon::connector: got daemon with pid: 55
2024-06-21T10:53:34.001+0000 [DEBUG] turborepo_lib::daemon::connector: creating AbsoluteSystemPath("/tmp/turbod/f53b52ad6d21cceb")
2024-06-21T10:53:34.001+0000 [DEBUG] turborepo_lib::daemon::connector: watching AbsoluteSystemPath("/tmp/turbod/f53b52ad6d21cceb")
2024-06-21T10:53:34.001+0000 [DEBUG] turborepo_lib::daemon::connector: creating AbsoluteSystemPath("/tmp/turbod/f53b52ad6d21cceb")
2024-06-21T10:53:34.001+0000 [DEBUG] turborepo_lib::daemon::connector: watching AbsoluteSystemPath("/tmp/turbod/f53b52ad6d21cceb")
2024-06-21T10:53:34.001+0000 [DEBUG] turborepo_lib::daemon::connector: connecting to socket: /tmp/turbod/f53b52ad6d21cceb/turbod.sock
2024-06-21T10:53:34.001+0000 [DEBUG] turborepo_lib::daemon::connector: socket error: transport error

[Cypher1]

I'll add that even with --no-daemon the channels are always acting 'closed' for me

turbo --no-daemon watch typecheck
• Packages in scope: @<org>/<pkg1>, @<org>/<pkg2>, @<org>/<pkg3>, @<org>/<pkg4>, @<org>/<pkg5>
• Running typecheck in 5 packages
• Remote caching disabled
  × failed to connect to daemon
  ╰─▶ server is unavailable: channel closed

 turbo daemon logs    
2024-06-22T04:05:32.604752Z ERROR turborepo_lib::daemon::server: package changes stream closed: channel closed
2024-06-22T04:27:16.227724Z  WARN turborepo_filewatch: encountered error watching filesystem enumerating recursive watch: IO error for operation on /home/jaypratt/Code/skfl/.git/.gitstatus.kKDIEm/a/1: No such file or directory (os error 2)
2024-06-22T04:27:18.363161Z ERROR turborepo_lib::daemon::server: package changes stream closed: channel closed
2024-06-22T04:27:40.715940Z ERROR turborepo_lib::daemon::server: package changes stream closed: channel closed
2024-06-22T04:27:51.961664Z ERROR turborepo_lib::daemon::server: package changes stream closed: channel closed

And then, if I manually clean

❯ turbo daemon clean                 
Done

❯ turbo watch typecheck            
• Packages in scope: ...
• Running typecheck in 5 packages
• Remote caching disabled
  × discovery failed: bad grpc status code: The operation
  │ was cancelled

Fwiw, non-watch commands succeed

❯ turbo typecheck
....successful logs....

❯ echo "$?"                            
0

[Cypher1]

Okay! Progress:

It looks like turbo watch <job> is quitting if any of the jobs fail!

So, if I add extra jobs of the form

// in package.json
"typecheckTrue": "npm run typecheck || true",

// In turbo.json
"typecheckTrue": { ... copy of the definition for "typecheck" },

I get the expected behaviour! (except that I get ticks for every item in the watch view)

[aaronadamsCA]

I believe I may have now worked around the issue by installing the Turborepo LSP extension for Visual Studio Code. I can't say for sure that's why turbo watch has begun to work reliably for me, but it seems like the most likely answer, and worth a shot for anyone else who uses Code.

[Mati365]

I have a similar issue, but it always happens after the first run of watch command. I'm unable to reproduce that on second run, for example.

[NicholasLYang]

Gotcha. I'll try to reproduce and fix these issues. I will note that turbo watch is not compatible with --no-daemon, since we use the daemon to watch your files and produce change events.

If I have it correct, there's a few issues at hand here:

• Running on /var vs /dev @ericanderson
• Running in a docker container @gegenschall
• Watch quitting when jobs fail @Cypher1
• Maybe other generic connection issue? @dBianchii @Mati365

Is that an accurate summary?

[ericanderson]

For mine, I am running on MacOS. When I said my "dev" drive I meant my literal other drive/partition that is mounted at /Volumes/git. For me, this is important because its case-sensitive (and a faster volume than my normal drive). I am not having problems if I run from my home directory, which is on the same volume/partition as /var. I get the errors when running from /Volumes/git which is a different partition/volume/mount point.

[uncvrd]

I've run in to this fairly reliably (not consistent enough for repro though) when i leave the turbo watch [cmds] running in my terminal and then my computer goes to sleep. If I wake up my computer and try to make a change to my code I receive

unable to connect to daemon after 3 retries

My command is turbo watch dev pkg-dev

Which does the following in my monorepo:

• Two persistent Fastify servers with tsx watch --clear-screen=false src/server.ts (maybe i should use turbo daemon to watch instead of using tsx watch?)
• One persistent NextJS app with next dev
• 6 packages watched with the turbo daemon tsup

If I try to run my turbo watch [cmd] again after this fails, all my processes in the terminal just hang. I have to go to my activity monitor where i normally see 2 node processes still active that I have to force quit. I then can run my command again and it works

The turbo daemon logs don't seem too helpful at the moment but here you go

2024-06-25T08:23:41.397334Z  WARN turborepo_lib::package_changes_watcher: hashes are the same, no need to rerun
2024-06-25T08:23:41.551849Z  WARN turborepo_lib::package_changes_watcher: changed_files: {AnchoredSystemPathBuf("apps/web/.next/cache/webpack/client-development/20.pack.gz_")}
2024-06-25T08:23:41.551857Z  WARN turborepo_lib::package_changes_watcher: changed_packages: Ok(Some({WorkspacePackage { name: Other("foundry-client"), path: AnchoredSystemPathBuf("apps/web") }}))
2024-06-25T08:23:41.551987Z  WARN turborepo_lib::package_changes_watcher: hashes are the same, no need to rerun
2024-06-25T08:23:42.552668Z  WARN turborepo_lib::package_changes_watcher: changed_files: {AnchoredSystemPathBuf("apps/web/.next/cache/webpack/client-development/20.pack.gz"), AnchoredSystemPathBuf("apps/web/.next/cache/webpack/client-development/8.pack.gz"), AnchoredSystemPathBuf("apps/web/.next/cache/webpack/client-development/12.pack.gz_"), AnchoredSystemPathBuf("apps/web/.next/cache/webpack/client-development/8.pack.gz_"), AnchoredSystemPathBuf("apps/web/.next/cache/webpack/client-development/index.pack.gz.old"), AnchoredSystemPathBuf("apps/web/.next/cache/webpack/client-development/index.pack.gz"), AnchoredSystemPathBuf("apps/web/.next/cache/webpack/client-development/20.pack.gz_"), AnchoredSystemPathBuf("apps/web/.next/cache/webpack/client-development/5.pack.gz_"), AnchoredSystemPathBuf("apps/web/.next/cache/webpack/client-development/0.pack.gz"), AnchoredSystemPathBuf("apps/web/.next/cache/webpack/client-development/0.pack.gz_"), AnchoredSystemPathBuf("apps/web/.next/cache/webpack/client-development/12.pack.gz"), AnchoredSystemPathBuf("apps/web/.next/cache/webpack/client-development/index.pack.gz_"), AnchoredSystemPathBuf("apps/web/.next/cache/webpack/client-development/5.pack.gz")}
2024-06-25T08:23:42.552677Z  WARN turborepo_lib::package_changes_watcher: changed_packages: Ok(Some({WorkspacePackage { name: Other("foundry-client"), path: AnchoredSystemPathBuf("apps/web") }}))
2024-06-25T08:23:42.565069Z  WARN turborepo_lib::package_changes_watcher: hashes are the same, no need to rerun
2024-06-25T17:14:53.505513Z  WARN turborepo_lib::package_changes_watcher: changed_files: {AnchoredSystemPathBuf("packages/trpc/src/scripts/migrate.ts")}
2024-06-25T17:14:53.505556Z  WARN turborepo_lib::package_changes_watcher: changed_packages: Ok(Some({WorkspacePackage { name: Other("@foundrydev/trpc"), path: AnchoredSystemPathBuf("packages/trpc") }}))
2024-06-25T17:14:53.954880Z ERROR turborepo_lib::daemon::server: package changes stream closed: channel closed

Hopefully this helps a little bit

[warjiang]

turbo daemon clean

@Cypher1
You're right, execute npx turbo daemon logs , I also find the similar logs:

And execute npx turbo daemon clean to clean the daemon manully. After that ,the watch mode works~ 🚀

[amarpatel]

[...] npx turbo daemon clean to clean the daemon manully. After that ,the watch mode works~ 🚀

Just experienced this; running npx turbo daemon clean allowed me to start a turbo watch job again 🤝

[NicholasLYang]

@uncvrd, how long are you letting your computer sleep? We do have an inactivity timeout for the daemon, so if you're leaving watch mode open for multiple hours, it's expected that the daemon should stop.

[uncvrd]

@NicholasLYang it would be overnight. In V1 versions of turbo i could just leave the dev mode running, and continue where I left off the next morning.

I've updated to the latest version of turbo v2.0.6 and this morning I keep randomly getting

The logs from the corresponding file change show

2024-07-03T19:02:30.688061Z  WARN turborepo_lib::package_changes_watcher: changed_files: {AnchoredSystemPathBuf("apps/web/components/Table/FileNameCell.tsx")}
2024-07-03T19:02:30.688125Z  WARN turborepo_lib::package_changes_watcher: changed_packages: Ok(Some({WorkspacePackage { name: Other("foundry-client"), path: AnchoredSystemPathBuf("apps/web") }}))
2024-07-03T19:02:32.986589Z  WARN turborepo_lib::package_changes_watcher: changed_files: {AnchoredSystemPathBuf("apps/web/.next/trace")}
2024-07-03T19:02:32.986599Z  WARN turborepo_lib::package_changes_watcher: changed_packages: Ok(Some({WorkspacePackage { name: Other("foundry-client"), path: AnchoredSystemPathBuf("apps/web") }}))
2024-07-03T19:02:32.986796Z  WARN turborepo_lib::package_changes_watcher: hashes are the same, no need to rerun

Not sure if helpful though.

really wish I could understand what triggers it but it has happened twice in the last few hours. Normally my changes are hot reloaded when updating a file but other times i get this cancelled error. If I can reliably repro I'll share a codesandbox but nothing yet still...:/

This is the turbo dev command i run for my NextJS app:

"dev": {
            "cache": false,
            "env": ["VAULT_ADDR", "VAULT_ENV", "VAULT_TOKEN"],
            "persistent": true
        },

Which runs this npm script:

"dev": "NODE_OPTIONS=\"--max_old_space_size=6144\" next dev -H foundry-dev.ac",

---

Okay hypothesis that might help...maybe?

Since my turbo task is defined as persistent i find it interesting that the turbo daemon logs are recording that files are being changed. Shouldn't file changes be ignored with turbo watch if persistent is true?

[NicholasLYang]

So the way turbo watch works is that it separates out your persistent tasks and runs them normally. File changes are only used to restart non-persistent tasks. If you're only running persistent tasks, i.e. dev, then you might not need watch mode. Regular turbo run should work just fine.

Nonetheless, there does seem to be an issue with the daemon either crashing or ending up in an invalid state. Continuing to investigate that...

[uncvrd]

Hey @NicholasLYang thanks for the response! I have a previous comment above that mentions that I use tsup for watching my packages to rebuild so I need to use the watch

#8491 (comment)

No worries though let me know if there's anything i can test on my end to help

[NicholasLYang]

Ah gotcha, sorry for missing that. I'm investigating the watch issues right now. Do you happen to have an open source repository that I can use to test?

[uncvrd]

All good! Since I haven't been able to repro consistently, nothing to provide unfortunately. Sorry :/ If I can lock it down then yes, would be happy to provide a repro

[JDaoMothership]

+1 for this happening intermittently as well. Hard to get a true repro since it is pretty random 😭

[dobrac]

+1 from me as well. I don’t know how to reproduce it either :/ For now, I might just add the state when it happens:

• lot of changes happening quite quickly and requiring to recompile a lot of packages from the dependency graph
• changes happening while compilation from another run is happening
• possible recompilation of a package happening twice
• quite interconnected dependency graph

[knowlesc]

+1 from us also. Sadly unable to find a way to reliably reproduce, but it is causing significant disruptions in our development workflow.

Similar constraints that @dobrac mentioned above. If there's anything else we find out I'll add here, but for now we aren't sure what to do other than just deal with it.

[sanjeethboddi]

Any updates on this?

[nimeshmaharjan1]

man this is just boring

[PatrykKuniczak]

It's so freaking annoying, let's do sth with this, pls.

[sttuartt]

We are seeing this issue 100% of the time we run turbo watch dev on a Mac case-sensitive volume.

This only happens for us on a Mac case-sensitive volume.

If the repo is copied to a non case-sensitive volume, this error does not occur.

Looking forward to a patch for this.

[ericanderson]

My issue is a Mac on a case sensitive volume as well

[PatrykKuniczak]

I Have it on Windows 11

[njbair]

+1 on Mac on case-sensitive volume; works fine when copied to the system drive.

[sanjeethboddi]

+1 on Mac on case-sensitive volume; works fine when copied to the system drive.

[polvallverdu]

+1 Having same result on Windows with WSL, and on Linux (Linux Mint 22). Using the same project in both OS.

---

Edit: I managed to fix it. I'm using a docker compose file to spin up a db locally. I was saving the data on ./data in the same folder as the monorepo. After running docker compose up, everything broke. Running docker compose down -v to delete everything, included volumes, removing the entire folder (as root), cleaning the project (deleting all .turbo/ and node_modules), and cleaning the turbo daemon, seems to solve it for me.

[PatrykKuniczak]

Next month with that error, hymm

[elliason]

Hi, I am also seeing this issue in my monorepo project, if I run the turbo watch inside node:20 docker container.
I have a packages folder where are 71 packages. Each of the packages has a "build" task, that uses tsc to compile.
I have a root package.json file with the task:
"watch:packages": "turbo watch --filter={./packages/*/**} build"
If I run the container in interactive mode, and then I run "pnpm watch:packages" it works just fine.
If I run the commnad in container via docker run ... bash -c "pnpm watch:packages" I got the error:

2024-09-22T13:59:16.061+0200 [DEBUG] turborepo_repository::package_json: loading package.json from /var/www/html/package.json
2024-09-22T13:59:16.062+0200 [DEBUG] turborepo_telemetry::config: Telemetry config path: /home/node/.config/turborepo/telemetry.json
2024-09-22T13:59:16.062+0200 [DEBUG] turborepo_lib::daemon::connector: looking for pid in lockfile: AbsoluteSystemPathBuf("/tmp/turbod/69dd2b02e1f3a659/turbod.pid")
2024-09-22T13:59:16.063+0200 [DEBUG] turborepo_lib::daemon::connector: found pid: 67
2024-09-22T13:59:16.063+0200 [DEBUG] turborepo_lib::daemon::connector: got daemon with pid: 67
2024-09-22T13:59:16.063+0200 [DEBUG] turborepo_lib::daemon::connector: creating AbsoluteSystemPath("/tmp/turbod/69dd2b02e1f3a659")
2024-09-22T13:59:16.063+0200 [DEBUG] turborepo_lib::daemon::connector: watching AbsoluteSystemPath("/tmp/turbod/69dd2b02e1f3a659")
2024-09-22T13:59:16.063+0200 [DEBUG] turborepo_lib::daemon::connector: creating AbsoluteSystemPath("/tmp/turbod/69dd2b02e1f3a659")
2024-09-22T13:59:16.063+0200 [DEBUG] turborepo_lib::daemon::connector: watching AbsoluteSystemPath("/tmp/turbod/69dd2b02e1f3a659")
2024-09-22T13:59:16.064+0200 [DEBUG] turborepo_lib::daemon::connector: connecting to socket: /tmp/turbod/69dd2b02e1f3a659/turbod.sock
2024-09-22T13:59:16.064+0200 [DEBUG] tower::buffer::worker: "processing request"
2024-09-22T13:59:16.065+0200 [DEBUG] turborepo_lib::daemon::connector: connected in 2498µs
2024-09-22T13:59:16.065+0200 [DEBUG] turborepo_lib::run::builder: running in daemon mode
2024-09-22T13:59:16.065+0200 [DEBUG] turborepo_repository::discovery: discovering packages using caching strategy
2024-09-22T13:59:16.065+0200 [DEBUG] turborepo_repository::discovery: discovering packages using primary strategy
2024-09-22T13:59:16.065+0200 [DEBUG] turborepo_lib::run::package_discovery: discovering packages using daemon
2024-09-22T13:59:16.065+0200 [DEBUG] tower::buffer::worker: "processing request"
2024-09-22T13:59:16.097+0200 [DEBUG] tower::buffer::worker: buffer closing; waking pending tasks
  x discovery failed: bad grpc status code: The operation was cancelled

Not sure why is this happening.

[trekie86]

Yet another +1 on MacOS case-sensitive volume. I move it to another location and works just fine.