Incremental compilation doesn't work in dropbox

[aetilley]

I'm sorry, I don't have time to create a minimal example for this, but I will keep the following commit unsquashed for a little while anyway:
aetilley/cool_rust@29e7613

Meta

rustc --version --verbose:

rustc 1.83.0 (90b35a623 2024-11-26)
binary: rustc
commit-hash: 90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf
commit-date: 2024-11-26
host: aarch64-apple-darwin
release: 1.83.0
LLVM version: 19.1.1

Error output

arthurtilley@Arthurs-MacBook-Pro-5 cool_rust % cargo check
    Checking cool_rust v0.1.0 (/Users/arthurtilley/Library/CloudStorage/Dropbox/PersonalDrop/DropDev/cool_rust)
thread 'rustc' panicked at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/compiler/rustc_middle/src/query/on_disk_cache.rs:520:5:
assertion `left == right` failed
  left: 4
 right: 0
stack backtrace:
   0:        0x11060b0ac - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h7adefaad4a31afc0
   1:        0x10da6ebf4 - core::fmt::write::h381c0b0ce6ab972a
   2:        0x1105ff040 - std::io::Write::write_fmt::h75af97148630d8d3
   3:        0x11060af6c - std::sys::backtrace::BacktraceLock::print::h8baf33e22611de71
   4:        0x11060d440 - std::panicking::default_hook::{{closure}}::h369c7295ef58c5b1
   5:        0x11060d28c - std::panicking::default_hook::h50746358288a9d6a
   6:        0x10e5fb6f8 - <alloc[9e098254554c5838]::boxed::Box<rustc_driver_impl[2ba876bd46efdf49]::install_ice_hook::{closure#0}> as core[6ba870000645607f]::ops::function::Fn<(&dyn for<'a, 'b> core[6ba870000645607f]::ops::function::Fn<(&'a std[18ac8b95e3b389da]::panic::PanicHookInfo<'b>,), Output = ()> + core[6ba870000645607f]::marker::Send + core[6ba870000645607f]::marker::Sync, &std[18ac8b95e3b389da]::panic::PanicHookInfo)>>::call
   7:        0x11060dccc - std::panicking::rust_panic_with_hook::h7d795911432661cb
   8:        0x11060d948 - std::panicking::begin_panic_handler::{{closure}}::h36f15310ecbde379
   9:        0x11060b554 - std::sys::backtrace::__rust_end_short_backtrace::heed121414170e0c7
  10:        0x11060d60c - _rust_begin_unwind
  11:        0x112cf8b54 - core::panicking::panic_fmt::h17b1b80ec02ffd19
  12:        0x112cf8ee8 - core::panicking::assert_failed_inner::h13f6fb8338d443dd
  13:        0x112ea2588 - core[6ba870000645607f]::panicking::assert_failed::<u64, u64>
  14:        0x10fd45fd4 - <rustc_middle[5138617138d99ce9]::query::on_disk_cache::OnDiskCache>::load_indexed::<core[6ba870000645607f]::option::Option<rustc_span[1fbc2a1e28a50ad9]::def_id::DefId>>
  15:        0x10fbb5bac - rustc_query_impl[54d0809e421041ba]::plumbing::try_load_from_disk::<core[6ba870000645607f]::option::Option<rustc_span[1fbc2a1e28a50ad9]::def_id::DefId>>
  16:        0x10fc0a7b8 - <rustc_query_impl[54d0809e421041ba]::query_impl::associated_type_for_effects::dynamic_query::{closure#6} as core[6ba870000645607f]::ops::function::FnOnce<(rustc_middle[5138617138d99ce9]::ty::context::TyCtxt, &rustc_span[1fbc2a1e28a50ad9]::def_id::DefId, rustc_query_system[f55e8beb91aab0dd]::dep_graph::serialized::SerializedDepNodeIndex, rustc_query_system[f55e8beb91aab0dd]::dep_graph::graph::DepNodeIndex)>>::call_once
  17:        0x10fb222cc - rustc_query_system[f55e8beb91aab0dd]::query::plumbing::try_execute_query::<rustc_query_impl[54d0809e421041ba]::DynamicConfig<rustc_query_system[f55e8beb91aab0dd]::query::caches::DefIdCache<rustc_middle[5138617138d99ce9]::query::erase::Erased<[u8; 8usize]>>, false, false, false>, rustc_query_impl[54d0809e421041ba]::plumbing::QueryCtxt, true>
  18:        0x10fd18f80 - rustc_query_impl[54d0809e421041ba]::query_impl::associated_type_for_effects::get_query_incr::__rust_end_short_backtrace
  19:        0x110528400 - rustc_ty_utils[cb1fa69e8b366b18]::assoc::associated_item_def_ids
  20:        0x10fbbd8c4 - rustc_query_impl[54d0809e421041ba]::plumbing::__rust_begin_short_backtrace::<rustc_query_impl[54d0809e421041ba]::query_impl::associated_item_def_ids::dynamic_query::{closure#2}::{closure#0}, rustc_middle[5138617138d99ce9]::query::erase::Erased<[u8; 16usize]>>
  21:        0x10fc05788 - <rustc_query_impl[54d0809e421041ba]::query_impl::associated_item_def_ids::dynamic_query::{closure#2} as core[6ba870000645607f]::ops::function::FnOnce<(rustc_middle[5138617138d99ce9]::ty::context::TyCtxt, rustc_span[1fbc2a1e28a50ad9]::def_id::DefId)>>::call_once
  22:        0x10fb18600 - rustc_query_system[f55e8beb91aab0dd]::query::plumbing::try_execute_query::<rustc_query_impl[54d0809e421041ba]::DynamicConfig<rustc_query_system[f55e8beb91aab0dd]::query::caches::DefIdCache<rustc_middle[5138617138d99ce9]::query::erase::Erased<[u8; 16usize]>>, false, false, false>, rustc_query_impl[54d0809e421041ba]::plumbing::QueryCtxt, true>
  23:        0x10fb0e09c - rustc_query_system[f55e8beb91aab0dd]::query::plumbing::force_query::<rustc_query_impl[54d0809e421041ba]::DynamicConfig<rustc_query_system[f55e8beb91aab0dd]::query::caches::DefIdCache<rustc_middle[5138617138d99ce9]::query::erase::Erased<[u8; 16usize]>>, false, false, false>, rustc_query_impl[54d0809e421041ba]::plumbing::QueryCtxt>
  24:        0x10fb88c34 - <rustc_query_impl[54d0809e421041ba]::plumbing::query_callback<rustc_query_impl[54d0809e421041ba]::query_impl::associated_item_def_ids::QueryType>::{closure#0} as core[6ba870000645607f]::ops::function::FnOnce<(rustc_middle[5138617138d99ce9]::ty::context::TyCtxt, rustc_query_system[f55e8beb91aab0dd]::dep_graph::dep_node::DepNode)>>::call_once
  25:        0x10fc76168 - <rustc_query_system[f55e8beb91aab0dd]::dep_graph::graph::DepGraphData<rustc_middle[5138617138d99ce9]::dep_graph::DepsType>>::try_mark_previous_green::<rustc_query_impl[54d0809e421041ba]::plumbing::QueryCtxt>
  26:        0x10fc75ef8 - <rustc_query_system[f55e8beb91aab0dd]::dep_graph::graph::DepGraphData<rustc_middle[5138617138d99ce9]::dep_graph::DepsType>>::try_mark_green::<rustc_query_impl[54d0809e421041ba]::plumbing::QueryCtxt>
  27:        0x10fb12ff0 - rustc_query_system[f55e8beb91aab0dd]::query::plumbing::ensure_must_run::<rustc_query_impl[54d0809e421041ba]::DynamicConfig<rustc_query_system[f55e8beb91aab0dd]::query::caches::DefIdCache<rustc_middle[5138617138d99ce9]::query::erase::Erased<[u8; 8usize]>>, false, false, false>, rustc_query_impl[54d0809e421041ba]::plumbing::QueryCtxt>
  28:        0x10fd18490 - rustc_query_impl[54d0809e421041ba]::query_impl::associated_items::get_query_incr::__rust_end_short_backtrace
  29:        0x10e870108 - rustc_hir_analysis[5678387cd3e92ec4]::collect::lower_item
  30:        0x10e860d60 - <rustc_hir_analysis[5678387cd3e92ec4]::collect::CollectItemTypesVisitor as rustc_hir[627793020afa6ea6]::intravisit::Visitor>::visit_item
  31:        0x10e94eac0 - rustc_hir_analysis[5678387cd3e92ec4]::check::wfcheck::check_well_formed
  32:        0x10fbbbeec - rustc_query_impl[54d0809e421041ba]::plumbing::__rust_begin_short_backtrace::<rustc_query_impl[54d0809e421041ba]::query_impl::check_well_formed::dynamic_query::{closure#2}::{closure#0}, rustc_middle[5138617138d99ce9]::query::erase::Erased<[u8; 1usize]>>
  33:        0x10fbfbff8 - <rustc_query_impl[54d0809e421041ba]::query_impl::check_well_formed::dynamic_query::{closure#2} as core[6ba870000645607f]::ops::function::FnOnce<(rustc_middle[5138617138d99ce9]::ty::context::TyCtxt, rustc_span[1fbc2a1e28a50ad9]::def_id::LocalDefId)>>::call_once
  34:        0x10fb67ad4 - rustc_query_system[f55e8beb91aab0dd]::query::plumbing::try_execute_query::<rustc_query_impl[54d0809e421041ba]::DynamicConfig<rustc_query_system[f55e8beb91aab0dd]::query::caches::VecCache<rustc_span[1fbc2a1e28a50ad9]::def_id::LocalDefId, rustc_middle[5138617138d99ce9]::query::erase::Erased<[u8; 1usize]>>, false, false, false>, rustc_query_impl[54d0809e421041ba]::plumbing::QueryCtxt, true>
  35:        0x10fd2f8a0 - rustc_query_impl[54d0809e421041ba]::query_impl::check_well_formed::get_query_incr::__rust_end_short_backtrace
  36:        0x10e89820c - <rustc_data_structures[4ea10da2abf0fcab]::sync::parallel::ParallelGuard>::run::<core[6ba870000645607f]::result::Result<(), rustc_span[1fbc2a1e28a50ad9]::ErrorGuaranteed>, rustc_data_structures[4ea10da2abf0fcab]::sync::parallel::disabled::try_par_for_each_in<&[rustc_hir[627793020afa6ea6]::hir::ItemId], rustc_span[1fbc2a1e28a50ad9]::ErrorGuaranteed, <rustc_middle[5138617138d99ce9]::hir::ModuleItems>::par_items<rustc_hir_analysis[5678387cd3e92ec4]::check::wfcheck::check_mod_type_wf::{closure#0}>::{closure#0}>::{closure#0}::{closure#0}::{closure#0}>
  37:        0x10e95774c - rustc_hir_analysis[5678387cd3e92ec4]::check::wfcheck::check_mod_type_wf
  38:        0x10fbbbec8 - rustc_query_impl[54d0809e421041ba]::plumbing::__rust_begin_short_backtrace::<rustc_query_impl[54d0809e421041ba]::query_impl::check_mod_type_wf::dynamic_query::{closure#2}::{closure#0}, rustc_middle[5138617138d99ce9]::query::erase::Erased<[u8; 1usize]>>
  39:        0x10fbfbe30 - <rustc_query_impl[54d0809e421041ba]::query_impl::check_mod_type_wf::dynamic_query::{closure#2} as core[6ba870000645607f]::ops::function::FnOnce<(rustc_middle[5138617138d99ce9]::ty::context::TyCtxt, rustc_span[1fbc2a1e28a50ad9]::def_id::LocalModDefId)>>::call_once
  40:        0x10fb452d0 - rustc_query_system[f55e8beb91aab0dd]::query::plumbing::try_execute_query::<rustc_query_impl[54d0809e421041ba]::DynamicConfig<rustc_query_system[f55e8beb91aab0dd]::query::caches::DefaultCache<rustc_span[1fbc2a1e28a50ad9]::def_id::LocalModDefId, rustc_middle[5138617138d99ce9]::query::erase::Erased<[u8; 1usize]>>, false, false, false>, rustc_query_impl[54d0809e421041ba]::plumbing::QueryCtxt, true>
  41:        0x10fd1d13c - rustc_query_impl[54d0809e421041ba]::query_impl::check_mod_type_wf::get_query_incr::__rust_end_short_backtrace
  42:        0x10e899004 - <rustc_data_structures[4ea10da2abf0fcab]::sync::parallel::ParallelGuard>::run::<(), rustc_data_structures[4ea10da2abf0fcab]::sync::parallel::disabled::par_for_each_in<&[rustc_hir[627793020afa6ea6]::hir_id::OwnerId], <rustc_middle[5138617138d99ce9]::hir::map::Map>::par_for_each_module<rustc_hir_analysis[5678387cd3e92ec4]::check_crate::{closure#0}::{closure#0}>::{closure#0}>::{closure#0}::{closure#0}::{closure#0}>
  43:        0x10e965e18 - <rustc_session[b1216d5a4a65f438]::session::Session>::time::<(), rustc_hir_analysis[5678387cd3e92ec4]::check_crate::{closure#0}>
  44:        0x10e8d6214 - rustc_hir_analysis[5678387cd3e92ec4]::check_crate
  45:        0x10ef434b0 - rustc_interface[b7ee42a0f694265f]::passes::analysis
  46:        0x10fbbf2fc - rustc_query_impl[54d0809e421041ba]::plumbing::__rust_begin_short_backtrace::<rustc_query_impl[54d0809e421041ba]::query_impl::analysis::dynamic_query::{closure#2}::{closure#0}, rustc_middle[5138617138d99ce9]::query::erase::Erased<[u8; 1usize]>>
  47:        0x10fc0e5b4 - <rustc_query_impl[54d0809e421041ba]::query_impl::analysis::dynamic_query::{closure#2} as core[6ba870000645607f]::ops::function::FnOnce<(rustc_middle[5138617138d99ce9]::ty::context::TyCtxt, ())>>::call_once
  48:        0x10fb2805c - rustc_query_system[f55e8beb91aab0dd]::query::plumbing::try_execute_query::<rustc_query_impl[54d0809e421041ba]::DynamicConfig<rustc_query_system[f55e8beb91aab0dd]::query::caches::SingleCache<rustc_middle[5138617138d99ce9]::query::erase::Erased<[u8; 1usize]>>, false, false, false>, rustc_query_impl[54d0809e421041ba]::plumbing::QueryCtxt, true>
  49:        0x10fd0be80 - rustc_query_impl[54d0809e421041ba]::query_impl::analysis::get_query_incr::__rust_end_short_backtrace
  50:        0x10e64aca0 - <rustc_middle[5138617138d99ce9]::ty::context::GlobalCtxt>::enter::<rustc_driver_impl[2ba876bd46efdf49]::run_compiler::{closure#0}::{closure#1}::{closure#5}, core[6ba870000645607f]::result::Result<(), rustc_span[1fbc2a1e28a50ad9]::ErrorGuaranteed>>
  51:        0x10e64a3c8 - <rustc_interface[b7ee42a0f694265f]::interface::Compiler>::enter::<rustc_driver_impl[2ba876bd46efdf49]::run_compiler::{closure#0}::{closure#1}, core[6ba870000645607f]::result::Result<core[6ba870000645607f]::option::Option<rustc_interface[b7ee42a0f694265f]::queries::Linker>, rustc_span[1fbc2a1e28a50ad9]::ErrorGuaranteed>>
  52:        0x10e6408b4 - rustc_span[1fbc2a1e28a50ad9]::create_session_globals_then::<core[6ba870000645607f]::result::Result<(), rustc_span[1fbc2a1e28a50ad9]::ErrorGuaranteed>, rustc_interface[b7ee42a0f694265f]::util::run_in_thread_with_globals<rustc_interface[b7ee42a0f694265f]::interface::run_compiler<core[6ba870000645607f]::result::Result<(), rustc_span[1fbc2a1e28a50ad9]::ErrorGuaranteed>, rustc_driver_impl[2ba876bd46efdf49]::run_compiler::{closure#0}>::{closure#1}, core[6ba870000645607f]::result::Result<(), rustc_span[1fbc2a1e28a50ad9]::ErrorGuaranteed>>::{closure#0}::{closure#0}::{closure#0}>
  53:        0x10e6300ac - std[18ac8b95e3b389da]::sys::backtrace::__rust_begin_short_backtrace::<rustc_interface[b7ee42a0f694265f]::util::run_in_thread_with_globals<rustc_interface[b7ee42a0f694265f]::interface::run_compiler<core[6ba870000645607f]::result::Result<(), rustc_span[1fbc2a1e28a50ad9]::ErrorGuaranteed>, rustc_driver_impl[2ba876bd46efdf49]::run_compiler::{closure#0}>::{closure#1}, core[6ba870000645607f]::result::Result<(), rustc_span[1fbc2a1e28a50ad9]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[6ba870000645607f]::result::Result<(), rustc_span[1fbc2a1e28a50ad9]::ErrorGuaranteed>>
  54:        0x10e647f48 - <<std[18ac8b95e3b389da]::thread::Builder>::spawn_unchecked_<rustc_interface[b7ee42a0f694265f]::util::run_in_thread_with_globals<rustc_interface[b7ee42a0f694265f]::interface::run_compiler<core[6ba870000645607f]::result::Result<(), rustc_span[1fbc2a1e28a50ad9]::ErrorGuaranteed>, rustc_driver_impl[2ba876bd46efdf49]::run_compiler::{closure#0}>::{closure#1}, core[6ba870000645607f]::result::Result<(), rustc_span[1fbc2a1e28a50ad9]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[6ba870000645607f]::result::Result<(), rustc_span[1fbc2a1e28a50ad9]::ErrorGuaranteed>>::{closure#1} as core[6ba870000645607f]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
  55:        0x110618050 - std::sys::pal::unix::thread::Thread::new::thread_start::h44ebc83f14222f58
  56:        0x1989d1f94 - __pthread_joiner_wake

error: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.83.0 (90b35a623 2024-11-26) running on aarch64-apple-darwin

note: compiler flags: --crate-type lib -C embed-bitcode=no -C debuginfo=2 -C split-debuginfo=unpacked -C incremental=[REDACTED]

note: some of the compiler flags provided by cargo are hidden

query stack during panic:

thread 'rustc' has overflowed its stack
fatal runtime error: stack overflow
error: could not compile `cool_rust` (lib)

Caused by:
  process didn't exit successfully: `/Users/arthurtilley/.rustup/toolchains/stable-aarch64-apple-darwin/bin/rustc --crate-name cool_rust --edition=2021 src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=173 --crate-type lib --emit=dep-info,metadata -C embed-bitcode=no -C debuginfo=2 -C split-debuginfo=unpacked --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values())' -C metadata=a633a5e12806e4c7 -C extra-filename=-a633a5e12806e4c7 --out-dir /Users/arthurtilley/Library/CloudStorage/Dropbox/PersonalDrop/DropDev/cool_rust/target/debug/deps -C incremental=/Users/arthurtilley/Library/CloudStorage/Dropbox/PersonalDrop/DropDev/cool_rust/target/debug/incremental -L dependency=/Users/arthurtilley/Library/CloudStorage/Dropbox/PersonalDrop/DropDev/cool_rust/target/debug/deps --extern lalrpop_util=/Users/arthurtilley/Library/CloudStorage/Dropbox/PersonalDrop/DropDev/cool_rust/target/debug/deps/liblalrpop_util-17b04e3143ce965c.rmeta --extern logos=/Users/arthurtilley/Library/CloudStorage/Dropbox/PersonalDrop/DropDev/cool_rust/target/debug/deps/liblogos-d4e436ddf730bfdc.rmeta` (signal: 6, SIGABRT: process abort signal)
arthurtilley@Arthurs-MacBook-Pro-5 cool_rust % 

Backtrace

<backtrace>

[aetilley]

Note, updating to rustc 1.85.0 (4d91de4 2025-02-17) seems to fix it.

[EDIT]. Actually no, I started working on it a little more after the update and I crashed again. Resubmitted with the new meta at #138203

[kpreid]

Note, updating to rustc 1.85.0 (4d91de4 2025-02-17) seems to fix it.

What's going on here is that clearing the cache is a good way to hide bugs involving reading from cache (and updating the compiler effectively does that). Which makes these bugs nastily hard to reproduce — a working set of reproduction steps will need to involve something like "compile this code, make this 1-line change, compile it again, see ICE".

[aetilley]

(Included backtrace in the new issue (#138203). Note that there seems to be a difference between cargo build and cargo check. Only the latter produces the crash.

[saethlin]

Only the latter produces the crash.

That's because there are separate artifacts for build and check. Only one set has been corrupted.

Based on the fact that you are running into a lot of crashes on an uncomplicated codebase while trying to do this all in dropbox, I am inclined to blame dropbox. The incremental compilation system relies on hard links and file locking being implemented correctly, and if they are not, the variety of crashes you're getting is what would result.

[Noratrieb]

Builds generally don't like being run on a cloud file system, no matter which language or cloud file system.
I recommend you move your development directories to a local file system.
You can periodically manually copy your directories to Dropbox (where you can then also remove the target directory) or use a dedicated code hosting platform like GitHub.

[aetilley]

I did not know that.

Well I moved the project out of Dropbox and haven't seen the crash again, so that was probably the issue.

[Noratrieb]

Maybe it would make sense to document it more clearly that weird filesystems are not supported by incremental compilation. So we have at least something to point to that explains it in more detail

[saethlin]

The problem there would be communicating to the people who actually need to know that weird filesystems may not work, and also defining what weird means. I'm sure people successfully use incr comp on all sorts of filesystems and we just don't hear about it because why would we.