encountered incremental compilation error with evaluate_obligation(d87a56c5dc6db6e-e9735b6f63d47303)

[scovich]

Summary

Encountered the following while working on a very large local change. It is reproducible in the sense that applying the following change triggers the crash and backing out the change avoids it:

  impl Channel {
-     async fn exchange<Out: Send + 'static, In: Send + 'static, P>(
+     async fn exchange<Out: Send + 'static, In: Send + 'static, P: Into<Request>>(
          &self,
          outbound: Out,
          make_request: impl FnOnce(Out, Reply<In>) -> P,
-     ) -> DeltaResult<In>
-     where
-         P: Into<Request>,
-     {
+     ) -> DeltaResult<In> {
          self.0
              .request(|reply| make_request(outbound, reply).into())
              .await
      }
  }

I've got an AI agent trying to minimize the repro in a separate worktree so I can keep coding. I'll post back here if they get something shareable.

Version

clippy 0.1.94 (4a4ef493e3 2026-03-02)

Error output

Backtrace

-*- mode: compilation; default-directory: "~/duckdb_delta/delta-kernel-rs/.worktrees/kernel-coroutines/" -*-
Compilation started at Wed Sep 16 13:33:01

cd ~/duckdb_delta/delta-kernel-rs/.worktrees/kernel-coroutines/ && cargo clippy --workspace --all-features --tests --benches
  Blocking waiting for file lock on build directory
  Checking delta_kernel v0.28.0 (/Users/ryan.johnson/duckdb_delta/delta-kernel-rs/.worktrees/kernel-coroutines/kernel)
error: internal compiler error: encountered incremental compilation error with evaluate_obligation(d87a56c5dc6db6e-e9735b6f63d47303)
|
= note: please follow the instructions below to create a bug report with the provided information
= note: for incremental compilation bugs, having a reproduction is vital
= note: an ideal reproduction consists of the code before and some patch that then triggers the bug when applied and compiled again
= note: as a workaround, you can run `cargo clean -p delta_kernel` or `cargo clean` to allow your project to compile


thread 'rustc' (74258840) panicked at /rustc-dev/4a4ef493e3a1488c6e321570238084b38948f6db/compiler/rustc_query_system/src/query/plumbing.rs:743:9:
Found unstable fingerprints for evaluate_obligation(d87a56c5dc6db6e-e9735b6f63d47303): Ok(EvaluatedToOkModuloRegions)
stack backtrace:
 0:        0x114147634 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::hcde33371d8bba08b
 1:        0x1113981fc - core::fmt::write::h8a52dffc4df6d016
 2:        0x114101260 - std::io::Write::write_fmt::h2fd0abc6933d662d
 3:        0x114119ea4 - std::panicking::default_hook::{{closure}}::h1af7615be47c43ab
 4:        0x114119dac - std::panicking::default_hook::hdcc779fdbf67ae29
 5:        0x111fe3378 - std[b190ad38c85ed59a]::panicking::update_hook::<alloc[cec09d4d3689febe]::boxed::Box<rustc_driver_impl[185e3967a9919f9d]::install_ice_hook::{closure#1}>>::{closure#0}
 6:        0x11411a2d0 - std::panicking::panic_with_hook::h455d3ea9bab4bcbd
 7:        0x114119f4c - std::panicking::panic_handler::{{closure}}::h2068a0d88ec25e56
 8:        0x1141139e4 - std::sys::backtrace::__rust_end_short_backtrace::h7bea34f9056001ee
 9:        0x1140f764c - __rustc[4425a7e20b4c8619]::rust_begin_unwind
10:        0x116e7f120 - core::panicking::panic_fmt::hc9b4228954207328
11:        0x116f58380 - rustc_query_system[23a19f823f15ce6c]::query::plumbing::incremental_verify_ich_failed::<rustc_middle[1dd1fc2248f63b22]::ty::context::TyCtxt>
12:        0x113619c6c - rustc_query_system[23a19f823f15ce6c]::query::plumbing::incremental_verify_ich::<rustc_middle[1dd1fc2248f63b22]::ty::context::TyCtxt, rustc_middle[1dd1fc2248f63b22]::query::erase::Erased<[u8; 2usize]>>
13:        0x1135c8348 - rustc_query_system[23a19f823f15ce6c]::query::plumbing::try_execute_query::<rustc_query_impl[def6bb11db1f9526]::DynamicConfig<rustc_query_system[23a19f823f15ce6c]::query::caches::DefaultCache<rustc_type_ir[2c11913eb1716827]::canonical::CanonicalQueryInput<rustc_middle[1dd1fc2248f63b22]::ty::context::TyCtxt, rustc_middle[1dd1fc2248f63b22]::ty::ParamEnvAnd<rustc_middle[1dd1fc2248f63b22]::ty::predicate::Predicate>>, rustc_middle[1dd1fc2248f63b22]::query::erase::Erased<[u8; 2usize]>>, false, false, false>, rustc_query_impl[def6bb11db1f9526]::plumbing::QueryCtxt, true>
14:        0x1136ed634 - rustc_query_impl[def6bb11db1f9526]::query_impl::evaluate_obligation::get_query_incr::__rust_end_short_backtrace
15:        0x113e585e4 - <rustc_infer[6c4201f777a84977]::infer::InferCtxt as rustc_trait_selection[e1fbad59407fbe56]::traits::query::evaluate_obligation::InferCtxtExt>::evaluate_obligation
16:        0x113e589d0 - <rustc_infer[6c4201f777a84977]::infer::InferCtxt as rustc_trait_selection[e1fbad59407fbe56]::traits::query::evaluate_obligation::InferCtxtExt>::evaluate_obligation_no_overflow
17:        0x113dd37bc - <rustc_trait_selection[e1fbad59407fbe56]::traits::fulfill::FulfillProcessor>::process_trait_obligation
18:        0x113e744cc - <rustc_trait_selection[e1fbad59407fbe56]::traits::fulfill::FulfillProcessor as rustc_data_structures[fafc8b92dea7f144]::obligation_forest::ObligationProcessor>::process_obligation
19:        0x113c39a28 - <rustc_data_structures[fafc8b92dea7f144]::obligation_forest::ObligationForest<rustc_trait_selection[e1fbad59407fbe56]::traits::fulfill::PendingPredicateObligation>>::process_obligations::<rustc_trait_selection[e1fbad59407fbe56]::traits::fulfill::FulfillProcessor>
20:        0x113e66acc - <rustc_trait_selection[e1fbad59407fbe56]::traits::fulfill::FulfillmentContext<rustc_infer[6c4201f777a84977]::traits::engine::ScrubbedTraitError> as rustc_infer[6c4201f777a84977]::traits::engine::TraitEngine<rustc_infer[6c4201f777a84977]::traits::engine::ScrubbedTraitError>>::try_evaluate_obligations
21:        0x113f01704 - <rustc_trait_selection[e1fbad59407fbe56]::traits::fulfill::FulfillmentContext<rustc_infer[6c4201f777a84977]::traits::engine::ScrubbedTraitError> as rustc_infer[6c4201f777a84977]::traits::engine::TraitEngine<rustc_infer[6c4201f777a84977]::traits::engine::ScrubbedTraitError>>::evaluate_obligations_error_on_ambiguity
22:        0x113f19294 - <rustc_trait_selection[e1fbad59407fbe56]::traits::engine::ObligationCtxt>::make_canonicalized_query_response::<()>
23:        0x113f8a714 - rustc_traits[1771b8103676f9bd]::type_op::type_op_prove_predicate
24:        0x1135816cc - rustc_query_impl[def6bb11db1f9526]::plumbing::__rust_begin_short_backtrace::<rustc_query_impl[def6bb11db1f9526]::query_impl::type_op_prove_predicate::dynamic_query::{closure#2}::{closure#0}, rustc_middle[1dd1fc2248f63b22]::query::erase::Erased<[u8; 8usize]>>
25:        0x11386e1cc - <rustc_query_impl[def6bb11db1f9526]::query_impl::type_op_prove_predicate::dynamic_query::{closure#2} as core[a936a58af82ee723]::ops::function::FnOnce<(rustc_middle[1dd1fc2248f63b22]::ty::context::TyCtxt, rustc_type_ir[2c11913eb1716827]::canonical::CanonicalQueryInput<rustc_middle[1dd1fc2248f63b22]::ty::context::TyCtxt, rustc_middle[1dd1fc2248f63b22]::ty::ParamEnvAnd<rustc_middle[1dd1fc2248f63b22]::traits::query::type_op::ProvePredicate>>)>>::call_once
26:        0x1135cc4d4 - rustc_query_system[23a19f823f15ce6c]::query::plumbing::try_execute_query::<rustc_query_impl[def6bb11db1f9526]::DynamicConfig<rustc_query_system[23a19f823f15ce6c]::query::caches::DefaultCache<rustc_type_ir[2c11913eb1716827]::canonical::CanonicalQueryInput<rustc_middle[1dd1fc2248f63b22]::ty::context::TyCtxt, rustc_middle[1dd1fc2248f63b22]::ty::ParamEnvAnd<rustc_middle[1dd1fc2248f63b22]::traits::query::type_op::ProvePredicate>>, rustc_middle[1dd1fc2248f63b22]::query::erase::Erased<[u8; 8usize]>>, false, false, false>, rustc_query_impl[def6bb11db1f9526]::plumbing::QueryCtxt, true>
27:        0x1136fad90 - rustc_query_impl[def6bb11db1f9526]::query_impl::type_op_prove_predicate::get_query_incr::__rust_end_short_backtrace
28:        0x113e5973c - <rustc_middle[1dd1fc2248f63b22]::traits::query::type_op::ProvePredicate as rustc_trait_selection[e1fbad59407fbe56]::traits::query::type_op::QueryTypeOp>::perform_query
29:        0x1117fb6e0 - <rustc_borrowck[9cea798e70066837]::type_check::TypeChecker>::fully_perform_op::<(), rustc_middle[1dd1fc2248f63b22]::ty::ParamEnvAnd<rustc_middle[1dd1fc2248f63b22]::traits::query::type_op::ProvePredicate>>
30:        0x11194d1b4 - <rustc_borrowck[9cea798e70066837]::type_check::TypeChecker>::normalize_and_prove_instantiated_predicates
31:        0x111a2b130 - <rustc_borrowck[9cea798e70066837]::type_check::TypeChecker as rustc_middle[1dd1fc2248f63b22]::mir::visit::Visitor>::visit_const_operand
32:        0x111a25f4c - <rustc_borrowck[9cea798e70066837]::type_check::TypeChecker as rustc_middle[1dd1fc2248f63b22]::mir::visit::Visitor>::visit_operand
33:        0x111a27ba4 - <rustc_borrowck[9cea798e70066837]::type_check::TypeChecker as rustc_middle[1dd1fc2248f63b22]::mir::visit::Visitor>::visit_terminator
34:        0x111a218ac - <rustc_borrowck[9cea798e70066837]::type_check::TypeChecker as rustc_middle[1dd1fc2248f63b22]::mir::visit::Visitor>::visit_body
35:        0x1119cce64 - rustc_borrowck[9cea798e70066837]::type_check::type_check
36:        0x11193fa94 - <rustc_borrowck[9cea798e70066837]::root_cx::BorrowCheckRootCtxt>::do_mir_borrowck
37:        0x11192a6dc - rustc_borrowck[9cea798e70066837]::mir_borrowck
38:        0x11357d0a0 - rustc_query_impl[def6bb11db1f9526]::plumbing::__rust_begin_short_backtrace::<rustc_query_impl[def6bb11db1f9526]::query_impl::mir_borrowck::dynamic_query::{closure#2}::{closure#0}, rustc_middle[1dd1fc2248f63b22]::query::erase::Erased<[u8; 8usize]>>
39:        0x1136132f4 - rustc_query_system[23a19f823f15ce6c]::query::plumbing::try_execute_query::<rustc_query_impl[def6bb11db1f9526]::DynamicConfig<rustc_data_structures[fafc8b92dea7f144]::vec_cache::VecCache<rustc_span[40bc6320bb32000f]::def_id::LocalDefId, rustc_middle[1dd1fc2248f63b22]::query::erase::Erased<[u8; 8usize]>, rustc_query_system[23a19f823f15ce6c]::dep_graph::graph::DepNodeIndex>, false, false, false>, rustc_query_impl[def6bb11db1f9526]::plumbing::QueryCtxt, true>
40:        0x1136d8330 - rustc_query_impl[def6bb11db1f9526]::query_impl::mir_borrowck::get_query_incr::__rust_end_short_backtrace
41:        0x11290c118 - <rustc_middle[1dd1fc2248f63b22]::ty::context::TyCtxt>::par_hir_body_owners::<rustc_interface[8731d2cac67c9b74]::passes::run_required_analyses::{closure#1}::{closure#0}>::{closure#0}
42:        0x112954040 - rustc_interface[8731d2cac67c9b74]::passes::analysis
43:        0x113582e98 - rustc_query_impl[def6bb11db1f9526]::plumbing::__rust_begin_short_backtrace::<rustc_query_impl[def6bb11db1f9526]::query_impl::analysis::dynamic_query::{closure#2}::{closure#0}, rustc_middle[1dd1fc2248f63b22]::query::erase::Erased<[u8; 0usize]>>
44:        0x11359c5f4 - rustc_query_system[23a19f823f15ce6c]::query::plumbing::try_execute_query::<rustc_query_impl[def6bb11db1f9526]::DynamicConfig<rustc_query_system[23a19f823f15ce6c]::query::caches::SingleCache<rustc_middle[1dd1fc2248f63b22]::query::erase::Erased<[u8; 0usize]>>, false, false, false>, rustc_query_impl[def6bb11db1f9526]::plumbing::QueryCtxt, true>
45:        0x11370a298 - rustc_query_impl[def6bb11db1f9526]::query_impl::analysis::get_query_incr::__rust_end_short_backtrace
46:        0x111f96b40 - rustc_interface[8731d2cac67c9b74]::passes::create_and_enter_global_ctxt::<core[a936a58af82ee723]::option::Option<rustc_interface[8731d2cac67c9b74]::queries::Linker>, rustc_driver_impl[185e3967a9919f9d]::run_compiler::{closure#0}::{closure#2}>
47:        0x111fe1a9c - rustc_interface[8731d2cac67c9b74]::interface::run_compiler::<(), rustc_driver_impl[185e3967a9919f9d]::run_compiler::{closure#0}>::{closure#1}
48:        0x111fd7d9c - std[b190ad38c85ed59a]::sys::backtrace::__rust_begin_short_backtrace::<rustc_interface[8731d2cac67c9b74]::util::run_in_thread_with_globals<rustc_interface[8731d2cac67c9b74]::util::run_in_thread_pool_with_globals<rustc_interface[8731d2cac67c9b74]::interface::run_compiler<(), rustc_driver_impl[185e3967a9919f9d]::run_compiler::{closure#0}>::{closure#1}, ()>::{closure#0}, ()>::{closure#0}::{closure#0}, ()>
49:        0x111fe8ff4 - <std[b190ad38c85ed59a]::thread::lifecycle::spawn_unchecked<rustc_interface[8731d2cac67c9b74]::util::run_in_thread_with_globals<rustc_interface[8731d2cac67c9b74]::util::run_in_thread_pool_with_globals<rustc_interface[8731d2cac67c9b74]::interface::run_compiler<(), rustc_driver_impl[185e3967a9919f9d]::run_compiler::{closure#0}>::{closure#1}, ()>::{closure#0}, ()>::{closure#0}::{closure#0}, ()>::{closure#1} as core[a936a58af82ee723]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
50:        0x114110530 - std::sys::thread::unix::Thread::new::thread_start::hb5c7d2a38fab9e31
51:        0x18eeb1c58 - __pthread_cond_wait

error: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust-clippy/issues/new?template=ice.yml

note: rustc 1.94.0 (4a4ef493e 2026-03-02) running on aarch64-apple-darwin

note: compiler flags: --crate-type lib -C embed-bitcode=no -C debuginfo=2 -C split-debuginfo=unpacked -C incremental=[REDACTED]

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
#0 [evaluate_obligation] evaluating trait selection obligation `coroutine::kernel::workflow::WorkflowImpl<{async closure@kernel/src/checkpoint/mod.rs:384:31: 384:51}, checkpoint::CheckpointWriter>: coroutine::kernel::workflow::Workflow`
#1 [type_op_prove_predicate] evaluating `type_op_prove_predicate` `ProvePredicate { predicate: Binder { value: TraitPredicate(<coroutine::kernel::workflow::WorkflowImpl<{async closure@kernel/src/checkpoint/mod.rs:384:31: 384:51}, checkpoint::CheckpointWriter> as coroutine::kernel::workflow::Workflow>, polarity:Positive), bound_vars: [] } }`
... and 2 other queries... use `env RUST_BACKTRACE=1` to see the full query stack
note: Clippy version: clippy 0.1.94 (4a4ef493e3 2026-03-02)

error: internal compiler error: reentrant incremental verify failure, suppressing message

error: could not compile `delta_kernel` (lib) due to 2 previous errors

Compilation exited abnormally with code 101 at Wed Sep 16 13:33:07

[scovich]

The agent is not off to a good start...

OpenAI flagged this request for potential high-risk cybersecurity activity. Please try a less sensitive prompt. Learn more here.

[scovich]

Minimal repro here:

Details

//! Incremental ICE on rustc 1.94.0: moving `P: Into<Request>` from a `where`
//! clause onto the generic parameter leaves an unstable
//! `evaluate_obligation` fingerprint (`EvaluatedToOkModuloRegions`) for
//! `WorkflowImpl<{async closure}, u8>: Workflow`.
//!
//! ```text
//! rustc --edition 2021 --crate-type lib -C incremental=incr src/lib.rs
//! rustc --edition 2021 --crate-type lib -C incremental=incr --cfg post src/lib.rs
//! ```

#![allow(dead_code, unexpected_cfgs)]

use std::future::Future;
use std::marker::PhantomData;

struct Reply<In>(PhantomData<fn() -> In>);

enum Request {
    Read(u8, Reply<u8>),
}

struct Channel;

impl Channel {
    async fn request<In>(&self, make_request: impl FnOnce(Reply<In>) -> Request) -> Result<In, ()> {
        let _ = make_request;
        Err(())
    }

    #[cfg(not(post))]
    async fn exchange<Out, In, P>(
        &self,
        outbound: Out,
        make_request: impl FnOnce(Out, Reply<In>) -> P,
    ) -> Result<In, ()>
    where
        P: Into<Request>,
    {
        self.request(|reply| make_request(outbound, reply).into())
            .await
    }

    #[cfg(post)]
    async fn exchange<Out, In, P: Into<Request>>(
        &self,
        outbound: Out,
        make_request: impl FnOnce(Out, Reply<In>) -> P,
    ) -> Result<In, ()> {
        self.request(|reply| make_request(outbound, reply).into())
            .await
    }

    async fn read(&self) -> Result<u8, ()> {
        self.exchange(0u8, Request::Read).await
    }
}

struct WorkflowImpl<F, O>(PhantomData<(F, O)>);

impl<F, O> WorkflowImpl<F, O> {
    fn new(_: F) -> Self {
        Self(PhantomData)
    }
}

trait Workflow: Send + Sized + 'static {
    type Output: Send + 'static;
}

impl<F, O> Workflow for WorkflowImpl<F, O>
where
    F: Send + 'static + for<'a> RunWithLifetime<'a, O>,
    O: Send + 'static,
{
    type Output = O;
}

trait RunWithLifetime<'a, O> {
    type Future: Future<Output = Result<O, ()>> + Send + 'a;
}

impl<'a, F, Fut, O> RunWithLifetime<'a, O> for F
where
    F: FnOnce(&'a Channel) -> Fut,
    Fut: Future<Output = Result<O, ()>> + Send + 'a,
{
    type Future = Fut;
}

fn drive<W: Workflow>(_: W) -> Result<W::Output, ()> {
    Err(())
}

pub fn try_new() -> Result<u8, ()> {
    drive(WorkflowImpl::new(async |channel: &Channel| {
        channel.read().await
    }))
}

[scovich]

Repro also triggers ICE on

• 1.97.1 8bab26f4f (2026-07-14)
• 1.98.1 (2026-09-01)

But it does not trigger on

• 1.100.0-nightly (215a8af4b, 2026-09-15)

[CommanderStorm]

I think this is another instance of rust-lang/rust#140793

We are in luck that this will be (at least the issue is labeled by this) fixed by the next solver 🎉
All hail the next solver!

See https://blog.rust-lang.org/2026/08/21/enabling-next-solver-on-nightly/ for more context