refactor(tests): adding KMS test material generator with isolated tests infrastructure

[maksymsur]

Description of changes

This PR replaces Docker Compose-based integration tests with native isolated tests that run directly without Docker, improving test speed, reliability, and developer experience.

What Changed

Test Migration - Complete List

Migrates all Docker-based tests to native isolated execution:

1. Core Service Library Tests (core/service/src/client/tests/)

Centralized Isolated Tests (5 tests):

• test_central_health_endpoint_availability_isolated - Health endpoint verification
• test_central_close_after_drop_isolated - Server shutdown on drop
• test_largecipher_isolated - Large ciphertext handling (slow_tests)
• test_insecure_central_dkg_backup_isolated - DKG backup/restore
• test_insecure_central_autobackup_after_deletion_isolated - Auto-backup after deletion

Threshold Isolated Tests (14 tests):

• test_insecure_dkg_isolated - Insecure DKG with Test params (4 parties)
• default_insecure_dkg_isolated - Insecure DKG with Default params (4 parties)
• secure_threshold_keygen_isolated - Secure keygen with preprocessing (PRSS)
• secure_threshold_keygen_crash_online_isolated - Crash recovery during online phase
• secure_threshold_keygen_crash_preprocessing_isolated - Crash recovery during preprocessing
• test_insecure_threshold_decompression_keygen_isolated - Decompression key generation
• test_threshold_health_endpoint_availability_isolated - Health endpoint verification
• test_threshold_close_after_drop_isolated - Server shutdown on drop
• test_threshold_shutdown_isolated - Graceful shutdown
• nightly_test_insecure_threshold_dkg_backup_isolated - DKG backup/restore (nightly)
• nightly_test_insecure_threshold_autobackup_after_deletion_isolated - Auto-backup after deletion (nightly)
• test_insecure_threshold_crs_backup_isolated - CRS backup/restore
• test_threshold_restore_from_backup_isolated - Threshold backup/restore
• test_threshold_restore_from_backup_with_custodian_isolated - Custodian-based restore

Example Tests (4 tests):

• test_centralized_isolated_example - Centralized test pattern demo
• test_threshold_isolated_example - Threshold test pattern demo (4 parties)
• test_different_material_types - Material type validation
• test_material_validation - Material existence validation

2. CLI Integration Tests (core-client/tests/integration_tests.rs)

Centralized CLI Tests (4 tests):

1. test_centralized_insecure - Keygen + decryption workflow
2. test_centralized_crsgen_secure - CRS generation
3. test_centralized_restore_from_backup - Backup/restore flow
4. test_centralized_custodian_backup - Custodian backup (5 custodians)

Threshold CLI Tests (11 tests):

1. test_threshold_insecure - Keygen + decryption (PRSS-enabled, 4 parties)
2. test_threshold_concurrent_crs - Concurrent CRS generation (4 parties)
3. nightly_tests_threshold_sequential_crs - Sequential CRS generation (4 parties)
4. test_threshold_restore_from_backup - Backup/restore flow (4 parties)
5. test_threshold_custodian_backup - Custodian backup (5 custodians, 4 parties)
6. nightly_tests_threshold_sequential_preproc_keygen - Sequential preprocessing (PRSS, 4 parties)
7. test_threshold_concurrent_preproc_keygen - Concurrent preprocessing (PRSS, 4 parties)
8. full_gen_tests_default_threshold_sequential_preproc_keygen - Full keygen with Default params (PRSS, 4 parties)
9. full_gen_tests_default_threshold_sequential_crs - Full CRS with Default params (4 parties)
10. test_threshold_mpc_context_init - MPC context initialization (4 parties, renamed from test_threshold_mpc_context)
11. test_threshold_mpc_context_switch_6 - MPC context switching with 6 parties (PRSS-enabled)

Test Migration - Key Generation

Migrated all 6 threshold key generation tests to isolated infrastructure:

Migrated Tests:

1. test_insecure_dkg → test_insecure_dkg_isolated
2. default_insecure_dkg → default_insecure_dkg_isolated
3. secure_threshold_keygen_test → secure_threshold_keygen_isolated
4. secure_threshold_keygen_test_crash_online → secure_threshold_keygen_crash_online_isolated
5. secure_threshold_keygen_test_crash_preprocessing → secure_threshold_keygen_crash_preprocessing_isolated
6. test_insecure_threshold_decompression_keygen → test_insecure_threshold_decompression_keygen_isolated

File Reorganization:

• key_gen_tests.rs → key_gen_helpers.rs (renamed to reflect purpose as helper library)
• Original test functions marked as #[ignore] with migration notes
• Helper functions preserved for use by integration tests (nightly, custodian backup, reshare)

Key Features:

• Decompression key test: Generates 2 insecure keysets, then secure decompression key between them
• Crash recovery tests: Simulate party crashes during preprocessing and online phases
• PRSS support: All secure tests properly configured with run_prss: true
• Proper configuration: Uses KeySetConfig and KeySetAddedInfo for decompression keys

File Changes

Core Changes:

• core-client/tests/integration_test.rs - Deleted (Docker-based)
• core-client/tests/integration_tests.rs - Added (Native isolated, 15 CLI tests)
• core/service/src/client/tests/centralized/*_isolated.rs - Added (5 tests)
• core/service/src/client/tests/threshold/key_gen_tests_isolated.rs - Added (6 keygen tests)
• core/service/src/client/tests/threshold/key_gen_helpers.rs - Renamed from key_gen_tests.rs (helper functions only)
• core/service/src/client/tests/threshold/misc_tests_isolated.rs - Added (3 tests)
• core/service/src/client/tests/threshold/restore_from_backup_tests_isolated.rs - Added (2 tests)
• core/service/src/client/tests/threshold/misc_tests.rs - Updated (removed 2 redundant tests)
• core/service/src/client/tests/threshold/restore_from_backup_tests.rs - Deleted (all tests redundant)
• core/service/src/client/tests/threshold/mod.rs - Updated (module declarations)
• core/service/src/client/tests/threshold/common.rs - Added threshold_key_gen_secure_isolated helper
• core/service/src/client/tests/isolated_test_example.rs - Added (4 example tests)
• core/service/src/util/key_setup/test_material_spec.rs - Added (test material specification system)
• core/service/src/util/key_setup/test_material_manager.rs - Added (material copying utilities)
• tools/generate-test-material/ - Complete pre-generation tool with CLI

CI/CD Changes:

• .github/workflows/main.yml - Updated test matrix with required features
• .github/workflows/common-testing.yml - Added test material generation + kms-custodian build
• Makefile - Added test material generation and isolated test targets:
  • generate-test-material-all - Generate all material (testing + default)
  • generate-test-material-testing - Generate fast testing material only
  • generate-test-material-default - Generate production-like material
  • validate-test-material - Validate existing material
  • clean-test-material - Clean generated material
  • test-isolated - Run all isolated tests
  • test-isolated-centralized - Run centralized tests only
  • test-isolated-threshold - Run threshold tests only
  • test-isolated-integration - Run CLI integration tests
• backward-compatibility/generate-v0.13.0/Cargo.toml - Updated to current branch commit

How It Works

Test Execution Flow

Native isolated tests run KMS servers as in-process native Rust processes (no Docker), with automatic lifecycle management:

┌─────────────────────────────────────────────────────────────┐
│ Test Function Execution                                      │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  1. Setup Phase                                              │
│     setup_isolated_centralized_cli_test("my_test")          │
│     ├─ Creates TempDir with isolated material               │
│     ├─ Spawns KMS server (native Rust process)              │
│     │  └─ Server listens on 127.0.0.1:54321 (dynamic port)  │
│     └─ Generates config file pointing to server             │
│                                                              │
│  2. CLI Command Execution                                    │
│     ├─ kms-core-client --config client_config.toml          │
│     ├─ Reads config → connects to 127.0.0.1:54321           │
│     └─ Sends gRPC request to running server                 │
│                                                              │
│  3. Server Processing                                        │
│     ├─ Receives gRPC request                                │
│     ├─ Performs operation (keygen, decrypt, etc.)           │
│     └─ Returns gRPC response                                │
│                                                              │
│  4. Test Validation                                          │
│     └─ assert!(output.status.success())                     │
│                                                              │
│  5. Automatic Cleanup (RAII)                                │
│     ├─ _server dropped → server.assert_shutdown()           │
│     ├─ Sends shutdown signal → server stops                 │
│     ├─ Verifies ports closed                                │
│     └─ material_dir dropped → temp directory deleted        │
│                                                              │
└─────────────────────────────────────────────────────────────┘

Key Points:

• Native Process: KMS server runs as Rust process (not Docker container)
• Dynamic Ports: Each test gets unique port (no conflicts)
• RAII Cleanup: Server and temp directory auto-cleanup via Rust Drop trait
• Real gRPC: CLI sends actual gRPC requests to running server
• Full Isolation: Each test completely isolated (own server, own material, own temp dir)

Test Architecture

Each test uses TestMaterialManager to copy pre-generated cryptographic material into isolated temporary directories:

#[tokio::test]
async fn test_centralized_insecure() -> Result<()> {
    // Setup: Creates isolated temp dir with pre-generated material
    // test_name: "centralized_insecure" - identifier for logging/debugging
    // Returns: (TempDir, ServerHandle, PathBuf)
    let (material_dir, _server, config_path) =
        setup_isolated_centralized_cli_test("centralized_insecure").await?;
    //                                      ^^^^^^^^^^^^^^^^^^^^^^
    //                                      Test identifier (appears in logs/errors)
    //   ^^^^^^^^^^^^  ^^^^^^^ ^^^^^^^^^^^
    //   |             |       └─ PathBuf: Path to generated CLI config file
    //   |             |              Contains server address (127.0.0.1:port)
    //   |             |              Used with --config flag in CLI commands
    //   |             |
    //   |             └─ ServerHandle: Running KMS server (CRITICAL!)
    //   |                  - Server runs in background as native process
    //   |                  - Listens on dynamically allocated port
    //   |                  - Underscore prefix = "unused but must keep alive"
    //   |                  - Dropping this = server shuts down immediately
    //   |                  - Auto-shutdown via RAII when function exits
    //   |
    //   └─ TempDir: Isolated test directory (auto-cleanup on drop)
    //        - Contains all test material (keys, config, etc.)
    //        - Unique per test (no conflicts)
    //        - Automatically deleted when dropped
    
    let keys_folder = material_dir.path();  // &Path to test material directory
    
    // Test: Run actual CLI commands against running server
    // CLI connects to server via config_path (contains server address)
    let key_id = insecure_key_gen_isolated(&config_path, keys_folder).await?;
    integration_test_commands_isolated(&config_path, keys_folder, key_id).await?;
    
    // Cleanup: Automatic via RAII
    // - _server dropped → server.assert_shutdown() → server stops
    // - material_dir dropped → temp directory deleted
    Ok(())
}

Key Features:

• Native Process: KMS server runs as Rust process (not Docker container)
• Isolated Material: Each test gets its own temporary directory with test material
• Dynamic Ports: Server uses dynamically allocated port (no conflicts between parallel tests)
• RAII Cleanup: Server and temp directory automatically cleaned up via Rust Drop trait
• Real CLI Testing: CLI commands unchanged (testing actual CLI functionality against real server)

Threshold Tests (Multi-Party):

For threshold tests, the setup returns a HashMap<u32, ServerHandle> instead of a single ServerHandle:

let (material_dir, _servers, config_path) =
    setup_isolated_threshold_cli_test("threshold_test").await?;
//                   ^^^^^^^^
//                   HashMap<u32, ServerHandle> - one server per party
//                   - Key: party_id (0, 1, 2, 3 for 4-party setup)
//                   - Value: ServerHandle for that party's server
//                   - All servers run simultaneously as native processes
//                   - Each party has unique ports (service + MPC)
//                   - All auto-shutdown when HashMap is dropped

Each party runs as a separate native process with its own ports, simulating a real distributed threshold setup.

┌─────────────────────────────────────────────────────────────────┐
│ tokio runtime is used for simulation                             │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  Main Test Task                                                  │
│  ├─ setup_threshold_isolated(4 parties)                         │
│  │                                                               │
│  └─ Spawns 4 async tasks (tokio::spawn):                        │
│      ├─ Task 1: Party 1 Server (service port + MPC port)        │
│      ├─ Task 2: Party 2 Server (service port + MPC port)        │
│      ├─ Task 3: Party 3 Server (service port + MPC port)        │
│      └─ Task 4: Party 4 Server (service port + MPC port)        │
│                                                                  │
│  All tasks run concurrently on tokio thread pool                │
│  (NOT separate OS threads - async tasks)                       │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Test Material Generation

The test infrastructure includes a comprehensive pre-generation system that creates all cryptographic material needed for tests.

Test Material Tool (tools/generate-test-material/)

A standalone CLI tool that generates cryptographic material for isolated tests:

Key Features:

• Material Types:

  • testing - Fast generation with Test parameters (default for local dev)
  • default - Production-like generation with Default parameters (slower, for comprehensive testing)
  • all - Generates both testing and default material

• Key Types Generated:

  • ClientKeys - FHE client keys
  • SigningKeys - Client signing keys
  • ServerSigningKeys - Server signing keys
  • FheKeys - Complete FHE key sets
  • CrsKeys - Common Reference String keys
  • PublicKeys - Public key components
  • DecompressionKeys - Decompression keys (slow_tests only)

Note: PRSS (Pseudo-Random Secret Sharing) setup is not pre-generated. It's created at runtime when KMS servers start with run_prss: true in the test configuration.

• Party Configurations:
  • Centralized: 1 party
  • Threshold: 4 parties (threshold=2)
  • Custom: Configurable party count and threshold

Makefile Targets

# Generate all material (testing + default)
make generate-test-material-all

# Generate fast testing material only (recommended for local dev)
make generate-test-material-testing

# Generate production-like material with Default parameters
make generate-test-material-default

# Validate existing material
make validate-test-material

# Clean generated material
make clean-test-material

Tool Usage

# Direct cargo usage
cargo run -p generate-test-material -- [OPTIONS]

# Options:
#   --output <DIR>     Output directory (default: ./test-material)
#   --verbose          Enable verbose output
#   <COMMAND>          testing | default | all | validate | clean

# Examples:
cargo run -p generate-test-material -- --output ./test-material --verbose testing
cargo run -p generate-test-material --features slow_tests -- --output ./test-material all

CI Integration

CI generates test material before running tests to avoid generation overhead:

# Step 1: Generate test material (runs once)
- name: Generate Test Material
  working-directory: .  # Must run from workspace root
  run: cargo run -p generate-test-material -- --output ./test-material --verbose testing

# Step 2: Build required binaries
- name: Build kms-custodian binary
  run: cargo build --package kms --bin kms-custodian

# Step 3: Run tests (uses pre-generated material)
- name: Run Tests
  run: cargo nextest run --profile ci --no-fail-fast ${ARGS_TESTS}

Material Structure

test-material/
├── centralized/
│   └── testing/           # Fast test parameters
│       ├── client_keys/
│       ├── signing_keys/
│       ├── server_signing_keys/
│       └── fhe_keys/
└── threshold/
    ├── testing/           # Fast test parameters
    │   ├── client_keys/
    │   ├── signing_keys/
    │   ├── server_signing_keys/
    │   ├── fhe_keys/
    │   ├── crs_keys/
    │   └── public_keys/
    └── default/           # Production-like parameters (slow_tests)
        └── ... (same structure)

Note: PRSS setup is not stored in test-material. It's generated at runtime by KMS servers when tests configure run_prss: true.

CI Configuration

Updated test matrix to include required features and proper test filtering:

Integration Tests (2 parallel jobs):

# Threshold tests
--features k8s_tests,testing -- threshold --skip centralized --skip full_gen_tests --skip nightly --skip k8s_ --skip isolated_test_example

# Centralized tests  
--features k8s_tests,testing -- centralized --skip threshold --skip full_gen_tests --skip nightly --skip k8s_ --skip isolated_test_example

Feature Flags Explained:

1. k8s_tests - PRSS Test Gating

   • Purpose: Gates tests requiring PRSS (Pseudo-Random Secret Sharing) network coordination
   • Why needed: PRSS tests require sequential execution and stable network environment
   • Usage: #[cfg_attr(not(feature = "k8s_tests"), ignore)] on PRSS tests
   • When enabled: Only in K8s CI environment with proper network isolation
   • Affected tests: 5 PRSS tests (keygen, preprocessing, MPC context)

2. testing - Test Helper Functions

   • Purpose: Enables test-only helper functions and utilities
   • Why needed: Security - prevents test utilities from being compiled in production
   • Usage: #[cfg(any(test, feature = "testing"))] on helper modules
   • Examples: TestMaterialManager, setup_isolated_* functions, test material generation
   • Required for: Compiling test code that uses feature-gated functions

3. Feature flags - Two scenarios:

   Scenario A: Run all tests (including PRSS):

   # Both features required
   cargo test --test integration_tests --features k8s_tests,testing -- --test-threads=1

   • k8s_tests: Enables PRSS tests (5 tests with network coordination)
   • testing: Compiles test helper functions
   • --test-threads=1: Sequential execution for PRSS tests

   Scenario B: Run non-PRSS tests only:

   # Only testing feature needed (PRSS tests will be ignored)
   cargo test --test integration_tests --features testing

   • PRSS tests automatically skipped (gated by k8s_tests)
   • Faster execution, can run in parallel

4. Why both features?

   • Without testing: Test code won't compile (helper functions are feature-gated)
   • Without k8s_tests: PRSS tests will be ignored (safe for local development)
   • With both: Full test suite runs (CI environment)

5. Party counts: Most tests use 4 parties, one test uses 6 parties (MPC context switch)

Other Skips:

• isolated_test_example - Demonstration tests (not part of test suite)
• k8s_ - Kubernetes cluster tests (run separately in kind-testing.yml)
• nightly - Slow comprehensive tests (run only in scheduled builds)
• full_gen_tests - Full parameter tests (run only in scheduled builds)

Unit Tests:

--features k8s_tests,testing -- --skip centralized --skip threshold --skip isolated_test_example --skip k8s_

Nightly Tests (comprehensive, scheduled only):

--features k8s_tests,testing -- --skip k8s_ --skip isolated_test_example

• Runs ALL tests including nightly_* and full_gen_tests_*
• No filtering by test category (centralized/threshold)

Running Tests

Local Development

# Generate test material (required first time)
make generate-test-material-testing

# Run all tests
make test-isolated

# Run specific categories
make test-isolated-centralized   # Centralized tests only
make test-isolated-threshold     # Threshold tests only
make test-isolated-integration   # Integration tests only

# Run with PRSS tests (requires k8s_tests feature)
cargo test --test integration_tests --features k8s_tests,testing -- --test-threads=1

Developer Guide

Writing New Tests

Centralized Test:

#[tokio::test]
async fn test_my_feature() -> Result<()> {
    // Setup isolated centralized KMS server
    // test_name: Used for logging/debugging (appears in tracing output)
    // Returns: (TempDir, ServerHandle, PathBuf)
    let (material_dir, _server, config_path) =
        setup_isolated_centralized_cli_test("my_test").await?;
    //                                      ^^^^^^^^^ 
    //                                      Test identifier for logging/debugging
    //   ^^^^^^^^^^^^  ^^^^^^^ ^^^^^^^^^^^
    //   |             |       └─ Path to CLI config file (for --config flag)
    //   |             └─ Running KMS server (auto-shutdown on drop)
    //   └─ Isolated temp directory with test material (auto-cleanup)
    
    // Run CLI commands against the isolated KMS server
    let output = Command::new(env!("CARGO_BIN_EXE_kms-core-client"))
        .args(["--config", config_path.to_str().unwrap()])
        .args(["my-command"])
        .output()?;
    
    assert!(output.status.success());
    Ok(())
}

Threshold Test:

#[tokio::test]
async fn test_my_threshold_feature() -> Result<()> {
    // Setup isolated threshold KMS cluster (4 parties, Default FHE params)
    // Returns: (TempDir, HashMap<u32, ServerHandle>, PathBuf)
    let (material_dir, _servers, config_path) =
        setup_isolated_threshold_cli_test_default("my_test", 4).await?;
    //   ^^^^^^^^^^^^  ^^^^^^^^ ^^^^^^^^^^^
    //   |             |        └─ Path to CLI config file (for --config flag)
    //   |             └─ HashMap of 4 running KMS servers (party_id -> ServerHandle)
    //   └─ Isolated temp directory with test material (auto-cleanup)
    
    // Run CLI commands against threshold KMS cluster
    // CLI automatically communicates with all 4 parties via config
    Ok(())
}

PRSS Test (Sequential, K8s CI Only):

#[tokio::test]
#[serial]  // Sequential execution required (PRSS network coordination)
#[cfg_attr(not(feature = "k8s_tests"), ignore)]  // Only runs in K8s CI
async fn test_my_prss_feature() -> Result<()> {
    // Setup isolated threshold KMS cluster with PRSS enabled
    // Returns: (TempDir, HashMap<u32, ServerHandle>, PathBuf)
    let (material_dir, _servers, config_path) =
        setup_isolated_threshold_cli_test_with_prss("my_test", 4).await?;
    //   ^^^^^^^^^^^^  ^^^^^^^^ ^^^^^^^^^^^
    //   |             |        └─ Path to CLI config file
    //   |             └─ HashMap of 4 KMS servers with PRSS initialized
    //   └─ Isolated temp directory with PRSS material
    
    // Run preprocessing/keygen operations requiring PRSS
    // PRSS enables secure distributed key generation
    Ok(())
}

Issue ticket number and link

PR Checklist

I attest that all checked items are satisfied. Any deviation is clearly justified above.

• Title follows conventional commits (e.g. chore: ...).
• Tests added for every new pub item and test coverage has not decreased.
• Public APIs and non-obvious logic documented; unfinished work marked as TODO(#issue).
• unwrap/expect/panic only in tests or for invariant bugs (documented if present).
• No dependency version changes OR (if changed) only minimal required fixes.
• No architectural protocol changes OR linked spec PR/issue provided.
• No breaking deployment config changes OR devops label + infra notified + infra-team reviewer assigned.
• No breaking gRPC / serialized data changes OR commit marked with ! and affected teams notified.
• No modifications to existing versionized structs OR backward compatibility tests updated.
• No critical business logic / crypto changes OR ≥2 reviewers assigned.
• No new sensitive data fields added OR Zeroize + ZeroizeOnDrop implemented.
• No new public storage data OR data is verifiable (signature / digest).
• No unsafe; if unavoidable: minimal, justified, documented, and test/fuzz covered.
• Strongly typed boundaries: typed inputs validated at the edge; no untyped values or errors cross modules.
• Self-review completed.

Dependency Update Questionnaire (only if deps changed or added)

Answer in the Cargo.toml next to the dependency (or here if updating):

1. Ownership changes or suspicious concentration?
2. Low popularity?
3. Unusual version jump?
4. Lacking documentation?
5. Missing CI?
6. No security / disclosure policy?
7. Significant size increase?

More details and explanations for the checklist and dependency updates can be found in CONTRIBUTING.md

[github-actions]

Consolidated Tests Results 2025-11-27 - 07:36:15

Test Results

5 failed
59 passed

Details

64 tests
not captured
junit-to-ctrf
main test-reporter #1501
refactor(tests): adding KMS test material generator with isolated tests infrastructure #270

Failed Tests

kms client::tests::threshold::user_decryption_tests::default_user_decryption_threshold_with_crash::case_1::secure_1_true
kms client::tests::threshold::user_decryption_tests::default_user_decryption_threshold::case_2::secure_1_true
kms client::tests::threshold::user_decryption_tests::default_user_decryption_threshold_with_crash::case_1::secure_2_false
kms client::tests::threshold::user_decryption_tests::default_user_decryption_threshold_precompute_sns::case_1::secure_1_true
kms client::tests::threshold::user_decryption_tests::default_user_decryption_threshold::case_1::secure_1_true

test-reporter: Run #1501

Tests 📝	Passed ✅	Failed ❌	Skipped ⏭️	Pending ⏳	Other ❓	Flaky 🍂	Duration ⏱️
64	59	5	0	0	0	0	not captured

❌ Some tests failed!

Name	Failure Message
❌ client::tests::threshold::user_decryption_tests::default_user_decryption_threshold_with_crash::case_1::secure_1_true	thread 'client::tests::threshold::user_decryption_tests::default_user_decryption_threshold_with_crash::case_1::secure_1_true' (35659) panicked at core/service/src/client/tests/threshold/user_decryption_tests.rs:751:18
❌ client::tests::threshold::user_decryption_tests::default_user_decryption_threshold::case_2::secure_1_true	thread 'client::tests::threshold::user_decryption_tests::default_user_decryption_threshold::case_2::secure_1_true' (35657) panicked at core/service/src/client/tests/threshold/user_decryption_tests.rs:715:26
❌ client::tests::threshold::user_decryption_tests::default_user_decryption_threshold_with_crash::case_1::secure_2_false	thread 'client::tests::threshold::user_decryption_tests::default_user_decryption_threshold_with_crash::case_1::secure_2_false' (35658) panicked at core/service/src/client/tests/threshold/user_decryption_tests.rs:774:18
❌ client::tests::threshold::user_decryption_tests::default_user_decryption_threshold_precompute_sns::case_1::secure_1_true	thread 'client::tests::threshold::user_decryption_tests::default_user_decryption_threshold_precompute_sns::case_1::secure_1_true' (35660) panicked at core/service/src/client/tests/threshold/user_decryption_tests.rs:715:26
❌ client::tests::threshold::user_decryption_tests::default_user_decryption_threshold::case_1::secure_1_true	thread 'client::tests::threshold::user_decryption_tests::default_user_decryption_threshold::case_1::secure_1_true' (35666) panicked at core/service/src/client/tests/threshold/user_decryption_tests.rs:715:26

Tests

View All Tests

Test Name	Status	Flaky	Duration
client::tests::threshold::user_decryption_tests::default_user_decryption_threshold::case_1::secure_1_true	✅		6m 47s
client::tests::threshold::user_decryption_tests::default_user_decryption_threshold::case_2::secure_1_true	✅		11.4s
client::tests::threshold::user_decryption_tests::default_user_decryption_threshold_precompute_sns::case_1::secure_1_true	✅		12.0s
client::tests::threshold::user_decryption_tests::default_user_decryption_threshold_with_crash::case_1::secure_1_true	✅		12.4s
client::tests::threshold::user_decryption_tests::default_user_decryption_threshold_with_crash::case_1::secure_2_false	✅		12.4s
test_threshold_custodian_backup	✅		1.5s
test_threshold_restore_from_backup	✅		4.0s
test_threshold_concurrent_crs	✅		10m 33s
test_threshold_insecure	✅		14m 22s
test_threshold_mpc_context_switch_6	✅		16m 21s
test_threshold_mpc_context_init	✅		16m 27s
test_threshold_concurrent_preproc_keygen	✅		20m 27s
client::tests::threshold::user_decryption_tests::default_user_decryption_threshold::case_1::secure_1_true	✅		6m 47s
client::tests::threshold::user_decryption_tests::default_user_decryption_threshold::case_2::secure_1_true	✅		11.4s
client::tests::threshold::user_decryption_tests::default_user_decryption_threshold_precompute_sns::case_1::secure_1_true	✅		12.0s
client::tests::threshold::user_decryption_tests::default_user_decryption_threshold_with_crash::case_1::secure_1_true	✅		12.4s
client::tests::threshold::user_decryption_tests::default_user_decryption_threshold_with_crash::case_1::secure_2_false	✅		12.4s
client::tests::threshold::user_decryption_tests::default_user_decryption_threshold::case_1::secure_1_true	✅		6m 47s
client::tests::threshold::user_decryption_tests::default_user_decryption_threshold::case_2::secure_1_true	✅		11.4s
client::tests::threshold::user_decryption_tests::default_user_decryption_threshold_precompute_sns::case_1::secure_1_true	✅		12.0s
client::tests::threshold::user_decryption_tests::default_user_decryption_threshold_with_crash::case_1::secure_1_true	✅		12.4s
client::tests::threshold::user_decryption_tests::default_user_decryption_threshold_with_crash::case_1::secure_2_false	✅		12.4s
client::tests::threshold::user_decryption_tests::default_user_decryption_threshold::case_1::secure_1_true	✅		6m 47s
client::tests::threshold::user_decryption_tests::default_user_decryption_threshold::case_2::secure_1_true	✅		11.4s
client::tests::threshold::user_decryption_tests::default_user_decryption_threshold_precompute_sns::case_1::secure_1_true	✅		12.0s
client::tests::threshold::user_decryption_tests::default_user_decryption_threshold_with_crash::case_1::secure_1_true	✅		12.4s
client::tests::threshold::user_decryption_tests::default_user_decryption_threshold_with_crash::case_1::secure_2_false	✅		12.4s
client::tests::threshold::user_decryption_tests::default_user_decryption_threshold_with_crash::case_1::secure_1_true	❌		7m 24s
client::tests::threshold::user_decryption_tests::default_user_decryption_threshold::case_2::secure_1_true	❌		7m 41s
client::tests::threshold::user_decryption_tests::default_user_decryption_threshold_with_crash::case_1::secure_2_false	❌		7m 41s
client::tests::threshold::user_decryption_tests::default_user_decryption_threshold_precompute_sns::case_1::secure_1_true	❌		7m 48s
client::tests::threshold::user_decryption_tests::default_user_decryption_threshold::case_1::secure_1_true	❌		7m 58s
test_centralized_custodian_backup	✅		619ms
test_centralized_insecure	✅		1.8s
test_centralized_crsgen_secure	✅		2.4s
test_centralized_restore_from_backup	✅		2.4s
tests::test_parse_hex	✅		3ms
tests::test_invalid_hex	✅		3ms
tests::test_core_client_config	✅		4ms
tests::test_eip712_sigs	✅		1.3s
test_backward_compatibility_kms_grpc	✅		3ms
test_backward_compatibility_threshold_fhe	✅		49ms
test_backward_compatibility_kms	✅		241ms
identifiers::tests::test_invalid_id_all_zeros	✅		4ms
identifiers::tests::test_id_type_conversions	✅		4ms
identifiers::tests::request_id_ordering	✅		4ms
rpc_types::tests::idempotent_plaintext	✅		4ms
rpc_types::tests::test_abi_encoding_fhevm	✅		4ms
identifiers::tests::test_v1_request_id_with_whitespace	✅		4ms
identifiers::tests::test_valid_hex_characters	✅		4ms
identifiers::tests::test_request_id_random	✅		4ms
identifiers::tests::test_v1_request_id_conversion	✅		5ms
identifiers::tests::test_invalid_v1_request_id	✅		5ms
identifiers::tests::test_key_id_from_str	✅		5ms
identifiers::tests::test_v1_request_id_with_prefix	✅		5ms
identifiers::tests::test_invalid_hex_length	✅		5ms
rpc_types::tests::test_eip712_verification	✅		5ms
identifiers::tests::test_v1_request_id_to_u128_conversion	✅		5ms
identifiers::tests::test_key_id_protobuf_conversion	✅		6ms
rpc_types::tests::test_enum_default	✅		3ms
rpc_types::tests::test_request_id	✅		3ms
rpc_types::tests::test_request_id_raw_string	✅		3ms
rpc_types::tests::test_old_fhe_type_enum_compatibility	✅		3ms
rpc_types::tests::test_types_plaintext_ser	✅		3ms

_{🍂 No flaky tests in this run.}

Github Test Reporter by CTRF 💚

🔄 This comment has been updated

[maksymsur]

I temporarily triggered nightly k8s tests in chore(ci): temporary enable k8s tests which will be reverted once I see that these tests successfully executed within this PR