Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Derive Clone for Keypair? #76

Closed
Sushisource opened this issue Mar 14, 2019 · 4 comments · Fixed by #249 · May be fixed by #179
Closed

Derive Clone for Keypair? #76

Sushisource opened this issue Mar 14, 2019 · 4 comments · Fixed by #249 · May be fixed by #179

Comments

@Sushisource
Copy link

Is there any particular reason that the Keypair struct doesn't derive Clone? I have a spot where I need to clone one, and I've resorted to doing to_bytes->from_bytes which feels less than great.

If there's not a particular reason, I'd be happy to make a PR. Thanks!

@isislovecruft
Copy link
Member

It's my understanding (please correct me if wrong!) that if we implement Clone then the Drop methods will still be recursively called. If so, it should be safe to implement Clone for Keypair (with the caveat that programmers who explicitly do this will temporarily have more copies of the SecretKey sitting around in memory.. which.. isn't the best). I'd like to ensure in some sense that programmers are disinclined to do this somehow, but I'm not sure how best to, which is why I've not a #[derive(Clone)] for SecretKey thus far.

@Sushisource
Copy link
Author

@isislovecruft I figured that wanting to avoid extra copies in memory was probably the reason.

I can't think of any really great mechanism for that, besides maybe not actually implementing the Clone trait, and just having a special function for it, but that has a variety of associated issues.

🤷‍♂️ The other way to look at it is people who want to clone one are gonna do it anyway (like I did), so just make things clear in the docs and leave it at that?

@trevyn
Copy link

trevyn commented Mar 9, 2021

I'd like to ensure in some sense that programmers are disinclined to do this somehow, but I'm not sure how best to, which is why I've not a #[derive(Clone)] for SecretKey thus far.

I think you could put this behind an opt-in crate-level feature (say, cloneable_secret_key) that would make programmers think twice about enabling it. If this sounds reasonable, I could verify that the approach works and make a PR.

ckamm added a commit to ckamm/solana that referenced this issue Jun 27, 2022
This was not implemented upstream in ed25519-dalek to force everyone to
think twice before creating another copy of a potentially sensitive
private key in memory.

See dalek-cryptography/ed25519-dalek#76

However, there are now 9 instances of
  Keypair::from_bytes(&keypair.to_bytes())
in the solana codebase and it would be preferable to have a function.

In particular since this also comes up when writing programs and can
cause users to either start messing with lifetimes or discover the
from_bytes() workaround themselves.

This patch opts to not implement the Clone trait. This avoids automatic
use in order to preserve some of the original "let developers think
twice about this" intention.
ckamm added a commit to ckamm/solana that referenced this issue Jun 28, 2022
This was not implemented upstream in ed25519-dalek to force everyone to
think twice before creating another copy of a potentially sensitive
private key in memory.

See dalek-cryptography/ed25519-dalek#76

However, there are now 9 instances of
  Keypair::from_bytes(&keypair.to_bytes())
in the solana codebase and it would be preferable to have a function.

In particular since this also comes up when writing programs and can
cause users to either start messing with lifetimes or discover the
from_bytes() workaround themselves.

This patch opts to not implement the Clone trait. This avoids automatic
use in order to preserve some of the original "let developers think
twice about this" intention.
ckamm added a commit to ckamm/solana that referenced this issue Jun 29, 2022
This was not implemented upstream in ed25519-dalek to force everyone to
think twice before creating another copy of a potentially sensitive
private key in memory.

See dalek-cryptography/ed25519-dalek#76

However, there are now 9 instances of
  Keypair::from_bytes(&keypair.to_bytes())
in the solana codebase and it would be preferable to have a function.

In particular since this also comes up when writing programs and can
cause users to either start messing with lifetimes or discover the
from_bytes() workaround themselves.

This patch opts to not implement the Clone trait. This avoids automatic
use in order to preserve some of the original "let developers think
twice about this" intention.
ckamm added a commit to ckamm/solana that referenced this issue Jul 13, 2022
This was not implemented upstream in ed25519-dalek to force everyone to
think twice before creating another copy of a potentially sensitive
private key in memory.

See dalek-cryptography/ed25519-dalek#76

However, there are now 9 instances of
  Keypair::from_bytes(&keypair.to_bytes())
in the solana codebase and it would be preferable to have a function.

In particular since this also comes up when writing programs and can
cause users to either start messing with lifetimes or discover the
from_bytes() workaround themselves.

This patch opts to not implement the Clone trait. This avoids automatic
use in order to preserve some of the original "let developers think
twice about this" intention.
ckamm added a commit to ckamm/solana that referenced this issue Jul 19, 2022
This was not implemented upstream in ed25519-dalek to force everyone to
think twice before creating another copy of a potentially sensitive
private key in memory.

See dalek-cryptography/ed25519-dalek#76

However, there are now 9 instances of
  Keypair::from_bytes(&keypair.to_bytes())
in the solana codebase and it would be preferable to have a function.

In particular since this also comes up when writing programs and can
cause users to either start messing with lifetimes or discover the
from_bytes() workaround themselves.

This patch opts to not implement the Clone trait. This avoids automatic
use in order to preserve some of the original "let developers think
twice about this" intention.
CriesofCarrots pushed a commit to solana-labs/solana that referenced this issue Aug 6, 2022
* Keypair: implement clone()

This was not implemented upstream in ed25519-dalek to force everyone to
think twice before creating another copy of a potentially sensitive
private key in memory.

See dalek-cryptography/ed25519-dalek#76

However, there are now 9 instances of
  Keypair::from_bytes(&keypair.to_bytes())
in the solana codebase and it would be preferable to have a function.

In particular since this also comes up when writing programs and can
cause users to either start messing with lifetimes or discover the
from_bytes() workaround themselves.

This patch opts to not implement the Clone trait. This avoids automatic
use in order to preserve some of the original "let developers think
twice about this" intention.

* Use Keypair::clone
apfitzge pushed a commit to apfitzge/agave that referenced this issue Aug 9, 2022
* Keypair: implement clone()

This was not implemented upstream in ed25519-dalek to force everyone to
think twice before creating another copy of a potentially sensitive
private key in memory.

See dalek-cryptography/ed25519-dalek#76

However, there are now 9 instances of
  Keypair::from_bytes(&keypair.to_bytes())
in the solana codebase and it would be preferable to have a function.

In particular since this also comes up when writing programs and can
cause users to either start messing with lifetimes or discover the
from_bytes() workaround themselves.

This patch opts to not implement the Clone trait. This avoids automatic
use in order to preserve some of the original "let developers think
twice about this" intention.

* Use Keypair::clone
xiangzhu70 pushed a commit to xiangzhu70/solana that referenced this issue Aug 17, 2022
* Keypair: implement clone()

This was not implemented upstream in ed25519-dalek to force everyone to
think twice before creating another copy of a potentially sensitive
private key in memory.

See dalek-cryptography/ed25519-dalek#76

However, there are now 9 instances of
  Keypair::from_bytes(&keypair.to_bytes())
in the solana codebase and it would be preferable to have a function.

In particular since this also comes up when writing programs and can
cause users to either start messing with lifetimes or discover the
from_bytes() workaround themselves.

This patch opts to not implement the Clone trait. This avoids automatic
use in order to preserve some of the original "let developers think
twice about this" intention.

* Use Keypair::clone
buffalu added a commit to jito-foundation/jito-solana that referenced this issue Aug 23, 2022
* chore: increase timeout limit for coverage test

* chore: install NDK r21 in runtime (#26916)

chore: install ndk 21

* Revert "chore: increase timeout limit for coverage test" (#26917)

This reverts commit b6ae6c1fe17e4b64c5051c651ca2585e4f55468c.

* core: fix double-readlock in replay_stage (#26052)

* Revert "Enable QUIC client by default. Add arg to disable QUIC client… (#26913)

Revert "Enable QUIC client by default. Add arg to disable QUIC client. (#26879)"

This reverts commit 4c297500959368342a94e953f59e28de60195dd5.

* Make `solana-ledger-tool` run AccountsBackgroundService (#26914)

Prior to this change, long running commands like `solana-ledger-tool
verify` would OOM due to AccountsDb cleanup not happening.

Co-authored-by: Michael Vines <[email protected]>

* spl: Bump token to 3.5.0 and ata to 1.1.0 (#26921)

* Remove v1.9 backport actions, add v1.11 backport actions

* restore BackportAssignee definition

* Remove runtime dependency from solana-transaction-status (#26930)

* Move RewardType out of runtime

* Move collect_token_balances to solana-ledger

* Remove solana-runtime dependency

* Bench tps: refactor client creation (#26862)

* relax Sized restriction functions using client

* extract function to build client

* protect access to rent paying account vec (#26919)

* fix version of spl-token to prevent conflict (#26947)

* Remove bank test_max_accounts_data_size_exceeded() (#26772)

bank: remove test_max_accounts_data_size_exceeded()

* Remove accounts data size checks in blockstore_processor (#26776)

* Add minor version clean up tasks to release doc (#26954)

* add program account to bpf loader close instruction parser (#26926)

* Io stats v2 (#26898)

* Use sysfs instead of procfs for disk stats

* Filter map to filter dmcrypt and mdraid volumes

* Unit test cover different kernel formats

* Refactor QUIC new connection handler function (#26855)

* Refactor QUIC new connection handler function

* cleanup setup_connection

* more cleanup

* Support jsonParsed address lookup table accounts (#26723)

Parse address lookup table accounts

* Fix sol_get_processed_sibling_instruction on 32-bit hosts (#26522)

* Refactor: Add `RuntimeConfig` field to Bank (#26946)

* Refactor: Simplify arguments for bank constructor methods

* Refactor: Add RuntimeConfig to Bank fields

* Arc wrap runtime_config

* Arc wrap all runtime config usages

* Remove Copy trait derivation from RuntimeConfig

* Remove some arc wrapping

* chore: bump indicatif from 0.16.2 to 0.17.0 (#26890)

* chore: bump indicatif from 0.16.2 to 0.17.0

Bumps [indicatif](https://github.com/console-rs/indicatif) from 0.16.2 to 0.17.0.
- [Release notes](https://github.com/console-rs/indicatif/releases)
- [Commits](https://github.com/console-rs/indicatif/compare/0.16.2...0.17.0)

---
updated-dependencies:
- dependency-name: indicatif
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

* Accommodate api changes

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>
Co-authored-by: Tyera Eulberg <[email protected]>

* Unpin tokio for non-rpc crates (#26957)

* fix: fixed the incorrect/unknown redirects

* bpf-loader: make syscalls pub (#26918)

* transaction-status, storage-proto: add compute_units_consumed (#26528)

* transaction-status, storage-proto: add compute_units_consumed

* fix bpf test

Co-authored-by: Justin Starry <[email protected]>

* Keypair: implement clone() (#26248)

* Keypair: implement clone()

This was not implemented upstream in ed25519-dalek to force everyone to
think twice before creating another copy of a potentially sensitive
private key in memory.

See https://github.com/dalek-cryptography/ed25519-dalek/issues/76

However, there are now 9 instances of
  Keypair::from_bytes(&keypair.to_bytes())
in the solana codebase and it would be preferable to have a function.

In particular since this also comes up when writing programs and can
cause users to either start messing with lifetimes or discover the
from_bytes() workaround themselves.

This patch opts to not implement the Clone trait. This avoids automatic
use in order to preserve some of the original "let developers think
twice about this" intention.

* Use Keypair::clone

* Bump version to v1.12 (#26967)

* Bump sbf-tools version to v1.29

* Switch to cargo-build-sbf for building tests in programs/bpf

* added arg --rpc-max-request-payload-size to validator (#26377)

* added ability to pass --rpc-max-request-payload-size to validator

* fixed lint errors

* more lint fix

* patch

Co-authored-by: ultd <ultd>
Co-authored-by: Justin Starry <[email protected]>

* A more convenient store-tool (#26796)

* Use new_from_file_unchecked - don't sanitize input length for appendvec file

* Exit-early on completely zeroed accounts

* Fix programs/bpf bpf_c and bpf_rust features to work independently

`cargo test --features="bpf_c"` fails to compile without this fix.

* Do not check accounts data size in InvokeContext (#26773)

* Fix windows release builds (#26986)

* Don't try to build protobuf-src on windows

* Set protoc envar

* chore: bump const_format from 0.2.25 to 0.2.26 (#26985)

Bumps [const_format](https://github.com/rodrimati1992/const_format_crates) from 0.2.25 to 0.2.26.
- [Release notes](https://github.com/rodrimati1992/const_format_crates/releases)
- [Changelog](https://github.com/rodrimati1992/const_format_crates/blob/master/Changelog.md)
- [Commits](https://github.com/rodrimati1992/const_format_crates/commits/0.2.26)

---
updated-dependencies:
- dependency-name: const_format
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Delete files older than the lowest_cleanup_slot in LedgerCleanupService::cleanup_ledger (#26651)

#### Problem
LedgerCleanupService requires compactions to propagate & digest range-delete tombstones
to eventually reclaim disk space.

#### Summary of Changes
This PR makes LedgerCleanupService::cleanup_ledger delete any file whose slot-range is
older than the lowest_cleanup_slot.  This allows us to reclaim disk space more often with
fewer IOps.  Experimental results on mainnet validators show that the PR can effectively
reduce 33% to 40% ledger disk size.

* Bail out of execute_batches() early for empty batches slice (#26932)

The caller of execute_batches() that assembles batches may call this
function with empty batches; we know we can bail early in this scenario.

* chore: bump rustversion from 1.0.7 to 1.0.9 (#26984)

* chore: bump rustversion from 1.0.7 to 1.0.9

Bumps [rustversion](https://github.com/dtolnay/rustversion) from 1.0.7 to 1.0.9.
- [Release notes](https://github.com/dtolnay/rustversion/releases)
- [Commits](https://github.com/dtolnay/rustversion/compare/1.0.7...1.0.9)

---
updated-dependencies:
- dependency-name: rustversion
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>

* chore: bump pickledb from 0.4.1 to 0.5.1 (#25511)

* chore: bump pickledb from 0.4.1 to 0.5.1

Bumps [pickledb](https://github.com/seladb/pickledb-rs) from 0.4.1 to 0.5.1.
- [Release notes](https://github.com/seladb/pickledb-rs/releases)
- [Commits](https://github.com/seladb/pickledb-rs/commits/0.5.1)

---
updated-dependencies:
- dependency-name: pickledb
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Add yaml feature declaration

* Turn off default-features

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Tyera Eulberg <[email protected]>

* docs: updated absolute routes to local routes

* Add API docs for secp256k1_instruction and secp256k1_recover (#26065)

* Add API docs for secp256k1_instruction and secp256k1_recover

* typo

* Remove unused variable from secp256k1 program test

* Bump solana_bpf_rust_secp256k1_recover ix count

Co-authored-by: Tyera Eulberg <[email protected]>

* Refactor cargo-build-sbf integration tests using assert_cmd crate

* chore: bump async-trait from 0.1.56 to 0.1.57 (#27003)

* chore: bump async-trait from 0.1.56 to 0.1.57

Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.56 to 0.1.57.
- [Release notes](https://github.com/dtolnay/async-trait/releases)
- [Commits](https://github.com/dtolnay/async-trait/compare/0.1.56...0.1.57)

---
updated-dependencies:
- dependency-name: async-trait
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>

* chore: bump sha3 from 0.10.1 to 0.10.2 (#27002)

* chore: bump sha3 from 0.10.1 to 0.10.2

Bumps [sha3](https://github.com/RustCrypto/hashes) from 0.10.1 to 0.10.2.
- [Release notes](https://github.com/RustCrypto/hashes/releases)
- [Commits](https://github.com/RustCrypto/hashes/compare/sha3-v0.10.1...sha3-v0.10.2)

---
updated-dependencies:
- dependency-name: sha3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

* Revert sha3 bump in zk-token-sdk

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>
Co-authored-by: Tyera Eulberg <[email protected]>

* chore: bump bytecount from 0.6.2 to 0.6.3 (#27014)

Bumps [bytecount](https://github.com/llogiq/bytecount) from 0.6.2 to 0.6.3.
- [Release notes](https://github.com/llogiq/bytecount/releases)
- [Commits](https://github.com/llogiq/bytecount/commits)

---
updated-dependencies:
- dependency-name: bytecount
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore: bump prost-types from 0.11.0 to 0.11.1 (#27013)

* chore: bump prost-types from 0.11.0 to 0.11.1

Bumps [prost-types](https://github.com/tokio-rs/prost) from 0.11.0 to 0.11.1.
- [Release notes](https://github.com/tokio-rs/prost/releases)
- [Commits](https://github.com/tokio-rs/prost/compare/v0.11.0...prost-types-0.11.1)

---
updated-dependencies:
- dependency-name: prost-types
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>

* Revert "Remove resolver=2 from Cargo.toml and add it to the Windows build" (#27011)

Revert "Remove resolver=2 from Cargo.toml and add it to the Windows build (#26706)"

This reverts commit 2f6f5b11dae9389c92125b24b03fe07157aca8ed.

* Add `Signers` impls for `Arc<dyn Signer>` (#27000)

* Add `Signers` impls for `Arc<dyn Signer>`

* Reformat

* client: Use async TPU client in sync TPU client by sharing tokio runtime (#26996)

* Make the sync tpu client use the async tpu client

* Try to fix CI errors

* Fix formatting

* Make rpc_client::get_nonblocking_client public only in the crate

* Save work

* Temporary hack to test sharing runtime between tpu_client and rpc_client

* [WIP] Copy rpc client

* Fix build

* Small refactoring

* Remove copies

* Refactor access to RPC client fields

* Change `clone_inner_client` to `get_inner_client`

Co-authored-by: Ryan Leung <[email protected]>

* Patch crossbeam-epoch to avoid overhead (#26555)

* tracer-packet-stats reporting should not reset id (#27012)

* adds number of coding shreds to broadcast metrics (#27006)

* ledger-tool: support Geyser accounts updates (#26909)

* Set receive_window per quic connection (#26936)

This change sets the receive_window for non-staked node to 1 * PACKET_DATA_SIZE, and maps the staked nodes's connection's receive_window between 1.2 * PACKET_DATA_SIZE to 10 * PACKET_DATA_SIZE based on the stakes.

The changes is based on Quinn library change to support per connection receive_window tweak at the server side. quinn-rs/quinn#1393

* chore: bump semver from 1.0.10 to 1.0.13 (#27016)

* chore: bump semver from 1.0.10 to 1.0.13

Bumps [semver](https://github.com/dtolnay/semver) from 1.0.10 to 1.0.13.
- [Release notes](https://github.com/dtolnay/semver/releases)
- [Commits](https://github.com/dtolnay/semver/compare/1.0.10...1.0.13)

---
updated-dependencies:
- dependency-name: semver
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>

* Fix typo in test function (#27031)

* Correct StakeInstruction::DeactivateDelinquent instruction type

* Fix VoteInstruction order (#27035)

* Feature: disable CPI setting `is_executable` and `rent_epoch` (#26987)

* Adds the feature disable_cpi_setting_executable_and_rent_epoch.

* Adds the feature gate for disable_cpi_setting_executable_and_rent_epoch.

* Removes TEST_EXECUTABLE_LAMPORTS.

* Test that is_executable and rent_epoch are ignored.

* Increase timeout to reduce the flakyness of rpc signature receving test (#27008)

* Increase timeout to reduce the flakyness of rpc signature receving test

* Minor fmt fix

* Different staked vs unstaked chunks_received (#27033)

* Different staked vs unstaked chunks_received

* Suppress a clippy warning

* Implement nonblocking version of BlockhashQuery (#27040)

* Move vote program state and instructions to solana-program

* ancestor hashes socket ping/pong support (#26866)

* chore: bump serde from 1.0.138 to 1.0.143 (#27015)

* chore: bump serde from 1.0.138 to 1.0.143

Bumps [serde](https://github.com/serde-rs/serde) from 1.0.138 to 1.0.143.
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.138...v1.0.143)

---
updated-dependencies:
- dependency-name: serde
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>

* Separate file for ImmutableDeserializedPacket type (#26951)

* Explorer: Refactor parsed transaction handling (#27050)

* Fix windows build after crossbeam-epoch patch (#27052)

* Fix quic client on TestValidator, alternative (#27046)

Add new method to enable custom offset

* chore: bump serde_json from 1.0.81 to 1.0.83 (#27036)

* chore: bump serde_json from 1.0.81 to 1.0.83

Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.81 to 1.0.83.
- [Release notes](https://github.com/serde-rs/json/releases)
- [Commits](https://github.com/serde-rs/json/compare/v1.0.81...v1.0.83)

---
updated-dependencies:
- dependency-name: serde_json
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>

* docs: fix typo in return-data.md (#27056)

langauge -> language

* `solana-validator monitor` how displays slot and gossip stake % while waiting for supermajority

* chore: bump dialoguer from 0.10.1 to 0.10.2 (#27054)

* chore: bump dialoguer from 0.10.1 to 0.10.2

Bumps [dialoguer](https://github.com/mitsuhiko/dialoguer) from 0.10.1 to 0.10.2.
- [Release notes](https://github.com/mitsuhiko/dialoguer/releases)
- [Changelog](https://github.com/mitsuhiko/dialoguer/blob/master/CHANGELOG.md)
- [Commits](https://github.com/mitsuhiko/dialoguer/compare/v0.10.1...v0.10.2)

---
updated-dependencies:
- dependency-name: dialoguer
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>

* chore: bump once_cell from 1.12.0 to 1.13.0 (#27049)

* chore: bump once_cell from 1.12.0 to 1.13.0

Bumps [once_cell](https://github.com/matklad/once_cell) from 1.12.0 to 1.13.0.
- [Release notes](https://github.com/matklad/once_cell/releases)
- [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md)
- [Commits](https://github.com/matklad/once_cell/compare/v1.12.0...v1.13.0)

---
updated-dependencies:
- dependency-name: once_cell
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>

* Fixed a cargo warning on specifing quinn dependencies (#27057)

* feat: handle `loadedAddresses` field in tx meta responses (#27065)

feat: handle loadedAddresses field in tx meta responses

* chore: bump serde_bytes from 0.11.6 to 0.11.7 (#27062)

* chore: bump serde_bytes from 0.11.6 to 0.11.7

Bumps [serde_bytes](https://github.com/serde-rs/bytes) from 0.11.6 to 0.11.7.
- [Release notes](https://github.com/serde-rs/bytes/releases)
- [Commits](https://github.com/serde-rs/bytes/compare/0.11.6...0.11.7)

---
updated-dependencies:
- dependency-name: serde_bytes
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>

* chore: annotate more types as deprecated (#27067)

* chore: bump libc from 0.2.126 to 0.2.129 (#27063)

* chore: bump libc from 0.2.126 to 0.2.129

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.126 to 0.2.129.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.126...0.2.129)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>

* Fix duplicate / incorrect docs in solana_sdk by removing the solana_program::* import (#26588)

* Make solana_sdk imports from solana_program explicit.

* Adjust imports

* chore: bump goauth from 0.13.0 to 0.13.1 (#27066)

* chore: bump goauth from 0.13.0 to 0.13.1

Bumps [goauth](https://github.com/durch/rust-goauth) from 0.13.0 to 0.13.1.
- [Release notes](https://github.com/durch/rust-goauth/releases)
- [Commits](https://github.com/durch/rust-goauth/commits)

---
updated-dependencies:
- dependency-name: goauth
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>

* Fix rust flags handling in cargo-build-sbf

* chore: bump tokio-tungstenite from 0.17.1 to 0.17.2 (#27069)

* chore: bump tokio-tungstenite from 0.17.1 to 0.17.2

Bumps [tokio-tungstenite](https://github.com/snapview/tokio-tungstenite) from 0.17.1 to 0.17.2.
- [Release notes](https://github.com/snapview/tokio-tungstenite/releases)
- [Changelog](https://github.com/snapview/tokio-tungstenite/blob/master/CHANGELOG.md)
- [Commits](https://github.com/snapview/tokio-tungstenite/compare/v0.17.1...v0.17.2)

---
updated-dependencies:
- dependency-name: tokio-tungstenite
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>

* Fix local cluster tests for QUIC usage (#27071)

* removes buffering when generating coding shreds in broadcast (#25807)

Given the 32:32 erasure recovery schema, current implementation requires
exactly 32 data shreds to generate coding shreds for the batch (except
for the final erasure batch in each slot).
As a result, when serializing ledger entries to data shreds, if the
number of data shreds is not a multiple of 32, the coding shreds for the
last batch cannot be generated until there are more data shreds to
complete the batch to 32 data shreds. This adds latency in generating
and broadcasting coding shreds.

In addition, with Merkle variants for shreds, data shreds cannot be
signed and broadcasted until coding shreds are also generated. As a
result *both* code and data shreds will be delayed before broadcast if
we still require exactly 32 data shreds for each batch.

This commit instead always generates and broadcast coding shreds as soon
as there any number of data shreds available. When serializing entries
to shreds:
* if the number of resulting data shreds is less than 32, then more
  coding shreds will be generated so that the resulting erasure batch
  has the same recovery probabilities as a 32:32 batch.
* if the number of data shreds is more than 32, then the data shreds are
  split uniformly into erasure batches with _at least_ 32 data shreds in
  each batch. Each erasure batch will have the same number of code and
  data shreds.

For example:
* If there are 19 data shreds, 27 coding shreds are generated. The
  resulting 19(data):27(code) erasure batch has the same recovery
  probabilities as a 32:32 batch.
* If there are 107 data shreds, they are split into 3 batches of 36:36,
  36:36 and 35:35 data:code shreds each.

A consequence of this change is that code and data shreds indices will
no longer align as there will be more coding shreds than data shreds
(not only in the last batch in each slot but also in the intermediate
ones);

* feat: support minContextSlot in getParsedAccountInfo method (#27084)

* Bump rust-rocksdb to 0.19.0 tag (#26949)

* chore: bump chrono from 0.4.19 to 0.4.21 (#27076)

* chore: bump chrono from 0.4.19 to 0.4.21

Bumps [chrono](https://github.com/chronotope/chrono) from 0.4.19 to 0.4.21.
- [Release notes](https://github.com/chronotope/chrono/releases)
- [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md)
- [Commits](https://github.com/chronotope/chrono/compare/v0.4.19...v0.4.21)

---
updated-dependencies:
- dependency-name: chrono
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>

* chore: bump crossbeam-channel from 0.5.5 to 0.5.6 (#27072)

* chore: bump crossbeam-channel from 0.5.5 to 0.5.6

Bumps [crossbeam-channel](https://github.com/crossbeam-rs/crossbeam) from 0.5.5 to 0.5.6.
- [Release notes](https://github.com/crossbeam-rs/crossbeam/releases)
- [Changelog](https://github.com/crossbeam-rs/crossbeam/blob/master/CHANGELOG.md)
- [Commits](https://github.com/crossbeam-rs/crossbeam/compare/crossbeam-channel-0.5.5...crossbeam-channel-0.5.6)

---
updated-dependencies:
- dependency-name: crossbeam-channel
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>

* Connect to RPC nodes in parallel w/ reduced timeout (#26892)

* Connect to RPC nodes in parallel w/ reduced timeout

* allow staked nodes weight override (#26870)

* Allowed staked nodes weight override (#26407)

* Allowed staked nodes weight override, passing only HashMap over to core module

Co-authored-by: Ondra Chaloupka <[email protected]>

* add subcommand to set randomized compute-unit-price to transactions. (#26891)

* add subcommand to set randomized compute-unit-price to transactions.

* add compute-unit-limit to limit additional cost from prioritization.

* increase funding if use_randomized_compute_unit_price is enabled.

* Handle JsonRpcService startup failure (#27075)

* chore: bump predicates from 2.0.3 to 2.1.1 (#27087)

Bumps [predicates](https://github.com/assert-rs/predicates-rs) from 2.0.3 to 2.1.1.
- [Release notes](https://github.com/assert-rs/predicates-rs/releases)
- [Changelog](https://github.com/assert-rs/predicates-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/assert-rs/predicates-rs/compare/v2.0.3...v2.1.1)

---
updated-dependencies:
- dependency-name: predicates
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Fix local-cluster for QUIC more (#27096)

* chore: bump js-sys from 0.3.58 to 0.3.59 (#27088)

* chore: bump js-sys from 0.3.58 to 0.3.59

Bumps [js-sys](https://github.com/rustwasm/wasm-bindgen) from 0.3.58 to 0.3.59.
- [Release notes](https://github.com/rustwasm/wasm-bindgen/releases)
- [Changelog](https://github.com/rustwasm/wasm-bindgen/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rustwasm/wasm-bindgen/commits)

---
updated-dependencies:
- dependency-name: js-sys
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>

* Allow overriding the runtime transaction account lock limit (#26948)

* Add --transaction-account-lock-limit cli arg to test-validator

* Allow overriding the tx account lock limit in ProgramTest

* Explorer: Display address lookup table instruction type (#27106)

* chore: restructure transaction files

* chore: restucture message files

* chore: restructure program files

* chore: restructure utils code

* Fix test_accounts_data_size_and_resize_transactions (#27105)

* adjusts max coding shreds per slot (#27083)

As a consequence of removing buffering when generating coding shreds:
https://github.com/solana-labs/solana/pull/25807
more coding shreds are generated than data shreds, and so
MAX_CODE_SHREDS_PER_SLOT needs to be adjusted accordingly.

The respective value is tied to ERASURE_BATCH_SIZE.

* Reduce quic multi write test packets count (#27074)

* Reduce counts of packets to reduce time taken to send all of them

* Reduce count to 3000

* Sleep between vote refreshes (#27115)

* Sleep between vote refreshes in unit test

* cli: Require `--bypass-warning` flag to close program accounts (#27108)

* cli: Display warning when closing program accounts

* Fix parsing tests

* clean-feature: `default_units_per_instruction` (#27101)

clean-feature: default_units_per_instruction

* Bench tps add nonce flag (#27030)

* add durable nonce option

* make blockhash thread optional

* add nonce test to bench-tps

* clean feature: `request_units_deprecated` (#27102)

clean feature: request_units_deprecated

* don't log when there is no work to do for combining ancient slots (#26925)

* feat: add getAddressLookupTable method to Connection (#27127)

* Add fallback for ledger-tool commands to create new column families (#26565)

RocksDB settings include an option to create_if_missing, which will
create missing columns or the entire rocksdb directory if starting from
scratch. However, create_if_missing functionality only works if the
session has Primary (read+write) access. Many ledger-tool commands only
need Secondary (read-only) access to the database, so these commands are
unable to open the Blockstore when a column must be added.

This change detects when Secondary access open fails due to missing
column(s) or files, opens the database temporarily with Primary access,
and then reattempts to open the database Secondary access.

* explorer: Bump @solana/web3.js to v1.53.0 (#27128)

* chore:(deps): bump react-dom and @types/react-dom in /explorer (#27129)

Bumps [react-dom](https://github.com/facebook/react/tree/HEAD/packages/react-dom) and [@types/react-dom](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/react-dom). These dependencies needed to be updated together.

Updates `react-dom` from 18.1.0 to 18.2.0
- [Release notes](https://github.com/facebook/react/releases)
- [Changelog](https://github.com/facebook/react/blob/main/CHANGELOG.md)
- [Commits](https://github.com/facebook/react/commits/v18.2.0/packages/react-dom)

Updates `@types/react-dom` from 18.0.3 to 18.0.6
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/react-dom)

---
updated-dependencies:
- dependency-name: react-dom
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: "@types/react-dom"
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore:(deps): bump prettier from 2.5.1 to 2.7.1 in /explorer (#27130)

Bumps [prettier](https://github.com/prettier/prettier) from 2.5.1 to 2.7.1.
- [Release notes](https://github.com/prettier/prettier/releases)
- [Changelog](https://github.com/prettier/prettier/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prettier/prettier/compare/2.5.1...2.7.1)

---
updated-dependencies:
- dependency-name: prettier
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Explorer: Add details page for address lookup table accounts (#27133)

* chore:(deps): bump @testing-library/jest-dom from 5.16.1 to 5.16.5 in /explorer (#27131)

chore:(deps): bump @testing-library/jest-dom in /explorer

Bumps [@testing-library/jest-dom](https://github.com/testing-library/jest-dom) from 5.16.1 to 5.16.5.
- [Release notes](https://github.com/testing-library/jest-dom/releases)
- [Changelog](https://github.com/testing-library/jest-dom/blob/main/CHANGELOG.md)
- [Commits](https://github.com/testing-library/jest-dom/compare/v5.16.1...v5.16.5)

---
updated-dependencies:
- dependency-name: "@testing-library/jest-dom"
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore: add constant for pubkey byte length (#27134)

* chore: bump @babel/preset-env from 7.18.0 to 7.18.10 in /web3.js (#27138)

Bumps [@babel/preset-env](https://github.com/babel/babel/tree/HEAD/packages/babel-preset-env) from 7.18.0 to 7.18.10.
- [Release notes](https://github.com/babel/babel/releases)
- [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md)
- [Commits](https://github.com/babel/babel/commits/v7.18.10/packages/babel-preset-env)

---
updated-dependencies:
- dependency-name: "@babel/preset-env"
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* test-validator: improve multi-value arg help output (#26650)

* Bumps solana_rbpf to v0.2.32 (#27059)

* wait for bg hash calc to complete before 'calculate_capitalization' (#27145)

* Add more sysvar API docs (#26849)

* Add more sysvar API docs

* Remove println from examples

* Update sdk/program/src/clock.rs

Co-authored-by: Tyera Eulberg <[email protected]>

* Update sdk/program/src/clock.rs

Co-authored-by: Tyera Eulberg <[email protected]>

* Update sdk/program/src/clock.rs

Co-authored-by: Tyera Eulberg <[email protected]>

* Update sdk/program/src/clock.rs

Co-authored-by: Tyera Eulberg <[email protected]>

* Update sdk/program/src/clock.rs

Co-authored-by: Tyera Eulberg <[email protected]>

* Update sdk/program/src/clock.rs

Co-authored-by: Tyera Eulberg <[email protected]>

* Update sdk/program/src/clock.rs

Co-authored-by: Tyera Eulberg <[email protected]>

* Update sdk/program/src/clock.rs

Co-authored-by: Tyera Eulberg <[email protected]>

* Fix docs for ACCOUNT_STORAGE_OVERHEAD

* Update sdk/program/src/epoch_schedule.rs

Co-authored-by: Tyera Eulberg <[email protected]>

* Update sdk/program/src/sysvar/slot_hashes.rs

Co-authored-by: Tyera Eulberg <[email protected]>

* Update sdk/program/src/sysvar/slot_history.rs

Co-authored-by: Tyera Eulberg <[email protected]>

* Update sdk/program/src/sysvar/slot_history.rs

Co-authored-by: Tyera Eulberg <[email protected]>

* Update sdk/program/src/sysvar/mod.rs

Co-authored-by: Tyera Eulberg <[email protected]>

* Fix docs for DEFAULT_LEADER_SCHEDULE_SLOT_OFFSET

* Fix recent_blockhash short description

* Fix whitespace

Co-authored-by: Tyera Eulberg <[email protected]>

* Remove the deprecated `KeyedAccount` interface (#27147)

* Removes the deprecated KeyedAccount interface.

* Removes outdated example code.

* adds Shred{Code,Data}::SIZE_OF_HEADERS trait constants (#27144)

* Update quinn lib to 0.8.4 (#27119)

* Add stats for readonly cache evicts (#26938)

* add stats for readonly cache evicts

* bump up account cache to 400M

* aggregate num_evicts in the loop

* chore: bump serial_test from 0.8.0 to 0.9.0 (#27097)

Bumps [serial_test](https://github.com/palfrey/serial_test) from 0.8.0 to 0.9.0.
- [Release notes](https://github.com/palfrey/serial_test/releases)
- [Commits](https://github.com/palfrey/serial_test/compare/v0.8.0...v0.9.0)

---
updated-dependencies:
- dependency-name: serial_test
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [docs] Docs sidebar/navbar restructure (#27005)

* fix: minor title changes

* fix: updated structure of 2 pages

* fix: proposals page and submitting one

added proposals page and made it clear how to submit one

* fix: minor update to docs home page

* fix: footer links

added more footer links and minor restructure

* fix: sidebar restructure

* fix: removed duplicate geyser link

* fix: sidebars and navbar

final separation of sidebars and updated navbar

* fix: formatting for the navbar icons

* fix: changes some sidebar links to refs

* style: changed order of footer's SPL link

* style: removed comment and fixed whitespace check

* chore: upload test results to buildkite and datadog (#27139)

* export test-stable result

* export test-stable-perf result

* export test-local-cluster result

* export test-local-cluster-flakey result

* export test-local-cluster-slow-1 result

* export test-local-cluster-slow-2 result

* export test-docs result

* export test-stable-bpf result

* upload test result to buildkite and datadog

* Remove `fn slot_deltas()` from StatusCache (#26931)

* renames size_of_erasure_encoded_slice to ShredCode::capacity (#27157)

Maintain symmetry between code and data shreds:
* ShredData::capacity -> data buffer capacity
* ShredCode::capacity -> erasure code capacity

* store hash calc failures in a separate folder by slot (#27146)

* store hash calc failures in a separate folder by slot

* Update runtime/src/accounts_db.rs

Co-authored-by: Brooks Prumo <[email protected]>

Co-authored-by: Brooks Prumo <[email protected]>

* remove abort() from test-validator (#27124)

* chore: bump bytes from 1.1.0 to 1.2.1 (#27172)

* chore: bump bytes from 1.1.0 to 1.2.1

Bumps [bytes](https://github.com/tokio-rs/bytes) from 1.1.0 to 1.2.1.
- [Release notes](https://github.com/tokio-rs/bytes/releases)
- [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md)
- [Commits](https://github.com/tokio-rs/bytes/compare/v1.1.0...v1.2.1)

---
updated-dependencies:
- dependency-name: bytes
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>

* Share Ancestors API get with contains_key (#27161)

consolidate similar fns

* Rename to `MAX_BLOCK_ACCOUNTS_DATA_SIZE_DELTA` (#27175)

* chore: bump libc from 0.2.129 to 0.2.131 (#27162)

* chore: bump libc from 0.2.129 to 0.2.131

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.129 to 0.2.131.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.129...0.2.131)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>

* reverts wide fanout in broadcast when the root node is down (#26359)

A change included in
https://github.com/solana-labs/solana/pull/20480
was that when the root node in turbine broadcast tree is down, the
leader will broadcast the shred to all nodes in the first layer.
The intention was to mitigate the impact of dead nodes on shreds
propagation, because if the root node is down, then the entire cluster
will miss out the shred.
On the other hand, if x% of stake is down, this will cause 200*x% + 1
packets/shreds ratio at the broadcast stage which might contribute to
line-rate saturation and packet drop.
To avoid this bandwidth saturation issue, this commit reverts that logic
and always broadcasts shreds from the leader only to the root node.
As before we rely on erasure codes to recover shreds lost due to staked
nodes being offline.

* add getTokenLargestAccounts rpc method to rust client (#26840)

* add get token largest accounts rpc call to client

* split to include with commitment

* Bump spl-token-2022 (#27181)

* Bump token-2022 to 0.4.3

* Allow cargo to bump stuff to v1.11.5

* VoteProgram.safeWithdraw function to safeguard against accidental vote account closures (#26586)

feat: safe withdraw function

Co-authored-by: aschonfeld <[email protected]>

* chore: bump futures from 0.3.21 to 0.3.23 (#27182)

* chore: bump futures from 0.3.21 to 0.3.23

Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.21 to 0.3.23.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.21...0.3.23)

---
updated-dependencies:
- dependency-name: futures
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>

* chore: bump nix from 0.24.2 to 0.25.0 (#27179)

* chore: bump nix from 0.24.2 to 0.25.0

Bumps [nix](https://github.com/nix-rust/nix) from 0.24.2 to 0.25.0.
- [Release notes](https://github.com/nix-rust/nix/releases)
- [Changelog](https://github.com/nix-rust/nix/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nix-rust/nix/compare/v0.24.2...v0.25.0)

---
updated-dependencies:
- dependency-name: nix
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>

* Parse ConfidentialTransaction instructions (#26825)

Parse ConfidentialTransfer instructions

* snapshots: serialize version file first (#27192)

serialize version file first

* serialize incremental_snapshot_hash (#26839)

* serialize incremental_snapshot_hash

* pr feedback

* derives Error trait for ClusterInfoError and core::result::Error (#27208)

* Add clean_accounts_for_tests() (#27200)

* Rust v1.63.0 (#27148)

* Upgrade to Rust v1.63.0

* Add nightly_clippy_allows

* Resolve some new clippy nightly lints

* Increase QUIC packets completion timeout

Co-authored-by: Michael Vines <[email protected]>

* docs: updated "transaction fees" page (#26861)

* docs: transaction fees, compute units, compute budget

* docs: added messages definition

* Revert "docs: added messages definition"

This reverts commit 3c56156dfaaf17158c5eafbc5877080a83607a06.

* docs: added messages definition

* Update docs/src/transaction_fees.md

Co-authored-by: Jacob Creech <[email protected]>

* fix: updates from feedback

Co-authored-by: Jacob Creech <[email protected]>

* sdk: Fix args after "--" in build-bpf and test-bpf (#27221)

* Flaky Unit Test test_rpc_subscriptions (#27214)

Increase unit test timeout from 5 seconds to 10 seconds

* chore: only buildkite pipelines use sccache in docker-run.sh (#27204)

chore: only buildkite ci use sccache

* clean feature: `prevent_calling_precompiles_as_programs` (#27100)

* clean feature: prevent_calling_precompiles_as_programs

* fix tests

* fix test

* remove comment

* fix test

* feedback

* Add get_account_with_commitment to BenchTpsClient (#27176)

* Fix a corner-case panic in get_entries_in_data_block() (#27195)

#### Problem
get_entries_in_data_block() panics when there's inconsistency between
slot_meta and data_shred.

However, as we don't lock on reads, reading across multiple column families is
not atomic (especially for older slots) and thus does not guarantee consistency
as the background cleanup service could purge the slot in the middle.  Such
panic was reported in #26980 when the validator serves a high load of RPC calls.

#### Summary of Changes
This PR makes get_entries_in_data_block() panic only when the inconsistency
between slot-meta and data-shred happens on a slot older than lowest_cleanup_slot.

* Verify snapshot slot deltas (#26666)

* store-tool: log lamports for each account (#27168)

log lamports for each account

* add an assert for a debug feature to avoid wasted time (#27210)

* remove redundant call that bumps age to future (#27215)

* Use from_secs api to create duration (#27222)

use from_secs api to create duration

* reorder slot # in debug hash data path (#27217)

* create helper fn for clarity (#27216)

* Verifying snapshot bank must always specify the snapshot slot (#27234)

* Remove `Bank::ensure_no_storage_rewards_pool()` (#26468)

* cli: Add subcommands for address lookup tables (#27123)

* cli: Add subcommand for creating address lookup tables

* cli: Add additional subcommands for address lookup tables

* short commands

* adds hash domain to ping-pong protocol (#27193)

In order to maintain backward compatibility, for now the responding node
will hash the token both with and without domain so that the other node
will accept the response regardless of its upgrade status.
Once the cluster has upgraded to the new code, we will remove the legacy
domain = false case.

* Revert "Rust v1.63.0 (#27148)" (#27245)

This reverts commit a2e7bdf50ac5e1d4c633f64f6362028b4164c003.

* correct double negation (#27240)

* Enable QUIC client by default. Add arg to disable QUIC client. (Forward port #26927) (#27194)

Enable QUIC client by default. Add arg to disable QUIC client.

* Enable QUIC client by default. Add arg to disable QUIC client.
* Deprecate --disable-quic-servers arg
* Add #[ignore] annotation to failing tests

* slots_connected: check if the range is connected (>= ending_slot) (#27152)

* create-snapshot check if snapshot slot exists (#27153)

* Add Bank::clean_accounts_for_tests() (#27209)

* Call `AccountsDb::shrink_all_slots()` directly (#27235)

* add ed25519_program to built-in instruction cost list (#27199)

* add ed25519_program to built-in instruction cost list

* Remove unnecessary and stale comment

* simple refactorings to disk idx (#27238)

* add _inclusive for clarity (#27239)

* eliminate unnecessary ZERO_RAW_LAMPORTS_SENTINEL (#27218)

* make test code more clear (#27260)

* banking stage: actually aggregate tracer packet stats (#27118)

* aggregated_tracer_packet_stats_option was alwasys None

* Actually accumulate tracer packet stats

* Refactor epoch reward 1 (#27253)

* refactor: extract store_stake_accounts fn

* clippy: slice

Co-authored-by: haoran <haoran@mbook>

* recovers merkle shreds from erasure codes (#27136)

The commit
* Identifies Merkle shreds when recovering from erasure codes and
  dispatches specialized code to reconstruct shreds.
* Coding shred headers are added to recovered erasure shards.
* Merkle tree is reconstructed for the erasure batch and added to
  recovered shreds.
* The common signature (for the root of Merkle tree) is attached to all
  recovered shreds.

* Simplify `Bank::clean_accounts()` by removing params (#27254)

* Account files remove (#26910)

* Create a new function cleanup_accounts_paths, a trivial change

* Remove account files asynchronously

* Update and simplify the implementation after the validator test runs.

* Fixes after testing on the dev device

* Discard tokio.  Use thread instead

* Fix comments format

* Fix config type to pass the github test

* Fix failed tests.  Handle the case of non-existing path

* Final cleanup, addressing the review comments
Avoided OsString.
Made the function more generic with "impl AsRef<Path>"

Co-authored-by: Jeff Washington <[email protected]>

* Refactor: Flattens `TransactionContext::instruction_trace` (#27109)

* Flattens TransactionContext::instruction_trace.

* Stop the search at transaction level.

* Renames get_instruction_context_at => get_instruction_context_at_nesting_level.

* Removes TransactionContext::get_instruction_trace().
Adds TransactionContext::get_instruction_trace_length() and TransactionContext::get_instruction_context_at_index().

* Have TransactionContext::instruction_accounts_lamport_sum() accept an iterator instead of a slice.

* Removes instruction_trace from ExecutionRecord.

* make InstructionContext::new() private

* Parallel insertion of dirty store keys during clean (#27058)

parallelize dirty store key insertion

* Refactor epoch reward 2 (#27257)

* refactor: extract store_stake_accounts fn

* refactor: extract store_vote_account fn

* clippy: slice

* clippy: slice

* fix merge error

Co-authored-by: haoran <haoran@mbook>

* Standardize thread names

Tenets:
1. Limit thread names to 15 characters
2. Prefix all Solana-controlled threads with "sol"
3. Use Camel case. It's more character dense than Snake or Kebab case

* cleanup comment on filter_zero_lamport_clean_for_incremental_snapshots (#27273)

* remove inaccurate log (#27255)

* patches metrics for invalid cached vote/stake accounts (#27266)

patches invalid cached vote/stake accounts metrics

Invalid cached vote accounts is overcounting actual mismatches, and
invalid cached stake accounts is undercounting.

* Refactor epoch reward 3 (#27259)

* refactor: extract store_stake_accounts fn

* refactor: extract store_vote_account fn

* refactor: extract reward history update fn

* clippy: slice

* clippy: slice

Co-authored-by: haoran <haoran@mbook>

* building

* Update `solana deploy` subcommand to warn non-upgradable (#27264)

Update subcommand text to warn deploy deprecated

Update the about text for `solana deploy` to warn this is only for non-upgradeable deploys. Fixes #27228

* Refactor epoch reward 4 (#27261)

* refactor: extract store_stake_accounts fn

* refactor: extract store_vote_account fn

* refactor: extract reward history update fn

* remove avg point value from pay_valiator fn. not used

* clippy: slice

* clippy: slice

* remove abort() from test-validator (#27124)

* chore: bump bytes from 1.1.0 to 1.2.1 (#27172)

* chore: bump bytes from 1.1.0 to 1.2.1

Bumps [bytes](https://github.com/tokio-rs/bytes) from 1.1.0 to 1.2.1.
- [Release notes](https://github.com/tokio-rs/bytes/releases)
- [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md)
- [Commits](https://github.com/tokio-rs/bytes/compare/v1.1.0...v1.2.1)

---
updated-dependencies:
- dependency-name: bytes
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>

* Share Ancestors API get with contains_key (#27161)

consolidate similar fns

* Rename to `MAX_BLOCK_ACCOUNTS_DATA_SIZE_DELTA` (#27175)

* chore: bump libc from 0.2.129 to 0.2.131 (#27162)

* chore: bump libc from 0.2.129 to 0.2.131

Bumps [libc](https://github.com/rust-lang/libc) from 0.2.129 to 0.2.131.
- [Release notes](https://github.com/rust-lang/libc/releases)
- [Commits](https://github.com/rust-lang/libc/compare/0.2.129...0.2.131)

---
updated-dependencies:
- dependency-name: libc
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>

* reverts wide fanout in broadcast when the root node is down (#26359)

A change included in
https://github.com/solana-labs/solana/pull/20480
was that when the root node in turbine broadcast tree is down, the
leader will broadcast the shred to all nodes in the first layer.
The intention was to mitigate the impact of dead nodes on shreds
propagation, because if the root node is down, then the entire cluster
will miss out the shred.
On the other hand, if x% of stake is down, this will cause 200*x% + 1
packets/shreds ratio at the broadcast stage which might contribute to
line-rate saturation and packet drop.
To avoid this bandwidth saturation issue, this commit reverts that logic
and always broadcasts shreds from the leader only to the root node.
As before we rely on erasure codes to recover shreds lost due to staked
nodes being offline.

* add getTokenLargestAccounts rpc method to rust client (#26840)

* add get token largest accounts rpc call to client

* split to include with commitment

* Bump spl-token-2022 (#27181)

* Bump token-2022 to 0.4.3

* Allow cargo to bump stuff to v1.11.5

* VoteProgram.safeWithdraw function to safeguard against accidental vote account closures (#26586)

feat: safe withdraw function

Co-authored-by: aschonfeld <[email protected]>

* chore: bump futures from 0.3.21 to 0.3.23 (#27182)

* chore: bump futures from 0.3.21 to 0.3.23

Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.21 to 0.3.23.
- [Release notes](https://github.com/rust-lang/futures-rs/releases)
- [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.21...0.3.23)

---
updated-dependencies:
- dependency-name: futures
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>

* chore: bump nix from 0.24.2 to 0.25.0 (#27179)

* chore: bump nix from 0.24.2 to 0.25.0

Bumps [nix](https://github.com/nix-rust/nix) from 0.24.2 to 0.25.0.
- [Release notes](https://github.com/nix-rust/nix/releases)
- [Changelog](https://github.com/nix-rust/nix/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nix-rust/nix/compare/v0.24.2...v0.25.0)

---
updated-dependencies:
- dependency-name: nix
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* [auto-commit] Update all Cargo lock files

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot-buildkite <[email protected]>

* Parse ConfidentialTransaction instructions (#26825)

Parse ConfidentialTransfer instructions

* snapshots: serialize version file first (#27192)

serialize version file first

* serialize incremental_snapshot_hash (#26839)

* serialize incremental_snapshot_hash

* pr feedback

* derives Error trait for ClusterInfoError and core::result::Error (#27208)

* Add clean_accounts_for_tests() (#27200)

* Rust v1.63.0 (#27148)

* Upgrade to Rust v1.63.0

* Add nightly_clippy_allows

* Resolve some new clippy nightly lints

* Increase QUIC packets completion timeout

Co-authored-by: Michael Vines <[email protected]>

* docs: updated "transaction fees" page (#26861)

* docs: transaction fees, compute units, compute budget

* docs: added messages definition

* Revert "docs: added messages definition"

This reverts commit 3c56156dfaaf17158c5eafbc5877080a83607a06.

* docs: added messages definition

* Update docs/src/transaction_fees.md

Co-authored-by: Jacob Creech <[email protected]>

* fix: updates from feedback

Co-authored-by: Jacob Creech <[email protected]>

* sdk: Fix args after "--" in build-bpf and test-bpf (#27221)

* Flaky Unit Test test_rpc_subscriptions (#27214)

Increase unit test timeout from 5 seconds to 10 seconds

* chore: only buildkite pipelines use sccache in docker-run.sh (#27204)

chore: only buildkite ci use sccache

* clean feature: `prevent_calling_precompiles_as_programs` (#27100)

* clean feature: prevent_calling_precompiles_as_programs

* fix tests

* fix test

* remove comment

* fix test

* feedback

* Add get_account_with_commitment to BenchTpsClient (#27176)

* Fix a corner-case panic in get_entries_in_data_block() (#27195)

#### Problem
get_entries_in_data_block() panics when there's inconsistency between
slot_meta and data_shred.

However, as we don't lock on reads, reading across multiple column families is
not atomic (especially for older slots) and thus does not guarantee consistency
as the background cleanup service could purge the slot in the middle.  Such
panic was reported in #26980 when the validator serves a high load of RPC calls.

#### Summary of Changes
This PR makes get_entries_in_data_block() panic only when the inconsistency
between slot-meta and data-shred happens on a slot older than lowest_cleanup_slot.

* Verify snapshot slot deltas (#26666)

* store-tool: log lamports for each account (#27168)

log lamports for each account

* add an assert for a debug feature to avoid wasted time (#27210)

* remove redundant call that bumps age to future (#27215)

* Use from_secs api to create duration (#27222)

use from_secs api to create duration

* reorder slot # in debug hash data path (#27217)

* create helper fn for clarity (#27216)

* Verifying snapshot bank must always specify the snapshot slot (#27234)

* Remove `Bank::ensure_no_storage_rewards_pool()` (#26468)

* cli: Add subcommands for address lookup tables (#27123)

* cli: Add subcommand for creating address lookup tables

* cli: Add additional subcommands for address lookup tables

* short commands

* adds hash domain to ping-pong protocol (#27193)

In order to maintain backward compatibility, for now the responding node
will hash the token both with and without domain so that the other node
will accept the response regardless of its upgrade status.
Once the cluster has upgraded to the new code, we will remove the legacy
domain = false case.

* Revert "Rust v1.63.0 (#27148)" (#27245)

This reverts commit a2e7bdf50ac5e1d4c633f64f6362028b4164c003.

* correct double negation (#27240)

* Enable QUIC client by default. Add arg to disable QUIC client. (Forward port #26927) (#27194)

Enable QUIC client by default. Add arg to disable QUIC client.

* Enable QUIC client by default. Add arg to disable QUIC client.
* Deprecate --disable-quic-servers arg
* Add #[ignore] annotation to failing tests

* slots_connected: check if the range is connected (>= ending_slot) (#27152)

* create-snapshot check if snapshot slot exists (#27153)

* Add Bank::clean_accounts_for_tests() (#27209)

* Call `AccountsDb::shrink_all_slots()` directly (#27235)

* add ed25519_program to built-in instruction cost list (#27199)

* add ed25519_program to built-in instruction cost list

* Remove unnecessary and stale comment

* simple refactorings to disk idx (#27238)

* add _inclusive for clarity (#27239)

* eliminate unnecessary ZERO_RAW_LAMPORTS_SENTINEL (#27218)

* make test code more clear (#27260)

* banking stage: actually aggregate tracer packet stats (#27118)

* aggregated_tracer_packet_stats_option was alwasys None

* Actually accumulate tracer packet stats

* Refactor epoch reward 1 (#27253)

* refactor: extract store_stake_accounts fn

* clippy: slice

Co-authored-by: haoran <haoran@mbook>

* recovers merkle shreds from erasure codes (#27136)

The commit
* Identifies Merkle shreds when recovering from erasure codes and
  dispatches specialized code to reconstruct shreds.
* Coding shred headers are added to recovered erasure shards.
* Merkle tree is reconstructed for the erasure batch and added to
  recovered shreds.
* The common signature (for the root of Merkle tree) is attached to all
  recovered shreds.

* Simplify `Bank::clean_accounts()` by removing params (#27254)

* Account files remove (#26910)

* Create a new function cleanup_accounts_paths, a trivial change

* Remove account files asynchronously

* Update and simplify the implementation after the validator test runs.

* Fixes after testing on the dev device

* Discard tokio.  Use thread instead

* Fix comments format

* Fix config type to pass the github test

* Fix failed tests.  Handle the case of non-existing path

* Final cleanup, addressing the review comments
Avoided OsString.
Made the function more generic with "impl AsRef<Path>"

Co-authored-by: Jeff Washington <[email protected]>

* Refactor: Flattens `TransactionContext::instruction_trace` (#27109)

* Flattens TransactionContext::instruction_trace.

* Stop the search at transaction level.

* Renames get_instruction_context_at => get_instruction_context_at_nesting_level.

* Removes TransactionContext::get_instruction_trace().
Adds TransactionContext::get_instruction_trace_length() and TransactionContext::get_instruction_context_at_index().

* Have TransactionContext::instruction_accounts_lamport_sum() accept an iterator instead of a slice.

* Removes instruction_trace from ExecutionRecord.

* make InstructionContext::new() private

* Parallel insertion of dirty store keys during clean (#27058)

parallelize dirty store key insertion

* Refactor epoch reward 2 (#27257)

* refactor: extract store_stake_accounts fn

* refactor: extract store_vote_account fn

* clippy: slice

* clippy: slice

* fix merge error

Co-authored-by: haoran <haoran@mbook>

* Standardize thread names

Tenets:
1. Limit thread names to 15 characters
2. Prefix all Solana-controlled threads with "sol"
3. Use Camel case. It's more character dense than Snake or Kebab case

* cleanup comment on filter_zero_lamport_clean_for_incremental_snapshots (#27273)

* remove inaccurate log (#27255)

* patches metrics for invalid cached vote/stake accounts (#27266)

patches invalid cached vote/stake accounts metrics

Invalid cached vote accounts is overcounting actual mismatches, and
invalid cached stake accounts is undercounting.

* Refactor epoch reward 3 (#27259)

* refactor: extract store_stake_accounts fn

* refactor: extract store_vote_account fn

* refactor: extract reward history update fn

* clippy: slice

* clippy: slice

Co-authored-by: haoran <haoran@mbook>

*…
@tarcieri
Copy link
Contributor

tarcieri commented Jan 5, 2023

Added in #249

@tarcieri tarcieri closed this as completed Jan 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants