Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pallet_contracts] Add support for transient storage in contracts host functions #4566

Merged
merged 70 commits into from
Jul 16, 2024

Conversation

smiasojed
Copy link
Contributor

@smiasojed smiasojed commented May 24, 2024

Introduce transient storage, which behaves identically to regular storage but is kept only in memory and discarded after every transaction. This functionality is similar to the TSTORE and TLOAD operations used in Ethereum.

The following new host functions have been introduced:
get_transient_storage
set_transient_storage
take_transient_storage
clear_transient_storage
contains_transient_storage
Note: These functions are declared as unstable and thus are not activated.

@smiasojed smiasojed added the T7-smart_contracts This PR/Issue is related to smart contracts. label May 24, 2024
@smiasojed smiasojed requested a review from ascjones June 4, 2024 08:07
substrate/frame/contracts/src/lib.rs Outdated Show resolved Hide resolved
substrate/frame/contracts/src/wasm/runtime.rs Outdated Show resolved Hide resolved
substrate/frame/contracts/src/wasm/runtime.rs Outdated Show resolved Hide resolved
substrate/frame/contracts/src/transient_storage.rs Outdated Show resolved Hide resolved
substrate/frame/contracts/src/lib.rs Outdated Show resolved Hide resolved
substrate/frame/contracts/src/transient_storage.rs Outdated Show resolved Hide resolved
substrate/frame/contracts/src/transient_storage.rs Outdated Show resolved Hide resolved
@smiasojed
Copy link
Contributor Author

bot bench substrate-pallet --pallet=pallet_contracts

@command-bot
Copy link

command-bot bot commented Jun 18, 2024

@smiasojed https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6494983 was started for your command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=dev --target_dir=substrate --pallet=pallet_contracts. Check out https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/pipelines?page=1&scope=all&username=group_605_bot to know what else is being executed currently.

Comment bot cancel 1-8c3cb4fc-c31f-4ea7-93cb-5a956db63737 to cancel this command or bot cancel to cancel all commands in this pull request.

@command-bot
Copy link

command-bot bot commented Jun 18, 2024

@smiasojed Command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=dev --target_dir=substrate --pallet=pallet_contracts has finished. Result: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6494983 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6494983/artifacts/download.

@smiasojed
Copy link
Contributor Author

bot bench substrate-pallet --pallet=pallet_contracts

@command-bot
Copy link

command-bot bot commented Jun 18, 2024

@smiasojed https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6497164 was started for your command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=dev --target_dir=substrate --pallet=pallet_contracts. Check out https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/pipelines?page=1&scope=all&username=group_605_bot to know what else is being executed currently.

Comment bot cancel 2-2506ce03-9cd3-46a1-b7a6-3a10d6498720 to cancel this command or bot cancel to cancel all commands in this pull request.

let value = Some(vec![42u8; max_value_len as _]);
let mut setup = CallSetup::<T>::default();
let (mut ext, _) = setup.ext();
CallSetup::<T>::with_transient_storage(&mut ext, T::MaxTransientStorageSize::get())?;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would be a bit nicer to have with_transient_storage be a method of the builder

let mut setup = CallSetup::<T>::default();
setup.transient_storage_size(T::MaxTransientStorageSize::get())
let (mut ext, _) = setup.ext();

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment on lines 165 to 166
let amount = ext.transient_storage().meter().current().amount;
let limit = ext.transient_storage().meter().current().limit;
Copy link
Contributor

@pgherveou pgherveou Jul 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe

let &MeterEntry { amount, limit } = ext.transient_storage().meter().current();

Comment on lines +258 to +259
// cost_write!(name, a, b, c) -> T::WeightInfo::name(a, b, c).saturating_add(T::WeightInfo::rollback_transient_storage())
// .saturating_add(T::WeightInfo::set_transient_storage_full().saturating_sub(T::WeightInfo::set_transient_storage_empty())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not clear to me why the cost is full - empty, can we explain the reasoning here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

full - empty represents the weight of adding an item to a full BTreeMap. All transient storage host function benchmarks are done with an empty transient storage (BTreeMap), so this represents the worst-case scenario.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

follow up question, should we refund the cost of rollback_transient_storage in the happy path since you pre-charge it

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still not clear to me why we need to add the weight of
T::WeightInfo::set_transient_storage_full() - T::WeightInfo::set_transient_storage_empty()
when we do a ClearTransientStorage for example

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The clear_transient_storage function modifies the storage. The benchmark is currently done for one item in the BTreeMap, but we would like to consider the worst-case scenario, where the modification is done on a fully filled BTreeMap.

I am treating it as a constant cost, similar to DBWrite, which is 100µs.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sense then, we should add that to the comment to make it clearer

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added

Copy link
Contributor

@pgherveou pgherveou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm as well, added a few more nits and a question

Copy link
Member

@xermicus xermicus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

substrate/frame/contracts/uapi/src/host.rs Show resolved Hide resolved
@athei
Copy link
Member

athei commented Jul 8, 2024

Can you do one additional thing in this PR: Since we are bumping the major version of the uapi crate it would be a good time to also seal the HostFn trait. The only reason we have to do a major bump is because we forgot to prevent downstream crates to implement it.

@paritytech-cicd-pr
Copy link

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-clippy
Logs: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6660740

@smiasojed
Copy link
Contributor Author

bot bench substrate-pallet --pallet=pallet_contracts

@command-bot
Copy link

command-bot bot commented Jul 9, 2024

@smiasojed https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6662227 was started for your command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=dev --target_dir=substrate --pallet=pallet_contracts. Check out https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/pipelines?page=1&scope=all&username=group_605_bot to know what else is being executed currently.

Comment bot cancel 1-3a0acf30-2f24-4ad3-9b64-1421719f53f6 to cancel this command or bot cancel to cancel all commands in this pull request.

@command-bot
Copy link

command-bot bot commented Jul 9, 2024

@smiasojed Command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" --subcommand=pallet --runtime=dev --target_dir=substrate --pallet=pallet_contracts has finished. Result: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6662227 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6662227/artifacts/download.

@smiasojed smiasojed added this pull request to the merge queue Jul 16, 2024
Merged via the queue into master with commit eac2a22 Jul 16, 2024
155 of 161 checks passed
@smiasojed smiasojed deleted the sm/contracts-tstore branch July 16, 2024 11:40
ordian added a commit that referenced this pull request Jul 17, 2024
* master:
  add elastic scaling MVP guide (#4663)
  Send PeerViewChange with high priority (#4755)
  [ci] Update forklift in CI image (#5032)
  Adjust base value for statement-distribution regression tests (#5028)
  [pallet_contracts] Add support for transient storage in contracts host functions (#4566)
  [1 / 5] Optimize logic for gossiping assignments (#4848)
  Remove `pallet-getter` usage from pallet-session (#4972)
  command-action: added scoped permissions to the github tokens (#5016)
  net/litep2p: Propagate ValuePut events to the network backend (#5018)
  rpc: add back rpc logger (#4952)
  Updated substrate-relay version for tests (#5017)
  Remove most all usage of `sp-std` (#5010)
  Use sp_runtime::traits::BadOrigin (#5011)
jpserrat pushed a commit to jpserrat/polkadot-sdk that referenced this pull request Jul 18, 2024
…t functions (paritytech#4566)

Introduce transient storage, which behaves identically to regular
storage but is kept only in memory and discarded after every
transaction. This functionality is similar to the `TSTORE` and `TLOAD`
operations used in Ethereum.

The following new host functions have been introduced:
`get_transient_storage`
`set_transient_storage`
`take_transient_storage`
`clear_transient_storage`
`contains_transient_storage`
Note: These functions are declared as `unstable` and thus are not
activated.

---------

Co-authored-by: command-bot <>
Co-authored-by: PG Herveou <[email protected]>
Co-authored-by: Alexander Theißen <[email protected]>
ordian added a commit that referenced this pull request Jul 18, 2024
* master: (125 commits)
  add elastic scaling MVP guide (#4663)
  Send PeerViewChange with high priority (#4755)
  [ci] Update forklift in CI image (#5032)
  Adjust base value for statement-distribution regression tests (#5028)
  [pallet_contracts] Add support for transient storage in contracts host functions (#4566)
  [1 / 5] Optimize logic for gossiping assignments (#4848)
  Remove `pallet-getter` usage from pallet-session (#4972)
  command-action: added scoped permissions to the github tokens (#5016)
  net/litep2p: Propagate ValuePut events to the network backend (#5018)
  rpc: add back rpc logger (#4952)
  Updated substrate-relay version for tests (#5017)
  Remove most all usage of `sp-std` (#5010)
  Use sp_runtime::traits::BadOrigin (#5011)
  network/tx: Ban peers with tx that fail to decode (#5002)
  Try State Hook for Bounties (#4563)
  [statement-distribution] Add metrics for distributed statements in V2 (#4554)
  added sync command (#4818)
  Bridges V2 refactoring backport and `pallet_bridge_messages` simplifications (#4935)
  xcm-executor: Improve logging (#4996)
  Remove usage of `sp-std` on templates (#5001)
  ...
TarekkMA pushed a commit to moonbeam-foundation/polkadot-sdk that referenced this pull request Aug 2, 2024
…t functions (paritytech#4566)

Introduce transient storage, which behaves identically to regular
storage but is kept only in memory and discarded after every
transaction. This functionality is similar to the `TSTORE` and `TLOAD`
operations used in Ethereum.

The following new host functions have been introduced:
`get_transient_storage`
`set_transient_storage`
`take_transient_storage`
`clear_transient_storage`
`contains_transient_storage`
Note: These functions are declared as `unstable` and thus are not
activated.

---------

Co-authored-by: command-bot <>
Co-authored-by: PG Herveou <[email protected]>
Co-authored-by: Alexander Theißen <[email protected]>
ordian added a commit that referenced this pull request Aug 6, 2024
* master: (130 commits)
  add elastic scaling MVP guide (#4663)
  Send PeerViewChange with high priority (#4755)
  [ci] Update forklift in CI image (#5032)
  Adjust base value for statement-distribution regression tests (#5028)
  [pallet_contracts] Add support for transient storage in contracts host functions (#4566)
  [1 / 5] Optimize logic for gossiping assignments (#4848)
  Remove `pallet-getter` usage from pallet-session (#4972)
  command-action: added scoped permissions to the github tokens (#5016)
  net/litep2p: Propagate ValuePut events to the network backend (#5018)
  rpc: add back rpc logger (#4952)
  Updated substrate-relay version for tests (#5017)
  Remove most all usage of `sp-std` (#5010)
  Use sp_runtime::traits::BadOrigin (#5011)
  network/tx: Ban peers with tx that fail to decode (#5002)
  Try State Hook for Bounties (#4563)
  [statement-distribution] Add metrics for distributed statements in V2 (#4554)
  added sync command (#4818)
  Bridges V2 refactoring backport and `pallet_bridge_messages` simplifications (#4935)
  xcm-executor: Improve logging (#4996)
  Remove usage of `sp-std` on templates (#5001)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
T7-smart_contracts This PR/Issue is related to smart contracts.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants