Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AddRemoveDC nemesis should create keyspace with tablets disabled or change rf for new dc before decommission #9861

Open
aleksbykov opened this issue Jan 20, 2025 · 4 comments · May be fixed by #10028
Open

AddRemoveDC nemesis should create keyspace with tablets disabled or change rf for new dc before decommission #9861

aleksbykov opened this issue Jan 20, 2025 · 4 comments · May be fixed by #10028
Assignees
@timtimb0t

Comments

@aleksbykov
Copy link
Contributor

With th latest schlla master AddRemoveDC nemesis start failing, because:

  • tablets enabled by default for every keyspace, and could be disabled during create/alter keyspace
  • node could be decommissioned if rf of keyspace == number of nodes

AddRemoveDC nemesis first create new keyspace keyspace_new_dc without any options for tablets. So tablets are enabled for it.
and then create new dc with single node and alter the keyspace with rf=1 for new dc. This cause that decommission of new node failed because tablets enabled and rf = num of nodes in DC.

More info scylladb/scylladb#22261

AddRemoveDC should be fixed with:

  • create kyespace with disable tablets explicitly.
  • before decomission the new added node change rf=0 for new dc.
@aleksbykov aleksbykov mentioned this issue Jan 20, 2025
Closed
2 tasks
@temichus temichus changed the title AddRemoveDC nemesis should create keyspace with tablets disaled or change rf for new dc before decommission AddRemoveDC nemesis should create keyspace with tablets disabled or change rf for new dc before decommission Jan 27, 2025
@timtimb0t
Copy link
Contributor

Packages

Scylla version: 2025.2.0~dev-20250131.e1b1a2068a4d with build-id 33145e2edb15d3b9de68da4b5144fe5f9774e65e

Kernel Version: 6.8.0-1021-azure

Installation details

Cluster size: 4 nodes (Standard_L16s_v3)

Scylla Nodes used in this run:

  • longevity-tls-1tb-7d-master-db-node-0b41f758-eastus-7 (null | 10.0.0.14) (shards: 14)
  • longevity-tls-1tb-7d-master-db-node-0b41f758-eastus-6 (null | 10.0.0.8) (shards: 14)
  • longevity-tls-1tb-7d-master-db-node-0b41f758-eastus-5 (null | 10.0.0.14) (shards: 14)
  • longevity-tls-1tb-7d-master-db-node-0b41f758-eastus-4 (null | 10.0.0.8) (shards: 14)
  • longevity-tls-1tb-7d-master-db-node-0b41f758-eastus-3 (null | 10.0.0.7) (shards: 14)
  • longevity-tls-1tb-7d-master-db-node-0b41f758-eastus-2 (null | 10.0.0.6) (shards: 14)
  • longevity-tls-1tb-7d-master-db-node-0b41f758-eastus-1 (null | 10.0.0.5) (shards: 14)

OS / Image: /subscriptions/6c268694-47ab-43ab-b306-3c5514bc4112/resourceGroups/SCYLLA-IMAGES/providers/Microsoft.Compute/images/scylla-2025.2.0-dev-x86_64-2025-02-01T01-56-30 (azure: undefined_region)

Test: longevity-1tb-5days-azure-test
Test id: 0b41f758-f4ae-4f75-934c-2ac0dc1a8af4
Test name: scylla-master/tier1/longevity-1tb-5days-azure-test
Test method: longevity_test.LongevityTest.test_custom_time
Test config file(s):

Logs and commands
  • Restore Monitor Stack command: $ hydra investigate show-monitor 0b41f758-f4ae-4f75-934c-2ac0dc1a8af4
  • Restore monitor on AWS instance using Jenkins job
  • Show all stored logs command: $ hydra investigate show-logs 0b41f758-f4ae-4f75-934c-2ac0dc1a8af4

Logs:

Jenkins job URL
Argus

@timtimb0t
Copy link
Contributor

Packages

Scylla version: 2025.2.0~dev-20250131.e1b1a2068a4d with build-id 33145e2edb15d3b9de68da4b5144fe5f9774e65e

Kernel Version: 6.8.0-1021-aws

Installation details

Cluster size: 5 nodes (i4i.2xlarge)

Scylla Nodes used in this run:

  • longevity-parallel-topology-schema--db-node-81141112-9 (54.77.210.57 | 10.4.8.76) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-8 (54.75.253.121 | 10.4.11.198) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-72 (99.80.25.8 | 10.4.10.176) (shards: -1)
  • longevity-parallel-topology-schema--db-node-81141112-71 (52.17.212.217 | 10.4.11.229) (shards: -1)
  • longevity-parallel-topology-schema--db-node-81141112-70 (54.76.166.195 | 10.4.8.179) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-7 (52.19.223.87 | 10.4.11.128) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-69 (54.73.234.125 | 10.4.8.165) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-68 (54.246.211.94 | 10.4.8.171) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-67 (54.220.59.9 | 10.4.9.203) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-66 (52.31.10.28 | 10.4.10.213) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-65 (63.34.217.55 | 10.4.11.166) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-64 (52.49.86.220 | 10.4.9.214) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-63 (52.215.16.6 | 10.4.8.84) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-62 (3.254.34.244 | 10.4.10.250) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-61 (52.210.216.52 | 10.4.8.52) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-60 (99.81.19.58 | 10.4.8.104) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-6 (34.252.209.22 | 10.4.10.248) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-59 (34.246.159.212 | 10.4.8.132) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-58 (99.80.25.240 | 10.4.11.159) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-57 (34.249.66.109 | 10.4.11.27) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-56 (52.214.19.239 | 10.4.10.217) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-55 (54.194.136.147 | 10.4.10.82) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-54 (18.202.120.5 | 10.4.9.139) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-53 (63.35.11.126 | 10.4.8.210) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-52 (52.214.116.148 | 10.4.8.252) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-51 (54.74.106.212 | 10.4.11.5) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-50 (46.137.90.59 | 10.4.9.193) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-5 (52.50.106.13 | 10.4.8.5) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-49 (52.213.121.55 | 10.4.8.196) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-48 (34.240.161.101 | 10.4.10.89) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-47 (52.50.156.90 | 10.4.11.254) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-46 (54.194.101.161 | 10.4.10.139) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-45 (46.137.161.67 | 10.4.11.8) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-44 (54.220.239.109 | 10.4.8.168) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-43 (54.155.140.172 | 10.4.9.251) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-42 (99.80.26.189 | 10.4.9.144) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-41 (52.19.135.51 | 10.4.10.150) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-40 (46.137.115.167 | 10.4.8.55) (shards: -1)
  • longevity-parallel-topology-schema--db-node-81141112-4 (34.255.142.240 | 10.4.10.36) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-39 (52.48.36.177 | 10.4.10.163) (shards: -1)
  • longevity-parallel-topology-schema--db-node-81141112-38 (46.137.122.7 | 10.4.8.34) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-37 (54.77.244.122 | 10.4.10.40) (shards: -1)
  • longevity-parallel-topology-schema--db-node-81141112-36 (52.214.202.135 | 10.4.11.92) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-35 (46.137.122.7 | 10.4.9.153) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-34 (99.80.45.217 | 10.4.10.207) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-33 (54.76.43.237 | 10.4.11.28) (shards: -1)
  • longevity-parallel-topology-schema--db-node-81141112-32 (54.217.246.224 | 10.4.9.124) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-31 (54.228.21.249 | 10.4.8.244) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-30 (54.77.46.114 | 10.4.8.134) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-3 (52.212.67.19 | 10.4.11.45) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-29 (18.200.70.143 | 10.4.11.133) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-28 (54.171.250.176 | 10.4.9.217) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-27 (54.154.93.116 | 10.4.9.179) (shards: -1)
  • longevity-parallel-topology-schema--db-node-81141112-26 (52.30.10.60 | 10.4.8.52) (shards: -1)
  • longevity-parallel-topology-schema--db-node-81141112-25 (52.19.177.200 | 10.4.9.180) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-24 (34.249.199.114 | 10.4.11.77) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-23 (3.255.64.46 | 10.4.10.207) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-22 (54.75.115.161 | 10.4.10.224) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-21 (54.155.248.86 | 10.4.8.189) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-20 (34.250.240.239 | 10.4.11.37) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-2 (52.211.20.155 | 10.4.8.179) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-19 (99.81.47.180 | 10.4.11.180) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-18 (54.154.73.143 | 10.4.9.33) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-17 (54.78.242.176 | 10.4.9.105) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-16 (34.250.111.172 | 10.4.8.42) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-15 (52.211.112.228 | 10.4.9.233) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-14 (46.51.174.164 | 10.4.10.191) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-13 (52.30.192.33 | 10.4.10.80) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-12 (34.254.84.184 | 10.4.9.255) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-11 (176.34.93.192 | 10.4.10.95) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-10 (63.32.167.57 | 10.4.10.92) (shards: 7)
  • longevity-parallel-topology-schema--db-node-81141112-1 (54.194.180.172 | 10.4.10.26) (shards: 7)

OS / Image: ami-0ce83b2d09a87141b (aws: undefined_region)

Test: longevity-schema-topology-changes-12h-test
Test id: 81141112-f4de-4601-b386-bf4d3214a167
Test name: scylla-master/tier1/longevity-schema-topology-changes-12h-test
Test method: longevity_test.LongevityTest.test_custom_time
Test config file(s):

Logs and commands
  • Restore Monitor Stack command: $ hydra investigate show-monitor 81141112-f4de-4601-b386-bf4d3214a167
  • Restore monitor on AWS instance using Jenkins job
  • Show all stored logs command: $ hydra investigate show-logs 81141112-f4de-4601-b386-bf4d3214a167

Logs:

Jenkins job URL
Argus

@timtimb0t
Copy link
Contributor

Packages

Scylla version: 2025.2.0~dev-20250131.e1b1a2068a4d with build-id 33145e2edb15d3b9de68da4b5144fe5f9774e65e

Kernel Version: 6.8.0-1021-aws

Installation details

Cluster size: 4 nodes (i4i.4xlarge)

Scylla Nodes used in this run:

  • longevity-50gb-12h-master-db-node-0e39a2c2-7 (3.255.219.172 | 10.4.10.42) (shards: 12)
  • longevity-50gb-12h-master-db-node-0e39a2c2-6 (3.253.52.241 | 10.4.11.55) (shards: 8)
  • longevity-50gb-12h-master-db-node-0e39a2c2-5 (34.240.172.11 | 10.4.8.29) (shards: 12)
  • longevity-50gb-12h-master-db-node-0e39a2c2-4 (54.75.144.239 | 10.4.10.249) (shards: 9)
  • longevity-50gb-12h-master-db-node-0e39a2c2-3 (54.78.238.249 | 10.4.10.96) (shards: 13)
  • longevity-50gb-12h-master-db-node-0e39a2c2-2 (3.248.223.234 | 10.4.8.192) (shards: 11)
  • longevity-50gb-12h-master-db-node-0e39a2c2-1 (3.255.155.85 | 10.4.9.199) (shards: 10)

OS / Image: ami-0ce83b2d09a87141b (aws: undefined_region)

Test: longevity-150gb-asymmetric-cluster-12h-test
Test id: 0e39a2c2-45d0-4eae-a5c6-03a72fd9b0b1
Test name: scylla-master/tier1/longevity-150gb-asymmetric-cluster-12h-test
Test method: longevity_test.LongevityTest.test_custom_time
Test config file(s):

Logs and commands
  • Restore Monitor Stack command: $ hydra investigate show-monitor 0e39a2c2-45d0-4eae-a5c6-03a72fd9b0b1
  • Restore monitor on AWS instance using Jenkins job
  • Show all stored logs command: $ hydra investigate show-logs 0e39a2c2-45d0-4eae-a5c6-03a72fd9b0b1

Logs:

Jenkins job URL
Argus

@timtimb0t
Copy link
Contributor

Packages

Scylla version: 2025.2.0~dev-20250131.e1b1a2068a4d with build-id 33145e2edb15d3b9de68da4b5144fe5f9774e65e

Kernel Version: 6.8.0-1021-gcp

Installation details

Cluster size: 5 nodes (n2-highmem-16)

Scylla Nodes used in this run:

  • longevity-large-partitions-200k-pks-db-node-9b896dc6-0-7 (35.243.142.13 | 10.142.0.22) (shards: 14)
  • longevity-large-partitions-200k-pks-db-node-9b896dc6-0-6 (35.227.116.77 | 10.142.0.13) (shards: 14)
  • longevity-large-partitions-200k-pks-db-node-9b896dc6-0-5 (34.23.125.229 | 10.142.0.232) (shards: 14)
  • longevity-large-partitions-200k-pks-db-node-9b896dc6-0-4 (34.138.173.105 | 10.142.0.231) (shards: 14)
  • longevity-large-partitions-200k-pks-db-node-9b896dc6-0-3 (35.231.188.62 | 10.142.0.222) (shards: 14)
  • longevity-large-partitions-200k-pks-db-node-9b896dc6-0-2 (35.185.115.142 | 10.142.0.217) (shards: 14)
  • longevity-large-partitions-200k-pks-db-node-9b896dc6-0-1 (34.74.108.228 | 10.142.0.209) (shards: 14)

OS / Image: https://www.googleapis.com/compute/v1/projects/scylla-images/global/images/scylla-2025-2-0-dev-x86-64-2025-02-01t01-49-11 (gce: undefined_region)

Test: longevity-large-partition-200k-pks-4days-gce-test
Test id: 9b896dc6-6b52-4122-8dda-2ab71aa49e0f
Test name: scylla-master/tier1/longevity-large-partition-200k-pks-4days-gce-test
Test method: longevity_large_partition_test.LargePartitionLongevityTest.test_large_partition_longevity
Test config file(s):

Logs and commands
  • Restore Monitor Stack command: $ hydra investigate show-monitor 9b896dc6-6b52-4122-8dda-2ab71aa49e0f
  • Restore monitor on AWS instance using Jenkins job
  • Show all stored logs command: $ hydra investigate show-logs 9b896dc6-6b52-4122-8dda-2ab71aa49e0f

Logs:

Jenkins job URL
Argus

@timtimb0t timtimb0t linked a pull request Feb 10, 2025 that will close this issue
Draft
2 tasks
@pehala pehala mentioned this issue Feb 13, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@timtimb0t timtimb0t

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(nemesis): replication factor set to 0 before decommission timtimb0t/scylla-cluster-tests
3 participants
@temichus @aleksbykov @timtimb0t