2023-08 CrateDB 5.4.md

Ingestion speed results CrateDB 5.4

This page documents ingest benchmark results from our one million rows/s blog post. While the above-linked article regularly gets updated to reflect most recent version of CrateDB, hardware architecture, etc., historic benchmark results are documented here.

Setup

• CrateDB version: 5.4.0

• EC2 instance type: m6in.4xlarge

• Operating system: Amazon Linux 2023

• Terraform config:

   module "cratedb-cluster" {
       source = "[email protected]:crate/crate-terraform.git//aws"
  
       region = "eu-west-1"
  
       vpc_id             = "vpc-..."
       subnet_ids         = ["subnet-..."]
       availability_zones = ["eu-west-1b"]
  
       ssh_keypair = "cratedb_terraform"
       ssh_access  = true
  
       instance_type         = "m6in.4xlarge"
       instance_architecture = "x86_64"
  
       # The size of the disk storing CrateDB's data directory
       disk_size_gb = 100
  
       disk_iops       = 5000
       disk_throughput = 400 # MiB/s
  
       # CrateDB-specific configuration
       crate = {
           # Java Heap size in GB available to CrateDB
           heap_size_gb = 40
  
           cluster_name = "cratedb-cluster"
  
           # The number of nodes the cluster will consist of
           cluster_size = 1 # increase to scale the cluster
  
           ssl_enable = true
       }
  
       enable_utility_vm      = true
       load_balancer_internal = true
  
       utility_vm = {
           instance_type         = "m6in.4xlarge"
           instance_architecture = "x86_64"
           disk_size_gb          = 50
       }
   }
  
   output "cratedb" {
       value     = module.cratedb-cluster
       sensitive = true
   }

Results

Without replication

Nodes	Cluster size increase	Throughput (rows/s)	Throughput increase absolute (rows/s)	Throughput increase relative	Primary shards	Processes	Concurrent requests
1		293,147			16	1	16
2	+100%	493,723	+200,575	+68%	24	2	12
3	+50%	676,781	+183,058	+37%	32	3	11
4	+33%	856,480	+179,680	+27%	40	4	10
5	+25%	1,043,659	+187,198	+22%	48	5	9

With 1 replica

Nodes	Cluster size increase	Throughput (rows/s)	Throughput increase absolute (rows/s)	Throughput increase relative	Primary shards	Processes	Concurrent requests
2		286,341			12	2	12
3	+50%	403,630	+117,288	+41%	16	3	11
4	+33%	510,086	+106,456	+26%	20	4	10
5	+25%	629,568	+119,482	+23%	24	5	9