Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mempool housekeeping #2137

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Mempool housekeeping #2137

wants to merge 4 commits into from

Conversation

edmundnoble
Copy link
Contributor

Slight performance improvements across the board, and simplifications to the getBlock code.

The following benchmarks include those added in #2136.

benchmarking mempool/mempoolGetBlock 1					   benchmarking mempool/mempoolGetBlock 1
time		     9.744 μs   (9.636 μs .. 9.842 μs)			   time			7.930 μs	  (7.840 μs .. 8.014 μs)
		     0.999 R²   (0.999 R² .. 1.000 R²)			   			0.999 R²	  (0.999 R² .. 1.000 R²)
mean		     9.644 μs   (9.565 μs .. 9.725 μs)			   mean			7.883 μs	  (7.829 μs .. 7.971 μs)
std dev		     216.9 ns	(193.2 ns .. 254.1 ns)			   std dev		171.9 ns   (123.9 ns .. 219.1 ns)
variance introduced by outliers: 22% (moderately inflated)		   variance introduced by outliers: 21% (moderately inflated)
									   
benchmarking mempool/mempoolGetBlock 16					   benchmarking mempool/mempoolGetBlock 16
time		     108.6 μs   (108.5 μs .. 108.8 μs)			   time			98.82 μs	  (98.69 μs .. 98.96 μs)
		     1.000 R²   (1.000 R² .. 1.000 R²)			   			1.000 R²	  (1.000 R² .. 1.000 R²)
mean		     108.8 μs   (108.7 μs .. 109.1 μs)			   mean			98.86 μs	  (98.73 μs .. 98.99 μs)
std dev		     700.5 ns	(550.3 ns .. 861.0 ns)			   std dev		559.5 ns   (463.4 ns .. 686.7 ns)
									   
benchmarking mempool/mempoolGetBlock 64					   benchmarking mempool/mempoolGetBlock 64
time		     446.3 μs   (443.6 μs .. 449.1 μs)			   time			397.2 μs	  (393.8 μs .. 400.8 μs)
		     0.999 R²   (0.999 R² .. 1.000 R²)			   			0.999 R²	  (0.998 R² .. 0.999 R²)
mean		     439.7 μs   (434.6 μs .. 443.4 μs)			   mean			398.0 μs	  (394.5 μs .. 400.6 μs)
std dev		     17.03 μs   (13.73 μs .. 21.47 μs)			   std dev		12.57 μs	  (9.862 μs .. 15.72 μs)
variance introduced by outliers: 36% (moderately inflated)		   variance introduced by outliers: 28% (moderately inflated)
									   
benchmarking mempool/mempoolGetBlock 256				   benchmarking mempool/mempoolGetBlock 256
time		     2.172 ms	(2.163 ms .. 2.181 ms)			   time			1.807 ms   (1.725 ms .. 1.856 ms)
		     1.000 R²   (1.000 R² .. 1.000 R²)			   			0.991 R²	  (0.983 R² .. 0.998 R²)
mean		     2.159 ms	(2.134 ms .. 2.170 ms)			   mean			1.850 ms   (1.826 ms .. 1.868 ms)
std dev		     65.99 μs   (27.29 μs .. 119.3 μs)			   std dev		86.48 μs	  (51.67 μs .. 123.8 μs)
variance introduced by outliers: 21% (moderately inflated)		   variance introduced by outliers: 39% (moderately inflated)
									   
benchmarking mempool/mempoolGetBlock 1024				   benchmarking mempool/mempoolGetBlock 1024
time		     8.962 ms	(8.908 ms .. 9.005 ms)			   time			7.657 ms   (7.443 ms .. 7.797 ms)
		     1.000 R²   (1.000 R² .. 1.000 R²)			   			0.991 R²	  (0.981 R² .. 0.998 R²)
mean		     8.908 ms	(8.870 ms .. 8.946 ms)			   mean			7.725 ms   (7.578 ms .. 7.824 ms)
std dev		     138.8 μs   (98.92 μs .. 221.2 μs)			   std dev		393.0 μs	  (244.6 μs .. 558.2 μs)
									   variance introduced by outliers: 33% (moderately inflated)
benchmarking mempool/mempoolGetBlock 4096				   
time		     31.65 ms	(29.22 ms .. 35.49 ms)			   benchmarking mempool/mempoolGetBlock 4096
		     0.964 R²   (0.940 R² .. 0.990 R²)			   time			30.34 ms   (28.44 ms .. 32.00 ms)
mean		     33.68 ms	(32.26 ms .. 34.73 ms)			   			0.978 R²	  (0.964 R² .. 0.989 R²)
std dev		     2.999 ms	(2.359 ms .. 3.446 ms)			   mean			28.93 ms   (28.08 ms .. 30.01 ms)
variance introduced by outliers: 45% (moderately inflated)		   std dev		2.503 ms   (2.227 ms .. 2.694 ms)
									   variance introduced by outliers: 45% (moderately inflated)
benchmarking mempool/mempoolGetBlockHalfExcludedHashes 1		   
time		     9.658 μs   (9.567 μs .. 9.768 μs)			   benchmarking mempool/mempoolGetBlockHalfExcludedHashes 1
		     0.999 R²   (0.999 R² .. 0.999 R²)			   time			8.029 μs	  (7.914 μs .. 8.145 μs)
mean		     9.697 μs   (9.623 μs .. 9.798 μs)			   			0.999 R²	  (0.998 R² .. 0.999 R²)
std dev		     261.1 ns	(223.0 ns .. 313.1 ns)			   mean			8.026 μs	  (7.940 μs .. 8.130 μs)
variance introduced by outliers: 29% (moderately inflated)		   std dev		237.9 ns   (203.9 ns .. 279.7 ns)
									   variance introduced by outliers: 33% (moderately inflated)
benchmarking mempool/mempoolGetBlockHalfExcludedHashes 16		   
time		     61.47 μs   (61.34 μs .. 61.63 μs)			   benchmarking mempool/mempoolGetBlockHalfExcludedHashes 16
		     1.000 R²   (1.000 R² .. 1.000 R²)			   time			53.72 μs	  (53.59 μs .. 53.89 μs)
mean		     61.51 μs   (61.41 μs .. 61.63 μs)			   			1.000 R²	  (1.000 R² .. 1.000 R²)
std dev		     402.1 ns	(326.3 ns .. 494.8 ns)			   mean			53.96 μs	  (53.85 μs .. 54.11 μs)
									   std dev		491.9 ns   (419.5 ns .. 597.8 ns)
benchmarking mempool/mempoolGetBlockHalfExcludedHashes 64		   
time		     220.5 μs   (218.9 μs .. 222.0 μs)			   benchmarking mempool/mempoolGetBlockHalfExcludedHashes 64
		     1.000 R²   (1.000 R² .. 1.000 R²)			   time			197.6 μs	  (196.6 μs .. 198.6 μs)
mean		     220.5 μs   (219.5 μs .. 221.5 μs)			   			1.000 R²	  (1.000 R² .. 1.000 R²)
std dev		     4.027 μs   (3.674 μs .. 4.507 μs)			   mean			197.2 μs	  (196.6 μs .. 197.7 μs)
variance introduced by outliers: 13% (moderately inflated)		   std dev		2.144 μs	  (1.922 μs .. 2.470 μs)
									   
benchmarking mempool/mempoolGetBlockHalfExcludedHashes 256		   benchmarking mempool/mempoolGetBlockHalfExcludedHashes 256
time		     1.097 ms	(1.069 ms .. 1.114 ms)			   time			887.0 μs	  (866.5 μs .. 898.5 μs)
		     0.994 R²   (0.990 R² .. 0.997 R²)			   			0.996 R²	  (0.990 R² .. 0.999 R²)
mean		     1.089 ms	(1.066 ms .. 1.105 ms)			   mean			896.1 μs	  (882.7 μs .. 903.4 μs)
std dev		     71.62 μs   (56.18 μs .. 89.37 μs)			   std dev		41.61 μs	  (26.87 μs .. 61.21 μs)
variance introduced by outliers: 58% (severely inflated)		   variance introduced by outliers: 42% (moderately inflated)
									   
benchmarking mempool/mempoolGetBlockHalfExcludedHashes 1024		   benchmarking mempool/mempoolGetBlockHalfExcludedHashes 1024
time		     4.532 ms	(4.341 ms .. 4.709 ms)			   time			4.125 ms   (4.105 ms .. 4.138 ms)
		     0.990 R²   (0.985 R² .. 0.996 R²)			   			1.000 R²	  (1.000 R² .. 1.000 R²)
mean		     4.708 ms	(4.635 ms .. 4.766 ms)			   mean			4.112 ms   (4.089 ms .. 4.128 ms)
std dev		     237.1 μs   (168.3 μs .. 310.5 μs)			   std dev		72.68 μs	  (53.92 μs .. 102.7 μs)
variance introduced by outliers: 35% (moderately inflated)		   
									   benchmarking mempool/mempoolGetBlockHalfExcludedHashes 4096
benchmarking mempool/mempoolGetBlockHalfExcludedHashes 4096		   time			16.28 ms   (16.17 ms .. 16.40 ms)
time		     18.79 ms	(18.19 ms .. 19.18 ms)			   			1.000 R²	  (0.999 R² .. 1.000 R²)
		     0.997 R²   (0.992 R² .. 1.000 R²)			   mean			16.08 ms   (15.94 ms .. 16.16 ms)
mean		     19.09 ms	(18.94 ms .. 19.19 ms)			   std dev		318.8 μs	  (220.0 μs .. 420.5 μs)
std dev		     431.3 μs   (223.9 μs .. 876.7 μs)			   
									   benchmarking mempool/mempoolInsertChecked
benchmarking mempool/mempoolInsertChecked				   time			3.438 ms   (3.291 ms .. 3.565 ms)
time		     3.979 ms	(3.959 ms .. 4.000 ms)			   			0.991 R²	  (0.983 R² .. 1.000 R²)
		     0.999 R²   (0.999 R² .. 1.000 R²)			   mean			3.535 ms   (3.487 ms .. 3.558 ms)
mean		     3.753 ms	(3.675 ms .. 3.820 ms)			   std dev		136.3 μs	  (74.83 μs .. 213.8 μs)
std dev		     289.9 μs   (248.5 μs .. 327.3 μs)			   variance introduced by outliers: 27% (moderately inflated)
variance introduced by outliers: 59% (severely inflated)		   
									   benchmarking mempool/mempoolInsert
benchmarking mempool/mempoolInsert					   time			2.324 ms   (2.203 ms .. 2.443 ms)
time		     2.459 ms	(2.384 ms .. 2.506 ms)			   			0.985 R²	  (0.982 R² .. 0.990 R²)
		     0.995 R²   (0.991 R² .. 0.999 R²)			   mean			2.454 ms   (2.400 ms .. 2.494 ms)
mean		     2.493 ms	(2.468 ms .. 2.509 ms)			   std dev		185.2 μs	  (160.7 μs .. 205.5 μs)
std dev		     86.76 μs   (54.30 μs .. 129.7 μs)			   variance introduced by outliers: 61% (severely inflated)
variance introduced by outliers: 24% (moderately inflated)		   
									   benchmarking mempool/mempoolInsertMultipleBatches
benchmarking mempool/mempoolInsertMultipleBatches			   time			2.880 ms   (2.841 ms .. 2.901 ms)
time		     3.169 ms	(3.162 ms .. 3.175 ms)			   			0.998 R²	  (0.993 R² .. 1.000 R²)
		     1.000 R²   (1.000 R² .. 1.000 R²)			   mean			2.893 ms   (2.856 ms .. 2.908 ms)
mean		     3.148 ms	(3.141 ms .. 3.154 ms)			   std dev		91.02 μs	  (37.01 μs .. 155.8 μs)
std dev		     25.35 μs   (19.68 μs .. 33.28 μs)			   variance introduced by outliers: 21% (moderately inflated)
									   
benchmarking mempool/mempoolInsertOverlappingBatches			   benchmarking mempool/mempoolInsertOverlappingBatches
time		     6.228 ms	(6.211 ms .. 6.245 ms)			   time			5.212 ms   (4.904 ms .. 5.590 ms)
		     1.000 R²   (1.000 R² .. 1.000 R²)			   			0.981 R²	  (0.975 R² .. 0.989 R²)
mean		     6.159 ms	(6.141 ms .. 6.173 ms)			   mean			5.753 ms   (5.613 ms .. 5.829 ms)
std dev		     60.10 μs   (51.21 μs .. 75.98 μs)			   std dev		338.7 μs	  (225.3 μs .. 456.8 μs)
									   variance introduced by outliers: 43% (moderately inflated)
benchmarking mempool/mempoolAddToBadList				   
time		     604.7 μs   (598.0 μs .. 609.3 μs)			   benchmarking mempool/mempoolAddToBadList
		     0.998 R²   (0.997 R² .. 0.999 R²)			   time			406.3 μs	  (399.6 μs .. 413.0 μs)
mean		     598.1 μs   (589.3 μs .. 602.1 μs)			   			0.997 R²	  (0.996 R² .. 0.998 R²)
std dev		     22.89 μs   (16.52 μs .. 29.98 μs)			   mean			395.5 μs	  (389.0 μs .. 401.7 μs)
variance introduced by outliers: 32% (moderately inflated)		   std dev		22.53 μs	  (18.14 μs .. 26.19 μs)
									   variance introduced by outliers: 53% (severely inflated)
benchmarking mempool/mempoolPrune					   
time		     157.6 μs   (157.1 μs .. 158.1 μs)			   benchmarking mempool/mempoolPrune
		     1.000 R²   (1.000 R² .. 1.000 R²)			   time			136.6 μs	  (135.7 μs .. 137.4 μs)
mean		     157.3 μs   (156.7 μs .. 157.7 μs)			   			0.999 R²	  (0.999 R² .. 1.000 R²)
std dev		     1.068 μs   (844.6 ns .. 1.319 μs)			   mean			136.1 μs	  (135.2 μs .. 136.9 μs)
									   std dev		1.645 μs	  (1.314 μs .. 2.073 μs)
benchmarking mempool/mempoolPruneExpired				   
time		     144.4 μs   (142.8 μs .. 145.8 μs)			   benchmarking mempool/mempoolPruneExpired
		     0.999 R²   (0.999 R² .. 1.000 R²)			   time			121.1 μs	  (119.6 μs .. 122.8 μs)
mean		     145.1 μs   (143.8 μs .. 146.2 μs)			   			0.999 R²	  (0.998 R² .. 0.999 R²)
std dev		     2.832 μs   (1.911 μs .. 3.855 μs)			   mean			122.0 μs	  (120.2 μs .. 123.4 μs)
variance introduced by outliers: 11% (moderately inflated)		   std dev		3.456 μs	  (2.819 μs .. 4.310 μs)
									   variance introduced by outliers: 21% (moderately inflated)
1,684,442,511,504 bytes allocated in the heap				   
 170,324,438,352 bytes copied during GC					   1,618,662,106,032 bytes allocated in the heap
      18,384,320 bytes maximum residency (4813 sample(s))		    172,170,485,256 bytes copied during GC
	 883,840 bytes maximum slop					   	 17,075,496 bytes maximum residency (4833 sample(s))
	      90 MiB total memory in use (0 MiB lost due to fragmentation) 	    938,128 bytes maximum slop
									   		 88 MiB total memory in use (0 MiB lost due to fragmentation)
				     Tot time (elapsed)	 Avg pause  Max pa 
  Gen  0     1953839 colls, 1953839 par	  101.266s  52.339s	0.0000s	   					Tot time (elapsed)  Avg pause  Max pa
  Gen  1      4813 colls,  4812 par   37.794s  10.378s	   0.0022s    0.00   Gen  0	2060662 colls, 2060662 par   100.899s  50.663s	   0.0000s
									     Gen  1	 4833 colls,  4832 par	 37.172s  10.490s     0.0022s	 0.00
  Parallel GC work balance: 27.34% (serial 0%, perfect 100%)		   
									     Parallel GC work balance: 27.09% (serial 0%, perfect 100%)
  TASKS: 18 (1 bound, 17 peak workers (17 total), using -N8)		   
									     TASKS: 18 (1 bound, 17 peak workers (17 total), using -N8)
  SPARKS: 38 (38 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled)	   
									     SPARKS: 38 (38 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled)
  INIT	  time	  0.001s  (  0.001s elapsed)				   
  MUT	  time	307.980s  (258.445s elapsed)				     INIT    time    0.002s  (	0.001s elapsed)
  GC	  time	139.060s  ( 62.717s elapsed)				     MUT     time  323.885s  (270.747s elapsed)
  EXIT	  time	  0.000s  (  0.000s elapsed)				     GC	     time  138.071s  ( 61.154s elapsed)
  Total	  time	447.042s  (321.163s elapsed)				     EXIT    time    0.000s  (	0.000s elapsed)
									     Total   time  461.959s  (331.902s elapsed)
  Alloc rate	5,469,330,952 bytes per MUT second			   
									     Alloc rate	   4,997,640,485 bytes per MUT second
  Productivity	68.9% of total user, 80.5% of total elapsed		   
									     Productivity  70.1% of total user, 81.6% of total elapsed

@edmundnoble
List sort by gas price instead of timsort
52562fa 
The idea here is, if there are more transactions than we want to look
at, we only sort the ones we want to look at.
Change-Id: Id00000007aeb15e3560626a104e1bc0e734cd051
@edmundnoble
Use Map instead of HashMap in the mempool
993f6fd 
Change-Id: Id0000000b84389feac559725b4c3cffb48df193d
@edmundnoble
use withoutKeys to remove seen txs
8f9e38c
@edmundnoble
Simplify getBlock implementation
44c657a 
Change-Id: Id00000004718d11610b5847fe38f1df56c71b80d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@edmundnoble