Upgrade RocksDB to version 9.2.1

[dlg99]

Descriptions of the changes in this PR:

Fix #4409

Motivation

Upgrade RocksDB to v.9.2.1 to pick up latest performance and stability improvements and fixes in deleteRanges.
Hopefully deleteRanges() is stable enough for us to try again perf improvements #3653 that were reverted previously.

Changes

Changed RocksDB version, updated code to use changed API.

[hangc0276]

Have you verified any compatible issues from 6.16.x to 9.2.1 and 7.x to 9.2.1?

[dlg99]

Have you verified any compatible issues from 6.16.x to 9.2.1 and 7.x to 9.2.1?

@hangc0276 I fixed/updated backward compat tests, that's it. If these aren't enough we need to invest some time into these kinds of tests

[dlg99]

@hangc0276 I rebased after the merge of downgrade compat tests; upgrade to 9.2.1 doesn't downgrade nicely. I am not yet sure what can be configured to support compatibility.

[lhotari]

@hangc0276 I rebased after the merge of downgrade compat tests; upgrade to 9.2.1 doesn't downgrade nicely. I am not yet sure what can be configured to support compatibility.

In RocksDB 9.0.0 release notes there's "format_version=6 is the new default setting in BlockBasedTableOptions, for more robust data integrity checking. DBs and SST files written with this setting cannot be read by RocksDB versions before 8.6.0."

@dlg99 Perhaps setting dbStorage_rocksDB_format_version=5 instead of dbStorage_rocksDB_format_version=2 could help solve this issue? It seems that format_version=2 is really old and doesn't support many optimizations.

[lhotari]

I created an issue #4479 about upgrading the default format_version to 5.

[lhotari]

I created PR #4480 to upgrade default format_version to 5.

[hangc0276]

@lhotari Have you tested the downgrade? We set the format_version to 2 by default, and it may not impacted by version 6

[lhotari]

@lhotari Have you tested the downgrade? We set the format_version to 2 by default, and it may not impacted by version 6

@hangc0276 The format_version setting was missing for conf/ledger_metadata_rocksdb.conf. that's been a problem previously. I haven't tested it yet.

[hangc0276]

@lhotari Have you tested the downgrade? We set the format_version to 2 by default, and it may not impacted by version 6

@hangc0276 The format_version setting was missing for conf/ledger_metadata_rocksdb.conf. that's been a problem previously. I haven't tested it yet.

@lhotari It is set to 2

• bookkeeper/conf/entry_location_rocksdb.conf.default

  Lines 62 to 63 in 8890a61

  	# set by jni: tableOptions.setFormatVersion
  	format_version=2

• https://github.com/apache/pulsar/blob/7636e8989f4d3fc24fce69a356d54e1c550945ed/conf/entry_location_rocksdb.conf#L61-L62

[lhotari]

@lhotari It is set to 2

• bookkeeper/conf/entry_location_rocksdb.conf.default

  Lines 62 to 63 in 8890a61

  	# set by jni: tableOptions.setFormatVersion
  	format_version=2

• https://github.com/apache/pulsar/blob/7636e8989f4d3fc24fce69a356d54e1c550945ed/conf/entry_location_rocksdb.conf#L61-L62

@hangc0276 Yes, you are right that format_version should already be set to 2.
I was assuming that the ledger metadata database would have been the problem. However, when conf/entry_location_rocksdb.conf is missing, the format_version set in code would get used.
(logic

bookkeeper/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/storage/ldb/KeyValueStorageRocksDB.java

Lines 126 to 140 in f048e7a

	if (dbConfigType == DbConfigType.EntryLocation) {
	dbFilePath = conf.getEntryLocationRocksdbConf();
	} else if (dbConfigType == DbConfigType.LedgerMetadata) {
	dbFilePath = conf.getLedgerMetadataRocksdbConf();
	} else {
	dbFilePath = conf.getDefaultRocksDBConf();
	}
	log.info("Searching for a RocksDB configuration file in {}", dbFilePath);
	if (Paths.get(dbFilePath).toFile().exists()) {
	log.info("Found a RocksDB configuration file and using it to initialize the RocksDB");
	db = initializeRocksDBWithConfFile(basePath, subPath, dbConfigType, conf, readOnly, dbFilePath);
	} else {
	log.info("Haven't found the file and read the configuration from the main bookkeeper configuration");
	db = initializeRocksDBWithBookieConf(basePath, subPath, dbConfigType, conf, readOnly);
	}

)
By default in BK, the file is missing since the provide file is conf/entry_location_rocksdb.conf.default and that has no impact.

However, since format_version 2 is ancient, you never know if it starts causing problems when continuing to use it with RocskDB 9.x+. I haven't yet investigated the problem.