Issue 1626: Add tests for LedgerUnderredplicationManager#markLedgerUnderreplicatedAsync #1629

…pache#4057) ### Motivation Rack-aware placement policies enable preference for bookies that reside in the same rack as the bookie client. - Initiate local node by resolving the rack information https://github.com/apache/bookkeeper/blob/5b5c05331757e7356579076970e61f119f5d34ae/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/RackawareEnsemblePlacementPolicyImpl.java#L207-L215 - When generating new ensembles for a ledger, the selecting algorithm will set the localNode's rack to `curRack` and select one bookie from `curRack` first https://github.com/apache/bookkeeper/blob/5b5c05331757e7356579076970e61f119f5d34ae/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/RackawareEnsemblePlacementPolicyImpl.java#L420-L426 However, when resolving the local node's rack information, we use IP to resolve the rack name, which is unfriendly with k8s deployment. https://github.com/apache/bookkeeper/blob/5b5c05331757e7356579076970e61f119f5d34ae/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/RackawareEnsemblePlacementPolicyImpl.java#L209 In k8s deployment, we usually use the hostname as bookieId and Pulsar broker name instead of IP, because the IP will be changed when the pods migrated to other nodes. ### Modification In order not to bring break change to the current behavior, I introduced a flag `useHostnameResolveLocalNodePlacementPolicy` in the BookKeeper client configuration to control whether to use the hostname to resolve the bookie client's local node rack information. The flag is `false` by default, which is the same behavior as the current logic. Due to this PR doesn't introduce any break changes, I think we can cherry-pick it back to the patch releases (branch-4.14, branch-4.15 and branch-4.16)

### Motivation Master issue: apache#4036 ### Changes Set the `numberOfEntriesToReplicate` to 0 when ledger is empty

…che#4102) BP-64: Improve the description of 47 commands for bin/bookkeeper (apache#4102)

…ntPolicy#newEnsemble (apache#4106) Descriptions of the changes in this PR: If the `perRegionPlacement` available bookies are empty, do not pick a bookie from it.

### Motivation OWASP dependency check reports CVE-2023-44981 for Zookeeper. ### Changes Upgrade Zookeeper to 3.8.3. Release notes: https://zookeeper.apache.org/doc/r3.8.3/releasenotes.html Also upgrade snappy-java to 1.1.10.5 since Zookeeper depends on that library.

### Motivation In `conf/bk_server.conf`, the document and default value of `diskUsageWarnThreshold` and `diskUsageLwmThreshold` is not synced with the code. ### Changes - Update the document and default value of `diskUsageWarnThreshold` and `diskUsageLwmThreshold`

### Motivation Introduce the batch read API to improve read performance.

### Motivation BookKeeper 4.16.2 is complied for M1 and can not run on x86_64 Linux if you enabled direct IO. We need to update the release note to let users know the risks of this version ### Changes Update release note for 4.16.2

### Motivation Generate 4.14.8 website docs

### Motivation While shouldRolloverJournal is true，the QueueEntry qe is not added to the toFlush queue but we decrease the JournalQueueSize incorrectly. ### Changes Decrease the JournalQueueSize after correct NonNull check of qe.

### Motivation #### [CVE-2022-45146](https://www.cve.org/CVERecord?id=CVE-2022-45146) Detailed paths Introduced through: org.apache.distributedlog:[email protected] › org.apache.distributedlog:[email protected] › org.apache.bookkeeper:[email protected] › org.bouncycastle:[email protected] Fixed in org.bouncycastle:[email protected] ### Changes Upgrade the org.bouncycastle:bc-fips dependency from 1.0.2.3 to 1.0.2.4

…che#4121) --- ### Motivation In the cleanup ledger manager, we just record every future and remove them when the future complete. Seems we don't need to print the logs

…e#3877) ### Motivation Improve the Http Rest API documentation

### Motivation The AutoRecovery would move the data to another alive bookie to ensure the data replicas. So it will read data from the alive bookie at the ensemble, then write the data to the new bookie. When reading data, it will choose the bookie in the ensemble randomly, so it may choose the bookie that may be already shut down, and then print many warm logs. If we make stickyReadSEnabled=true, it will read the data from the alive bookie continually, and won't print any warm log, and the stickyReadS can improve hit rate of the bookie server read cache, speed the read operation.

* Fix no known bookies after reset racks for all BKs

9.4.51 needs to be updated due to - https://nvd.nist.gov/vuln/detail/CVE-2023-40167 - https://security.snyk.io/vuln/SNYK-JAVA-ORGECLIPSEJETTY-5958847 Co-authored-by: Anup Ghatage <[email protected]>

…eNotInWriteSet (apache#4110) Master Issue: apache#4109 ### Motivation Fix Flaky-test: HandleFailuresTest.testHandleFailureBookieNotInWriteSet When we call `b1Delay.completeExceptionally(new BKException.BKWriteException())` at line480(code1), the `preWriteHook` will complete with exception and then do some actions in the choosen thread(code2), e.g., put the failure bookie to `delayedWriteFailedBookies`(code3). So the `delayedWriteFailedBookies` update is async. However, `lh.appendAsync("entry2".getBytes()))`(Line483 in Code4) is invoked in main thread. So when `appendAsync` execute, the `delayedWriteFailedBookies` could be not updated yet, and `changeInProgress.complete(null)`(code5) will never be invoked.

### Motivation Make AuditorBookieTest#waitForNewAuditor stronger.

### Motivation When migrating the TestReplicationWorker to pulsar(apache/pulsar#21188), the CI always failed. We need to reset the config after test passing

Descriptions of the changes in this PR: ### Motivation Fix a slow gc thread shutdown when compacting. The problem here is that the stop flag `running` has been moved after `compacting.compareAndSet`. When `running` is not set to false, entry log continues to compact one after one and the shutdown thread is hard to set the `compacting`. ### Changes Set `running` to false first and then check `compacting`. Master Issue: apache#4126

…pache#4144) ### Motivation The `BookieZKExpireTest.testBookieServerZKSessionExpireBehaviour` is a flaky test and the root cause is described in apache#3418. There is a race condition in the zookeeper reconnection and listener and it is an expected behavior in Bookie. We created one issue to track this apache#4142. We can skip this flaky test to unblock the pending PRs.

* fix calculate checkSum when using Java9IntHash * - * improve performence * add test * edit test * rename test * fix license * address comments

Update StatsProvider URL Descriptions of the changes in this PR: 404 not found ### Motivation Fix URL

…he#4138) The **ledgersCount.incrementAndGet** should be called in **setExplicitLac** , just like in **setFenced** and **setLimbo**.

…ache#4152)

)

…che#4156) * fix defult value https://bookkeeper.apache.org/docs/reference/config/#statistics * Update bk_server.conf

### Motivation The auditor didn't close the ledger manager and ledger under replication manager the thread will leak. Run the test `AuditorPeriodicBookieCheckTest` and get the stack. ## Before this PR: ``` Full thread dump Java HotSpot(TM) 64-Bit Server VM (11.0.15+8-LTS-149 mixed mode): Threads class SMR info: _java_thread_list=0x0000600003e44460, length=16, elements={ 0x00007fad34009000, 0x00007fad45032800, 0x00007fad14813800, 0x00007fad44010800, 0x00007fad44011800, 0x00007fad45037800, 0x00007fad3400a000, 0x00007fad45812800, 0x00007fad34944000, 0x00007fad4585f800, 0x00007fad45456000, 0x00007fad4444c000, 0x00007fad3431b800, 0x00007face5825800, 0x00007face5860800, 0x00007face4009800 } "main" #1 prio=5 os_prio=31 cpu=1801.68ms elapsed=20.85s tid=0x00007fad34009000 nid=0x2a03 waiting on condition [0x000000030a10d000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park([email protected]/Native Method) - parking to wait for <0x000000079f26a8a0> (a java.util.concurrent.FutureTask) at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/LockSupport.java:234) at java.util.concurrent.FutureTask.awaitDone([email protected]/FutureTask.java:444) at java.util.concurrent.FutureTask.get([email protected]/FutureTask.java:203) at org.junit.internal.runners.statements.FailOnTimeout.getResult(FailOnTimeout.java:141) at org.junit.internal.runners.statements.FailOnTimeout.evaluate(FailOnTimeout.java:127) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69) at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38) at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11) at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35) at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:235) at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54) Locked ownable synchronizers: - None "Reference Handler" #2 daemon prio=10 os_prio=31 cpu=2.14ms elapsed=20.82s tid=0x00007fad45032800 nid=0x4803 waiting on condition [0x000000030a823000] java.lang.Thread.State: RUNNABLE at java.lang.ref.Reference.waitForReferencePendingList([email protected]/Native Method) at java.lang.ref.Reference.processPendingReferences([email protected]/Reference.java:241) at java.lang.ref.Reference$ReferenceHandler.run([email protected]/Reference.java:213) Locked ownable synchronizers: - None "Finalizer" #3 daemon prio=8 os_prio=31 cpu=1.33ms elapsed=20.82s tid=0x00007fad14813800 nid=0x5503 in Object.wait() [0x000000030a926000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait([email protected]/Native Method) - waiting on <0x00000007801b6f18> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove([email protected]/ReferenceQueue.java:155) - waiting to re-lock in wait() <0x00000007801b6f18> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove([email protected]/ReferenceQueue.java:176) at java.lang.ref.Finalizer$FinalizerThread.run([email protected]/Finalizer.java:170) Locked ownable synchronizers: - None "Signal Dispatcher" #4 daemon prio=9 os_prio=31 cpu=2.69ms elapsed=20.81s tid=0x00007fad44010800 nid=0x5b03 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Service Thread" #5 daemon prio=9 os_prio=31 cpu=0.03ms elapsed=20.81s tid=0x00007fad44011800 nid=0x7403 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread0" #6 daemon prio=9 os_prio=31 cpu=1986.68ms elapsed=20.81s tid=0x00007fad45037800 nid=0x5d03 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE No compile task Locked ownable synchronizers: - None "C1 CompilerThread0" apache#9 daemon prio=9 os_prio=31 cpu=769.61ms elapsed=20.81s tid=0x00007fad3400a000 nid=0x7003 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE No compile task Locked ownable synchronizers: - None "Sweeper thread" apache#10 daemon prio=9 os_prio=31 cpu=0.08ms elapsed=20.81s tid=0x00007fad45812800 nid=0x6e03 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Common-Cleaner" apache#11 daemon prio=8 os_prio=31 cpu=11.62ms elapsed=20.73s tid=0x00007fad34944000 nid=0x6b03 in Object.wait() [0x000000030b049000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait([email protected]/Native Method) - waiting on <0x00000007801b7550> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove([email protected]/ReferenceQueue.java:155) - waiting to re-lock in wait() <0x00000007801b7550> (a java.lang.ref.ReferenceQueue$Lock) at jdk.internal.ref.CleanerImpl.run([email protected]/CleanerImpl.java:148) at java.lang.Thread.run([email protected]/Thread.java:834) at jdk.internal.misc.InnocuousThread.run([email protected]/InnocuousThread.java:134) Locked ownable synchronizers: - None "Monitor Ctrl-Break" apache#12 daemon prio=5 os_prio=31 cpu=21.16ms elapsed=20.65s tid=0x00007fad4585f800 nid=0x6a03 runnable [0x000000030b14c000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0([email protected]/Native Method) at java.net.SocketInputStream.socketRead([email protected]/SocketInputStream.java:115) at java.net.SocketInputStream.read([email protected]/SocketInputStream.java:168) at java.net.SocketInputStream.read([email protected]/SocketInputStream.java:140) at sun.nio.cs.StreamDecoder.readBytes([email protected]/StreamDecoder.java:284) at sun.nio.cs.StreamDecoder.implRead([email protected]/StreamDecoder.java:326) at sun.nio.cs.StreamDecoder.read([email protected]/StreamDecoder.java:178) - locked <0x00000007801b81e8> (a java.io.InputStreamReader) at java.io.InputStreamReader.read([email protected]/InputStreamReader.java:181) at java.io.BufferedReader.fill([email protected]/BufferedReader.java:161) at java.io.BufferedReader.readLine([email protected]/BufferedReader.java:326) - locked <0x00000007801b81e8> (a java.io.InputStreamReader) at java.io.BufferedReader.readLine([email protected]/BufferedReader.java:392) at com.intellij.rt.execution.application.AppMainV2$1.run(AppMainV2.java:56) Locked ownable synchronizers: - None "Log4j2-TF-2-Scheduled-1" apache#13 daemon prio=5 os_prio=31 cpu=3.80ms elapsed=19.18s tid=0x00007fad45456000 nid=0x8c03 waiting on condition [0x000000030bb6a000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park([email protected]/Native Method) - parking to wait for <0x00000007805c1f78> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/LockSupport.java:234) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos([email protected]/AbstractQueuedSynchronizer.java:2123) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/ScheduledThreadPoolExecutor.java:1182) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/ScheduledThreadPoolExecutor.java:899) at java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628) at java.lang.Thread.run([email protected]/Thread.java:834) Locked ownable synchronizers: - None "Time-limited test" apache#15 daemon prio=5 os_prio=31 cpu=1344.20ms elapsed=18.79s tid=0x00007fad4444c000 nid=0x9107 waiting on condition [0x000000030bd70000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep([email protected]/Native Method) at org.apache.bookkeeper.replication.AuditorPeriodicBookieCheckTest.tearDown(AuditorPeriodicBookieCheckTest.java:81) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0([email protected]/Native Method) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke([email protected]/NativeMethodAccessorImpl.java:62) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke([email protected]/DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke([email protected]/Method.java:566) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run([email protected]/FutureTask.java:264) at java.lang.Thread.run([email protected]/Thread.java:834) Locked ownable synchronizers: - None "ZkLedgerManagerScheduler-48-1" apache#165 prio=5 os_prio=31 cpu=9.25ms elapsed=16.04s tid=0x00007fad3431b800 nid=0x25b03 waiting on condition [0x0000000314d1a000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park([email protected]/Native Method) - parking to wait for <0x000000079db6aed0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/AbstractQueuedSynchronizer.java:2081) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/ScheduledThreadPoolExecutor.java:1170) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/ScheduledThreadPoolExecutor.java:899) at java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run([email protected]/Thread.java:834) Locked ownable synchronizers: - None "ZkLedgerManagerScheduler-49-1" apache#166 prio=5 os_prio=31 cpu=1.01ms elapsed=16.04s tid=0x00007face5825800 nid=0x25903 waiting on condition [0x0000000314e1d000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park([email protected]/Native Method) - parking to wait for <0x000000079db6b3f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/AbstractQueuedSynchronizer.java:2081) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/ScheduledThreadPoolExecutor.java:1170) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/ScheduledThreadPoolExecutor.java:899) at java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run([email protected]/Thread.java:834) Locked ownable synchronizers: - None "ForkJoinPool.commonPool-worker-19" apache#187 daemon prio=5 os_prio=31 cpu=1.52ms elapsed=15.64s tid=0x00007face5860800 nid=0x2ca03 waiting on condition [0x0000000316259000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park([email protected]/Native Method) - parking to wait for <0x000000079ef3f6e8> (a java.util.concurrent.ForkJoinPool) at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:194) at java.util.concurrent.ForkJoinPool.runWorker([email protected]/ForkJoinPool.java:1628) at java.util.concurrent.ForkJoinWorkerThread.run([email protected]/ForkJoinWorkerThread.java:183) Locked ownable synchronizers: - None "Attach Listener" apache#313 daemon prio=9 os_prio=31 cpu=1.90ms elapsed=0.24s tid=0x00007face4009800 nid=0x3d53b waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "VM Thread" os_prio=31 cpu=122.89ms elapsed=20.84s tid=0x00007fad1405f000 nid=0x4e03 runnable "GC Thread#0" os_prio=31 cpu=67.63ms elapsed=20.85s tid=0x00007fad3480c000 nid=0x3b03 runnable "GC Thread#1" os_prio=31 cpu=60.78ms elapsed=20.16s tid=0x00007fad45203800 nid=0x8233 runnable "GC Thread#2" os_prio=31 cpu=56.71ms elapsed=20.16s tid=0x00007fad45039800 nid=0x8403 runnable "GC Thread#3" os_prio=31 cpu=59.80ms elapsed=20.16s tid=0x00007fad45341800 nid=0xa703 runnable "GC Thread#4" os_prio=31 cpu=60.21ms elapsed=20.16s tid=0x00007fad45342000 nid=0x8503 runnable "GC Thread#5" os_prio=31 cpu=60.89ms elapsed=20.16s tid=0x00007fad4536d000 nid=0xa503 runnable "GC Thread#6" os_prio=31 cpu=57.91ms elapsed=20.16s tid=0x00007fad4594a800 nid=0x8803 runnable "GC Thread#7" os_prio=31 cpu=56.48ms elapsed=20.16s tid=0x00007fad4594b800 nid=0xa403 runnable "GC Thread#8" os_prio=31 cpu=56.38ms elapsed=20.16s tid=0x00007fad4536e000 nid=0x8a03 runnable "G1 Main Marker" os_prio=31 cpu=2.03ms elapsed=20.85s tid=0x00007fad3482d000 nid=0x3a03 runnable "G1 Conc#0" os_prio=31 cpu=12.67ms elapsed=20.85s tid=0x00007fad4500d800 nid=0x5403 runnable "G1 Conc#1" os_prio=31 cpu=14.15ms elapsed=18.88s tid=0x00007fad45470800 nid=0x8d4f runnable "G1 Refine#0" os_prio=31 cpu=2.03ms elapsed=20.84s tid=0x00007fad3492d800 nid=0x4103 runnable "G1 Young RemSet Sampling" os_prio=31 cpu=5.41ms elapsed=20.84s tid=0x00007fad14008800 nid=0x5103 runnable "VM Periodic Task Thread" os_prio=31 cpu=14.86ms elapsed=20.64s tid=0x00007fad3494b000 nid=0x680f waiting on condition JNI global refs: 20, weak refs: 0 ``` ## After this PR: ``` Full thread dump Java HotSpot(TM) 64-Bit Server VM (11.0.15+8-LTS-149 mixed mode): Threads class SMR info: _java_thread_list=0x00006000020b8540, length=14, elements={ 0x00007ff536808800, 0x00007ff53311f800, 0x00007ff533021800, 0x00007ff53485f000, 0x00007ff53500f000, 0x00007ff535010000, 0x00007ff53500c000, 0x00007ff53400d800, 0x00007ff536861800, 0x00007ff536053000, 0x00007ff53621c800, 0x00007ff536178800, 0x00007ff5357e2000, 0x00007ff4d3009800 } "main" #1 prio=5 os_prio=31 cpu=1768.54ms elapsed=23.32s tid=0x00007ff536808800 nid=0x2a03 waiting on condition [0x000000030630b000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park([email protected]/Native Method) - parking to wait for <0x000000079f31ab80> (a java.util.concurrent.FutureTask) at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/LockSupport.java:234) at java.util.concurrent.FutureTask.awaitDone([email protected]/FutureTask.java:444) at java.util.concurrent.FutureTask.get([email protected]/FutureTask.java:203) at org.junit.internal.runners.statements.FailOnTimeout.getResult(FailOnTimeout.java:141) at org.junit.internal.runners.statements.FailOnTimeout.evaluate(FailOnTimeout.java:127) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69) at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38) at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11) at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35) at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:235) at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54) Locked ownable synchronizers: - None "Reference Handler" #2 daemon prio=10 os_prio=31 cpu=1.79ms elapsed=23.29s tid=0x00007ff53311f800 nid=0x4d03 waiting on condition [0x0000000306a21000] java.lang.Thread.State: RUNNABLE at java.lang.ref.Reference.waitForReferencePendingList([email protected]/Native Method) at java.lang.ref.Reference.processPendingReferences([email protected]/Reference.java:241) at java.lang.ref.Reference$ReferenceHandler.run([email protected]/Reference.java:213) Locked ownable synchronizers: - None "Finalizer" #3 daemon prio=8 os_prio=31 cpu=1.34ms elapsed=23.29s tid=0x00007ff533021800 nid=0x5503 in Object.wait() [0x0000000306b24000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait([email protected]/Native Method) - waiting on <0x00000007801c5b68> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove([email protected]/ReferenceQueue.java:155) - waiting to re-lock in wait() <0x00000007801c5b68> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove([email protected]/ReferenceQueue.java:176) at java.lang.ref.Finalizer$FinalizerThread.run([email protected]/Finalizer.java:170) Locked ownable synchronizers: - None "Signal Dispatcher" #4 daemon prio=9 os_prio=31 cpu=1.44ms elapsed=23.27s tid=0x00007ff53485f000 nid=0x5b03 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Service Thread" #5 daemon prio=9 os_prio=31 cpu=0.03ms elapsed=23.27s tid=0x00007ff53500f000 nid=0x7503 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "C2 CompilerThread0" #6 daemon prio=9 os_prio=31 cpu=1882.60ms elapsed=23.27s tid=0x00007ff535010000 nid=0x7403 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE No compile task Locked ownable synchronizers: - None "C1 CompilerThread0" apache#9 daemon prio=9 os_prio=31 cpu=768.63ms elapsed=23.27s tid=0x00007ff53500c000 nid=0x6003 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE No compile task Locked ownable synchronizers: - None "Sweeper thread" apache#10 daemon prio=9 os_prio=31 cpu=0.05ms elapsed=23.27s tid=0x00007ff53400d800 nid=0x6203 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Common-Cleaner" apache#11 daemon prio=8 os_prio=31 cpu=8.06ms elapsed=23.20s tid=0x00007ff536861800 nid=0x6403 in Object.wait() [0x0000000307245000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait([email protected]/Native Method) - waiting on <0x00000007801c6678> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove([email protected]/ReferenceQueue.java:155) - waiting to re-lock in wait() <0x00000007801c6678> (a java.lang.ref.ReferenceQueue$Lock) at jdk.internal.ref.CleanerImpl.run([email protected]/CleanerImpl.java:148) at java.lang.Thread.run([email protected]/Thread.java:834) at jdk.internal.misc.InnocuousThread.run([email protected]/InnocuousThread.java:134) Locked ownable synchronizers: - None "Monitor Ctrl-Break" apache#12 daemon prio=5 os_prio=31 cpu=19.84ms elapsed=23.11s tid=0x00007ff536053000 nid=0x6c03 runnable [0x0000000307348000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0([email protected]/Native Method) at java.net.SocketInputStream.socketRead([email protected]/SocketInputStream.java:115) at java.net.SocketInputStream.read([email protected]/SocketInputStream.java:168) at java.net.SocketInputStream.read([email protected]/SocketInputStream.java:140) at sun.nio.cs.StreamDecoder.readBytes([email protected]/StreamDecoder.java:284) at sun.nio.cs.StreamDecoder.implRead([email protected]/StreamDecoder.java:326) at sun.nio.cs.StreamDecoder.read([email protected]/StreamDecoder.java:178) - locked <0x00000007801c7860> (a java.io.InputStreamReader) at java.io.InputStreamReader.read([email protected]/InputStreamReader.java:181) at java.io.BufferedReader.fill([email protected]/BufferedReader.java:161) at java.io.BufferedReader.readLine([email protected]/BufferedReader.java:326) - locked <0x00000007801c7860> (a java.io.InputStreamReader) at java.io.BufferedReader.readLine([email protected]/BufferedReader.java:392) at com.intellij.rt.execution.application.AppMainV2$1.run(AppMainV2.java:56) Locked ownable synchronizers: - None "Log4j2-TF-2-Scheduled-1" apache#13 daemon prio=5 os_prio=31 cpu=14.05ms elapsed=21.67s tid=0x00007ff53621c800 nid=0x9c03 waiting on condition [0x0000000307d66000] java.lang.Thread.State: TIMED_WAITING (parking) at jdk.internal.misc.Unsafe.park([email protected]/Native Method) - parking to wait for <0x00000007805a1378> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/LockSupport.java:234) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos([email protected]/AbstractQueuedSynchronizer.java:2123) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/ScheduledThreadPoolExecutor.java:1182) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/ScheduledThreadPoolExecutor.java:899) at java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/ThreadPoolExecutor.java:1054) at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1114) at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628) at java.lang.Thread.run([email protected]/Thread.java:834) Locked ownable synchronizers: - None "Time-limited test" apache#15 daemon prio=5 os_prio=31 cpu=1353.43ms elapsed=21.31s tid=0x00007ff536178800 nid=0x9603 waiting on condition [0x0000000307f6c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep([email protected]/Native Method) at org.apache.bookkeeper.replication.AuditorPeriodicBookieCheckTest.tearDown(AuditorPeriodicBookieCheckTest.java:81) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0([email protected]/Native Method) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke([email protected]/NativeMethodAccessorImpl.java:62) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke([email protected]/DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke([email protected]/Method.java:566) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run([email protected]/FutureTask.java:264) at java.lang.Thread.run([email protected]/Thread.java:834) Locked ownable synchronizers: - None "ForkJoinPool.commonPool-worker-19" apache#187 daemon prio=5 os_prio=31 cpu=1.50ms elapsed=18.22s tid=0x00007ff5357e2000 nid=0x2d003 waiting on condition [0x0000000312455000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park([email protected]/Native Method) - parking to wait for <0x000000079f052a68> (a java.util.concurrent.ForkJoinPool) at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:194) at java.util.concurrent.ForkJoinPool.runWorker([email protected]/ForkJoinPool.java:1628) at java.util.concurrent.ForkJoinWorkerThread.run([email protected]/ForkJoinWorkerThread.java:183) Locked ownable synchronizers: - None "Attach Listener" apache#313 daemon prio=9 os_prio=31 cpu=1.42ms elapsed=0.25s tid=0x00007ff4d3009800 nid=0x2a767 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "VM Thread" os_prio=31 cpu=122.92ms elapsed=23.30s tid=0x00007ff536863800 nid=0x5003 runnable "GC Thread#0" os_prio=31 cpu=62.12ms elapsed=23.31s tid=0x00007ff536815000 nid=0x3403 runnable "GC Thread#1" os_prio=31 cpu=57.82ms elapsed=22.63s tid=0x00007ff53580c800 nid=0x8207 runnable "GC Thread#2" os_prio=31 cpu=56.34ms elapsed=22.63s tid=0x00007ff535ad9800 nid=0xa703 runnable "GC Thread#3" os_prio=31 cpu=59.35ms elapsed=22.63s tid=0x00007ff53516f000 nid=0xa603 runnable "GC Thread#4" os_prio=31 cpu=57.35ms elapsed=22.63s tid=0x00007ff536015000 nid=0x8503 runnable "GC Thread#5" os_prio=31 cpu=57.46ms elapsed=22.63s tid=0x00007ff5351b5000 nid=0x8603 runnable "GC Thread#6" os_prio=31 cpu=52.34ms elapsed=22.63s tid=0x00007ff5351b6000 nid=0x8803 runnable "GC Thread#7" os_prio=31 cpu=57.02ms elapsed=22.63s tid=0x00007ff5351b6800 nid=0x8a03 runnable "GC Thread#8" os_prio=31 cpu=58.19ms elapsed=22.63s tid=0x00007ff5351b7800 nid=0xa403 runnable "G1 Main Marker" os_prio=31 cpu=1.36ms elapsed=23.31s tid=0x00007ff53482e800 nid=0x3603 runnable "G1 Conc#0" os_prio=31 cpu=11.69ms elapsed=23.31s tid=0x00007ff53482f000 nid=0x5403 runnable "G1 Conc#1" os_prio=31 cpu=12.95ms elapsed=21.39s tid=0x00007ff53695b000 nid=0x9103 runnable "G1 Refine#0" os_prio=31 cpu=2.18ms elapsed=23.31s tid=0x00007ff53311e800 nid=0x5303 runnable "G1 Young RemSet Sampling" os_prio=31 cpu=6.12ms elapsed=23.31s tid=0x00007ff534854000 nid=0x4203 runnable "VM Periodic Task Thread" os_prio=31 cpu=17.11ms elapsed=23.11s tid=0x00007ff535888000 nid=0x8003 waiting on condition JNI global refs: 20, weak refs: 0 ``` ### You can see the executor `ZkLedgerManagerScheduler` shutdown correctly.

Descriptions of the changes in this PR: ### Motivation After the issue on disabled sticky reads, I'm adding this to improve documentation about performance on reading. ### Changes Master Issue: apache#4130

Signed-off-by: ZhangJian He <[email protected]>

…4063) * Entry write support local node region aware placement policy * address comments

… to the default provider (apache#4168) Descriptions of the changes in this PR: Add OpenTelemetry stats provider and change Prometheus stats provider to the default provider ### Motivation fix doc

### Motivation Fix stats providers in configuration

…he#4163) * Fix the autorecovery failed replicate by add entry fenced error --- ### Motivation apache#3783 changed the recovery add request flags. It causes when replication happening, if the ledger was fenced, the ledger can not be recovered. Because the recovery add flag is not added then the normal add is not allowed to a fenced ledger.

Descriptions of the changes in this PR: Fix Docker README.md

Co-authored-by: Matteo Merli <[email protected]> ### Motivation For the `LedgerMetadataIndex#removeDeletedLedgers` and `LedgerMetadataIndex#flush`, it will call ledgersDB sync whether the ledgersDB has changed or not. We can skip the sync call when nothing changed in the ledgersDB. ### Changes - Check whether pendingLedgersUpdates is empty in `flush()` and `pendingDeletedLedgers` is empty in removeDeletedLedgers - Move the `key.recycle()` in finally to cover keys leak when the ledgersDB operations throw an exception.

### Motivation Generate 4.15.5 website documents

…ache#4135) ### Motivation When we use DirectIO and a catch up read occurs, a memory leak occurs because the ByteBuf is not released. Because although in the `internalReadEntry` method in `DirectEntryLogger`, even if the upper layer `fillReadAheadCache` uses the `ReferenceCountUtil.release(entry)` method in the finally block to release ByteBuf, it will still cause an exception thrown by `internalReadEntry`, and the `entryLogger.readEntry` method happens to be in try. ..catch unexpected, causing ByteBuf to not be released correctly ### Changes Master Issue: apache#4134

Descriptions of the changes in this PR: ### Motivation Optimize shell log print. When we execute the bookkeeper shell , e.g., `bin/bookkeeper shell listbookies -a`, it will print a lot of redundant logs. And the key infomation is hard to find: <img width="1146" alt="image" src="https://github.com/apache/bookkeeper/assets/10233437/fe97c299-900b-4f71-babf-444000f60432"> ### Changes Changing the `log4j2.shell.xml` to only print the key infomation, other logs will be printed to the `bookkeeper-shell.log` file After changing: <img width="764" alt="image" src="https://github.com/apache/bookkeeper/assets/10233437/62f0acc9-15dd-4745-b43b-72cba97ccd8c">

Co-authored-by: fanjianye <[email protected]> ### Motivation fix a config error in RegionAware#initialize. There is double config ignoreLocalNodeInPlacementPolicy in RegionAware#initialize. Actually one of the config is useHostnameResolveLocalNodePlacementPolicy. ### change use correct config name

### Motivation fix apache#4096 ensemble change counter not inc when ensemble change . ### Changes inc counter when ensemble change.

…n. (apache#3876) ### Motivation Current HardLink will create a process to execute mv like command to rename file in compaction logic. maybe we can just use jdk api to do this with lower overhead. see javadoc: https://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html#createLink(java.nio.file.Path,%20java.nio.file.Path) ### Changes test if `Files.createLink` is available in hardlink static code block. if test fails means `Files.createLink` is not available. else will use `Files.createLink` when call createHardlink

…pache#4174) --- ###Motivation Allow to override the getUnderreplicatedFragments to change the customize the behaviour.

* Improve auto-recovery noise log when some bookie down. * Address comment.

### Motivation Now, when we shut down the auditor elector. The shutdown behavior via submitting a shutdown task to do it. In some cases, the follower auditor elector executor is always blocked due to waiting leader election, so the shutdown task will lie in the task queue forever, get no chance to execute. In the pulsar, the case happen. See apache/pulsar#21797 (comment) So in the auditor elector shutdown, if the remain task can't done in time, we should invoke shutdownNow to interrupt the blocked thread.

… batchReadRequest and batchReadResponse (apache#4177) ### Motivation This is the first PR for the batch read(apache#4051) feature. Protocol change: Support encoding and decoding for batchReadRequest and batchReadResponse

…dle batch read request (apache#4187) Descriptions of the changes in this PR: This is the second PR for the batch read(apache#4051) feature. Bookie server introduces the BatchedReadEntryProcessor to handle batch read request

)

### Modification Generate 4.16.4 website docs

### Motivation Generate BookKeeper 4.16.4 release note

Descriptions of the changes in this PR: 1. There are multiple spec.ports fields in sts yaml 2. The RUN command specified in dockerfile is not standardized Master Issue: apache#4185

### Motivation  If one ledger's ensemble is [bk0, bk1] and bk0 is down, the bookie client may send a read request to bk0 first then fail with the following errors, and resend the read request to bk1 in the end. ``` 2023-10-19T18:33:52,042 - ERROR - [BookKeeperClientWorker-OrderedExecutor-3-0:PerChannelBookieClient@563] - Cannot connect to 192.168.31.216:3181 as endpoint resolution failed (probably bookie is down) err org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId 192.168.31.216:3181, bookie does not exist or it is not running 2023-10-19T18:33:52,042 - INFO - [BookKeeperClientWorker-OrderedExecutor-3-0:DefaultBookieAddressResolver@77] - Cannot resolve 192.168.31.216:3181, bookie is unknown org.apache.bookkeeper.client.BKException$BKBookieHandleNotAvailableException: Bookie handle is not available 2023-10-19T18:33:52,042 - INFO - [BookKeeperClientWorker-OrderedExecutor-3-0:PendingReadOp$LedgerEntryRequest@223] - Error: Bookie handle is not available while reading L6 E40 from bookie: 192.168.31.216:3181 ``` One of the related issues is in the auto-recovery decommission and there is one PR in the BookKeeper repo: apache#4113 However, the bookie client already knows the bk0 is down and we should send the read request to bk1 first. So we can reorder the read request based on the known bookie list. If one bookie is lost, it will reorder the lost bookie to the end of the read list. ### Modifications  Enable the `reorderReadSequence` by default for auto-recovery.

### Motivation This is the fourth PR for the batch read(apache#4051) feature. Refactor read op, extract ReadOpBase. Introduce batchedReadOp.

### Motivation This is the fifth PR for the batch read(apache#4051) feature. LedgerHandle introduces batch read API. This PR is based on apache#4190, please merge it firstly.

…ith readerIndex > 0 (apache#4196) * Add a test that reproduces a bug in checksum calculation * Revert "Fixed unnecessary copy to heap (apache#2701)" changes to ByteBufList This partially reverts commit 3c9c710. * Remove CompositeBuffer unwrapping in DigestManager * Rename update -> internalUpdate so that unwrapping logic could be added to update * Remove unnecessary unwrapping logic in Java9IntHash * Add safe way to handle CompositeByteBuf * Add license header * Fix checkstyle * Refactor ByteBuf visitor solution * Fix checkstyle * Reformat * Refactor recursive visiting * Revisit equals, hashCode and toString * Refactor test case * Add support for UnpooledHeapByteBuf.getBytes which passes an array * Add support for visiting buffers backed by byte[] arrays - getBytes calls setBytes with a byte[] argument for heap ByteBufs * Move ByteBufVisitor to org.apache.bookkeeper.util package * Update javadoc * Refactor to use stateless visitor so that instance can be shared * Improve test so that a single scenario can be used for debugging * Fix bug in Java9IntHash calculation that assumed crc32c_update(x) == ~crc32c_update(~x) - Java9IntHash uses private methods from java.util.zip.CRC32C class, updateBytes and updateDirectByteBuffer. When inspecting the use and interface contract, it doesn't match how it is used in Java9IntHash. This PR addresses that by introducing a separate initial value for initializing the accumulated value so that the initial value could match the logic in java.util.zip.CRC32C.reset method. There's also a separate method for finalizing the accumulated value into a final checksum value. This is to match the java.util.zip.CRC32C.getValue method's logic (uses bitwise complement operator ~). - With a quick glance, it might appear that the previous logic is similar. However it isn't since I have a failing test which gets fixed with this change. I haven't yet added the Java9IntHash level unit test case to prove how it differs. It must be related to integer value overflow. For the CRC32C function, I believe it means that it cannot be assumed in all cases that func(x) == ~func(~x). That's the assumption that the previous code was making. It probably applies for many inputs, but not all. It would break in overflow cases. * Fix checkstyle * Fix checkstyle * Fix missing depth increment that prevents StackOverflowException * Properly handle the depth increase and decrease * Remove unnecessary condition * Use more efficient way to read bytes to the target array * Don't use ByteBufVisitor if it's not necessary * Revert "Fix bug in Java9IntHash calculation that assumed crc32c_update(x) == ~crc32c_update(~x)" This reverts commit 272e962. * Fix issue in resume byte[] version that was added - input and output should be complemented. explanation has been added to the resume ByteBuf method * Polish ByteBufVisitor - reuse GetBytesCallbackByteBuf instance for handling the root ByteBuf instance * Use extracted method * Fix bug with array handling * Polish ByteBufVisitor * Optimize the buffer copying in the case where array or memory address cannot be accessed - read-only buffers will need to be copied before reading - use ByteBuf.copy for direct buffers with pooled allocator when the algorithm can accept a memory address buffer - use the 64kB threadlocal byte[] buffer for copying all other inputs * Check if memory address is accepted * Improve comments about complement (current = ~current) in resume * Print thread dump when build is cancelled * Filter empty buffers and arrays in ByteBufVisitor

Bumps [jsoup](https://github.com/jhy/jsoup) from 1.14.3 to 1.15.3. - [Release notes](https://github.com/jhy/jsoup/releases) - [Changelog](https://github.com/jhy/jsoup/blob/master/CHANGES) - [Commits](jhy/jsoup@jsoup-1.14.3...jsoup-1.15.3) --- updated-dependencies: - dependency-name: org.jsoup:jsoup dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…4197) * Add support for Junit5 tests - upgrade surefire plugin - replace <forkMode>always</forkMode> with supported way in newer maven-surefire-plugin - maven-failsafe-plugin version must match maven-surefire-plugin version - remove invalid test case. Log4j is in the classpath for other reasons checked by running "mvn -pl tests/integration-tests-utils/pom.xml test -X" also verified by debugging that it's not in the system classpath, it is loaded by the MavenClassLoader - add assertj-core for fluent assertions in tests * Upgrade arquillian-junit-standalone * Fix error in pom.xml * Fix java.lang.ClassNotFoundException: javax.ws.rs.ProcessingException * Pick docker-java-api version that is compatible with arquillian-cube-docker * Fix snakeyaml issue * Fix running of bookkeeper in Docker in tests * Improve GitHub Actions CI workflow logging * Increase client tests timeout * Add script for picking fast ubuntu mirror for docker build * upgrade maven-dependency-plugin version (used for copying .tar.gz for docker image building) * Fix building of the docker image * Increase timeouts for integration tests * Use eclipse-temurin base image * Replace "FROM ubuntu:22.10" with "FROM ubuntu:22.04" to use LTS version of ubuntu 22.10 (kinetic) is EOL since July 20 2023 * Build docker image also for backward-compatibility-tests * Install gpg that is required by the scripts * Add UBUNTU_MIRROR * Remove redundancy from integration tests * Fix missing dependency * Fix classpath issues for arquillian-cube * Upgrade actions versions * Fix invalid references in native-io/pom.xml * Fix circe-checksum native library compilation * Migrate Gradle specific settings to Maven * Another attempt to disable backward-compat tests in integration test build job * Add working solution for skipping backward compat tests in the int tests * Attempt to fix issue in locating the file * Fix tar file extraction * Improve solution * Backward compat tests build commands without int tests * Copy tune-runner-vm and clean-disk from Pulsar CI and clean disk * Revisit skipping of backward compat tests once again * Use similar MAVEN_OPTS as there is in Pulsar * Fix directory * Add way to find git root directory * Revisit one more time * Add current-version-image back * Enable tests for backwardCompatTests * Don't create /ledgers in ZK if it already exists * Add uncaughtexceptionhandler * Add exclusions for reload4j * Use maven-surefire 2.8.1 for certain tests * Add failing test back in, also fails with surefire 2.8.1 * Exclude logging implementations * Revert "Use maven-surefire 2.8.1 for certain tests" This reverts commit 94e2b8e. * Exclude log4j * Add instructions how to run on Mac Apple Silicon * Find slf4j version * Block loading some classes * Change context classloader while creating new bookkeeper * Route commons logging to slf4j * Update shrinkwrap.version * Wait for ZK client to shutdown before returning from close * Upgrade Groovy version * Remove arquillian-junit-container which causes tests to be run twice * Skip backward compat tests unless -DbackwardCompatTests is passed * Delay closing of classloaders by 5 seconds * Fix checkstyle * Improve instructions for running tests * Upgrade Testcontainers and import the bom in dependencyManagement * Fix synchronization bug pointed out by spotbugs * upgrade markdown-link-check * Ignore checking Google calendar for dead links * Ignore anchors * Add 2 more patterns to dead link checker exclusions * Collect logs * Log integration tests to console to observe progress, disable retries * Use maven version provided by GitHub Action runner * Improve instructions for docker socket proxy * Add test timeout of 5 minutes for BookieShellTestBase * Fix test to match changes made in e41d672 * Upgrade mockito to 4.11.0 * Align hamcrest with junit * Revert "Upgrade mockito to 4.11.0" This reverts commit 55b7fcd. * Make JAVA_HOME optional * Tune Java's DNS cache timeouts for docker images * Use different directory for JDK 8 * Add tooling to containers that help debugging * Upgrade JMH version and configuration * Specify maven.compiler.release to fix issues with building with Java 11 * Remove gmavenplus plugin since there's already groovy-eclipse compiler in use * Upgrade apache-rat-plugin * Fix validation step * Don't import shaded class * Move freebuilder out of distributed libraries (there was invalid config for the annotation processor) * Remove use of shaded class in test * Upgrade junit5, assertj-core & awaitility to latest stable * Fix simpletest logic - LAC is updated asynchronously. Entries might be unconfirmed when read before ledger is closed. - The last entry id passed to the read method was invalid * Fix SimpleTestCommandTest --------- Co-authored-by: Matteo Merli <[email protected]>

…x-aarch_64 (apache#4204)

* Avoid system call to improve read performance. * Fix ci. * Add comments for getCurrentWritingLogId * Fix ci. * Consider compacting log. * Fix checkstyle. * Address the comment. * Address comment. * Address the comments. * Add tests. * Fix checkstyle. * address the comments. * Fix concurrency problem.

Fix implicit cast from long to int

* AutoRecovery support batch read * Fix check style * address comments

…pache#4213)

* Add test to cover Bookkeeper test. * fix ci. * Fix ci.

@deprecated

Descriptions of the changes in this PR: ## Journal settings | Parameter | Description | Default | --------- | ----------- | ------- | | journalDirectories | The directories to which Bookkeeper outputs its write-ahead log (WAL). Could define multi directories to store write head logs, separated by ','. For example: journalDirectories=/tmp/bk-journal1,/tmp/bk-journal2 If journalDirectories is set, bookies will skip journalDirectory and use this setting directory. | /tmp/bk-journal | | journalDirectory | @deprecated since 4.5.0, in favor of using `journalDirectories`. The directory to which Bookkeeper outputs its write-ahead log (WAL). | /tmp/bk-txn | ### Motivation update doc about conf#JournalDirectory 2 conf#JournalDirectories& fix listfilesondisc cmd description ### Changes Master Issue: apache#4183

### Motivation `setBatchReadEnabled` is duplicated, which breaks the compilation: ``` Error: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.12.1:compile (default-compile) on project bookkeeper-server: Compilation failure: Compilation failure: Error: /home/runner/work/bookkeeper/bookkeeper/bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ClientConfiguration.java:[2110,32] method setBatchReadEnabled(boolean) is already defined in class org.apache.bookkeeper.conf.ClientConfiguration ``` ### Changes Remove the duplicated method.

…apache#4219) ### Motivation In BenchReadThroughputLatency, it generates the ledger zk node path using `String.format("/ledgers/L%010d", ledger.get());` But if the user uses `HierarchicalLedgerManager`, `LegacyHierarchicalLedgerManager`, or `LongHierarchicalLedgerManager`, the ledger zk node path is not the same as the way in the BenchReadThroughputLatency. So we need support config the `genLedgerWay` to match the custom LedgerManager.

* add filename check for unTar * update code

* Fix uncontrolled data used in path expression * update code * update code

* fix error stack track may expose to external user * update code * fix check style

…4214)

) Co-authored-by: ZhangJian He <[email protected]>

### Motivation In the coding stage, we implements API not same with the doc, we should correct the doc to match the code.

…e#4220) Fix BenchReadThroughputLatency batch read can't stop problems.

* fix: set metrics endpoint content-type when bookie http server is enabled * fix: checkstyle * fix: use existing prometheus content type constant * chore: use minimal dependency / license fixes * Revert "chore: use minimal dependency / license fixes" This reverts commit 7daf8be. * Revert "fix: use existing prometheus content type constant" This reverts commit c265c49.

…ld (apache#4237)

…lyUpToLastAcknoledgedWrite (apache#4234) * Issue 4200: fix flaky test DeferredSyncTest.testForceWillAdvanceLacOnlyUpToLastAcknoledgedWrite * add a comment

* Bump org.apache.zookeeper:zookeeper from 3.8.3 to 3.8.4 Bumps org.apache.zookeeper:zookeeper from 3.8.3 to 3.8.4. --- updated-dependencies: - dependency-name: org.apache.zookeeper:zookeeper dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> * Fix license --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: chenhang <[email protected]>

…#4230) Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.14.9 to 1.15.6. - [Release notes](https://github.com/follow-redirects/follow-redirects/releases) - [Commits](follow-redirects/follow-redirects@v1.14.9...v1.15.6) --- updated-dependencies: - dependency-name: follow-redirects dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…e#4254) ### Motivation The implementation of is not consistent with the description. ### Changes fix doc. Master Issue: apache#4253

### Motivation I tried to use command `bin/bookkeeper shell listbookies -a` to get all bookies from localbookie, it raised error ``` ERROR Fail to process command 'list' org.apache.bookkeeper.client.BKException$ZKException: Error while using ZooKeeper at org.apache.bookkeeper.discover.ZKRegistrationClient.lambda$getChildren$4(ZKRegistrationClient.java:352) ~[bookkeeper-server-4.17.0-SNAPSHOT.jar:4.17.0-SNAPSHOT] at org.apache.bookkeeper.zookeeper.ZooKeeperClient$25$1.processResult(ZooKeeperClient.java:1177) ~[bookkeeper-server-4.17.0-SNAPSHOT.jar:4.17.0-SNAPSHOT] at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:668) ~[zookeeper-3.8.1.jar:3.8.1] at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:553) ~[zookeeper-3.8.1.jar:3.8.1] Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /ledgers/cookies at org.apache.zookeeper.KeeperException.create(KeeperException.java:118) ~[zookeeper-3.8.1.jar:3.8.1] at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.8.1.jar:3.8.1] at org.apache.bookkeeper.discover.ZKRegistrationClient.lambda$getChildren$4(ZKRegistrationClient.java:351) ~[bookkeeper-server-4.17.0-SNAPSHOT.jar:4.17.0-SNAPSHOT] ... 3 more ``` The reason is that no /ledgers/cookies are created in ZK when starting the local bk server. After applying this change, the command works.

### Motivation Creating the temp directory for unpacking the native library is failing for the affinity library. ### Changes Use `Files.createTempDirectory()` instead.

### Motivation & Changes - Upgrade vertx-core to 4.5.7 to address CVE-2024-1300 - Depends on merging apache#4264 before this PR (since Netty 4.1.108.Final is required for vertx-core 4.5.7)

### Motivation Java 22 has been released. Lombok 1.18.32 comes with Java 22 support. Lombok changelog: https://projectlombok.org/changelog ### Modifications Upgrade Lombok to 1.18.32

Signed-off-by: ZhangJian He <[email protected]>

…he#4266) ### Motivation - The http handler implementations in Bookkeeper aren't necessarily non-blocking. That's why they should be executed on the blocking thread pool in vertx. ### Changes Executing on blocking thread pool in vertx can be achieved by registering the handlers using the `blockingHandler` method instead of the `handler` method.

…ache#4239) Bumps [webpack-dev-middleware](https://github.com/webpack/webpack-dev-middleware) from 5.3.1 to 5.3.4. - [Release notes](https://github.com/webpack/webpack-dev-middleware/releases) - [Changelog](https://github.com/webpack/webpack-dev-middleware/blob/v5.3.4/CHANGELOG.md) - [Commits](webpack/webpack-dev-middleware@v5.3.1...v5.3.4) --- updated-dependencies: - dependency-name: webpack-dev-middleware dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…4268) Bumps [@babel/traverse](https://github.com/babel/babel/tree/HEAD/packages/babel-traverse) from 7.17.3 to 7.24.1. - [Release notes](https://github.com/babel/babel/releases) - [Changelog](https://github.com/babel/babel/blob/main/CHANGELOG.md) - [Commits](https://github.com/babel/babel/commits/v7.24.1/packages/babel-traverse) --- updated-dependencies: - dependency-name: "@babel/traverse" dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

### Motivation journalTime metric is unused anymore since apache#3843 ### Changes Remove journalTime Co-authored-by: gavingaozhangmin <[email protected]>

Bumps [postcss](https://github.com/postcss/postcss) from 8.4.8 to 8.4.38. - [Release notes](https://github.com/postcss/postcss/releases) - [Changelog](https://github.com/postcss/postcss/blob/main/CHANGELOG.md) - [Commits](postcss/postcss@8.4.8...8.4.38) --- updated-dependencies: - dependency-name: postcss dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [semver](https://github.com/npm/node-semver) from 5.7.1 to 5.7.2. - [Release notes](https://github.com/npm/node-semver/releases) - [Changelog](https://github.com/npm/node-semver/blob/v5.7.2/CHANGELOG.md) - [Commits](npm/node-semver@v5.7.1...v5.7.2) --- updated-dependencies: - dependency-name: semver dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [ua-parser-js](https://github.com/faisalman/ua-parser-js) from 0.7.31 to 0.7.37. - [Release notes](https://github.com/faisalman/ua-parser-js/releases) - [Changelog](https://github.com/faisalman/ua-parser-js/blob/master/CHANGELOG.md) - [Commits](faisalman/ua-parser-js@0.7.31...0.7.37) --- updated-dependencies: - dependency-name: ua-parser-js dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

) Bumps [@sideway/formula](https://github.com/sideway/formula) from 3.0.0 to 3.0.1. - [Release notes](https://github.com/sideway/formula/releases) - [Commits](hapijs/formula@v3.0.0...v3.0.1) --- updated-dependencies: - dependency-name: "@sideway/formula" dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Signed-off-by: ZhangJian He <[email protected]>

### Motivation Adding lmax disruptor in dependency allow user to config AsyncLogger in log4j2.xml like this ```xml <Loggers> <AsyncLogger name="async" level="info"> <AppenderRef ref="AsyncConsole"/> </AsyncLogger> </Loggers> ``` See also https://logging.apache.org/log4j/2.x/manual/async.html Signed-off-by: ZhangJian He <[email protected]>

Signed-off-by: ZhangJian He <[email protected]>

…4277) Clean up org.apache.bookkeeper.util package-info.java. This is no class in that package. Signed-off-by: ZhangJian He <[email protected]>

Signed-off-by: ZhangJian He <[email protected]>

* Add 4.17.0 website * Add release notes for BookKeeper 4.17.0

* [fix] ByteBuf release/retain incorrect * improve the code comment * fix other cases * modify the code comment * improve the code * improve the test * add description

…dd a new method with correct name (apache#4288) ### Motivation Fix some typos ### Changes - Deprecate method `setAllocatorPoolingConcurrenncy` due to typos and add a new method with correct name - Fix other internal typos

Signed-off-by: ZhangJian He <[email protected]>

…4301) Signed-off-by: ZhangJian He <[email protected]>

### Motivation When reformatting Java files, JAVADOC_STYLE cause the license header be treated as doc comments and it will be affected from: ```java /** * ... * with the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, * ... */ ``` to ```java /** * ... * with the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, * ... */ ``` Signed-off-by: ZhangJian He <[email protected]>

### Motivation Upgrade Bouncy Castle to 1.78 to address CVEs https://bouncycastle.org/releasenotes.html#r1rv78 - https://www.cve.org/CVERecord?id=CVE-2024-29857 (reserved) - https://security.snyk.io/vuln/SNYK-JAVA-ORGBOUNCYCASTLE-6613079 - https://www.cve.org/CVERecord?id=CVE-2024-30171 (reserved) - https://security.snyk.io/vuln/SNYK-JAVA-ORGBOUNCYCASTLE-6613076 - https://www.cve.org/CVERecord?id=CVE-2024-30172 (reserved) - https://security.snyk.io/vuln/SNYK-JAVA-ORGBOUNCYCASTLE-6612984 Signed-off-by: ZhangJian He <[email protected]>

…ageCollectorThread (apache#4296) ### Motivation Current scheduling mechanism for garbage collection uses scheduleAtFixedRate. This approach schedules the next execution without considering whether the current task has finished, potentially leading to overlapping executions if a task takes longer than expected. In my test environment, After task accumulation in gc thread pool, sometimes there is no entrylog to extract and no entrylogger to compact. But every round of gc, still need to compare ledger meta between local and metadata store (zk), which will result in highly frequently access to metadata store and each access will bring considerable unnecessary data flow. See https://lists.apache.org/thread/023vkc5rwyq0j776zcv8dtp7c8cml6vp Signed-off-by: ZhangJian He <[email protected]>

apache#4291) ### Motivation Clean up ostrich related code、docs ### Changes delete `ostrich` version define in `pom.xml`, wrong version docs. Signed-off-by: ZhangJian He <[email protected]>

### Motivation Fixed the JVM Direct Memory Metrics test failure on Windows due to differences in line separators across operating systems. ### Changes - Moved the ByteBuf allocation after metrics server started. - Changed the line separator in assertions from System.lineSeparator() to \n. Signed-off-by: ZhangJian He <[email protected]>

Signed-off-by: ZhangJian He <[email protected]>

…4290) Signed-off-by: ZhangJian He <[email protected]>

…ache#3864) According to `https://stackoverflow.com/questions/12808934/what-is-p99-latency`, the implementation for `percentile` in bookkeeper-benchmark is wrong. Signed-off-by: ZhangJian He <[email protected]> Co-authored-by: ZhangJian He <[email protected]>

…elp (apache#4241)

…er (apache#4311)

…apache#4313)

Signed-off-by: ZhangJian He <[email protected]>

…he#4310) ### Motivation See apache#995 ### Changes 1. use junit assert instead of java assert statement in test 2. delete a unused util class also uses assert statement Signed-off-by: ZhangJian He <[email protected]>

Signed-off-by: ZhangJian He <[email protected]>

Fix apache#2700 ## Changes - shade jar should include com.google.guava:failureaccess. - remove unused exclude dependency now. Signed-off-by: ZhangJian He <[email protected]>

…pache#4066)

### Changes 1. Added junit5 annotation `@DisabledOnOs(OS.WINDOWS)`(It's modern and better than junit4) to tests that use `suspend` and `resume`, effectively disabling these tests on windows. 2. Migrate these test files to junit5 See also https://lists.apache.org/thread/hlyc722gy45s3y61q7n79hdcr6cgrx7v Signed-off-by: ZhangJian He <[email protected]>

…yMode (apache#3972)

…od (apache#4303) ### Motivation remove duplicate forceGarbageCollection setValue, it is redundant and error-prone. It's already seted in https://github.com/apache/bookkeeper/blob/945cfa138f2b64f258d5b53a98505264085500f0/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java#L473-L478 Signed-off-by: ZhangJian He <[email protected]>

apache#3113) Master Issue: apache#3112 ### Motivation Fix the failed pid occupied check. we'll fail when use bookkeeper-daemon.sh start or stop bookie, after the last time we exit the bookie direct kill or the bookie occurred non-normal exit, then the bin/bookkeeper-bookie.pid are retained, and the pid in the file is occupied by the thread in other progress. ### Changes Change the pid occupied check from 'kill -0 $pid' to 'ps -p $pid'. The both will return true when the pid is occupied by one progress, but 'kill -0 $pid' will return true and the 'ps -p $pid' will return false when the pid is occupied by one progress's sub thread. StackOverflow discussion: https://stackoverflow.com/questions/30958964/why-do-kill-0-pid-echo-and-ps-ppid-echo-sometimes-differ Co-authored-by: nicklixinyang <[email protected]>

Signed-off-by: ZhangJian He <[email protected]>

…ache#4258) Signed-off-by: ZhangJian He <[email protected]> Co-authored-by: ZhangJian He <[email protected]>

…ache#4327) Signed-off-by: ZhangJian He <[email protected]>

…yServer (apache#4337) Signed-off-by: ZhangJian He <[email protected]>

### Motivation 4.17.0 doesn't show up in the releases page ### Changes Add 4.17.0

### Motivation The tests under `bookkeeper-server` were failing on Windows due to path handling and other compatibility issues. The aim of this PR is to ensure that all tests pass on Windows as well as other operating systems by making the necessary adjustments in the test code. ### Changes - **TestBKConfiguration**: Updated the comment style from a block comment to a single-line comment for `setZkRetryBackoffMaxRetries`. - **TestTLS**: - Modified `getResourcePath` to use `Paths.get()` which handles file paths in a cross-platform manner. - Replaced `assertTrue` with `assertFalse` followed by `isEmpty()` to check if ensembles are present, improving the readability. Signed-off-by: ZhangJian He <[email protected]>

…RegionAwareEnsemblePlacementPolicy beyond jdk11 (apache#4323) ### Motivation In JDK21, it's is no way to change private static final variable currently, before figure out how to replace, disable it first ``` Field logField = repp.getClass().getDeclaredField("LOG"); Logger mockLogger = mock(Logger.class); Field modifiers = Field.class.getDeclaredField("modifiers"); modifiers.setAccessible(true); modifiers.setInt(logField, logField.getModifiers() & ~Modifier.FINAL); ``` See also https://issues.apache.org/jira/browse/CASSANDRA-18181 ### Changes 1. Added junit5 annotation `@EnabledForJreRange(max = JRE.JAVA_17)`(It's modern and better than junit4) to tests that use `suspend` and `resume`, effectively disabling these tests on JDK versions higher than 17 where these methods are not supported. 2. BookieZKExpireTest should be annotated as junit5 test. 3. Make `BookKeeperClusterTestCase` compatible to junit5. Signed-off-by: ZhangJian He <[email protected]>

Signed-off-by: ZhangJian He <[email protected]>

### Motivation Fix build warnings, and `librt` is not used. ``` [INFO] --- nar-maven-plugin:3.5.2:nar-compile (default-nar-compile) @ circe-checksum --- [INFO] Compiling 2 native files [WARNING] No file matching patterns ("librt.a, librt.so) for library name "rt" was found. [INFO] 2 total files to be compiled. ``` Signed-off-by: shoothzj <[email protected]>

### Motivation `JAVA_HOME` env variable is not properly set. I have tested `ARG TARGETARCH` works well, when we run `docker build`, it will automatically set the value. error log below: ``` bookie3_1 | JAVA_HOME not set, using java from PATH. (/usr/bin/java) bookie1_1 | JAVA_HOME not set, using java from PATH. (/usr/bin/java) bookie3_1 | Unrecognized VM option 'PrintGCApplicationStoppedTime' bookie3_1 | Error: Could not create the Java Virtual Machine. ``` Signed-off-by: ZhangJian He <[email protected]>

### Motivation When using SLF4J 2.0.x, it's useful to use slf4j-bom to align slf4j library versions. This is necessary to ensure that logging doesn't break due to mixed slf4j library versions. Similarly, it's useful to use log4j-bom to align log4j library versions. SLF4J 2.0.x upgrade was made in PR apache#4248 . ### Changes - use slf4j-bom and log4j-bom in dependencyManagement

…pache#4332) Fix apache#2077 By the way, I have been tested, negative number will fail. ### Motivation In the existing tests for `ConcurrentLongHashSet` and `ConcurrentLongLongHashMap`, the method `Math.abs(random.nextLong())` is used to generate keys. However, `Math.abs` can return a negative number if `Long.MIN_VALUE` is generated, which is problematic for tests that assume non-negative keys. Additionally, the assertions in the tests are using the older JUnit 4 style, which is less readable and not as flexible as JUnit 5. ### Changes 1. **Key Generation Fix**: Replaced `Math.abs(random.nextLong())` with `ThreadLocalRandom.current().nextLong(Long.MAX_VALUE)` to ensure that all generated keys are non-negative. 2. **Assertion Updates**: Updated all assertions in the `ConcurrentLongHashSetTest` and `ConcurrentLongLongHashMapTest` to use JUnit 5 style. This includes using `assertEquals`, `assertTrue`, `assertFalse`, and `assertNull` for better readability and consistency. Signed-off-by: ZhangJian He <[email protected]>

Signed-off-by: ZhangJian He <[email protected]>

…ationException (apache#4347) ### Motivation JDK 17 and later throw `UnsupportedOperationException` for the `suspend` and `resume` methods, causing the `testConstructionZkDelay` and `testConstructionNotConnectedExplicitZk` tests to fail. See jdk21 daily build error: https://github.com/apache/bookkeeper/actions/runs/8962118072 ### Changes - Disabled the two affected test methods for JDK 17 and above using the `@EnabledForJreRange` annotation. - Transitioned to JUnit 5 annotations and assertions. ### Q&A Q: why there are so many line changes? like format change A: In junit5, the msg should be put in the latest param, related to some changes. Signed-off-by: ZhangJian He <[email protected]>

…ache#4256)

…he#4345)

…he#4340)

Signed-off-by: ZhangJian He <[email protected]>

…#4349) Signed-off-by: ZhangJian He <[email protected]>

Signed-off-by: ZhangJian He <[email protected]>

…he#4331) ### Changes 1. Remove empty module backward-compat yahoo-custom-version 2. fix a minor annotation in LedgerHandle Signed-off-by: ZhangJian He <[email protected]>

Signed-off-by: ZhangJian He <[email protected]>

…pact settings (apache#4355) ### Motivation The current configuration documentation for BookKeeper lacks some important parameters. Additionally, existing parameters are not optimally organized. This update: - Introduces missing parameters like `isThrottleByBytes`, `compactionRateByEntries`, and `compactionRateByBytes`. - Reorganizes the configuration parameters for better clarity. ### Changes - **Added:** New parameters that were missing in the configuration: - `compactionRateByEntries`: Set the rate at which compaction will read entries, measured in entries per second. - `compactionRateByBytes`: Set the rate at which compaction will read entries, measured in bytes per second. - **Reorganized:** Adjusted the placement of existing parameters for better logical grouping and readability: - `minorCompactionThreshold`: Adjusted position for clarity. - `minorCompactionInterval`: Moved for logical grouping. Signed-off-by: ZhangJian He <[email protected]> Co-authored-by: ZhangJian He <[email protected]>

Signed-off-by: ZhangJian He <[email protected]>

### Motivation Fix ![image](https://github.com/apache/bookkeeper/assets/10069311/c7256c52-ebb5-4577-935a-cccdd4625f47)

### Motivation In the RackawareEnsemblePlacementPolicyImpl#selectRandomInternal method, if bookieNode in the ensemble list is not added to the excludeBookies set and the diskWeight feature is enabled, this method will fall into an infinite loop. https://github.com/apache/bookkeeper/blob/3ed93a0342ccff8c30ea472d731a7b593b4d32b0/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/RackawareEnsemblePlacementPolicyImpl.java#L773-L777 The root cause is the newly selected bookies already exist in the ensemble list, leading the `ensemble.addNode(bookie)` method to return `false` and the while loop continues. https://github.com/apache/bookkeeper/blob/3ed93a0342ccff8c30ea472d731a7b593b4d32b0/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/RackawareEnsemblePlacementPolicyImpl.java#L841 In the next round of the loop, `bookiesSeenSoFar.size()` will never equal to `bookiesToSelectFrom.size()` due to the wRSelection set doesn't exclude the bookies in the ensemble set. https://github.com/apache/bookkeeper/blob/3ed93a0342ccff8c30ea472d731a7b593b4d32b0/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/RackawareEnsemblePlacementPolicyImpl.java#L815-L822 ### Changes Remove the following code to make sure the items in bookiesToSelectFrom are all included in `wRSelection` https://github.com/apache/bookkeeper/blob/3ed93a0342ccff8c30ea472d731a7b593b4d32b0/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/RackawareEnsemblePlacementPolicyImpl.java#L795-L797

### Motivation 4.14.0 introduce new conf `minorCompactionMaxTimeMillis` and `majorCompactionMaxTimeMillis`, but not recorded in the `upgrade.md`. ### Changes Record the conf change in 4.14.0. Co-authored-by: ZhangJian He <[email protected]>

…pache#4366) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: ZhangJian He <[email protected]>

Signed-off-by: ZhangJian He <[email protected]>

…rsions (apache#4365) ### Changes - Fix some versions wrong in the version docs. - Cherry-Pick the removal of twitter metrics library to 4.17.0 doc - Update the `latest_release` and `stable_release ` Signed-off-by: ZhangJian He <[email protected]>

Signed-off-by: ZhangJian He <[email protected]>

…ache#4017)

Signed-off-by: ZhangJian He <[email protected]>

…pache#4300) * Make tests fail * Fix ThreadRegistry#register to ensure correct Prom metrics * Change style to match BK standards * Fix tests --------- Co-authored-by: Nicolò Boschi <[email protected]>

Signed-off-by: ZhangJian He <[email protected]>

…pache#4293) ### Motivation This addresses the remaining gaps of apache#4289 in handling ByteBuf retain/release. This PR will also address the concern about NioBuffer lifecycle brought up in the review of the original PR review: apache#791 (comment) . This PR fixes several problems: * ByteString buffer lifecycle in client, follows ByteBufList lifecycle * ByteBufList lifecycle, moved to write promise * Calling of write promises in AuthHandler which buffers messages while authentication is in progress. It was ignoring the promises. ### Changes - add 2 callback parameters to writeAndFlush: cleanupActionFailedBeforeWrite and cleanupActionAfterWrite - use these callback actions for proper cleanup - extract a utility class ByteStringUtil for wrapping ByteBufList or ByteBuf as concatenated zero copy ByteString - properly handle releasing of ByteBufList in the write promise - properly handle calling promises that are buffered while authentication is in progress

### Motivation ByteBufVisitor added in apache#4196 contains some code that isn't used and covered by unit tests. It's better to remove such code since it makes it harder to reason about the solution. The ByteBufVisitor solution will unwrap all direct buffers that can be unwrapped without the code that is to be removed. I believe that I forgot this code from some earlier phase of the solution where it was necessary to include this. I ran some local tests and didn't see that the code was used for the original purpose that it was added for (supporting read-only buffers). ### Changes Remove the unused code.

…4387)

Fix [apache#4373](apache#4373) ### Motivation As shown in the issue. This is the first pr to fix the lost metric problem. And the other pr improve the metric in SingleThreadExecutor. Co-authored-by: fanjianye <[email protected]>

Signed-off-by: ZhangJian He <[email protected]>

### Motivation Introduce typos ci to avoid typos. See https://lists.apache.org/thread/04hqqcnkfc5189zsxj0s5wm37t2x7bky Signed-off-by: ZhangJian He <[email protected]>

### Motivation `integrate-tests` module depends on `arquillian-cube`, which are updates infrequently. Lastly release was in 2018, and it depends on a lot of dependency with CVEs, like `bcprov-jdk15on-1.64.jar` etc. But it still merging code frequently, my team will trying to ask for new release, I think we can remove tests module from owasp check for now. I run the check locally, fix 7 error, and no new errors generated compared to daily build result. #### Local Result ``` [ERROR] amqp-client-5.5.3.jar: CVE-2023-46120(7.5) [ERROR] jetcd-core-0.7.7.jar: CVE-2020-15113(7.1) [ERROR] jetcd-grpc-0.7.7.jar: CVE-2023-44487(7.5), CVE-2017-8359(9.8), CVE-2023-33953(7.5), CVE-2020-15113(7.1), CVE-2020-7768(9.8), CVE-2017-7861(9.8), CVE-2017-9431(9.8), CVE-2017-7860(9.8) [ERROR] okio-3.2.0.jar: CVE-2023-3635(7.5) ``` After refresh the suppression list(jetcd required jdk11, amqp is independent dependency from dropwizard metrics. There is only, I think we can fix it though upgrading OTEL version. ``` [ERROR] okio-3.2.0.jar: CVE-2023-3635(7.5) ``` #### Daily build result ``` Error: amqp-client-5.5.3.jar: CVE-2023-46120(7.5) Error: bcprov-jdk15on-1.64.jar: CVE-2024-29857(7.5), CVE-2024-34447(7.699999809265137) Error: jetcd-core-0.7.7.jar: CVE-2020-151[13](https://github.com/apache/bookkeeper/actions/runs/9239112134/job/25417858696#step:5:14)(7.1) Error: jetcd-grpc-0.7.7.jar: CVE-2023-44487(7.5), CVE-2017-8359(9.8), CVE-2023-33953(7.5), CVE-2020-15113(7.1), CVE-2020-7768(9.8), CVE-2017-7861(9.8), CVE-2017-9431(9.8), CVE-2017-7860(9.8) Error: okio-3.2.0.jar: CVE-2023-3635(7.5) Error: plexus-cipher-2.0.jar: CVE-2022-4244(7.5) Error: plexus-classworlds-2.7.0.jar: CVE-2022-4244(7.5) Error: plexus-component-annotations-2.1.0.jar: CVE-2022-4244(7.5) Error: plexus-interpolation-1.26.jar: CVE-2022-4244(7.5) Error: plexus-sec-dispatcher-2.0.jar: CVE-2022-4244(7.5) Error: snakeyaml-1.19.jar: CVE-20[17](https://github.com/apache/bookkeeper/actions/runs/9239112134/job/25417858696#step:5:18)-18640(7.5), CVE-2022-25857(7.5) ``` Signed-off-by: ZhangJian He <[email protected]>

### Changes - **circe-checksum module:** - Removed unused imports such as `java.nio.ByteBuffer` and `java.util.zip.Checksum` in `Java8IntHash.java`. - Removed `java.io.IOException` import in `NativeUtils.java`. - **microbenchmarks module:** - Modified the `pom.xml` to enforce Checkstyle by removing the `<skip>true</skip>` configuration. - Removed unused `ByteBufList` imports from `ProtocolBenchmark.java` and `DigestManagerBenchmark.java`. Signed-off-by: ZhangJian He <[email protected]>

…apache#4404) * Fix TLS stability issues with V2 protocol that caused data corruption - add the TLS handler after the FlushConsolidationHandler - This makes TLS connections from Pulsar Broker to Bookkeeper stable when bookkeeperUseV2WireProtocol=true is used - Fix test TestTLS for V2 - Fix inconsistency in client configuration in BookKeeperClusterTestCase

…pache#4406) Signed-off-by: ZhangJian He <[email protected]>

Signed-off-by: ZhangJian He <[email protected]>

…he#4408) Signed-off-by: ZhangJian He <[email protected]>

Signed-off-by: ZhangJian He <[email protected]>

…tPolicyCheckTest. (apache#4080)

Co-authored-by: ZhangJian He <[email protected]>

…pache#4305)

…pache#4306) * SingleDirectoryDbLedgerStorage skip optimistic cache put When tryOptimisticRead returns 0, the lock is acquired by another thread and we will fail the validation step. We can therefore skip the eager cache insertion.

### Motivation This PR addresses the issue where RocksDB configurations fail to correctly resolve paths on Windows systems in the `ServerConfiguration` class. ### Changes - Updated `ServerConfiguration.java` to utilize `java.nio.file.Paths` for path normalization and resolution. - Refactored the `getDefaultRocksDBConf`, `getEntryLocationRocksdbConf`, and `getLedgerMetadataRocksdbConf` methods to use a new method `getFilePath`. Signed-off-by: ZhangJian He <[email protected]>

…he#4416) Signed-off-by: ZhangJian He <[email protected]>

Main Issue: apache#4409 ### Motivation Upgrade RocksDB to version 7.10.2 ### Changes Upgrade RocksDB to version 7.10.2

Signed-off-by: ZhangJian He <[email protected]>

…ache#4415) Fix apache#3514 ### Motivation This configuration may have initially intended to add a switch in `slowBookies`, but it was overlooked. I reset the history to when the code was merged into the master branch; it is still only referenced in the test files. Furthermore, even if the test call is removed, the test cases still pass. Therefore, I think it is reasonable to remove it. Signed-off-by: ZhangJian He <[email protected]>

…ache#4419)

…pgrade cases, read check from old server (apache#3981) * Backwards compat tests: added new versions, pulsar upgrade cases, read check from old server * upgrade tests with crc32c * Fail integration tests if no tests found

…odeQL workflow (apache#4361) Co-authored-by: ZhangJian He <[email protected]>

Signed-off-by: ZhangJian He <[email protected]>

…pache#4432) Signed-off-by: zhiheng123 <[email protected]>

### Changes - Replaced `docker build` with `docker buildx build` to enhance cross-platform compatibility. - Modified Dockerfile commands to conditionally create user and group only if they do not already exist, preventing errors during the build process.

…reasons (apache#4433)

…e#4434)

…4435) Signed-off-by: ZhangJian He <[email protected]>

…pache#4426) ### Motivation [Netty 4.1.111.Final](https://netty.io/news/2024/06/11/4-1-111-Final.html) contains important fixes: - netty/netty#14086 - netty/netty#14093 On the Pulsar side these address Broker stability when TLS with Bookkeeper V2 protocol is used between Broker and Bookies. The Bookkeeper client didn't contain a workaround for the Netty SslHandler "feature" like Pulsar did. That's why Bookkeeper client was impacted. Netty 4.1.111.Final will address those stability issues with the Bookkeeper client when using V2 protocol over TLS. ### Changes - Upgrade Netty to 4.1.111.Final - Switch to use grpc-netty-shaded instead of grpc-netty - grpc-java is not compatible with Netty 4.1.111.Final. [grpc-java supports only specific Netty versions](https://github.com/grpc/grpc-java/blob/master/SECURITY.md#netty). The solution is to use grpc-netty-shaded instead of grpc-netty dependency. - Add module to shade jetcd-core partially so that it can work with grpc-netty-shaded. - jetcd-core library and it's transient dependency vertx-grpc has a direct dependency on grpc-netty and it doesn't work with grpc-netty-shaded without this solution. - this solution avoids the need to shade grpc libraries completely which would cause a lot of duplication and increase build times.

…pache#4436) Signed-off-by: ZhangJian He <[email protected]>

…sword (apache#4437) Signed-off-by: ZhangJian He <[email protected]>

)

Co-authored-by: qunzhong <[email protected]>

…#4443) Fix apache#1413 ## Changes change ledger state to close when close metadata

Signed-off-by: ZhangJian He <[email protected]>

)

…pache#4445) Signed-off-by: ZhangJian He <[email protected]>

Signed-off-by: ZhangJian He <[email protected]>

…che#4458) ### Motivation This PR updates the release scripts to ensure more robust and consistent release processes. The changes include modifying the git tagging instructions to prevent potential mis-tags and updating the Python scripts to use Python 3 explicitly, which aligns with best practices for Python development. ### Changes 1. **Release Guide Modification**: Using ^{} ensures that the tag is created for the commit itself, not rc tag to be deleted. 2. **Python Publishing Script**: Changed all instances of `python` to `python3`, modern operation systems default has `python3`, but not all has `python` Signed-off-by: ZhangJian He <[email protected]>

### Motivation Reference: https://adoptium.net/blog/2021/08/using-jlink-in-dockerfiles Reduce size: 398.23 MB -> 245.23 MB New image:https://hub.docker.com/layers/nodece/bk/4.17.1-jlink/images/sha256-73c8354a981d7dfefcb1247cf84f6c3e91db2342431749e528ba8468c65be1f5?context=explore Old image: https://hub.docker.com/layers/apache/bookkeeper/4.17.1/images/sha256-342f384bc21b4d31d5349f741cf831f3c5bb47dfea34557104528ccd9597810f?context=explore ### Changes - Using jlink to compress the JDK and then copy that to the base image

…kie is not available. (apache#4439) When the bookie is not available, the RackawareEnsemblePlacementPolicyImpl default rack will be `/default-region/default-rack`, it should be `/default-rack` for RackawareEnsemblePlacementPolicyImpl. There are some logs. ``` 2024-06-17T05:22:46,591+0000 [ReplicationWorker] ERROR org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy - Cannot resolve bookieId `test-bk-3:3181` to a network address, resolving as /default-region/default-rack org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId test-bk-3:3181, bookie does not exist or it is not running at org.apache.bookkeeper.client.DefaultBookieAddressResolver.resolve(DefaultBookieAddressResolver.java:66) ~[io.streamnative-bookkeeper-server-4.16.5.2.jar:4.16.5.2] at org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy.resolveNetworkLocation(TopologyAwareEnsemblePlacementPolicy.java:821) ~[io.streamnative-bookkeeper-server-4.16.5.2.jar:4.16.5.2] at org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy.createBookieNode(TopologyAwareEnsemblePlacementPolicy.java:811) ~[io.streamnative-bookkeeper-server-4.16.5.2.jar:4.16.5.2] at org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy.convertBookieToNode(TopologyAwareEnsemblePlacementPolicy.java:845) ~[io.streamnative-bookkeeper-server-4.16.5.2.jar:4.16.5.2] at org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy.convertBookiesToNodes(TopologyAwareEnsemblePlacementPolicy.java:837) ~[io.streamnative-bookkeeper-server-4.16.5.2.jar:4.16.5.2] at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.replaceBookie(RackawareEnsemblePlacementPolicyImpl.java:474) ~[io.streamnative-bookkeeper-server-4.16.5.2.jar:4.16.5.2] at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.replaceBookie(RackawareEnsemblePlacementPolicy.java:119) ~[io.streamnative-bookkeeper-server-4.16.5.2.jar:4.16.5.2] at org.apache.bookkeeper.client.BookKeeperAdmin.getReplacementBookiesByIndexes(BookKeeperAdmin.java:993) ~[io.streamnative-bookkeeper-server-4.16.5.2.jar:4.16.5.2] at org.apache.bookkeeper.client.BookKeeperAdmin.replicateLedgerFragment(BookKeeperAdmin.java:1025) ~[io.streamnative-bookkeeper-server-4.16.5.2.jar:4.16.5.2] at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:473) ~[io.streamnative-bookkeeper-server-4.16.5.2.jar:4.16.5.2] at org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:301) ~[io.streamnative-bookkeeper-server-4.16.5.2.jar:4.16.5.2] at org.apache.bookkeeper.replication.ReplicationWorker.run(ReplicationWorker.java:249) ~[io.streamnative-bookkeeper-server-4.16.5.2.jar:4.16.5.2] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.108.Final.jar:4.1.108.Final] at java.lang.Thread.run(Thread.java:840) ~[?:?] Caused by: org.apache.bookkeeper.client.BKException$BKBookieHandleNotAvailableException: Bookie handle is not available at org.apache.bookkeeper.discover.ZKRegistrationClient.getBookieServiceInfo(ZKRegistrationClient.java:226) ~[io.streamnative-bookkeeper-server-4.16.5.2.jar:4.16.5.2] at org.apache.bookkeeper.client.DefaultBookieAddressResolver.resolve(DefaultBookieAddressResolver.java:45) ~[io.streamnative-bookkeeper-server-4.16.5.2.jar:4.16.5.2] ... 13 more ```

…ache#4244) ### Motivation If the system property `readonlymode.enabled` is set to true on a ZooKeeper server, read-only mode is enabled. Data can be read from the server in read-only mode even if that server is split from the quorum. https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#Experimental+Options%2FFeatures To connect to the server in read-only mode, the client must also allow read-only mode. The `ZooKeeperClient` class in the bookkeeper repository also has an option called `allowReadOnlyMode`. https://github.com/apache/bookkeeper/blob/15171e1904f7196d8e9f4116ab2aecdf582e0032/bookkeeper-server/src/main/java/org/apache/bookkeeper/zookeeper/ZooKeeperClient.java#L219-L222 However, even if this option is set to true, the connection to the server in read-only mode will actually fail. The cause is in the `ZooKeeperWatcherBase` class. When the `ZooKeeperWatcherBase` class receives the `SyncConnected` event, it releases `clientConnectLatch` and assumes that the connection is complete. https://github.com/apache/bookkeeper/blob/15171e1904f7196d8e9f4116ab2aecdf582e0032/bookkeeper-server/src/main/java/org/apache/bookkeeper/zookeeper/ZooKeeperWatcherBase.java#L128-L144 However, if the server is in read-only mode, it will receive `ConnectedReadOnly` instead of `SyncConnected`. This causes the connection to time out without being completed. ### Changes Modified the switch statement in the `ZooKeeperWatcherBase` class to release `clientConnectLatch` when `ConnectedReadOnly` is received if the `allowReadOnlyMode` option is true. By the way, `allowReadOnlyMode` is never set to true in BookKeeper. So this change would be useless for BookKeeper. However, it is useful for Pulsar. Because Pulsar also uses `ZooKeeperWatcherBase` and needs to be able to connect to ZooKeeper in read-only mode. https://github.com/apache/pulsar/blob/cba1600d0f6a82f1ea194f3214a80f283fe8dc27/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/PulsarZooKeeperClient.java#L242-L244

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue 1626: Add tests for LedgerUnderredplicationManager#markLedgerUnderreplicatedAsync #1629

Issue 1626: Add tests for LedgerUnderredplicationManager#markLedgerUnderreplicatedAsync #1629

Commits on Jul 12, 2024

Issue 1626: Add tests for LedgerUnderredplicationManager#markLedgerUnderreplicatedAsync #1629

Are you sure you want to change the base?

Issue 1626: Add tests for LedgerUnderredplicationManager#markLedgerUnderreplicatedAsync #1629

Commits on Jul 12, 2024