-
Notifications
You must be signed in to change notification settings - Fork 36
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Hadoop 3.0 NN drops replicas silently #64
Comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In Hadoop 3.0/CDH5.7 and above,
HDFS-9260 (Improve the performance and GC friendliness of NameNode startup and full block reports) changed the internal representation of block replicas, as well as the block report processing logic in NameNode.
After HDFS-9260, NN expects block replicas to be reported in ascending order of block id. If a block id is not in order, NN discards it silently. Because simulated DataNode in Dynamometer uses hash map to store block replicas, the replicas are not reported in order. The Dynamometer cluster would then see missing blocks gradually increase several minutes after NN starts.
Suggest to change SimulatedBPStorage.blockMap to a TreeMap sorted by block id. Will supply a patch for the proposed change.
Credit: @fangyurao for identifying the issue, and help verifying the fix.
The text was updated successfully, but these errors were encountered: