Should we remove the use of versionHintFile from the entire FileSystemCatalog?

[BsoBird]

Feature Request / Improvement

For the past while, I've been doing some work related to fixing the correctness of fileSystemCatalog. Most of the problems were caused by incorrect version information being written to the versionHint file. This is because there is currently no way to guarantee that multiple file operations are atomic in the file system. If there is an interruption in the middle, we have all sorts of problems.

However, I noticed that versionHintFile is just an index file, and even without it, we can still read the latest version correctly. Since we only need to manipulate one file, we can ensure atomicity in this case.(Example: list all metadata and find max version, and using fs.rename to commit new version)

In my opinion, the introduction of the versionHint file not only doesn't greatly improve the overall efficiency of the task, but also introduces data inconsistencies. Should we remove the use of versionHintFile from the entire FileSystemCatalog?

Query engine

Spark

[BsoBird]

BTW, this PR has been on hold for several months now, if possible, can anyone continue to review this PR? Tks.
#9546

@RussellSpitzer @rdblue @nastra

[Fokko]

This would entail removing the whole HadoopCatalog. I think this is something that we can bring up with Iceberg 2.0

[BsoBird]

@Fokko
Sir, I don't quite understand, do you mean we will delete the whole hadoopCatalog? But we have a large number of customers who are using hadoopCatalog.

[BsoBird]

@Fokko
HadoopCatalog is working fine after fixing the problems associated with it, so why remove it?

[Fokko]

@BsoBird I'm sorry I misinterpreted what you're suggesting. What kind of alternative are you suggesting for the version hint file? One of the requirements of the File System Catalog is that the underlying file system is atomic. This is why other catalogs use an external service (Glue, Rest, etc) to ensure atomicity.

[BsoBird]

@Fokko We just need to drop the use of the versionHint file, and the hadoopcatalog is now atomic.

[BsoBird]

@Fokko
Because, for the file system catalog, the client's behaviour is currently to write to a temp file with a random ID and then rename the file to complete the commit. This is an atomic operation in hadoop or local filesystems. But the problem is that the previous author, in addition to the above behaviour, also wrote a versionHint file to record the latest version information. Since this involves two filesystem operations, the latest version of the rename is not necessarily the same as the latest version maintained in versionHint.

[BsoBird]

@Fokko
However, whether or not we use a versionHint file, we can find the latest version of the commit by some means. In other words, as long as the metadata file is renamed, the commit is actually successful. In this case, can we consider abandoning the use of the versionHint file and just use the metadata file to find the biggest version? The above logic is already done in the hadoopcatalog code.

The reason why people found it difficult to use hadoopcatalog before is that some of the processing logic was not rigorous enough. Once we fix those problems, hadoopcatalog is atomic.

[BsoBird]

In our production environment, we use hadoopcatalog heavily. After fixing the above issues, it performs very well.

[Fokko]

Thanks for the additional context. This is where I got confused:

we can find the latest version of the commit by some means

By some means, I assumed that you were hinting at an external service like a {REST,Hive,Glue} catalog. Because the FileIO does not allow list operations because that doesn't scale for object stores.

We cannot remove the versionHintFile since other systems might depend on it. If we want to do this, we should go through a deprecation cycle.

[BsoBird]

@Fokko

By some means, I assumed that you were hinting at an external service like a {REST,Hive,Glue} catalog. Because the FileIO does not allow list operations because that doesn't scale for object stores.

The current use of hadoopcatalog/fileSystemCatalog assumes that the file system provides atomic operations. Object stores do not have this feature.
As long as the file system meets this feature, we can find the latest metadata and parse out the latest version through the list operation. For object storage systems, we should avoid fs.rename. We should use fs.write to commit the metadata directly. This requires a new set of logic. But that's a different topic. (However, Some vendors do not provide synchronisation locking mechanisms for the object storage services they maintain.)

[BsoBird]

@Fokko

We cannot remove the versionHintFile since other systems might depend on it. If we want to do this, we should go through a deprecation cycle.

The versionHintFile is not that important in the current implementation of hadoopcatalog. If versionHintFile does not exist. It will list all metadataFiles to find the latest version information. This will slow down the submission slightly, but will not affect the correctness of the results.

The logic for parsing the latest commit-version with and without the versionHintFile is already implemented in the hadoopcatalog. All we need to do is remove the part about versionHintFile. The client will theoretically be unaware of this.

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.