Extra Metadata Entries

Adding Extra Metadata to Parquet Files

Overview

This functionality allows you to add extra metadata entries to the footer of a Parquet file. The metadata can be specified in a JSON configuration file and will be included in the Parquet file when it is generated.

How to Use

Full example here

Example:

{
  "fileName": "example_with_extra_metadata.parquet",
  "options": {
    "extraMetaData": {
      "author": "John Doe",
      "description": "Sample Parquet file with extra metadata",
      "createdDate": "2021-01-01"
    }
  }
}

Now when we check the metadata of the generated Parquet file, we will see the following entries:

KeyValue
key = description
value = Sample Parquet file with extra metadata
KeyValue
key = createdDate
value = 2023-10-01
KeyValue
key = author
value = John Doe