Skip to content

The parcel repository format

Jump to bottom
philipl edited this page Feb 14, 2014 · 12 revisions

To make parcels easily accessible to Cloudera Manager, we have a concept of a parcel repository. Like a yum or apt package repository, this is a remote location that CM can connect to do discover and download parcels.

Structure of a parcel repository

A parcel repository is very simple: It consists of a directory that contains a set of parcel files and a manifest.json file that describes them. This simple structure allows a repository to be constructed simply by publishing the directory through an http server

Example repository layout

  • manifest.json
  • CDH-5.0.0-0.cdh5b2.p0.27-el5.parcel
  • CDH-5.0.0-0.cdh5b2.p0.27-el6.parcel
  • CDH-5.0.0-0.cdh5b2.p0.27-precise.parcel
  • CDH-5.0.0-0.cdh5b2.p0.27-sles11.parcel
  • CDH-5.0.0-0.cdh5b2.p0.27-wheezy.parcel

Note that as we have one parcel for each supported distro, there is a separate entry in the manifest for each of them. CM takes care of only showing a single entry in its UI and auto-selecting the right parcel file based on the installed distro on each host in a cluster.

Example manifest.json 

{
  "lastUpdated": 13876052920000,
  "parcels": [
    {
      "parcelName": "HADOOP_LZO-0.4.15-1.gplextras.p0.39-sles11.parcel",
      "components": [
        { "name":    "hadoop-lzo-cdh4",
          "version": "0.4.15",
          "pkg_version": "0.4.15"
        }
        ,{ "name":    "hadoop-lzo-cdh4-mr1",
          "version": "0.4.15",
          "pkg_version": "0.4.15"
        }
        ,{ "name":    "impala-lzo",
          "version": "1.2.3",
          "pkg_version": "1.2.3"
        }
      ],
      "hash": "872e6b60b2ba0483f5249f9f0a093e64a1ac636e"
    },
    {
      "parcelName": "HADOOP_LZO-0.4.15-1.gplextras.p0.39-el5.parcel",
      "components": [
        { "name":    "hadoop-lzo-cdh4",
          "version": "0.4.15",
          "pkg_version": "0.4.15"
        }
        ,{ "name":    "hadoop-lzo-cdh4-debuginfo",
          "version": "0.4.15",
          "pkg_version": "0.4.15"
        }
        ,{ "name":    "hadoop-lzo-cdh4-mr1",
          "version": "0.4.15",
          "pkg_version": "0.4.15"
        }
        ,{ "name":    "impala-lzo",
          "version": "1.2.3",
          "pkg_version": "1.2.3"
        }
        ,{ "name":    "impala-lzo-debuginfo",
          "version": "1.2.3",
          "pkg_version": "1.2.3"
        }
      ],
      "hash": "66789d91506335d1627810c2f857ae5818bbfeba"
    },
    {
      "parcelName": "HADOOP_LZO-0.4.15-1.gplextras.p0.39-el6.parcel",
      "components": [
        { "name":    "hadoop-lzo-cdh4",
          "version": "0.4.15",
          "pkg_version": "0.4.15"
        }
        ,{ "name":    "hadoop-lzo-cdh4-debuginfo",
          "version": "0.4.15",
          "pkg_version": "0.4.15"
        }
        ,{ "name":    "hadoop-lzo-cdh4-mr1",
          "version": "0.4.15",
          "pkg_version": "0.4.15"
        }
        ,{ "name":    "impala-lzo",
          "version": "1.2.3",
          "pkg_version": "1.2.3"
        }
        ,{ "name":    "impala-lzo-debuginfo",
          "version": "1.2.3",
          "pkg_version": "1.2.3"
        }
      ],
      "hash": "272d4c6a5dcba9c1d551b6bd08d330d1f0613f70"
    },
    {
      "parcelName": "HADOOP_LZO-0.4.15-1.gplextras.p0.39-lucid.parcel",
      "components": [
        { "name":    "hadoop-lzo-cdh4",
          "version": "0.4.15-1.gplextras.p0.152~lucid-gplextras4.3.0",
          "pkg_version": "0.4.15-1.gplextras.p0.152~lucid-gplextras4.3.0"
        }
        ,{ "name":    "hadoop-lzo-cdh4-mr1",
          "version": "0.4.15-1.gplextras.p0.152~lucid-gplextras4.3.0",
          "pkg_version": "0.4.15-1.gplextras.p0.152~lucid-gplextras4.3.0"
        }
        ,{ "name":    "impala-lzo",
          "version": "1.2.3-1.gplextras.p0.216~lucid-gplextras4.3.0",
          "pkg_version": "1.2.3-1.gplextras.p0.216~lucid-gplextras4.3.0"
        }
      ],
      "hash": "525376985d5b6ad1fa4281c84f72f70ace07b63b"
    },
    {
      "parcelName": "HADOOP_LZO-0.4.15-1.gplextras.p0.39-squeeze.parcel",
      "components": [
        { "name":    "hadoop-lzo-cdh4",
          "version": "0.4.15-1.gplextras.p0.152~squeeze-gplextras4.3.0",
          "pkg_version": "0.4.15-1.gplextras.p0.152~squeeze-gplextras4.3.0"
        }
        ,{ "name":    "hadoop-lzo-cdh4-mr1",
          "version": "0.4.15-1.gplextras.p0.152~squeeze-gplextras4.3.0",
          "pkg_version": "0.4.15-1.gplextras.p0.152~squeeze-gplextras4.3.0"
        }
        ,{ "name":    "impala-lzo",
          "version": "1.2.3-1.gplextras.p0.216~squeeze-gplextras4.3.0",
          "pkg_version": "1.2.3-1.gplextras.p0.216~squeeze-gplextras4.3.0"
        }
      ],
      "hash": "dca12af8e1a0e4b67ed3fd7583e75528478664fa"
    },
    {
      "parcelName": "HADOOP_LZO-0.4.15-1.gplextras.p0.39-precise.parcel",
      "components": [
        { "name":    "hadoop-lzo-cdh4",
          "version": "0.4.15-1.gplextras.p0.152~precise-gplextras4.3.0",
          "pkg_version": "0.4.15-1.gplextras.p0.152~precise-gplextras4.3.0"
        }
        ,{ "name":    "hadoop-lzo-cdh4-mr1",
          "version": "0.4.15-1.gplextras.p0.152~precise-gplextras4.3.0",
          "pkg_version": "0.4.15-1.gplextras.p0.152~precise-gplextras4.3.0"
        }
        ,{ "name":    "impala-lzo",
          "version": "1.2.3-1.gplextras.p0.216~precise-gplextras4.3.0",
          "pkg_version": "1.2.3-1.gplextras.p0.216~precise-gplextras4.3.0"
        }
      ],
      "hash": "f5514a1cfa355b4a6d354934416b871f37b697e7"
    }
  ]
}

Building a manifest.json file

We recommend that you use the make_manifest.py tool, provided in this git repo, to build your manifest.json files. This automatically handles generating hashes and copying parcel metadata where appropriate.

Example

$ ./make_manifest.py /var/www/parcel-repo

Clone this wiki locally