url-metadata

Request an http(s) url and scrape its metadata. Many of the metadata fields returned are Open Graph Protocol (og:) so far.

Support also added for JSON-LD.

Under the hood, this package does some post-request processing on top of the request module.

If you want a new feature, please open an issue or pull request in GitHub.

Usage

To use in an npm/ Node.js project, install from your CLI:

$ npm install url-metadata --save`

Then in your project file (from example/basic.js):

const urlMetadata = require('url-metadata')
urlMetadata('http://bit.ly/2ePIrDy').then(
  function (metadata) { // success handler
    console.log(metadata)
  },
  function (error) { // failure handler
    console.log(error)
  })

If you'd like to override the default options (see below), pass in a second argument:

const urlMetadata = require('urlMetadata')
urlMetadata('http://bit.ly/2ePIrDy', {fromEmail: '[email protected]'}).then(...)

Options & Defaults

This package's default options are the values below that you may want to override:

{
  // custom name for the user agent and email that will make url request:
  userAgent: 'MetadataScraper',
  fromEmail: '[email protected]',

  // module will follow a maximum of 10 redirects
  maxRedirects: 10,

  // timeout in milliseconds, default below is 10 seconds:
  timeout: 10000,

  // number of characters to truncate description to:
  descriptionLength: 750,

  // force image urls in selected tags to use https,
  // valid for 'image', 'og:image' and 'og:image:secure_url' tags:
  ensureSecureImageRequest: true,

  // object containing key/value pairs used for `source` attribution;
  // defaults to empty object, see usage details below:
  sourceMap: {},

  // custom function to decode special-case encodings;
  // defaults to undefined:
  decode: undefined,

  // custom function to encode the metadata fields before they are returned;
  // defaults to undefined:
  encode: undefined,

  // custom headers
  customHeaders: {}
}

Option: Source Map

This module introduces and supports a metadata field called source. More details about source in the Returns section below. Example usage can be found in example/source-map.js

sourceMap is used to override the default source attribution behavior, which derives source from the web host that the url resolves to.

sourceMap is a simple object containing key/value pairs where the key is a YouTube username and the value is the source you'd like to attribute the content to (such as a domain, as in example below).

const options = {
  sourceMap: { 'the guardian': 'theguardian.com' }
}

If you'd like to extend this functionality beyond YouTube attribution, create an issue or pull request in GitHub.

Option: Decode

You can supply a custom function to decode the metadata scraped from the url. Example decoding of EUC-JP (Japanese) metadata can be found in example/decode.js.

If you pass in an options.decode() function, this module will force the request module to return the scraped metadata as a buffer to decode(). This module is not opinionated about what you do in the decode() function, only that it accepts a buffer as its argument and returns a string.

Option: Encode

You can supply a custom function to encode the metadata fields before they are returned from this module, see example/encode.js:

const options = {
  encode: function (value) {
    return encodeURIComponent(value).replace(/['*]/g, escape)
  }
}

Option: Custom Headers

You can specify any custom headers such as cookie, accept-language, accept-encoding, cache-control etc:

const options = {
  customHeaders: {
    'cookie': 'over18=1;',
    'accept-language': 'zh-TW,zh;q=0.9,en-NL;q=0.8,en;q=0.7,en-US;q=0.6'
  }
}

Returns

Returns a promise that gets resolved with the following url metadata if the url request response returns successfully. Note that the url field returned below will be the last hop in the request chain. So if you passed in a url that was generated by a link shortener, for example, you'll get back the final destination of the link as the url.

{
    'url': '',
    'canonical': '',
    'title': '',
    'image': '',
    'author': '',
    'description': '',
    'keywords': '',
    'source': '',
    'price': '',
    'priceCurrency': '',
    'availability': '',
    'robots': '',
    'og:url': '',
    'og:locale': '',
    'og:locale:alternate': '',
    'og:title': '',
    'og:type': '',
    'og:description': '',
    'og:determiner': '',
    'og:site_name': '',
    'og:image': '',
    'og:image:secure_url': '',
    'og:image:type': '',
    'og:image:width': '',
    'og:image:height': '',
    'twitter:title': '',
    'twitter:image': '',
    'twitter:image:alt': '',
    'twitter:card': '',
    'twitter:site': '',
    'twitter:site:id': '',
    'twitter:account_id': '',
    'twitter:creator': '',
    'twitter:creator:id': '',
    'twitter:player': '',
    'twitter:player:width': '',
    'twitter:player:height': '',
    'twitter:player:stream': '',
    'jsonld': {}
}

Additional fields are also returned if the url has an og:type set to article. These fields are:

{
  'article:published_time'     : '',
  'article:modified_time'      : '',
  'article:expiration_time'    : '',
  'article:author'             : '',
  'article:section'            : '',
  'article:tag'                : '',
  'og:article:published_time'  : '',
  'og:article:modified_time'   : '',
  'og:article:expiration_time' : '',
  'og:article:author'          : '',
  'og:article:section'         : '',
  'og:article:tag'             : ''
}

The source field

This module introduces and supports a metadata field called source which may be useful for attributing content hosted elsewhere to an original source.

The default behavior of this module is to derive the source field from the url's host (in cases where redirects take place before the last hop, the host is the last hop in the request chain):

metadata.set({ source: url.split('://')[1].split('/')[0] })

You may be able to override this default behavior with the sourceMap option.