Support non-MaxMind mmdb data layouts

[ckreibich]

@philrz reported on Slack that alternative geolocation databases use the MaxMind data format but differ in the yielded data layout, see brimdata/geoip-conn#46 for details (including sample data). For example:

$ mmdblookup --file ./GeoLite2-Country.mmdb --ip 8.8.8.8

  {
    "continent":
      {
        "code":
          "NA" <utf8_string>
        "geoname_id":
          6255149 <uint32>
        "names":
          {
            "de":
              "Nordamerika" <utf8_string>
            "en":
              "North America" <utf8_string>
            "es":
              "Norteamérica" <utf8_string>
            "fr":
              "Amérique du Nord" <utf8_string>
            "ja":
              "北アメリカ" <utf8_string>
            "pt-BR":
              "América do Norte" <utf8_string>
            "ru":
              "Северная Америка" <utf8_string>
            "zh-CN":
              "北美洲" <utf8_string>
          }
      }
    "country":
      {
        "geoname_id":
          6252001 <uint32>
        "iso_code":
          "US" <utf8_string>
        "names":
          {
            "de":
              "Vereinigte Staaten" <utf8_string>
            "en":
              "United States" <utf8_string>
            "es":
              "Estados Unidos" <utf8_string>
            "fr":
              "États Unis" <utf8_string>
            "ja":
              "アメリカ" <utf8_string>
            "pt-BR":
              "EUA" <utf8_string>
            "ru":
              "США" <utf8_string>
            "zh-CN":
              "美国" <utf8_string>
          }
      }
    "registered_country":
      {
        "geoname_id":
          6252001 <uint32>
        "iso_code":
          "US" <utf8_string>
        "names":
          {
            "de":
              "Vereinigte Staaten" <utf8_string>
            "en":
              "United States" <utf8_string>
            "es":
              "Estados Unidos" <utf8_string>
            "fr":
              "États Unis" <utf8_string>
            "ja":
              "アメリカ" <utf8_string>
            "pt-BR":
              "EUA" <utf8_string>
            "ru":
              "США" <utf8_string>
            "zh-CN":
              "美国" <utf8_string>
          }
      }
  }

$ mmdblookup --file ./country.mmdb --ip 8.8.8.8

  {
    "continent":
      "NA" <utf8_string>
    "continent_name":
      "North America" <utf8_string>
    "country":
      "US" <utf8_string>
    "country_name":
      "United States" <utf8_string>
  }

Since we currently hardwire the lookup paths, opening such DBs works, but the result value lookups fail.

We can support this by making those paths configurable (as long as the data still fit the respective records), perhaps with "profiles" for common vendors.

[abdullahdevrel]

Hi @ckreibich and @philrz,

I reached out to Phil about supporting our free IP database. We are really excited to see the Zeek team support IPinfo's data. I would like to suggest using the IP to Country ASN database instead of the IP to Country database. We observed that ASN tends to provide better contextual data to IP addresses. A combination of country and ASN data will be quite useful.

Download command:

curl -L https://ipinfo.io/data/free/country_asn.mmdb?token=<token> -o country_asn.mmdb

Again, we really appreciate seeing this being developed. Please let us know if we can help in any way.