Skip to content

API: Reports Query

Jump to bottom
elsif2 edited this page Aug 30, 2024 · 18 revisions

Reports API

An API to query the different reports received as well as to do basic queries of the data itself. This is meant as an optional replacement to the emails received with the report URL's. In all cases the queries and the data that is delivered is only from the reports that you would have normally received. You only get the data on the networks you are responsible for. You will not be able to get data on other networks or systems. Note refer to the API: Documentation pages for testing details and examples.

A complete list of report types and their schema is published at https://interchange.shadowserver.org/schema/reports.json.

Access to methods in this module are limited to members of the reports group.

Methods

  • device-info - Obtain device details for a given IP
  • list - List of actual reports that could be downloaded
  • query - Query the stored data
  • schema - Obtain JSON schema for a given report type
  • stats - Output stats
  • subscribed - List of reports that the user is subscribed to
  • types - List of all the types of reports that are available for the subscriber

reports/subscribed

Note that most organizations will only have a single list they are subscribed to and can get data on.

Fields:

apikey : string : Your API key

Response:

array of lists you are subscribed to

Example:

python3 call-api.py reports/subscribed {} pretty
[
    "united-states",
    "california",
]

reports/types

Fields:

    apikey  : string     : your api key

    date    : string     : date range matching query interface

    detail  : boolean    : include details (default false)

Response:

List of available types of reports.

Example 1:

python3 call-api.py reports/types {} pretty
[
   "blocklist",
   "device_id",
   "device_id6",
   "event4_honeypot_brute_force",
   "event4_honeypot_ddos_amp",
   "event4_honeypot_http_scan",
   "event4_microsoft_sinkhole_http",
   "event4_sinkhole",
   "event4_sinkhole_http",
   "scan6_telnet"
]

Example 2:

python3 call-api.py reports/types '{ "detail": true }' pretty
[
   {
      "url" : "https://www.shadowserver.org/what-we-do/network-reporting/accessible-ftp-report/",
      "type" : "scan_ftp",
      "description" : "Accessible FTP Service Results"
   },
   {
      "url" : "https://www.shadowserver.org/what-we-do/network-reporting/vulnerable-http-report/",
      "type" : "scan_http_vulnerable",
      "description" : "Vulnerable HTTP Results"
   },
   {
      "url" : "https://www.shadowserver.org/what-we-do/network-reporting/vulnerable-isakmp-report/",
      "type" : "scan_isakmp",
      "description" : "ISAKMP Vulnerability Scan Results"
   }
]

reports/list

Fields:

apikey  : string : Your API key
reports : list   : Report types to return (optional) - note that this is a list and must be bracketed ** [ ] **
date    : string : Date (YYYY-MM-DD) or range (YYYY-MM-DD:YYYY-MM-DD) (optional)
type    : string : Specific report to get download ID for
limit   : number : Limit the query to a specific number of records

Response:

array of up to 1,000 matching records

Use the id to download the CSV version of the report.

https://dl.shadowserver.org/uN6n7yZK90sdflkjdlLKTOkspksg?HjgX1lI_hAdsKGmVanG_Og

Please see the report-manager utility for an automation example.

Example request data:

python3 call-api.py reports/list '{"date":"2023-12-10", "limit":3}' pretty
[
   {
      "file" : "2023-12-10-blocklist-example-geo.csv",
      "id": "AuN6n7yZK90sdflkjdlLKTOkspksg?HjgX1lI_hAdsKHmVanG_Og"
      "report" : "example",
      "timestamp" : "2023-12-10",
      "type" : "blocklist",
      "url" : "https://www.shadowserver.org/what-we-do/network-reporting/blocklist-report/"
   },
   {
      "file" : "2023-12-10-compromised_website-example-geo.csv",
      "id": "ZjM0guFABt8KLHohi9s67lk23vjosu09lns934I5uqbP7KW6AHg",
      "report" : "example",
      "timestamp" : "2023-12-10",
      "type" : "compromised_website",
      "url" : "https://www.shadowserver.org/what-we-do/network-reporting/compromised-website-report/"
   },
   {
      "file" : "2023-12-10-device_id-example-geo.csv",
      "id": "3kkOMxJknsfe098yuwek5lnsvdoi0-9uwljIHOFR3ebOyf9YovjQ",
      "report" : "example",
      "timestamp" : "2023-12-10",
      "type" : "device_id",
      "url" : "https://www.shadowserver.org/what-we-do/network-reporting/device-identification-report/"
   }
]

An example specifying the report type:

python3 call-api.py reports/list '{"type":"hp_ics_scan", "date":"2020-10-27"}' pretty

[
    {
        "timestamp": "2020-10-27",
        "report": "example",
        "file": "2020-10-27-hp_ics_scan-example-geo.csv",
        "id": "60EL63IKLH90uLKJ3xcmXW93403mnkdsfiuIje-0nmdsseTXNP5EgDog"
    },
    {
        "report": "example",
        "timestamp": "2020-10-27",
        "id": "3lG4u-cHIOY9083oihdkUGIkhdsjuj38sdlkn0u92345AHyO9g5H5PK7A",
        "file": "2020-10-27-hp_ics_scan-example-geo.csv"
    }
]

reports/query

Returns a list of events matching the query parameters. This method is provided for research purposes and is not intended to replace the reports. The results returned provide less detail that what appears in the reports.

Fields:

apikey : string     : Your API key 
query  : dictionary : Search fields [any of the available fields from the reports].
sort   : string     : Ascending | descending 
date   : string     : Date (YYYY-MM-DD) or range (YYYY-MM-DD:YYYY-MM-DD) now may be used in place of a date 
facet  : string     : Returns the cardinality of each value of the given field sorted from highest to lowest
limit  : number     : Specify the number of records to pull
page   : number     : Default is 1; used to obtain additional pages of results

The query must contain one attribute that matches your organization's report filters.

Examples:

  • The Swiss CERT (GovCERT.ch) receives reports for all events for the country code 'CH'. In order for their query to be valid it must contain a matching attribute of "geo": "CH".
  • Cisco Systems receives reports filtered by ASN (109) and by domain name (com.cisco.). In order for their query to be valid it must contain "asn": 109 and/or "domain": "*.cisco.com".
  • Universität Konstanz receives reports filtered by IP (134.34.0.0/16). In order for their query to be valid it must contain an IP that matches their CIDR block "ip": "134.34.0.1" or the CIDR block "ip": "134.34.0.0/16".

Help:

python3 call-api.py reports/query '{"help":true}' pretty
[
    "agent",
    "application",
    "asn",
    "asn_name",
    "banner",
    "city",
    "county_fips",
    "county_name",
    "device_model",
    "device_sector",
    "device_type",
    "device_vendor",
    "device_version",
    "domain",
    "dst_asn",
    "dst_asn_name",
    "dst_city",
    "dst_county_fips",
    "dst_county_name",
    "dst_geo",
    "dst_ip",
    "dst_isp_name",
    "dst_latitude",
    "dst_longitude",
    "dst_naics",
    "dst_port",
    "dst_region",
    "dst_sector",
    "family",
    "geo",
    "infection",
    "ip",
    "isp_name",
    "latitude",
    "longitude",
    "md5",
    "naics",
    "port",
    "protocol",
    "referer",
    "region",
    "registrar",
    "sector",
    "sha1",
    "sha256",
    "sha512",
    "sid",
    "source",
    "source_url",
    "tag",
    "text",
    "timestamp",
    "tld",
    "type",
    "version"
]

Response:

array of up to 1,000 matching records

Example SSL Query by port:

python3 call-api.py reports/query '{"date":"2020-10-27", "query":{"geo":"us", "port":443}, "limit":1}' pretty
[
    {
        "port": "443",
        "tag": [
            "ssl"
        ],
        "region": "VIRGINIA",
        "protocol": "tcp",
        "isp_name": "MICROSOFT CORP",
        "county_fips": "51107",
        "naics": "334111",
        "infection": "ssl",
        "type": "scan",
        "asn": "8075",
        "asn_name": "MICROSOFT-CORP-MSN-A",
        "latitude": "39.03",
        "county_name": "LOUDOUN",
        "ip": "13.82.60.109",
        "rip": "109.60.82.13",
        "timestamp": "2020-10-27 02:59:32",
        "sid": "4096627",
        "geo": "US",
        "city": "ASHBURN",
        "longitude": "-77.49",
        "sector": "Critical Manufacturing"
    }
]

Example SSL query by tag:

python3 call-api.py reports/query '{"date":"2020-10-27", "query":{"geo":"us", "tag":"ssl"}, "limit":1}' pretty
[
    {
        "region": "VIRGINIA",
        "protocol": "tcp",
        "tag": [
            "ssl"
        ],
        "port": "443",
        "county_fips": "51107",
        "naics": "334111",
        "isp_name": "MICROSOFT CORP",
        "infection": "ssl",
        "type": "scan",
        "asn_name": "MICROSOFT-CORP-MSN-A",
        "latitude": "39.03",
        "asn": "8075",
        "county_name": "LOUDOUN",
        "rip": "109.60.82.13",
        "timestamp": "2020-10-27 02:59:32",
        "sid": "4096627",
        "ip": "13.82.60.109",
        "sector": "Critical Manufacturing",
        "geo": "US",
        "longitude": "-77.49",
        "city": "ASHBURN"
    }
]

Example using IP address:

python3 call-api.py reports/query '{"date":"2020-10-27", "query":{"geo":"us", "ip":"13.82.60.109"}, "limit":1}' pretty
[
    {
        "geo": "US",
        "longitude": "-77.49",
        "city": "ASHBURN",
        "sector": "Critical Manufacturing",
        "ip": "13.82.60.109",
        "timestamp": "2020-10-27 02:59:32",
        "rip": "109.60.82.13",
        "sid": "4096627",
        "county_name": "LOUDOUN",
        "asn": "8075",
        "latitude": "39.03",
        "asn_name": "MICROSOFT-CORP-MSN-A",
        "type": "scan",
        "infection": "ssl",
        "isp_name": "MICROSOFT CORP",
        "county_fips": "51107",
        "naics": "334111",
        "tag": [
            "ssl"
        ],
        "port": "443",
        "region": "VIRGINIA",
        "protocol": "tcp"
    }
]

Example by City:

python3 call-api.py reports/query '{"date":"2020-10-27", "query":{"geo":"us", "city":"ashburn"}, "limit":1}' pretty
[
    {
        "geo": "US",
        "rip": "193.74.6.52",
        "tag": [
            "http"
        ],
        "longitude": "-77.49",
        "infection": "http",
        "latitude": "39.03",
        "isp_name": "AMAZON TECHNOLOGIES INC.",
        "county_fips": "51107",
        "county_name": "LOUDOUN",
        "protocol": "tcp",
        "asn": "14618",
        "port": "8080",
        "sid": "4096627",
        "asn_name": "AMAZON-AES",
        "tld": "com",
        "city": "ASHBURN",
        "type": "scan",
        "ip": "52.6.74.193",
        "region": "VIRGINIA",
        "sector": "Communications",
        "naics": "454110",
        "timestamp": "2020-10-27 00:02:21",
        "domain": "crmapp005.anterasoftware.com"
    }
]

Example requiring a tag:

python3 call-api.py reports/query '{"date":"2020-10-27", "query":{"geo":"us", "require":"tag"}, "limit":1}' pretty
[
    {
        "rip": "229.156.120.34",
        "tag": [
            "http"
        ],
        "geo": "US",
        "county_fips": "29047",
        "isp_name": "GOOGLE INC.",
        "county_name": "CLAY",
        "latitude": "39.22",
        "infection": "http",
        "longitude": "-94.57",
        "asn_name": "GOOGLE",
        "sid": "4096627",
        "port": "8080",
        "asn": "15169",
        "protocol": "tcp",
        "city": "KANSAS CITY",
        "tld": "com",
        "region": "MISSOURI",
        "sector": "Defense Industrial Base",
        "ip": "34.120.156.229",
        "type": "scan",
        "naics": "519130",
        "timestamp": "2020-10-27 00:02:19",
        "domain": "229.156.120.34.bc.googleusercontent.com"
    }
]

Example of getting counts of different tags:

python3 call-api.py reports/query '{"date":"2020-10-27", "query":{"geo":"us", "require":"tag"}, "facet":"tag", "limit":9}' pretty

[
    {
        "count": 15203935,
        "tag": "ssl"
    },
    {
        "tag": "cwmp",
        "count": 11129661
    },
    {
        "tag": "ftp",
        "count": 2781725
    },
    {
        "tag": "http",
        "count": 2672067
    },
    {
        "tag": "ssl-poodle",
        "count": 1866075
    },
    {
        "count": 1519186,
        "tag": "isakmp"
    },
    {
        "tag": "quic",
        "count": 1212766
    },
    {
        "tag": "rdp",
        "count": 851989
    },
    {
        "count": 571483,
        "tag": "portmapper"
    }
]

Example getting tag counts over time. Note that since most of the data is repeated daily a count like this has little value:

python3 call-api.py reports/query '{"date":"2020-10-01:2020-10-28", "query":{"geo":"us", "require":"tag"}, "facet":"tag", "limit":9}' pretty

[
    {
        "tag": "ssl",
        "count": 322512588
    },
    {
        "count": 238483485,
        "tag": "cwmp"
    },
    {
        "tag": "ftp",
        "count": 57667266
    },
    {
        "count": 57308759,
        "tag": "http"
    },
    {
        "count": 39666402,
        "tag": "ssl-poodle"
    },
    {
        "tag": "isakmp",
        "count": 32107711
    },
    {
        "count": 30016842,
        "tag": "quic"
    },
    {
        "count": 16919098,
        "tag": "rdp"
    },
    {
        "tag": "portmapper",
        "count": 12755184
    }
]

reports/stats

An API option to allow looking through the history of the statistics of the different reports. Note that if the consumer is subscribed to more than one filter or report set, you will need to specify a filter using the report filter for that report set of statistics.

Fields:

apikey : string           : your api key

Optional Fields:

date   : string         : date or date range; default is the previous date
report : string or list : filter by report name
type   : string or list : filter by report type

Help

python3 call-api.py reports/stats '{"help":true}' pretty

[
    "date",
    "report",
    "type"
]

Returns

The date, type, and number of events for each report type as a two dimensional array.

Example:

python3 call-api.py reports/stats '{"type":"scan_exchange", "date":"2021-04-24:2021-04-26"}' pretty

[
    [
        "date",
        "type",
        "events"
    ],
    [
        "2021-04-24",
        "scan_exchange",
        2055
    ],
    [
        "2021-04-25",
        "scan_exchange",
        2034
    ],
    [
        "2021-04-26",
        "scan_exchange",
        2030
    ]
]

Different example of date filtering:

python3 call-api.py reports/stats '{"type":"scan_exchange", "date":"-3:now"}' pretty

[
    [
        "date",
        "type",
        "events"
    ],
    [
        "2021-04-24",
        "scan_exchange",
        2055
    ],
    [
        "2021-04-25",
        "scan_exchange",
        2034
    ],
    [
        "2021-04-26",
        "scan_exchange",
        2030
    ]
]

reports/device-info

Provides device details for a given IP.

Fields:

  • apikey : string : your api key
  • query : dictionary : search fields [ ip, asn, geo ]

The query must contain one attribute that matches your organization's report filters.

For example the CERT-LT who has a report filter for the country code 'LT' might request:

python3 call-api.py reports/device-info '{"query":{"geo":"lt","ip":"192.168.137.155"}}' pretty

Response:

{
   "device_type" : "router",
   "device_sector" : "consumer",
   "device_vendor" : "TP-Link"
}

reports/schema

Obtain JSON schema for a given report type.

Fields:

  • apikey : string : your api key
  • type : string : report type

Example:

python3 call-api.py reports/schema '{"type":"scan-snmp"}' pretty

{
   "protocol" : "string",
   "device_version" : "string",
   "naics" : "string",
   "ip" : "string",
   "hostname" : "string",
   "asn" : "string",
   "sic" : "string",
   "city" : "string",
   "device_type" : "string",
   "sysname" : "string",
   "timestamp" : "string",
   "device_model" : "string",
   "geo" : "string",
   "version" : "string",
   "sector" : "string",
   "region" : "string",
   "port" : "string",
   "device_sector" : "string",
   "sysdesc" : "string",
   "device_vendor" : "string"
}
Clone this wiki locally