Skip to content

call_api Documentation

Jump to bottom
elsif2 edited this page Feb 24, 2023 · 4 revisions

Introduction

There are two types of API’s that are available. The first are those that do not require any authentication and are free to use in any non-commercial product without a license but is limited to ten queries per second from any IP address.

The second type of API has both an API key as well as a secret key that is used to access the data that the API services. The maximum limit for a query is 10,000 results. The limit is due to how the underlying search engine (OpenSearch) is implemented.

There are details about each API separately to include which type each one is. This document goes through some of the basic testing API’s that are available for those API’s that require keys and secrets.

Programs

Here are two programs, one in Python and one in Perl to help with the execution and accessing the data through the API's:

For either of these programs to function, they expect a file called ~/.shadowserver.api to exist and to contain your API key and secret. As an example:

[api]
key = <<API-KEY>>
secret = <<SECRET>>
uri = https://transform.shadowserver.org/api2/

If an error like this occurs, it means that the query is going through a lot of data and the timeout in the program should be increased:

$ ./call-api.py reports/query '{"report":"united-states", \
                                            "date":"2020-10-27", \
                                            "query":{"city":"ashburn"}, "limit":3}' pretty
API Exception: The read operation timed out

On all the other API documentation pages, all examples will be used using these programs and not curl. Curl is only used on this page as an example of an alternative way of accessing the API's.

API Keys

The API requires that the client send their apikey as well as a hash based message authentication code (HMAC) of the request data created with a secret given to each client. Recalculating and comparing the hash allows the server to authenticate the user and to ensure that the request has not been tampered with.

API Endpoints

RESTful

The API is mostly RESTful with the following stipulations:

  • API calls must be made with HTTP POST.
  • All requests are JSON objects.
  • Each request requires an API key and HMAC header.
  • Any non-200 HTTP response is an error. Error details will be included in the response object.

Here is an example call using curl:

$ echo -n '{ "apikey": "0123456789" }' > req
$ hmac=`openssl sha256 -hmac 'secret' -hex < req | awk '{print $2}'`
$ curl -s -X POST -H HMAC2:$hmac --data-binary @req  https://transform.shadowserver.org/api2/test/ping
{ "pong": "2014-08-12 23:14:20" }

And an example using the call-api program:

$ ./call-api.py test/ping {} pretty
{
    "pong": "2020-10-28 21:38:49"
}

Note that in the curl example above, for each query you will need to regenerate the HMAC code. Al the curl examples have been made to look pretty when they will actually be transferred via a single line of JSON. Only because there is a pretty mode in the call-api code is it auto formatted to be human readable.

Modules

Test Methods

test/ping

Check your connection to the API server.

Fields:

apikey : string : your api key

Response:

pong : string : current timestamp in UTC

Example:

$ ./call-api.py test/ping {} pretty
{
    "pong": "2020-10-28 21:38:49"
}

Key Methods

key/info

Returns details about your apikey.

Fields:

apikey : string : your api key

Response:

user        : string : user name 
label       : string : key label 
domain      : string : request source domain, IP, or CIDR 
access      : list : list of access group names 
status      : string : key status 
query_count : int : current query count 
query_limit : int : maximum number of queries per month 
expiration  : string : expiration date (YYYY-MM-DD [HH:MM:SS] in UTC)`

Example:

$ ./call-api.py key/info {} pretty
[
    {
        "apikey": "<<API-KEY>>",
        "query_limit": "0",
        "label": "The Shadowserver Foundation",
        "user": "[email protected]",
        "access": [
            "reports"
        ],
        "status": "active",
        "query_count": "0"
    }
]

Logic

AND

Each query argument is combined with AND by default.

This will return results that match both asn:64512 and geo:US:

{ "query": { "asn": 64512, "geo": "US" } }

Example:

$ ./call-api.py reports/query '{"query":{"asn":64512,"geo":"US","date":"2023-02-14"},"limit":1}' pretty
[
    {
        "timestamp": "2023-02-14 01:41:51Z",
        "ip": "192.168.1.1",
        "protocol": "tcp",
        "port": "443",
        "asn": "64512",
        "geo": "US",
        "type": "scan",
        "infection": "ssl",
        "tag": [
            "http-favicon"
        ]
    }
]

OR

The pipe (|) operator is used in place of OR. This will return results that match type:sinkhole or type:honeypot:

{ "query": { "type": [ "|sinkhole", "|honeypot" ] } }

Example:

$ ./call-api.py reports/query '{"query":{"geo":"US", "type":["|sinkhole","|honeypot"],"date":"2023-02-14"},"limit":1}' pretty
[
    {
        "timestamp": "2023-02-14 00:02:24Z"
        "ip": "192.168.1.2",
        "protocol": "tcp",
        "port": "1579",
        "asn": "64512",
        "geo": "US",
        "type": "honeypot",
        "infection": "masterblaster",
        "tag": [
            "masterblaster"
        ]
    }
]

NOT

The exclamation (!) operator is used in place of NOT. This will return results that match asn:64512 and not geo:US:

{ "query": { "asn": 64512, "geo": "!US" } }

Wildcards

The asterix (*) operator can match zero or more characters, including an empty one.

This would match dhcp or dhcpd or dhcpdiscover:

{ "query": { "tag": "dhcp*" } }

Example:

$ ./call-api.py reports/query '{"query":{"geo":"US", "tag":"dhcp*","date":"2023-02-14"},"limit":1}' pretty
[
    {
        "timestamp": "2023-02-14 01:47:31Z",
        "ip": "192.168.1.3",
        "protocol": "udp",
        "asn": "64512",
        "geo": "US",  
        "type": "honeypot",
        "infection": "ddos-amplification",
        "tag": [
            "dhcpdiscover"
        ]
    }
]

The question mark (?) operator matches any single character.

This would match smtp or snmp:

{ "query": { "tag": "s??p" } }

Example:

$ ./call-api.py reports/query '{"query":{"geo":"US", "tag":"s??p","date":"2023-02-14"},"limit":1}' pretty
[
    {
        "timestamp": "2023-02-14 00:14:17Z",
        "ip": "192.168.1.4",
        "protocol": "tcp",
        "port": "25",
        "asn": "64512",
        "geo": "US",
        "type": "scan",
        "infection": "smtp",
        "tag": [
            "smtp"
        ]
    }
]

Regular expressions

Only fields specifically noted in the API documentation support regular expressions.

A regular expression is bracketed between percent (%) characters:

{ "query": { "tag": "%(snmp|smtp)%" } }

The regular expression must match the entire string.

The following operators are reserved and need to be escaped with a preceding backslash (\):

. ? + * | { } [ ] ( ) " \

Operators

  • The period (.) operator matches any character.
  • The question mark (?) operator matches any single character.
  • The plus (+) operator matches the preceding character one or more times.
  • The asterix (*) operator matches the preceding character zero or more times.
  • The curly brackets ({}) define the minimum and maximum time the preceding character can repeat.
  • The pipe (|) operator will match the longest pattern on either the left side OR the right side.
  • The parenthesis ( ... ) form a group to treat part of an expression as a single character.
  • The brackets [ ... ] define a character set. Inside brackets the - indicates a range unless the - is the first character or escaped. A ^ character in the bracket negates the character or range.

Help Features

Most of the API's have a help feature that will list out all the possible fields that are available to use in a query for that API. Eventually detailed descriptions will be added to the help over time.

Example:

$ ./call-api.py reports/query '{ "help":true }' pretty
[
    "agent",
    "application",
    "asn",
    "asn_name",
    "banner",
    "city",
    "county_fips",
    "county_name",
    ...
    "source_url",
    "tag",
    "text",
    "timestamp",
    "tld",
    "type",
    "version"
]
Clone this wiki locally