Skip to content

call_api Documentation

Jump to bottom
freed0 edited this page Sep 21, 2022 · 4 revisions

Last Updated: 2021-10-25

Introduction

There are two types of API’s that are available. The first are those that do not require any authentication and are free to use in any non-commercial product without a license but is limited to ten queries per second from any IP address.

The second type of API has both an API key as well as a secret key that is used to access the data that the API services. The maximum limit for a query is 10,000 results. The limit is due to how the underlying search engine (OpenSearch) is implemented.

There are details about each API separately to include which type each one is. This document goes through some of the basic testing API’s that are available for those API’s that require keys and secrets.

Programs

Here are two programs, one in Python and one in Perl to help with the execution and accessing the data through the API's:

For either of these programs to function, they expect a file called ~/.shadowserver.api to exist and to contain your API key and secret. As an example:

[api]
key = <<API-KEY>>
secret = <<SECRET>>
uri = https://transform.shadowserver.org/api2/

If an error like this occurs, it means that the query is going through a lot of data and the timeout in the program should be increased:

freed0@pit:~$ ./call-api.py reports/query '{"report":"united-states", \
                                            "date":"2020-10-27", \
                                            "query":{"city":"ashburn"}, "limit":3}' pretty
API Exception: The read operation timed out

On all the other API documentation pages, all examples will be used using these programs and not curl. Curl is only used on this page as an example of an alternative way of accessing the API's.

API Keys

The API requires that the client send their apikey as well as a hash based message authentication code (HMAC) of the request data created with a secret given to each client. Recalculating and comparing the hash allows the server to authenticate the user and to ensure that the request has not been tampered with.

API Endpoints

RESTful

The API is mostly RESTful with the following stipulations:

  • API calls must be made with HTTP POST.
  • All requests are JSON objects.
  • Each request requires an API key and HMAC header.
  • Any non-200 HTTP response is an error. Error details will be included in the response object.

Here is an example call using curl:

$ echo -n '{ "apikey": "0123456789" }' > req
$ hmac=`openssl sha256 -hmac 'secret' -hex < req | awk '{print $2}'`
$ curl -s -X POST -H HMAC2:$hmac --data-binary @req  https://transform.shadowserver.org/api2/test/ping
{ "pong": "2014-08-12 23:14:20" }

And an example using the call-api program:

freed0@pit:~$ ./call-api.py test/ping {} pretty
{
    "pong": "2020-10-28 21:38:49"
}

Note that in the curl example above, for each query you will need to regenerate the HMAC code. Al the curl examples have been made to look pretty when they will actually be transferred via a single line of JSON. Only because there is a pretty mode in the call-api code is it auto formatted to be human readable.

Modules

Test Methods

test/ping

Check your connection to the API server.

Fields:

apikey : string : your api key

Response:

pong : string : current timestamp in UTC

Example:

freed0@pit:~$ ./call-api.py test/ping {} pretty
{
    "pong": "2020-10-28 21:38:49"
}

Key Methods

key/info

Returns details about your apikey.

Fields:

apikey : string : your api key

Response:

user        : string : user name 
label       : string : key label 
domain      : string : request source domain, IP, or CIDR 
access      : list : list of access group names 
status      : string : key status 
query_count : int : current query count 
query_limit : int : maximum number of queries per month 
expiration  : string : expiration date (YYYY-MM-DD [HH:MM:SS] in UTC)`

Example:

freed0@pit:~$ ./call-api.py key/info {} pretty
[
    {
        "apikey": "<<API-KEY>>",
        "query_limit": "0",
        "label": "The Shadowserver Foundation",
        "user": "[email protected]",
        "access": [
            "reports"
        ],
        "status": "active",
        "query_count": "0"
    }
]

Help Features

Most of the API's have a help feature that will list out all the possible fields that are available to use in a query for that API. Eventually detailed descriptions will be added to the help over time.

Example:

lich@pit:~$ ./call-api.py reports/query '{ "help":true }' pretty
[
    "agent",
    "application",
    "asn",
    "asn_name",
    "banner",
    "city",
    "county_fips",
    "county_name",
    ...
    "source_url",
    "tag",
    "text",
    "timestamp",
    "tld",
    "type",
    "version"
]

CHANGELOG

  • 2021-10-25 - Added information on query limits
  • 2021-03-11 - Added in example to show the help pages for the API fields.
  • 2021-03-11 - Updated Perl and Python scripts to allow downloading of binary files
Clone this wiki locally