Exceeded a secondary rate limit. Please wait a few minutes before you try again.

[alkatrazz1993]

Select Topic Area

Question

Body

I collect metrics by number of (contributors, commits, releases, pullrequests, branches, issues, stargazers, subscribers, forks) by projects.

I'm making only GET requests to the github API with a token (5000 requests per hour).

All requests is unique by projects.

I'm doing multi-threaded only GET requests (simultaneous) in PHP with curl_multi - 10 simultaneous GET requests with an interval between each of 0.2 seconds.

Example my code with curl:

// Total of count unique urls - 870. They are broken into packs. Each packet contains unique 10 pieces urls.

// CURL OPTIONS for each url

$mh = curl_multi_init();

$curl_array = [];

// $urls - unique 10 pieces urls
foreach($urls as $i => $url) {
    $curl_array[$i] = curl_init($url);
    
    curl_setopt($curl_array[$i], CURLOPT_TIMEOUT , 60);
    curl_setopt($curl_array[$i], CURLOPT_CONNECTTIMEOUT, 60);
    curl_setopt($curl_array[$i], CURLOPT_FOLLOWLOCATION, 1);
    curl_setopt($curl_array[$i], CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($curl_array[$i], CURLOPT_SSL_VERIFYPEER, 0);
    curl_setopt($curl_array[$i], CURLOPT_CUSTOMREQUEST, 'GET');

    curl_setopt($handler, CURLOPT_USERAGENT, "alkatrazz1993"); //my github username
    curl_setopt($handler, CURLOPT_HTTPHEADER, [
         'Accept: application/vnd.github+json',
         'Authorization: Bearer ' . API_ACCESS_TOKEN,
    ]);

    curl_multi_add_handle($mh, $curl_array[$i]);
}

$running = NULL;
do {
      usleep(200000); // 0.2 seconds
      curl_multi_exec($mh, $running);
} while($running > 0);

Example 10 simultaneous GET requests with an interval between each of 0.2 seconds. They are always unique by projects:

1. https://api.github.com/repos/stakwork/sphinx-keysend-test/pulls?per_page=1
2. https://api.github.com/repos/nayutaco/ptarmigan/releases?per_page=1
3. https://api.github.com/repos/Blockstream/green_cli/releases?per_page=1
4. https://api.github.com/repos/KxSystems/javakdb/contributors?per_page=1&anon=true
5. https://api.github.com/repos/Blockstream/gdk/commits?per_page=1
6. https://api.github.com/repos/ZmnSCPxj/clboss/pulls?per_page=1
7. https://api.github.com/repos/TheRemote/RaspberryPiMinecraft/pulls?per_page=1
8. https://api.github.com/repos/lightninglabs/lightning-api/commits?per_page=1
9. https://api.github.com/repos/asoltys/coinos.io/contributors?per_page=1&anon=true
10. https://api.github.com/repos/ElementsProject/lightning-jukebox/releases?per_page=1

In total, I have 870 GET requests (unique url by project in the api that I access).

At the first processing of 870 GET requests, with the conditions described above, I receive the data successfully. After 5 minutes I start re-processing 870 GET requests and sometimes I get a 403 error in responce "You have exceeded a secondary rate limit. Please wait a few minutes before you try again." on multiple urls with header retry-after: 60 (in seconds).

This issue occurs when we access the github api on the server hosting provider GoDaddy (https://godaddy.com). When I access the github api locally on my computer, there is no problem of secondary restrictions never. We have no control over GoDaddy's server settings.

Please tell me, what interval of seconds it is allowed to make simultaneous unique GET requests, and how many simultaneous unique GET requests can be made so that this error does not occur?

[byrneh]

@alkatrazz1993 the published gudiance is #dealing-with-secondary-rate-limits.

[alkatrazz1993]

Thanks, yes I have read this guide and other github api guides on limits. But there is no answer to my question, so I asked it here.

[alkatrazz1993]

graphql helped me.

[SerenaIce]

Hi @alkatrazz1993,

I also encountered this error, but I've already used graphql. Could you please provide how did you solve that?