Google AJAX Image Search: "This API is no longer available"

[seriallos]

Looks like Google may have finally killed the API:

$ curl https://ajax.googleapis.com/ajax/services/search/images\?v\=1.0\&q\=testing
{"responseData": null, "responseDetails": "This API is no longer available.", "responseStatus": 403}%

[JasonMore]

Yeah they turned it off. The new api has 1000 req/day limit too :-( 👎

[wjr1985]

😢

[HaroldPutman]

So maybe we fallback to random cat picture http://lorempixel.com/200/200/cats/?

[jeffbyrnes]

Super sad.

[onlydole]

Are there any other services that could be used instead?

[afeld]

Some options:

• https://developers.google.com/custom-search/docs/overview
• https://datamarket.azure.com/dataset/bing/search
  • This is promising and I got it to work, but the docs are a bit all over the place.
  • Obtaining an Access Token
• http://www.programmableweb.com/category/images%2Bsearch/apis?category=19979%2C20055

[darrinholst]

I tried setting up CSE and failed miserably. Do you really have to specify the domains you want to search? This is a sad, sad day.

[seriallos]

Once you've created the custom search engine with one random domain in it, you can edit and choose "Search the entire web" (or something named like that) in the Basic tab. You can't do it when creating the CSE though.

[darrinholst]

oh nice, any tips on finding the key? I found the id.

[seriallos]

Google API Console. Might have to set up a new project and enable the CSE API.

https://code.google.com/apis/console

[darrinholst]

found it by stumbling around in there. Thinking about starting a side business to set this up for people 😵

[AngelMarquez]

Hey guys:

• Setting up CSE is detailed on this page in under 'CSE Setup Details' - https://github.com/hubot-scripts/hubot-google-images/
• Bing Search API allows 5,000 image searches per month. Perhaps we can modify the script to use the 100 free google searches each day then switch over to Bing after those are exhausted?

I know it's not perfect, but that's roughly 261 searches per day ... anyone game?

[sadatay]

I started a thread over in the corresponding lita plugin and I've been thinking of doing something similar to @AngelMarquez's suggestion, perhaps also with yahoo support. Basically start with google, fallback to bing, then fallback to yahoo which goes for a rate of $1.80 per 1000 requests (or $0.20 with advertisements). There is also pixabay which provides an unlimited (though rate limited to 2k requests per 30 minutes) image search api - although it's only public domain images. So for example things like "flower" have suitable results but "there's something about mary" has no results. So maybe first hit pixabay to filter out things that might be relatively common before going to the more expensive APIs for broader results.

My coffeescript game is weak so I dunno how much help I can be on the hubot side, but I'm "game" in the sense that I'm game to pursue this idea. It might be smart to put a little thought into the architecture so that we can sort of frame the individual apis as components, so maybe other apis can be easily mixed in or out at will. A generalized "image fetcher" plugin, i guess, with as many sources as people want to implement. There are certainly many possible configuration options when we start talking about an amalgamation of sources like this.

For what it's worth, I also explored scraping a bit today. It definitely violates Google's terms of service, so this wasn't something I planned to release publicly on GitHub, but if anybody is considering handrolling their own image scraping plugin for their chatbots, at least with ruby (for lita) I was not able to find anything suitable. Best I could get were the thumbnails - the whole "click an image to get options" thing means the actual sources of the images aren't really accessible on the page (as far as I can tell). And rigging up some jankiness with, say, phantomjs to go through clicking all the links and getting the sources would be prohibitively slow.

I really want this functionality, I'm sure many can attest that our team chats are less lively without it. I'm pretty sure what you're suggesting here is, for now, the most reasonable course of action.

[wjr1985]

I like the idea of Google first, falling back to Bing, but I can't seem to find an option to request animated images through Bing's image API. Anyone else have any luck? I tried things like Style:Animated, Style:AnimatedGIF, and Style:GIF set in ImageFilters, but I would get no results.

[HaroldPutman]

The same issue exists for Google CSE (see issue #13). We work around it by setting file type to GIF and adding animated to the query string. Not perfect, but better than nothing.

[nzifnab]

@AngelMarquez I'm trying to follow the README's instructions for setting up a CSE for use w/ HUBOT, but it says I need a CSE Key by going here: https://code.google.com/apis/console

But that URL redirects me to a google developer's console landing page with no clear direction of where to get that key from... I got the CSE ID, but not sure what key I'm supposed to use :/

EDIT: I figured it out, but the instructions could be clearer. You need to generate an API key in that console, and then go to the "Custom search API" link from the console page and "Enable" that API.

[AngelMarquez]

I have modified this script https://github.com/github/hubot-scripts/blob/master/src/scripts/bing-images.coffee a tiny bit and have it working by itself now. When I get some time (hopefully this weekend) i'll try to merge it into this script. The hard part will be cutting over after free searches are exhausted, then knowing when to switch back.

[wjr1985]

I gave integrating Bing as backup a shot as well, if anyone's interested: hubot-scripts/hubot-google-images@master...wjr1985:bing_fallback

It doesn't stop sending requests to Google when the quota has been exceeded, but it does fall back properly to Bing. I'm sure it could use some cleanup and an updated Readme, but it's working right now in my Hubot integration with Slack

[AngelMarquez]

@wjr1985 I love it!

I think we can just take your code and tweak it by adding a new skipGoogle var. When we have to resort to Bing just set skipGoogle to true and make a call tosetTimeout() to switch skipGoogle back to false in an hour. Not perfect, but it makes it so I'm not hammering Google and speeds up the searches overall.

[HaroldPutman]

@technicalpickles, @louim, @seriallos and other contributors... I'm interested in your thoughts about the Bing fallback. Since this is a "ship-with" command, is there any impetus to keep it simple? Should we spin off a more robust hubot-images command that could draw from a wide range of sources? Is there some more modern "middleware" way to keep google-images, bing-images, and potentially other *-images as separate commands and have hubot "load balance" between a selection of options?

[technicalpickles]

is there any impetus to keep it simple?

Simplicity should be kept in mind for default scripts, for sure.

Should we spin off a more robust hubot-images command that could draw from a wide range of sources?

That makes sense to me.

Is there some more modern "middleware" way to keep google-images, bing-images, and potentially other *-images as separate commands and have hubot "load balance" between a selection of options?

I'm not sure middleware will help here, as it applies to everything. It'd be possible for individual scripts to add a function like robot.googleImageSearch, and let hubot-images or whatever check for those to be present before using them.

[shana]

http://api.imgur.com/ might also be an option. There's a gallery search endpoint (http://api.imgur.com/endpoints/gallery#gallery-search). They have rate limits (12500 a day), and there's the possibility of having our app whitelisted if we ask nice.

[JasonMore]

@wjr1985 I maintain a fork of this code for another project, here is how I got gif working in bing [spoiler alert, its gross]

    if(animated) {
        // arg why?! no good api unfortunately
        qs.Query = `'${query} ".gif"'`;
    }

basically tacking a ".gif" to the end of my query

ex: pikachu ".gif"

[afeld]

Found another option for GIF search: http://api.riffsy.com/

[josephks]

@nzifnab How did you get the CSE Key? I still can't figure it out. I've generated a server key but what field from that JSON get stuck into HUBOT_GOOGLE_CSE_KEY ?

[nzifnab]

@josephks

First, make sure you created a project and generated an API key for it:

https://console.developers.google.com/apis/credentials

That might get you to the right page

After you have your project, you should be able to generate the server key (HUBOT_GOOGLE_CSE_KEY)

next, go here: https://console.developers.google.com/apis/library

You should see a big list of API links, in the search box type "Custom Search",

There should be one result for "Custom Search API", go into it and click "Enable"

I think when you define your custom search engine, it might give you the ID at that time. See this article: https://developers.google.com/custom-search/json-api/v1/overview?hl=en_US for info about creating your custom search engine, it does look like you'll get the ID once you've done that.

EDIT: You can create your custom search engine here: https://cse.google.com/manage/all

Make sure you let it "search the entire web but emphasize included sites" (and just leave included sites blank), and make sure "image search" is turned on.

There's a search engine ID button there that'll give you the ID to use in hubot when you're done.

[josephks]

When I generate the server key I get a json file. Am I supposed to stick all of that json in HUBOT_GOOGLE_CSE_KEY ?

[nzifnab]

er... no? What're you doing to get a JSON file?

[nzifnab]

If you go here: https://console.developers.google.com/apis/credentials

you should create a new project, then generate a "Server Key" and it should give you an API key that's just a string of characters, that's the HUBOT_GOOGLE_CSE_KEY

[josephks]

I see a button that says "Create credentials". After I click on that what do I click on?

[nzifnab]

"API Key", and then "Server Key"

[josephks]

Image search still isn't working :(

[nzifnab]

And you created a custom search engine and used the ID it provided for HUBOT_GOOGLE_CSE_ID?

[josephks]

Yes.

[murrant]

Is there a way to enable Custom Search API without putting in billing information?

[moktarul12]

try this

[moktarul12]

free google unofficial image search api. http://api.ababeen.com/
Demo: http://api.ababeen.com/api/demo.php

[theapache64]

I've developed an API named 'GPix' that can act as a API for the google images. Take a look at https://github.com/theapache64/GPix and if there's any error report at [email protected] . It's fresh and hot.!! enjoy! :)

[EugeneKovalev]

@theapache64 OMG That's awesome! Thank you! :)

[theapache64]

@cyberthrone3 You're most welcome. :)

[saggiyogesh]

@theapache64
Is there any daily quota for search ?

[moktarul12]

Try api.ababeen.com

…

On Nov 25, 2016 11:41 AM, "Yogesh Agrawal" ***@***.***> wrote: @theapache64 <https://github.com/theapache64> Is there any daily quota for search ? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://github.com/hubot-scripts/hubot-google-images/issues/29#issuecomment-262887357>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABOZW7ZPTXMmIKQ7yzAAqHCV69Ewfw7Kks5rBnv3gaJpZM4GtVbE> .

[theapache64]

@saggiyogesh It's UNLIMITED. All you want to have an api_key. :)

[theapache64]

I've developed an unofficial API for Google images. It's faster and simple. https://github.com/theapache64/GPix

[EugeneKovalev]

@theapache64 hello again! I'd like to tell you that i get the same new brand API key every day :) hope you can fix this

[theapache64]

@cyberthrone3 It's not a bug. I changed the algorithm to [one email]-[one-API-key] model. You can use your API key for multiple applications. There's no difference.

[theapache64]

@seriallos , @JasonMore , @wjr1985 , @HaroldPutman , @jeffbyrnes , @onlydole , @afeld , @darrinholst , @AngelMarquez , @sadatay , @nzifnab , @technicalpickles , @shana , @josephks , @murrant , @moktarul12 , @theapache64 , @cyberthrone3 , @saggiyogesh I've developed an unofficial API for Google images. It's faster and simple. https://github.com/theapache64/GPix . Please take a look and report if there's any issue. It's hot and fresh.. Enjoy. :)