Simbad.query_region returns None's when executed in for loop

[danielasmus]

Hello,

I am using python 2.7.11 with Ipython 3.2, astropy 2.0.6 and astroquery 0.3.9.dev0 (although when I install it, it say package astroquery-0.3.9.dev450.tar.gz) on a MAC OS 10.9.5, and I have the following issue:

I want to do a Simbad.query_region for a long list of objects, so I use a for loop to go over all the objects and do one query per object, and I test this for well-known objects and coordinates (nearby galaxies). Now, the queries return results for the first 10 objects in the loop but then only return "None" for all objects after. If I just execute the script again, now the queries are successful for the first 12 objects, and so on. Also, if I execute query_region commandys individually, it returns the object as expected.

Here is the code:

import numpy as np

from astroquery.simbad import Simbad
import astropy.coordinates as coord
import astropy.units as u


fin = "test.csv"

skiplines= 1
maxline = 10000000
radius = 6
rapos = 3

fi = open(fin, 'r')

i = 0

for line in fi:

    i = i + 1

    # --- skip the first line of the file because it contains the header
    if (i <= skiplines) | (i > maxline):
        continue

    # --- extract the coordinates
    temp = line.split(",")
    ra = temp[rapos]
    dec = temp[rapos+1]

    # --- transform coordinates into astropy format
    co = coord.SkyCoord(ra + " " + dec, frame='icrs', unit=(u.deg, u.deg))

    # --- do the simbad query around the coordinates
    result = Simbad.query_region(co, radius=radius*u.arcsec)

    print(i, temp[0], co.ra.value, co.dec.value, result)

fi.close

And the first 30 lines in the "test.csv":

Machine_Name,NED_Name,Alt_Name,RA2000,DEC2000
2MASXJ00004876-0709117,2MASX J00004876-0709117,,0.203208333333333,-7.15322222222222
2MASXJ00014596-7657144,FAIRALL 1203,Fairall 1203,0.441916666666667,-76.9539722222222
NGC7811,NGC 7811,,0.610291666666667,3.35191666666667
2MASXJ00032742+2739173,2MASX J00032742+2739173,,0.86425,27.65475
2MASXJ00040192+7019185,2MASX J00040192+7019185,,1.00816666666667,70.32175
Mrk0335,MRK 0335,,1.58133333333333,20.2029166666667
SDSSJ000911.57-003654.7,SPRC 001,,2.29825,-0.615194444444444
LEDA0001348,2MASX J00210753-1910056,,5.28141666666667,-19.1682222222222
LEDA0136991,2MASX J00253292+6821442,,6.38695833333333,68.3622777777778
LEDA0433346,2MASX J00264073-5309479,,6.66954166666667,-53.1632777777778
PG0026+129,PG 0026+129,PG0026+129,7.30708333333334,13.2677777777778
ESO112-G006,ESO 112- G 006,,7.682625,-59.0071944444444
2MASXJ00331831+6127433,2MASX J00331831+6127433,,8.326625,61.462
RHS03,2MASX J00341665-7905204,,8.56975,-79.0889722222222
2MASXJ00343284-0424117,2MASX J00343284-0424117,,8.63679166666666,-4.40330555555555
Z535-012,CGCG 535-012,,9.08741666666667,45.6650277777778
Mrk0344,MRK 0344,,9.633875,23.6133055555556
2MASXJ00423991+3017515,2MASX J00423991+3017515,,10.6662916666667,30.2976388888889
NGC0235A,NGC 0232 E,NGC 235A,10.7200416666667,-23.5410277777778
2MASXJ00430184+3017195,2MASX J00430184+3017195,,10.7577916666667,30.2887777777778
MCG-02-02-095,MCG -02-02-095,,10.7865416666667,-11.601
ESP39607,GALEXASC J004620.53-400548.7,,11.586,-40.0969722222222
Mrk0348,NGC 0262,Mrk 348,12.1964225,31.9569680555556
UGC00524,UGC 00524,,12.895875,29.40125
Mrk1148,MRK 1148,,12.9781666666667,17.4329166666667
2MASXJ00520383-2723488,2MASX J00520383-2723488,,13.0158333333333,-27.3968888888889
PG0052+251,PG 0052+251,PG0052+251,13.7171666666667,25.4275
Mrk0352,MRK 0352,,14.972,31.8269166666667
2MASXJ01003490-4752033,ESO 195-IG 021 NED03,,15.1456666666667,-47.8676388888889

I also attach both files as an archive:
Archive.zip

Any idea what could be the problem, and how I can get it working (I need to query for a few million objects in total, so I need a stable and reliable method).

Many thanks in advance for any help! Also my apologies if I missed some description/solution of this problem!

Best regards,

Daniel

[keflavich]

Look at Simbad.last_response when the result is None. In particular, you can print(Simbad.last_response.text) to see the exact result returned from SIMBAD. Since you're not receiving an error message, the query "succeeded" in some sense, but maybe they're returning junk?

[danielasmus]

Dear Adam,

many thanks for your suggestion. This is what I get:

In [6]: print(Simbad.last_response.text)

<title>403 Forbidden</title>

Forbidden

You don't have permission to access /simbad/sim-script on this server.

---

Apache/2.4.7 (Ubuntu) mod_jk/1.2.37 OpenSSL/1.0.1f Server at simbad.u-strasbg.fr Port 80

Which makes no sense to me? Any further ideas?

Cheers,

Daniel

[keflavich]

Uh, interesting. That does not make any sense to me. Can you access that URL via your browser? If not, raise a question with CDS.

[danielasmus]

OK, after talking with the people from CDS, I found the problem: The CDS server apparently only allows 6 queries per second. If there are more requests they are blocked. So, if I add a time delay between each query, it works without problem.

[bsipocz]

Actually the simbad module supports vectorized queries since #833, so what I would do is to factor out the astroquery call from your loop, and instead of creating single object SkyCoords build a vector one.