Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sources that support returnIdsOnly but not returnCountOnly #34

Open
albarrentine opened this issue Feb 23, 2017 · 1 comment
Open

Sources that support returnIdsOnly but not returnCountOnly #34

albarrentine opened this issue Feb 23, 2017 · 1 comment

Comments

@albarrentine
Copy link
Contributor

For the Wuhan, China data source (seem to be getting all the legacy servers lately), pyesridump was only pulling 1001 records when there were 3429 in the full data set.

From this line:

except EsriDownloadError:
, it looks like if the source doesn't support returnCountOnly, the bounding box is recursively subdivided into four quadrants (Quadtree-style) with a stopping condition when there are < maxRecords in a given quadrant.

  1. This should generally retrieve everything, except for the following test in _scrape_an_envelope:
    if len(features) == max_records:
    It appears the Wuhan source returns 1001 records where max_records is 1000, which executes the same code as if it had returned 999 results i.e. assumes the base case has been met and returns early. This could be fixed by changing the conditional to:
    if len(features) >= max_records
  2. With the new OID enumeration from Faster method for object ID enumeration for sources that do not support pagination #33, it might make sense to use the quadrant-based method as a fallback only if the source supports neither returnCountOnly nor returnIdsOnly. Otherwise OID enumeration should be fewer queries. Does that make sense or are there some other edge cases to consider?
@iandees
Copy link
Member

iandees commented Feb 28, 2017

@thatdatabaseguy can you make a PR for the change you mention in (1) above? That seems like a useful thing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants