Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CMR-10296: Adding arrayLimit to qs parser to increase the array limit without indices #389

Merged
merged 4 commits into from
Feb 5, 2025

Conversation

william-valencia
Copy link
Contributor

Overview

What is the feature?

Currently, the default array limit of the qs parser in express is set to 20. When arrays with more than 20 items are passed to CMR STAC as query parameters(for coodinates of an intersects parameter), array indices are added. The CMR STAC code cannot handle the array indices so it errors out.

What is the Solution?

Increasing the array limit of the default qs parser in express to 100. This will allow coordinates of up to 100 points.

What areas of the application does this impact?

CMR STAC

Testing

Reproduction steps

You can try the following on a browser locally or in PROD with the proper URL:
https://cmr.earthdata.nasa.gov/stac/LPCLOUD/search?datetime=2024-01-01T00:00:00Z/2024-01-31T23:59:59Z&collections[0]=HLSS30_2.0&intersects[type]=Polygon&intersects[coordinates][0][0][0]=-48.31876570764564&intersects[coordinates][0][0][1]=-24.13235588091331&intersects[coordinates][0][1][0]=-48.319182157887525&intersects[coordinates][0][1][1]=-24.135649639167937&intersects[coordinates][0][2][0]=-48.2842003375668&intersects[coordinates][0][2][1]=-24.159843492656776&intersects[coordinates][0][3][0]=-48.26087912402002&intersects[coordinates][0][3][1]=-24.122474097202897&intersects[coordinates][0][4][0]=-48.259074506305325&intersects[coordinates][0][4][1]=-24.110310854137083&intersects[coordinates][0][5][0]=-48.268652861869185&intersects[coordinates][0][5][1]=-24.095231896596758&intersects[coordinates][0][6][0]=-48.273511448024976&intersects[coordinates][0][6][1]=-24.09662582428146&intersects[coordinates][0][7][0]=-48.2842003375668&intersects[coordinates][0][7][1]=-24.086361092375142&intersects[coordinates][0][8][0]=-48.28600495528215&intersects[coordinates][0][8][1]=-24.081038315106454&intersects[coordinates][0][9][0]=-48.28142400262118&intersects[coordinates][0][9][1]=-24.076095538322363&intersects[coordinates][0][10][0]=-48.28364507057779&intersects[coordinates][0][10][1]=-24.070645589226345&intersects[coordinates][0][11][0]=-48.29280697589971&intersects[coordinates][0][11][1]=-24.079390744017203&intersects[coordinates][0][12][0]=-48.29655502807688&intersects[coordinates][0][12][1]=-24.07977095461004&intersects[coordinates][0][13][0]=-48.295583310845984&intersects[coordinates][0][13][1]=-24.08433339376434&intersects[coordinates][0][14][0]=-48.30169124772749&intersects[coordinates][0][14][1]=-24.08293933236135&intersects[coordinates][0][15][0]=-48.3040511324312&intersects[coordinates][0][15][1]=-24.0845868578446&intersects[coordinates][0][16][0]=-48.298914912781214&intersects[coordinates][0][16][1]=-24.08876894281238&intersects[coordinates][0][17][0]=-48.30058071374884&intersects[coordinates][0][17][1]=-24.089909487754852&intersects[coordinates][0][18][0]=-48.30641101713553&intersects[coordinates][0][18][1]=-24.086614552446946&intersects[coordinates][0][19][0]=-48.31140842003842&intersects[coordinates][0][19][1]=-24.096879264048766&intersects[coordinates][0][20][0]=-48.310020252565636&intersects[coordinates][0][20][1]=-24.10068080039386&intersects[coordinates][0][21][0]=-48.31876570764564&intersects[coordinates][0][21][1]=-24.13235588091331

Make sure the request doesn't error out with:
{"errors":["Oops! Something has gone wrong. We have been alerted and are working to resolve the problem. Please try your request again later."]}

Attachments

Please include relevant screenshots or files that would be helpful in reviewing and verifying this change.

Checklist

  • I have added automated tests that prove my fix is effective or that my feature works
  • [x ] New and existing unit tests pass locally with my changes
  • [x ] I have performed a self-review of my own code
  • [x ] I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • [x ] My changes generate no new warnings

@codecov-commenter
Copy link

codecov-commenter commented Feb 4, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 88.42%. Comparing base (568bb2c) to head (d7570da).

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #389      +/-   ##
==========================================
+ Coverage   88.39%   88.42%   +0.02%     
==========================================
  Files          24       24              
  Lines        1215     1218       +3     
  Branches      269      269              
==========================================
+ Hits         1074     1077       +3     
  Misses        141      141              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

src/app.ts Outdated
// Anything over 100 would error out because indices are added. See CMR-10296 and
// https://github.com/ljharb/qs for more details.
app.set("query parser", function (str: string) {
return qs.parse(str, { arrayLimit: 10000 });
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Array limit is 10,000 but, comment is 100?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry about that. Was testing 10000.

@william-valencia william-valencia merged commit 3d5d1bb into master Feb 5, 2025
5 checks passed
@william-valencia william-valencia deleted the CMR-10296 branch February 5, 2025 20:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants