Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SQL-281] CSV export #464

Open
wants to merge 51 commits into
base: main
Choose a base branch
from
Open

[SQL-281] CSV export #464

wants to merge 51 commits into from

Conversation

kelvinqian00
Copy link
Collaborator

@kelvinqian00 kelvinqian00 commented Feb 11, 2025 •
edited
Loading

Implement the exporting of all xAPI statements as CSV data. Both the CSV headers and the query params (based off the params for regular statement query) should be specified as parameters of the /admin/csv GET endpoint. Also adds a /admin/csv/auth endpoint to return a one-time JWT (since using /admin/csv as a download HTML anchor tag attribute means we cannot apply headers).

Adds a new LRSQL_STMT_GET_MAX_CSV to apply a limit to how many statements can be included in the CSV file.

TODO (for future tickets/PRs):

@kelvinqian00
Extract out query-many-statements helper function
a2c3ea1
@kelvinqian00
Additional refactors to statement query
6659d85
@kelvinqian00
Move path-related util functions to their own ns
e2c6079
@kelvinqian00
Make changes to path->jsonpath functions
92471f6
@kelvinqian00
Add required deps
994da69
@kelvinqian00
Add query-all-statements function
9fe638a
@kelvinqian00
Add statements to CSV util functions
247b66c
@kelvinqian00
Implement get-statements-csv protocol function
d60579a
@kelvinqian00
Remove unused require
df178d6
@kelvinqian00
Only put quotes in SQL JSON path strings when needed
40530f4
@kelvinqian00
Change some arg names
8d06741
@kelvinqian00
Implement CSV download interceptor and endpoint
1c742b3
@kelvinqian00
Disable JWT validation for /admin/csv endpoint
7ec555a
@kelvinqian00
Use next.jdbc/plan and test queries beyond :limit
5fce0e3
@kelvinqian00
Delete now-unused requires
131e2ef
@kelvinqian00
Add docstrings, specs, and refactors to query fn
ae04282
@kelvinqian00
Add with-open clause to interceptor
d311b07
@kelvinqian00
Make :limit an optional param so it can be dissoc'd
e766c21
@kelvinqian00
Remove stray semicolon from PG query
8ea1204
@kelvinqian00
Reinstate transaction in get-statements-csv
27bb811
@kelvinqian00
Extract bench inputs to support namespace
379933a
@kelvinqian00
Add tests for inserting and querying large amounts of data
0bb7191
@kelvinqian00
Move encoding functions to util namespace
83e7504
@kelvinqian00
Refactor query-statements-stream a bit
ce400e1
@kelvinqian00 kelvinqian00 marked this pull request as ready for review February 18, 2025 17:15
kelvinqian00 and others added 15 commits March 5, 2025 15:23
@kelvinqian00 kelvinqian00 mentioned this pull request Mar 10, 2025
Open
milt
milt approved these changes Mar 11, 2025
View reviewed changes
Copy link
Member

@milt milt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This works great, and I like the impl. Approved, pending the decision on where it goes!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@milt milt milt approved these changes

@invaliduser invaliduser Awaiting requested review from invaliduser

@cliffcaseyyet cliffcaseyyet Awaiting requested review from cliffcaseyyet

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@kelvinqian00 @milt