Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dataset statistics #5768

Draft
wants to merge 41 commits into
base: main
Choose a base branch
from
Draft

Dataset statistics #5768

wants to merge 41 commits into from

Conversation

YohannParis
Copy link
Member

@YohannParis YohannParis commented Dec 5, 2024 •
edited
Loading

Description

  • Add a pandas stat task from a Dataset
  • Updated DatasetColumn with a new property stats with proper object representation
  • Add a new Service DatasetStatistics to handle the task and update the dataset accordingly
  • Call the new service in two places, on upload of Dataset from the front-end and when adding a new file

@YohannParis YohannParis requested a review from kbirk December 5, 2024 22:00
@YohannParis YohannParis linked an issue Dec 5, 2024 that may be closed by this pull request
[TASK]: Get Dataset stats #5754
Open
YohannParis and others added 26 commits December 5, 2024 16:12
@YohannParis
Merge branch 'main' into dataset-statistics
000a912
@YohannParis
Merge branch 'main' into dataset-statistics
315ef25
@YohannParis
Merge branch 'main' into dataset-statistics
8397e13
@YohannParis
remove the unnecessary util file
88218d4
@YohannParis
Add messages and python/pandas type to ColumnType
3a67e93
@YohannParis
Merge branch 'main' into dataset-statistics
7785c63
@YohannParis
Create DatasetStatistics.java
72a5622
@YohannParis
Update DatasetStatistics.java
466c018
@YohannParis
Update DatasetService.java
9205194
@YohannParis
change when this is called
225f737
@YohannParis
cleaning up
b2c6efe
@YohannParis
Merge branch 'main' into dataset-statistics
dc68235
@YohannParis
add task notifications
8e32a59
@github-actions
chore: lint types
0424577
@YohannParis
Setup proper types for the column statistics
88d91b1
@YohannParis
Merge branch 'main' into dataset-statistics
9ca632d
@YohannParis
add the presignedURL
77e9de7
@YohannParis
Update DatasetStatistics.java
2774c28
@YohannParis
Do it properly
e583056
@YohannParis
Merge branch 'main' into dataset-statistics
e9e1106
@YohannParis
use the preseigned URL to pass to the task
ddf812c
@YohannParis
Check that the response might be null
61c6407
@YohannParis
better error checking
b5c701a
@YohannParis
improve readibility and format response properly
8133bac
@YohannParis
Update DatasetStatistics.java
384217f
@YohannParis
fix the issue with 🐍 to 🐫 case
37fa82e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kbirk kbirk Awaiting requested review from kbirk

@dgauldie dgauldie Awaiting requested review from dgauldie dgauldie will be requested when the pull request is marked ready for review dgauldie is a code owner

@dvince2 dvince2 Awaiting requested review from dvince2 dvince2 will be requested when the pull request is marked ready for review dvince2 is a code owner

@mwdchang mwdchang Awaiting requested review from mwdchang mwdchang will be requested when the pull request is marked ready for review mwdchang is a code owner

@jryu01 jryu01 Awaiting requested review from jryu01 jryu01 will be requested when the pull request is marked ready for review jryu01 is a code owner

@Tom-Szendrey Tom-Szendrey Awaiting requested review from Tom-Szendrey Tom-Szendrey will be requested when the pull request is marked ready for review Tom-Szendrey is a code owner

@shawnyama shawnyama Awaiting requested review from shawnyama shawnyama will be requested when the pull request is marked ready for review shawnyama is a code owner

@blanchco blanchco Awaiting requested review from blanchco blanchco will be requested when the pull request is marked ready for review blanchco is a code owner

@asylves1 asylves1 Awaiting requested review from asylves1 asylves1 will be requested when the pull request is marked ready for review asylves1 is a code owner

@mloppie mloppie Awaiting requested review from mloppie mloppie will be requested when the pull request is marked ready for review mloppie is a code owner

At least 2 approving reviews are required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[TASK]: Get Dataset stats
1 participant
@YohannParis