Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updates to automatic build process, data prepocessor apps and federates #9

Open
wants to merge 85 commits into
base: main
Choose a base branch
from
Open

Updates to automatic build process, data prepocessor apps and federates #9

wants to merge 85 commits into from

Conversation

sibyjackgrove
Copy link
Contributor

@sibyjackgrove sibyjackgrove commented Sep 27, 2024 •
edited
Loading

Adds the following features:

  1. Automatically builds the single-container application using GitHub links of applications specified through specification.json
  2. Uses version v2.0.2 of the oedisi library and version v0.4.1 of the oedi-example library.
  3. Refactored data preprocessing federates to use Keras 3 instead of tf.keras.
  4. Added additional data preprocessing federates: anomaly detection,solar disaggregation.
  5. Added script to generate solar node load profiles using solar der data and added DER data collected from ANL solar PV site.
  6. Additional documentation added for data preprocessing federates and automated build process.
  7. Updated CI scripts.

sibyjackgrove and others added 30 commits April 19, 2024 15:56
@sibyjackgrove
Update with changes from GitLab
88579c0
@sibyjackgrove
Fix errors due to previous commit
8847731
@sibyjackgrove
Add anomalydetection and update solar disaggregation
b1df5e0
@sibyjackgrove
Add anomalydetection
d033c90
@sibyjackgrove
Update with modules created for Keras 3 integration
5772429
@sibyjackgrove
Add notebook for dataimputation
2ba6cb8
@sibyjackgrove
Fix bugs and add requirements
ff778b9
@sibyjackgrove
Add notebook for PMU data imputation model training
eeda0f9
@sibyjackgrove
Update pmu notebook
2584c75
@sibyjackgrove
Add pmu data sample from ORNL
385a641
@sibyjackgrove
updates to data imputation simple demo
f873bb9
@sibyjackgrove
Updates to build.py to enable cloning of application repository
ee56062
@sibyjackgrove
More updates to build.py
769427e
@sibyjackgrove
Bug fix in git clone command
f2c31ae
@sibyjackgrove
Add method to modify application Dockerfile
a6f51f2
@sibyjackgrove
Clone application repo to tmp folder, remove redundant FROM statement…
378df5c 
…s in Dockerflie
@sibyjackgrove
Add specification.json file, reduce verbosity of git clone
69d42a5
@sibyjackgrove
Modififcations to allow datapreprocessor to be added to image
aabc32b
@sibyjackgrove
Modify datapreprocessor dockerfile
41ae146
@sibyjackgrove
Updates to fix WORKDIR problem + added oedisi to specification.json
8755c5c
@sibyjackgrove
Update datapreprocessor pip installs
bb113b2
change the datapreprocessor installation path and update code
6b5ffca
@sibyjackgrove
Remove references to sgidal-example
85f9610
@sibyjackgrove
Merge branch 'build_updates' of https://github.com/openEDI/oedi-si-si…
f80f13e 
…ngle-container into build_updates
@sibyjackgrove
Modified script to generate config_runner
1f03648
@sibyjackgrove
Modified Dockerfile so that it will install Python 3.11
1775f09
@sibyjackgrove
Change datapreprocessor location
6226069
@yim0331
change path for new datapreprocessor installation
da16848
@sibyjackgrove
Update paths in parse_config.py
e5ad4e1
@sibyjackgrove
Merging with Sam's changes
1d29718
@sibyjackgrove
Minor updates
a72bf43
@sibyjackgrove
Fix model loading bug for datapreprocessor, cleanup code
25a8419
@sibyjackgrove
Add trained model archives for data imputation and anomaly detection
0e9e2ef
@sibyjackgrove
Move trained models to model_archives
791d8d1
@sibyjackgrove
Minor updates
2b02e52
@sibyjackgrove
Refactor dataimputation federate, add anomalydetection federate+models
cc75712
@sibyjackgrove
Remove unloadble model, fix bugs in anomalydetction federate
c22572a
@sibyjackgrove
Bug fix in parse_config, add custom components.json and system.json
1aa1d46
@sibyjackgrove
Add dsse_ornl, refactoring
6b959c4
@sibyjackgrove
Bug fix for solar disagregation, update datapreprocessor README
d24a127
@sibyjackgrove
Update disaggregation trainining script
ab31268
@sibyjackgrove
Add solar disaggreation bilstm model, fix bugs in disagregation model…
083871f 
… training script
@sibyjackgrove
Cleanup dataimputation federate
c7a1ea7
@sibyjackgrove
Add solar disggregation federate
de6f927
@sibyjackgrove
Fix solardisaggregation federate bugs, add pre-trained diagg models
8dd24ee
@sibyjackgrove
Comment out unused code
2303868
@sibyjackgrove
Re-enable logic to add user dsse federate, Clean up parse_config
2778104
@sibyjackgrove
Refactoring + modifications to use der data
d4dd2fc
@sibyjackgrove
Add script to generate solar node load using operational der data
b492cd7
@sibyjackgrove
Add missing copy_statement.txt for pnnl dsse
7fdee27
@sibyjackgrove sibyjackgrove self-assigned this Sep 27, 2024
@sibyjackgrove sibyjackgrove marked this pull request as ready for review September 30, 2024 14:47
@sibyjackgrove sibyjackgrove added documentation Improvements or additions to documentation enhancement New feature or request labels Sep 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@karthikbalasu karthikbalasu Awaiting requested review from karthikbalasu

Assignees

@sibyjackgrove sibyjackgrove

Labels
documentation Improvements or additions to documentation enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@sibyjackgrove @yim0331