Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: ORNL/flowcept
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v0.0.114
Choose a base ref
...
head repository: ORNL/flowcept
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: main
Choose a head ref
Loading
Showing with 23,427 additions and 1,419 deletions.
  1. +0 −9 .flake8
  2. +0 −35 .github/workflows/bump-version.yml
  3. +35 −0 .github/workflows/checks.yml
  4. +0 −27 .github/workflows/code-formatting.yml
  5. +68 −11 .github/workflows/create-release-n-publish.yml
  6. +52 −0 .github/workflows/run-llm-tests.yml
  7. +64 −0 .github/workflows/run-tests-all-dbs.yml
  8. +44 −0 .github/workflows/run-tests-in-container.yml
  9. +73 −0 .github/workflows/run-tests-kafka.yml
  10. +59 −0 .github/workflows/run-tests-py313.yml
  11. +63 −0 .github/workflows/run-tests-simple.yml
  12. +97 −0 .github/workflows/run-tests.yml
  13. +0 −34 .github/workflows/run-unit-tests.yml
  14. +101 −0 .github/workflows/run_examples.sh
  15. +9 −25 .github/workflows/version_bumper.py
  16. +23 −5 .gitignore
  17. +28 −0 .readthedocs.yaml
  18. +48 −50 CONTRIBUTING.md
  19. +97 −0 Makefile
  20. +187 −40 README.md
  21. +0 −14 bin/clean_directory.sh
  22. +0 −57 bin/git_push.sh
  23. +26 −0 deployment/Dockerfile
  24. +37 −0 deployment/build-image.sh
  25. +63 −0 deployment/compose-kafka.yml
  26. +44 −0 deployment/compose-mofka.yml
  27. +31 −0 deployment/compose-mongo.yml
  28. +11 −18 deployment/compose.yml
  29. +10 −0 docs/api-reference.rst
  30. +26 −0 docs/conf.py
  31. +4 −0 docs/contributing.rst
  32. +70 −0 docs/getstarted.rst
  33. +12 −0 docs/index.rst
  34. +65 −0 examples/dask_example.py
  35. +76 −0 examples/distributed_consumer_example.py
  36. +19 −0 examples/instrumented_loop_example.py
  37. +22 −0 examples/instrumented_simple_example.py
  38. +35 −0 examples/llm_complex/custom_provenance_id_mapping.yaml
  39. +231 −0 examples/llm_complex/llm_dataprep.py
  40. +484 −0 examples/llm_complex/llm_main_example.py
  41. +256 −0 examples/llm_complex/llm_model.py
  42. +117 −0 examples/llm_complex/llm_test_runner.py
  43. +37 −0 examples/mlflow_example.py
  44. +116 −0 examples/single_layer_perceptron_example.py
  45. +147 −0 examples/tensorboard_example.py
  46. +55 −0 examples/unmanaged/main.py
  47. +29 −0 examples/unmanaged/simple_task.py
  48. +0 −5 extra_requirements/dev-requirements.txt
  49. +0 −6 extra_requirements/mlflow-requirements.txt
  50. +0 −1 extra_requirements/mongo-requirements.txt
  51. +0 −4 extra_requirements/tensorboard-requirements.txt
  52. +0 −1 extra_requirements/zambeze-requirements.txt
  53. +0 −3 flowcept/__init__.py
  54. +0 −28 flowcept/commons/flowcept_data_classes.py
  55. +0 −24 flowcept/commons/mq_dao.py
  56. +0 −9 flowcept/commons/vocabulary.py
  57. +0 −51 flowcept/configs.py
  58. +0 −56 flowcept/flowcept_consumer/doc_db/document_db_dao.py
  59. +0 −61 flowcept/flowcept_consumer/doc_db/document_inserter.py
  60. +0 −27 flowcept/flowcept_consumer/main.py
  61. +0 −85 flowcept/flowceptor/plugins/base_interceptor.py
  62. +0 −18 flowcept/flowceptor/plugins/base_settings_dataclasses.py
  63. 0 flowcept/flowceptor/plugins/mlflow/__init__.py
  64. +0 −74 flowcept/flowceptor/plugins/mlflow/mlflow_interceptor.py
  65. +0 −54 flowcept/flowceptor/plugins/settings_factory.py
  66. 0 flowcept/flowceptor/plugins/tensorboard/__init__.py
  67. +0 −18 flowcept/flowceptor/plugins/tensorboard/tensorboard_dataclasses.py
  68. +0 −102 flowcept/flowceptor/plugins/tensorboard/tensorboard_interceptor.py
  69. 0 flowcept/flowceptor/plugins/zambeze/__init__.py
  70. +0 −71 flowcept/flowceptor/plugins/zambeze/zambeze_interceptor.py
  71. 0 flowcept/unused_flowcept_server/__init__.py
  72. +0 −19 flowcept/unused_flowcept_server/app.py
  73. 0 flowcept/unused_flowcept_server/handlers/__init__.py
  74. 0 flowcept/unused_flowcept_server/handlers/zambeze/__init__.py
  75. 0 flowcept/unused_flowcept_server/handlers/zambeze/zambeze_handler.py
  76. +0 −5 flowcept/version.py
  77. +626 −0 notebooks/analytics.ipynb
  78. +1,339 −0 notebooks/dask.ipynb
  79. +169 −0 notebooks/dask_from_CLI.ipynb
  80. +453 −0 notebooks/mlflow.ipynb
  81. +9 −0 notebooks/reset_dask_nb_exec_counts.py
  82. +10 −0 notebooks/sample_data.csv
  83. +411 −0 notebooks/tensorboard.ipynb
  84. +112 −3 pyproject.toml
  85. +0 −4 requirements.txt
  86. +20 −0 resources/mofka/mofka-requirements.yaml
  87. +58 −0 resources/mofka/mofka_config.json
  88. +121 −0 resources/sample_settings.yaml
  89. +0 −41 resources/settings.yaml
  90. +24 −0 resources/simple_redis_consumer.py
  91. +0 −64 setup.py
  92. +93 −0 src/flowcept/__init__.py
  93. +23 −0 src/flowcept/analytics/__init__.py
  94. +271 −0 src/flowcept/analytics/analytics_utils.py
  95. +61 −0 src/flowcept/analytics/data_augmentation.py
  96. +105 −0 src/flowcept/analytics/plot.py
  97. +1 −0 src/flowcept/commons/__init__.py
  98. +84 −0 src/flowcept/commons/autoflush_buffer.py
  99. +1 −0 src/flowcept/commons/daos/__init__.py
  100. +1 −0 src/flowcept/commons/daos/docdb_dao/__init__.py
  101. +392 −0 src/flowcept/commons/daos/docdb_dao/docdb_dao_base.py
  102. +335 −0 src/flowcept/commons/daos/docdb_dao/lmdb_dao.py
  103. +964 −0 src/flowcept/commons/daos/docdb_dao/mongodb_dao.py
  104. +72 −0 src/flowcept/commons/daos/keyvalue_dao.py
  105. +1 −0 src/flowcept/commons/daos/mq_dao/__init__.py
  106. +205 −0 src/flowcept/commons/daos/mq_dao/mq_dao_base.py
  107. +100 −0 src/flowcept/commons/daos/mq_dao/mq_dao_kafka.py
  108. +82 −0 src/flowcept/commons/daos/mq_dao/mq_dao_mofka.py
  109. +76 −0 src/flowcept/commons/daos/mq_dao/mq_dao_redis.py
  110. +1 −0 src/flowcept/commons/flowcept_dataclasses/__init__.py
  111. +23 −0 src/flowcept/commons/flowcept_dataclasses/base_settings_dataclasses.py
  112. +154 −0 src/flowcept/commons/flowcept_dataclasses/task_object.py
  113. +107 −0 src/flowcept/commons/flowcept_dataclasses/telemetry.py
  114. +132 −0 src/flowcept/commons/flowcept_dataclasses/workflow_object.py
  115. +54 −0 src/flowcept/commons/flowcept_logger.py
  116. +73 −0 src/flowcept/commons/query_utils.py
  117. +52 −0 src/flowcept/commons/settings_factory.py
  118. +260 −0 src/flowcept/commons/utils.py
  119. +37 −0 src/flowcept/commons/vocabulary.py
  120. +236 −0 src/flowcept/configs.py
  121. +1 −0 src/flowcept/flowcept_api/__init__.py
  122. +300 −0 src/flowcept/flowcept_api/db_api.py
  123. +258 −0 src/flowcept/flowcept_api/flowcept_controller.py
  124. +649 −0 src/flowcept/flowcept_api/task_query_api.py
  125. +1 −0 src/flowcept/flowcept_webserver/__init__.py
  126. +28 −0 src/flowcept/flowcept_webserver/app.py
  127. +1 −0 src/flowcept/flowcept_webserver/resources/__init__.py
  128. +45 −0 src/flowcept/flowcept_webserver/resources/query_rsrc.py
  129. +27 −0 src/flowcept/flowcept_webserver/resources/task_messages_rsrc.py
  130. +1 −0 src/flowcept/flowceptor/__init__.py
  131. +1 −0 src/flowcept/flowceptor/adapters/__init__.py
  132. +112 −0 src/flowcept/flowceptor/adapters/base_interceptor.py
  133. +1 −0 src/flowcept/flowceptor/adapters/dask/__init__.py
  134. +24 −0 src/flowcept/flowceptor/adapters/dask/dask_dataclasses.py
  135. +215 −0 src/flowcept/flowceptor/adapters/dask/dask_interceptor.py
  136. +122 −0 src/flowcept/flowceptor/adapters/dask/dask_plugins.py
  137. +23 −0 src/flowcept/flowceptor/adapters/instrumentation_interceptor.py
  138. +14 −10 {flowcept/flowceptor/plugins → src/flowcept/flowceptor/adapters}/interceptor_state_manager.py
  139. +1 −0 src/flowcept/flowceptor/adapters/mlflow/__init__.py
  140. +7 −2 ...wcept/flowceptor/plugins → src/flowcept/flowceptor/adapters}/mlflow/interception_event_handler.py
  141. +57 −18 {flowcept/flowceptor/plugins → src/flowcept/flowceptor/adapters}/mlflow/mlflow_dao.py
  142. +12 −5 {flowcept/flowceptor/plugins → src/flowcept/flowceptor/adapters}/mlflow/mlflow_dataclasses.py
  143. +100 −0 src/flowcept/flowceptor/adapters/mlflow/mlflow_interceptor.py
  144. +1 −0 src/flowcept/flowceptor/adapters/tensorboard/__init__.py
  145. +24 −0 src/flowcept/flowceptor/adapters/tensorboard/tensorboard_dataclasses.py
  146. +117 −0 src/flowcept/flowceptor/adapters/tensorboard/tensorboard_interceptor.py
  147. +1 −0 src/flowcept/flowceptor/adapters/zambeze/__init__.py
  148. +12 −6 {flowcept/flowceptor/plugins → src/flowcept/flowceptor/adapters}/zambeze/zambeze_dataclasses.py
  149. +102 −0 src/flowcept/flowceptor/adapters/zambeze/zambeze_interceptor.py
  150. +1 −0 src/flowcept/flowceptor/consumers/__init__.py
  151. +156 −0 src/flowcept/flowceptor/consumers/consumer_utils.py
  152. +233 −0 src/flowcept/flowceptor/consumers/document_inserter.py
  153. +378 −0 src/flowcept/flowceptor/telemetry_capture.py
  154. +1 −0 src/flowcept/instrumentation/__init__.py
  155. +324 −0 src/flowcept/instrumentation/flowcept_loop.py
  156. +175 −0 src/flowcept/instrumentation/flowcept_task.py
  157. +544 −0 src/flowcept/instrumentation/flowcept_torch.py
  158. +127 −0 src/flowcept/instrumentation/task_capture.py
  159. +7 −0 src/flowcept/version.py
  160. 0 {flowcept/commons → tests/adapters}/__init__.py
  161. +45 −0 tests/adapters/dask_test_utils.py
  162. +184 −0 tests/adapters/test_dask.py
  163. +55 −0 tests/adapters/test_dask_with_context_mgmt.py
  164. +65 −0 tests/adapters/test_file_observer.py
  165. +101 −0 tests/adapters/test_mlflow.py
  166. +72 −40 tests/{plugins → adapters}/test_tensorboard.py
  167. 0 {flowcept/flowcept_consumer → tests/api}/__init__.py
  168. +127 −0 tests/api/db_api_test.py
  169. +118 −0 tests/api/flowcept_api_test.py
  170. +6,589 −0 tests/api/sample_data.json
  171. +1 −0 tests/api/sample_data_with_telemetry_and_rai.json
  172. +432 −0 tests/api/task_query_api_test.py
  173. +0 −26 tests/doc_db_inserter/doc_db_inserter.py
  174. +104 −0 tests/doc_db_inserter/doc_db_inserter_test.py
  175. 0 {flowcept/flowcept_consumer/doc_db → tests/instrumentation_tests}/__init__.py
  176. +29 −0 tests/instrumentation_tests/flowcept_explicit_tasks.py
  177. +155 −0 tests/instrumentation_tests/flowcept_loop_test.py
  178. +292 −0 tests/instrumentation_tests/flowcept_task_decorator_test.py
  179. 0 {flowcept/flowceptor → tests/instrumentation_tests/ml_tests}/__init__.py
  180. +240 −0 tests/instrumentation_tests/ml_tests/dl_trainer.py
  181. +61 −0 tests/instrumentation_tests/ml_tests/ml_decorator_dask_test.py
  182. +52 −0 tests/instrumentation_tests/ml_tests/ml_decorator_test.py
  183. 0 {flowcept/flowceptor/plugins → tests/misc_tests}/__init__.py
  184. +34 −0 tests/misc_tests/log_test.py
  185. +38 −0 tests/misc_tests/singleton_test.py
  186. +11 −0 tests/misc_tests/telemetry_test.py
  187. 0 tests/plugins/__init__.py
  188. +0 −81 tests/plugins/test_mlflow.py
  189. +0 −72 tests/plugins/test_zambeze.py
9 changes: 0 additions & 9 deletions .flake8

This file was deleted.

35 changes: 0 additions & 35 deletions .github/workflows/bump-version.yml

This file was deleted.

35 changes: 35 additions & 0 deletions .github/workflows/checks.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
name: Linter, formatter, and docs checks

on: pull_request

jobs:

checks:
runs-on: ubuntu-22.04
if: "!contains(github.event.head_commit.message, 'CI Bot')"
steps:
- uses: actions/checkout@v4

- name: Set up Python 3.12
uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: "pip"

- name: Install package and dependencies
run: |
pip install --upgrade pip
pip install ruff
pip install .[docs]
- name: Run linter and formatter checks using ruff
run: make checks

- name: Run HTML builder for Sphinx documentation
run: make docs

- name: Clean up
run: |
make clean
find /home/runner/runners/ -type f -name "*.log" -exec sh -c 'echo {}; >"{}"' \; || true
27 changes: 0 additions & 27 deletions .github/workflows/code-formatting.yml

This file was deleted.

79 changes: 68 additions & 11 deletions .github/workflows/create-release-n-publish.yml
Original file line number Diff line number Diff line change
@@ -1,40 +1,46 @@
name: Release and Publish
on:
push:
push: # It has to be push, otherwise error happens in code below.
branches: [ "main" ]
# branches: [ "main", "dev" ] # use this only to test the CI. If testing this CI, consider commenting out the automatic version updates and manually adjust the patch version.
#branches: [ "disabled" ]
jobs:
build:
name: Create Release and Publish
runs-on: ubuntu-latest
env:
MONGO_ENABLED: false
LMDB_ENABLED: true
timeout-minutes: 60
steps:
- name: Checkout code
uses: actions/checkout@v2
uses: actions/checkout@v4
with:
fetch-depth: 1

- name: Set up Python 3.10
uses: actions/setup-python@v3
uses: actions/setup-python@v5
with:
python-version: "3.10"
- name: Get branch names
id: branch-name
uses: tj-actions/branch-names@v6

- name: Update version.py
run: |
export PYTHONPATH=$PYTHONPATH:flowcept
export BRANCH_NAME="${{ steps.branch-name.outputs.current_branch }}"
python .github/workflows/version_bumper.py
- name: Commit new version
run: |
git config --global user.name 'Flowcept CI Bot'
git config --global user.email 'flowcept@users.noreply.github.com'
git branch
git add flowcept/version.py
git add src/flowcept/version.py
git commit -m "Flowcept CI Bot: bumping master version"
git push --force
- name: Get Latest PR and Create Release
run: |
export PYTHONPATH=$PYTHONPATH:flowcept
export CURRENT_VERSION=`python -c "from flowcept import __version__; print(__version__)"`
export CURRENT_VERSION=`python -c "f = open('src/flowcept/version.py'); exec(f.read()); print(locals()['__version__']); f.close()"`
echo $CURRENT_VERSION
REPOSITORY=${{ github.repository }}
ACCESS_TOKEN=${{ secrets.GITHUB_TOKEN }}
TARGET=${{ steps.branch-name.outputs.current_branch }}
@@ -59,12 +65,14 @@ jobs:
\"prerelease\": false}" \
-H "Authorization: Bearer ${ACCESS_TOKEN}" \
https://api.github.com/repos/${REPOSITORY}/releases
- name: Install pypa/build
run: >-
python -m
pip install
build
--user
- name: Build a binary wheel and a source tarball
run: >-
python -m
@@ -73,15 +81,64 @@ jobs:
--wheel
--outdir dist/
.
- name: Publish distribution to Test PyPI
uses: pypa/gh-action-pypi-publish@release/v1
with:
password: ${{ secrets.TEST_PYPI_API_TOKEN }}
repository_url: https://test.pypi.org/legacy/
verbose: true

- name: Publish distribution to PyPI
#if: startsWith(github.ref, 'refs/tags')
uses: pypa/gh-action-pypi-publish@release/v1
with:
password: ${{ secrets.PYPI_API_TOKEN }}
verbose: true

- name: Wait pypi do its thing
run: sleep 120

- name: Test pip install
run: pip install flowcept

- name: Print installed version
run: pip list | grep flowcept

- name: Test pip install one adapter
run: pip install flowcept[dask]

- name: Test pip install multiple adapters
run: pip install flowcept[mlflow,tensorboard]

- name: Install our dependencies
run: pip install flowcept[all]

- name: Install ml_dev dependencies
run: pip install flowcept[ml_dev]

- name: Pip list
run: pip list

- name: Start up services
run: make services

- name: Test with pytest
run: pytest

- name: Test notebooks
run: |
# export FLOWCEPT_SETTINGS_PATH=~/.flowcept/settings.yaml
python src/flowcept/flowcept_webserver/app.py &
sleep 3
pytest --nbmake "notebooks/" --nbmake-timeout=600 --ignore=notebooks/dask_from_CLI.ipynb
- name: Stop services
run: make services-stop

- name: Clean up
run: |
make clean
find /home/runner/runners/ -type f -name "*.log" -exec sh -c 'echo {}; >"{}"' \; || true
docker image prune -a -f
52 changes: 52 additions & 0 deletions .github/workflows/run-llm-tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
name: LLM Tests
on: [pull_request]

jobs:

build:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: [ "3.10", "3.11", "3.12" ]
env:
MONGO_ENABLED: true
LMDB_ENABLED: false
timeout-minutes: 60
if: "!contains(github.event.head_commit.message, 'CI Bot')"

steps:
- uses: actions/checkout@v4
with:
fetch-depth: 1

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: "pip"

- name: Show OS Info
run: '[[ "$OSTYPE" == "linux-gnu"* ]] && { echo "OS Type: Linux"; (command -v lsb_release &> /dev/null && lsb_release -a) || cat /etc/os-release; uname -r; } || [[ "$OSTYPE" == "darwin"* ]] && { echo "OS Type: macOS"; sw_vers; uname -r; } || echo "Unsupported OS type: $OSTYPE"'

- name: Start docker compose with redis
run: make services-mongo

- name: Upgrade pip
run: |
python -m pip install --upgrade pip
python --version
- name: Test LLM
run: bash .github/workflows/run_examples.sh examples true llm_complex/llm_test_runner.py

- name: Shut down docker compose
run: make services-stop-mongo

- name: Clean up
run: |
make clean
find /home/runner/runners/ -type f -name "*.log" -exec sh -c 'echo {}; >"{}"' \; || true
docker image prune -a -f
- name: List large files
run: find . -type f -exec du -h {} + | sort -h
64 changes: 64 additions & 0 deletions .github/workflows/run-tests-all-dbs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
name: With and Without Mongo at the same time
on: [pull_request]

jobs:

build:
runs-on: ubuntu-latest
timeout-minutes: 40
strategy:
matrix:
python-version: [ "3.10", "3.11", "3.12" ]
env:
MONGO_ENABLED: true
LMDB_ENABLED: true

if: "!contains(github.event.head_commit.message, 'CI Bot')"

steps:
- uses: actions/checkout@v4
with:
fetch-depth: 1

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: "pip"

- name: Show OS Info
run: '[[ "$OSTYPE" == "linux-gnu"* ]] && { echo "OS Type: Linux"; (command -v lsb_release &> /dev/null && lsb_release -a) || cat /etc/os-release; uname -r; } || [[ "$OSTYPE" == "darwin"* ]] && { echo "OS Type: macOS"; sw_vers; uname -r; } || echo "Unsupported OS type: $OSTYPE"'

- name: Start docker compose with redis
run: make services-mongo

- name: Upgrade pip
run: python -m pip install --upgrade pip

- name: Show Python version
run: python --version && pip --version

- name: Test examples
run: bash .github/workflows/run_examples.sh examples true # with mongo

- name: Install all dependencies
run: |
python -m pip install --upgrade pip
python -m pip install .[all]
python -m pip install .[ml_dev]
- name: List installed packages
run: pip list

- name: Test with pytest and redis
run: |
export MONGO_ENABLED=true
export LMDB_ENABLED=true
python -c "from flowcept.configs import MONGO_ENABLED, LMDB_ENABLED; print('MONGO?', MONGO_ENABLED); print('LMDB?', LMDB_ENABLED)"
make tests
- name: Test notebooks with pytest and redis
run: pytest --nbmake "notebooks/" --nbmake-timeout=600 --ignore="notebooks/dask_from_CLI.ipynb"

- name: Stop services
run: make services-stop-mongo
Loading