[pull] master from Netflix:master #104
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![pull[bot]](https://avatars.githubusercontent.com/in/12910?s=60&v=4){kind=small}
* [WIP] Very initial support for namespace packages for metaflow_extensions.
This is more to get an early flavor. Does not yet work.
* First working version
* Updates to extensions framework:
- Now with more debugging; set METAFLOW_DEBUG_EXT=1
- More validation to avoid unexpected issues (conflicts between
packages for example). Also verify if an "old" metaflow_extensions
is around. The user is warned with a comprehensible (hopefully) error
message.
- Support for packages in PYTHONPATH
- Updated test case
* Addressed comments
* Fix issue when there are no metaflow_extensions packages
* Forgot to commit a file that caused test failures
Also a bit more code cleanup to eliminate repetition
* Vendor importlib_metadata * format vendor.py with black * add attribution * adjust precommit hook * format
* Fix compute_resources to take into account string values * Always return strings. * additional unit tests Co-authored-by: Oleg Avdeev <[email protected]>
* multiple decorator support * Fixing comments * comment fix. * commet fix * allow multiple decorators of same type from cli * Stacked @card v1.2 : Customizing `@card` with `current.card` (#894) * allow passing userland `MetaflowCardComponents` - Added api for `current.cards` that handles multiple decorators * `current.cards` with `id` support. - Introduced `ALLOW_USER_COMPONENTS` attribute to `MetaflowCard` class - setting `ALLOW_USER_COMPONENTS=True` allow editable cards - Modified logic of `CardComponentCollector` - `CardComponentCollector.append` only accessible to default editble card * Added `customize=True` in `@card` - `customize=True` only allowed for one @card deco per @step - `customize=True` sets the card to be default editable - `current.card.append` appends to @card with `customize=True` * Stacked @card v1.2: Read cli changes for Supporting Multiple`@card`s (#895) * Added the `--id` option to `card view`/`card get` - modified datastore and fixed the exception class. - Allowing some ambiguity in pathspec argument for `get/view` command. * Added `id` argument to `get_cards` * Bringing `card list` cli functionality from test-suite branch - added functionality to list cards about a task and even list it as JSON. * changed hash checking logic. - instead of equating we check `startswith` * Added list many feature for `card list` - listing all cards from the latest run when card list is called with no argument * Added `card list` print formatting changes - `--as-json` works for many cards and single cards * Stacked @card v1.2 : New `MetaflowCardComponent`s (#896) * user-facing `MetaflowCardComponent`s - import via `from metaflow.cards import Artifact,..` * 7 Major Changes: - Removed `Section` component from `metaflow.cards` - Making user-facing component inherent to `UserComponent` which in turn inherits `MetaflowCardComponent` - Wrap all `UserComponents` inside a section after rendering everything per card - created a `render_safely` decorator to ensure fail-safe render of `UserComponent`s - removed code from component serializer which used internal components - Refactored some components that return render - Added docstrings to all components. * JS + CSS + Cards UI Build * Stacked @card v1.2 : Graph Related Changes to card cli (#911) * accomodating changes from #833 - Minor tweaks to `graph.py` * Stacked @card v1.2 : Namespace Packages with `@card` (#897) * setup import of cards from `metaflow_extensions` using `metaflow.extension_support` - Added import modules in `card_modules` - Added hook in card decorators to add custom packages * Added some debug statements for external imports. * Stacked @card v1.2 : Test cases for Multiple `@card`s (#898) * Multiple Cards Test Suite Mods (#27) - Added `list_cards` to `CliCheck` and `MetadataCheck` - Bringing #875 into the code - Added a card that prints taskspec with random number * Added test case for multiple cards * Added tests card, summary : - `current.cards['myid']` should be accessible when cards have an `id` argument in decorator - `current.cards.append` should not work when there are no single default editable card. - if a card has `ALLOW_USER_COMPONENTS=False` then it can still be edited via accessing it with `id` property. - adding arbitrary information to `current.cards.append` should not break user code. - Only cards with `ALLOW_USER_COMPONENTS=True` are considered default editable. - If a single @card decorator is present with `id` then it `current.cards.append` should still work - Access of `current.cards` with non existant id should not fail. - `current.cards.append` should be accessible to the card with `customize=True`. - * fixed `DefaultEditableCardTest` test case - Fixed comment - Fixed the `customize=True` test case. * ensure `test_pathspec_card` has no duplicates - ensure entropy of rendered information is high enough to not overwrite a file. * test case fix : `current.cards` to `current.card` * Added Test case to support import of cards. - Test case validates that broken card modules don't break metaflow - test case validates that we are able to import cards from metaflow_extensions - Test case validate that cards can be editable if they are importable. * Added Env var to tests to avoid warnings added to cards. * Added Test for card resume. * Stacked @card v1.2: Card Dev Docs (#899) Co-authored-by: Brendan Gibson <[email protected]> Co-authored-by: Brendan Gibson <[email protected]> Co-authored-by: adam <[email protected]> Co-authored-by: Romain Cledat <[email protected]> * Nit fix in error. * Fixing bug in decorator * whitespace. * Changing import scheme of warning variable. * fixing warning suppression env var setting Co-authored-by: Brendan Gibson <[email protected]> Co-authored-by: Brendan Gibson <[email protected]> Co-authored-by: adam <[email protected]> Co-authored-by: Romain Cledat <[email protected]>
Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.14.6 to 1.14.7. - [Release notes](https://github.com/follow-redirects/follow-redirects/releases) - [Commits](follow-redirects/follow-redirects@v1.14.6...v1.14.7) --- updated-dependencies: - dependency-name: follow-redirects dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [marked](https://github.com/markedjs/marked) from 4.0.8 to 4.0.10. - [Release notes](https://github.com/markedjs/marked/releases) - [Changelog](https://github.com/markedjs/marked/blob/master/.releaserc.json) - [Commits](markedjs/marked@v4.0.8...v4.0.10) --- updated-dependencies: - dependency-name: marked dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Removing some code bits. * removed `RENDER_TEMPLATE_PATH` * Refactoring redundant code.
* Add mamba option * black format * fixups * apply black * remove disable check * add support for micromamba * more changes Co-authored-by: jimgoo <[email protected]>
- Added a card to get the default card JSON object. - Added Base64 transform to js - Built JS + CSS/
* vend pylint * v * dependency fixes * update setup.py * formatting fixes * update tests * remove coverage * fix spurious commit * address comments * address black issues * address comments
* porting changes. * porting changes. * Fix * fix * Py3.5 accoomodating fix. * version fix. * test * Removing python version 2 * Nit fixes + add/remove py/R versions * Removing 3.11 py * testing GH action-reusabitity * Importing Testing. Deleting file post tests. * Testing fix * removing testing dummy file.
- files pointed by symlinks are now dereferenced by tar (no symlinks inside tar) - symlink cycles are detected
* Add README.md to PyPI package description * make black happy
* Don't explicitly break py2 support * Typo * Typo
…the Env Escape server (#960) We were currently not reflecting possible values passed using PYTHONPATH or programmatically added to the interpreter launching the environment escape server.
Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.14.7 to 1.14.9. - [Release notes](https://github.com/follow-redirects/follow-redirects/releases) - [Commits](follow-redirects/follow-redirects@v1.14.7...v1.14.9) --- updated-dependencies: - dependency-name: follow-redirects dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…#972) * Fix "Too many symbolic links" error when using Conda + Batch on MacOS * Ran black
Pushing directly per our conversation with Savin
When a module configured in the env escape was present in the local environment, we would raise an error. We now print a message and use the module present in the environment (not the environment escape). This allows the env escape modules to provide a "fallback" in case the module is not present but to still use the local module if present
The environment escape server would launch in the cwd. This could cause issues if, for example, a `metaflow` directory existed in that cwd (because it would then take precedence over the actual metaflow. To remediate this situation, the server is launched in the configuration directory (where we know what there is). Also took care of a quick annoyance of having to create an empty overrides.py file for everything. This is now no longer required
* Support session_vars and client_params in S3() This allows the user to pass down session variables and client parameters to set when getting a S3 client. * Addressed comments about default configuration
Co-authored-by: Preetam Joshi <[email protected]>
* Fixing s3util test caset * added comment Co-authored-by: Preetam Joshi <[email protected]>
Bumps [svelte](https://github.com/sveltejs/svelte) from 3.46.1 to 3.49.0. - [Release notes](https://github.com/sveltejs/svelte/releases) - [Changelog](https://github.com/sveltejs/svelte/blob/master/CHANGELOG.md) - [Commits](sveltejs/svelte@v3.46.1...v3.49.0) --- updated-dependencies: - dependency-name: svelte dependency-type: direct:development ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [terser](https://github.com/terser/terser) from 5.10.0 to 5.14.2. - [Release notes](https://github.com/terser/terser/releases) - [Changelog](https://github.com/terser/terser/blob/master/CHANGELOG.md) - [Commits](https://github.com/terser/terser/commits) --- updated-dependencies: - dependency-name: terser dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix r CI deps
Make the docstring format in public APIs compliant with the new API reference framework
In its current location, this could modify the sys.path of the current
running metaflow which could have nefarious consequences with the escape
hatch which uses sys.path to determine the outside environment's python path.
The following scenario would cause issues:
- metaflow is installed in the usual path on the system
- a conda environment was manually bootstrapped from a directory A.
+ at this point, sys.path starts with `A` and then contains the other
system includes
+ s3op.py is imported at some point by the Conda installer when it calls `get_many`
+ this modifies sys.path to insert, at the beginning, the parent of Metaflow; so
in this case, sys.path looks something like
['/apps/python3/lib/python3.7/site-packages', 'A', '/apps/python3/lib/python3.7/site-packages'...]
+ when the escape hatch trampolines are created, this sys.path is used to determine what the
sys.path for the outside interpreter is.
+ in A, we create:
* INFO
* metaflow
* metaflow_extensions
which properly describe the installation of metaflow
- when the escape hatch client runs, it runs in the conda environment and uses metaflow
created in A.
- when the client wants to start the server, this is where we run into issues because, at this
point, the server will use the PYTHONPATH which starts with '/apps/python3/lib/python3.7/site-packages'
in which it will find metaflow. It will therefore use that metaflow (which is the same
as the one linked in A) to start the server. This runs into issues though because A
is also in PYTHONPATH and so the extension support loader will also try to load `A/metaflow_extensions`.
This will cause issues if multiple extensions are installed there (it will complain about duplicate
configurations for example. The `INFO` file typically used to solve this problem is not read as it was
not present for the TL metaflow.
This patch simply moves the modification of sys.path to where it is actually needed and avoids polluting
sys.path when the module is simply included (and not called as a script).
* Airflow on Kubernetes minus Foreachs. - Support for all metaflow construct without foreach and sensors Squashed commit of the following: commit ef8b1e3 Author: Valay Dave <[email protected]> Date: Fri Jul 29 01:06:26 2022 +0000 Removed sernsors and banned foreach's commit 8d517c4 Author: Valay Dave <[email protected]> Date: Fri Jul 29 00:59:01 2022 +0000 commiting k8s related file from master. commit a7e1ecd Author: Valay Dave <[email protected]> Date: Fri Jul 29 00:54:45 2022 +0000 Uncommented code for foreach support with k8s KubernetesPodOperator version 4.2.0 renamed `resources` to `container_resources` - Check : (apache/airflow#24673) / - (apache/airflow@45f4290) This was done because `KubernetesPodOperator` didn't play nice with dynamic task mapping and they had to deprecate the `resources` argument. Hence the below codepath checks for the version of `KubernetesPodOperator` and then sets the argument. If the version < 4.2.0 then we set the argument as `resources`. If it is > 4.2.0 then we set the argument as `container_resources` The `resources` argument of KuberentesPodOperator is going to be deprecated soon in the future. So we will only use it for `KuberentesPodOperator` version < 4.2.0 The `resources` argument will also not work for foreach's. commit 2719f5d Author: Valay Dave <[email protected]> Date: Mon Jul 18 18:31:58 2022 +0000 nit fixes : - fixing comments. - refactor some variable/function names. commit 2079293 Author: Valay Dave <[email protected]> Date: Mon Jul 18 18:14:53 2022 +0000 change `token` to `production_token` commit 14aad5f Author: Valay Dave <[email protected]> Date: Mon Jul 18 18:11:56 2022 +0000 Refactored import Airflow Sensors. commit b1472d5 Author: Valay Dave <[email protected]> Date: Mon Jul 18 18:08:41 2022 +0000 new comment on `startup_timeout_seconds` env var. commit 6d81b75 Author: Valay Dave <[email protected]> Date: Mon Jul 18 18:06:09 2022 +0000 Removing traces of `@airflow_schedule_interval` commit 0673db7 Author: Valay Dave <[email protected]> Date: Thu Jul 14 12:43:08 2022 -0700 Foreach polish (valayDave#62) * Removing unused imports * Added validation logic for airflow version numbers with foreaches * Removed `airflow_schedule_interval` decorator. * Added production/deployment token related changes - Uses s3 as a backend to store the production token - Token used for avoiding nameclashes - token stored via `FlowDatastore` * Graph type validation for airflow foreachs - Airflow foreachs only support single node fanout. - validation invalidates graphs with nested foreachs * Added configuration about startup_timeout. * Added final todo on `resources` argument of k8sOp - added a commented code block - it needs to be uncommented when airflow releasese the patch for the op - Code seems feature complete keeping aside airflow patch commit 4b2dd12 Author: Valay Dave <[email protected]> Date: Thu Jul 7 19:33:07 2022 +0000 Removed retries from user-defaults. commit 0e87a97 Author: Valay Dave <[email protected]> Date: Wed Jul 6 16:29:33 2022 +0000 updated pod startup time commit fce2bd2 Author: Valay Dave <[email protected]> Date: Wed Jun 29 18:44:11 2022 +0000 Adding default 1 retry for any airflow worker. commit 5ef6bbc Author: Valay Dave <[email protected]> Date: Mon Jun 27 01:22:42 2022 +0000 Airflow Foreach Integration - Simple one node foreach-join support as gaurenteed by airflow - Fixed env variable setting issue - introduced MetaflowKuberentesOperator - Created a new operator to allow smootness in plumbing xcom values - Some todos commit d319fa9 Author: Valay Dave <[email protected]> Date: Fri Jun 24 21:12:09 2022 +0000 simplifying run-id macro. commit 0ffc813 Author: Valay Dave <[email protected]> Date: Fri Jun 24 11:51:42 2022 -0700 Refactored parameter macro settings. (valayDave#60) commit a3a4950 Author: Valay Dave <[email protected]> Date: Fri Jun 24 02:05:57 2022 +0000 added comment on need for `start_date` commit a3147be Author: Valay Dave <[email protected]> Date: Tue Jun 21 06:03:56 2022 +0000 Refactored an `id_creator` method. commit 04d7f20 Author: Valay Dave <[email protected]> Date: Tue Jun 21 05:52:05 2022 +0000 refactor : -`RUN_ID_LEN` to `RUN_HASH_ID_LEN` - `TASK_ID_LEN` to `TASK_ID_HASH_LEN` commit cde4605 Author: Valay Dave <[email protected]> Date: Tue Jun 21 05:48:55 2022 +0000 refactored an error string commit 1145818 Author: Valay Dave <[email protected]> Date: Mon Jun 20 22:42:36 2022 -0700 addressing savins comments. (#59) - Added many adhoc changes based for some comments. - Integrated secrets and `KUBERNETES_SECRETS` - cleaned up parameter setting - cleaned up setting of scheduling interval - renamed `AIRFLOW_TASK_ID_TEMPLATE_VALUE` to `AIRFLOW_TASK_ID` - renamed `AirflowSensorDecorator.compile` to `AirflowSensorDecorator.validate` - Checking if dagfile and flow file are same. - fixing variable names. - checking out `kubernetes_decorator.py` from master (6441ed5) - bug fixing secret setting in airflow. - simplified parameter type parsing logic - refactoring airflow argument parsing code. commit 83b20a7 Author: Valay Dave <[email protected]> Date: Mon Jun 13 14:02:57 2022 -0700 Addressing Final comments. (#57) - Added dag-run timeout. - airflow related scheduling checks in decorator. - Auto naming sensors if no name is provided - Annotations to k8s operators - fix: argument serialization for `DAG` arguments (method names refactored like `to_dict` became `serialize`) - annotation bug fix - setting`workflow-timeout` for only scheduled dags commit 4931f9c Author: Valay Dave <[email protected]> Date: Mon Jun 6 04:50:49 2022 +0000 k8s bug fix commit 200ae8e Author: Valay Dave <[email protected]> Date: Mon Jun 6 04:39:50 2022 +0000 removed un-used function commit 70e285e Author: Valay Dave <[email protected]> Date: Mon Jun 6 04:38:37 2022 +0000 Removed unused `sanitize_label` function commit 84fc622 Author: Valay Dave <[email protected]> Date: Mon Jun 6 04:37:34 2022 +0000 GPU support added + container naming same as argo commit c92280d Author: Valay Dave <[email protected]> Date: Mon Jun 6 04:25:17 2022 +0000 Refactored sensors to different files + bug fix - bug caused due `util.compress_list`. - The function doesn't play nice with strings with variety of characters. - Ensured that exceptions are handled appropriately. - Made new file for each sensor under `airflow.sensors` module. commit b72a1dc Author: Valay Dave <[email protected]> Date: Sat Jun 4 01:41:49 2022 +0000 ran black. commit 558c82f Author: Valay Dave <[email protected]> Date: Fri Jun 3 18:32:48 2022 -0700 Moving information from airflow_utils to compiler (#56) - commenting todos to organize unfinished changes. - some environment variables set via`V1EnvVar` - `client.V1ObjectFieldSelector` mapped env vars were not working in json form - Moving k8s operator import into its own function. - env vars moved. commit 9bb5f63 Author: Valay Dave <[email protected]> Date: Fri Jun 3 18:06:03 2022 +0000 added mising Run-id prefixes to variables. - merged `hash` and `dash_connect` filters. commit 37b5e6a Author: Valay Dave <[email protected]> Date: Fri Jun 3 18:00:22 2022 +0000 nit fix : variable name change. commit 660756f Author: Valay Dave <[email protected]> Date: Fri Jun 3 17:58:34 2022 +0000 nit fixes to dag.py's templating variables. commit 1202f5b Author: Valay Dave <[email protected]> Date: Fri Jun 3 17:56:53 2022 +0000 Fixed defaults passing - Addressed comments for airflow.py commit b9387dd Author: Valay Dave <[email protected]> Date: Fri Jun 3 17:52:24 2022 +0000 Following Changes: - Refactors setting scheduling interval - refactor dag file creating function - refactored is_active to is_paused_upon_creation - removed catchup commit 054e3f3 Author: Valay Dave <[email protected]> Date: Fri Jun 3 17:33:25 2022 +0000 Multiple Changes based on comments: 1. refactored `create_k8s_args` into _to_job 2. Addressed comments for snake casing 3. refactored `attrs` for simplicity. 4. refactored `metaflow_parameters` to `parameters`. 5. Refactored setting of `input_paths` commit d481b2f Author: Valay Dave <[email protected]> Date: Fri Jun 3 16:42:24 2022 +0000 Removed Sensor metadata extraction. commit d8e6ec0 Author: Valay Dave <[email protected]> Date: Fri Jun 3 16:30:34 2022 +0000 porting savin's comments - next changes : addressing comments. commit 3f2353a Merge: d370ffb c1ff469 Author: Valay Dave <[email protected]> Date: Thu Jul 28 23:52:16 2022 +0000 Merge branch 'master' into airflow commit d370ffb Merge: a82f144 e4eb751 Author: Valay Dave <[email protected]> Date: Thu Jul 14 19:38:48 2022 +0000 Merge branch 'master' into airflow commit a82f144 Merge: bdb1f0d 6f097e3 Author: Valay Dave <[email protected]> Date: Wed Jul 13 00:35:49 2022 +0000 Merge branch 'master' into airflow commit bdb1f0d Merge: 8511215 f9a4968 Author: Valay Dave <[email protected]> Date: Wed Jun 29 18:44:51 2022 +0000 Merge branch 'master' into airflow commit 8511215 Author: Valay Dave <[email protected]> Date: Tue Jun 21 02:53:11 2022 +0000 Bug fix from master merge. commit 90c06f1 Merge: 0fb73af 6441ed5 Author: Valay Dave <[email protected]> Date: Mon Jun 20 21:20:20 2022 +0000 Merge branch 'master' into airflow commit 0fb73af Author: Valay Dave <[email protected]> Date: Sat Jun 4 00:53:10 2022 +0000 squashing bugs after changes from master. commit 09c6ba7 Merge: 7bdf662 ffff49b Author: Valay Dave <[email protected]> Date: Sat Jun 4 00:20:38 2022 +0000 Merge branch 'master' into af-mmr commit 7bdf662 Author: Valay Dave <[email protected]> Date: Mon May 16 17:42:38 2022 -0700 Airflow sensor api (#3) * Fixed run-id setting - Change gaurentees that multiple dags triggered at same moment have unique run-id * added allow multiple in `Decorator` class * Airflow sensor integration. >> support added for : - ExternalTaskSensor - S3KeySensor - SqlSensor >> sensors allow multiple decorators >> sensors accept those arguments which are supported by airflow * Added `@airflow_schedule_interval` decorator * Fixing bug run-id related in env variable setting. commit 2604a29 Author: Valay Dave <[email protected]> Date: Thu Apr 21 18:26:59 2022 +0000 Addressed comments. commit 584e88b Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:33:55 2022 +0000 fixed printing bug commit 169ac15 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:30:59 2022 +0000 Option help bug fix. commit 6f8489b Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:25:54 2022 +0000 variable renamemetaflow_specific_args commit 0c779ab Merge: d299b13 5a61508 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:23:10 2022 +0000 Merge branch 'airflow-tests' into airflow commit 5a61508 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:22:54 2022 +0000 Removing un-used code / resolved-todos. commit d030830 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:06:03 2022 +0000 ran black, commit 2d1fc06 Merge: f2cb319 7921d13 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:04:19 2022 +0000 Merge branch 'master' into airflow-tests commit d299b13 Merge: f2cb319 7921d13 Author: Valay Dave <[email protected]> Date: Wed Apr 20 03:02:37 2022 +0000 Merge branch 'master' into airflow commit f2cb319 Author: Valay Dave <[email protected]> Date: Wed Apr 20 02:54:03 2022 +0000 reverting change. commit 05b9db9 Author: Valay Dave <[email protected]> Date: Wed Apr 20 02:47:41 2022 +0000 3 changes: - Removing s3 dep - remove uesless import - added `deployed_on` in dag file template commit c6afba9 Author: Valay Dave <[email protected]> Date: Fri Apr 15 22:50:52 2022 +0000 Fixed passing secrets with kubernetes. commit c3ce7e9 Author: Valay Dave <[email protected]> Date: Fri Apr 15 22:04:22 2022 +0000 Refactored code . - removed compute/k8s.py - Moved k8s code to airflow_compiler.py - ran isort to airflow_compiler.py commit d1c343d Author: Valay Dave <[email protected]> Date: Fri Apr 15 18:02:25 2022 +0000 Added validations about: - un-supported decorators - foreach Changed where validations are done to not save the package. commit 7b19f8e Author: Valay Dave <[email protected]> Date: Fri Apr 15 03:34:26 2022 +0000 Fixing mf log related bug - No double logging on metaflow. commit 4d1f6bf Author: Valay Dave <[email protected]> Date: Thu Apr 14 03:10:51 2022 +0000 Removed usless code WRT project decorator. commit 5ad9a39 Author: Valay Dave <[email protected]> Date: Thu Apr 14 03:03:19 2022 +0000 Remove readme. commit 60cb6a7 Author: Valay Dave <[email protected]> Date: Thu Apr 14 03:02:38 2022 +0000 Made file path required arguement. commit 9f0dc1b Author: Valay Dave <[email protected]> Date: Thu Apr 14 03:01:07 2022 +0000 changed `--is-active`->`--is-paused-upon-creation` - dags are active by default. commit 5b98f93 Author: Valay Dave <[email protected]> Date: Thu Apr 14 02:55:46 2022 +0000 shortened length of run-id and task-id hashes. commit e53426e Author: Valay Dave <[email protected]> Date: Thu Apr 14 02:41:32 2022 +0000 Removing un-used args. commit 72cbbfc Author: Valay Dave <[email protected]> Date: Thu Apr 14 02:39:59 2022 +0000 Moved exceptions to airflow compiler commit b2970dd Author: Valay Dave <[email protected]> Date: Thu Apr 14 02:33:02 2022 +0000 Changes based on PR comments: - removed airflow xcom push file , moved to decorator code - removed prefix configuration - nit fixes. commit 9e622ba Author: Valay Dave <[email protected]> Date: Mon Apr 11 20:39:00 2022 +0000 Removing un-used code paths + code cleanup commit 7425f62 Author: Valay Dave <[email protected]> Date: Mon Apr 11 19:45:04 2022 +0000 Fixing bug fix in schedule. commit eb775cb Author: Valay Dave <[email protected]> Date: Sun Apr 10 02:52:59 2022 +0000 Bug fixes WRT Kubernetes secrets + k8s deployments. - Fixing some error messages. - Added some comments. commit 04c92b9 Author: Valay Dave <[email protected]> Date: Sun Apr 10 01:20:53 2022 +0000 Added secrets support. commit 4a0a85d Author: Valay Dave <[email protected]> Date: Sun Apr 10 00:11:46 2022 +0000 Bug fix. commit af91099 Author: Valay Dave <[email protected]> Date: Sun Apr 10 00:03:34 2022 +0000 bug fix. commit c17f04a Author: Valay Dave <[email protected]> Date: Sat Apr 9 23:55:41 2022 +0000 Bug fix in active defaults. commit 0d37236 Author: Valay Dave <[email protected]> Date: Sat Apr 9 23:54:02 2022 +0000 @project, @schedule, default active dag support. - Added a flag to allow setting dag as active on creation - Airflow compatible schedule interval - Project name fixes. commit 5c97b15 Author: Valay Dave <[email protected]> Date: Thu Apr 7 21:15:18 2022 +0000 Max workers and worker pool support. commit 9c973f2 Author: Valay Dave <[email protected]> Date: Thu Apr 7 19:34:33 2022 +0000 Adding exceptions for missing features. commit 2a946e2 Author: Valay Dave <[email protected]> Date: Mon Mar 28 19:34:11 2022 +0000 2 changes : - removed hacky line - added support to directly throw dags in s3. commit e0772ec Author: Valay Dave <[email protected]> Date: Wed Mar 23 22:38:20 2022 +0000 fixing bugs in service account setting commit 874b94a Author: Valay Dave <[email protected]> Date: Sun Mar 20 23:49:15 2022 +0000 Added support for Branching with Airflow - remove `next` function in `AirflowTask` - `AirflowTask`s has no knowledge of next tasks. - removed todos + added some todos - Graph construction on airflow side using graph_structure datastructure. - graph_structure comes from`FlowGraph.output_steps()[1]` commit 8e9f649 Author: Valay Dave <[email protected]> Date: Sun Mar 20 02:33:04 2022 +0000 Added hacky line commit fd5db04 Author: Valay Dave <[email protected]> Date: Sun Mar 20 02:06:38 2022 +0000 Removed hacky line. commit 5b23eb7 Author: Valay Dave <[email protected]> Date: Sun Mar 20 01:44:57 2022 +0000 Added support for Parameters. - Supporting int, str, bool, float, JSONType commit c9378e9 Author: Valay Dave <[email protected]> Date: Sun Mar 20 00:14:10 2022 +0000 Removed todos + added some validation logic. commit 7250a44 Author: Valay Dave <[email protected]> Date: Sat Mar 19 23:45:15 2022 +0000 Fixing logs related change from master. commit d125978 Merge: 8cdac53 7e210a2 Author: Valay Dave <[email protected]> Date: Sat Mar 19 23:42:48 2022 +0000 Merge branch 'master' into aft-mm commit 8cdac53 Author: Valay Dave <[email protected]> Date: Sat Mar 19 23:36:47 2022 +0000 making changes sync with master commit 5a93d9f Author: Valay Dave <[email protected]> Date: Sat Mar 19 23:29:47 2022 +0000 Fixed bug when using catch + retry commit 62bc8df Author: Valay Dave <[email protected]> Date: Sat Mar 19 22:58:37 2022 +0000 Changed retry setting. commit 563a200 Author: Valay Dave <[email protected]> Date: Sat Mar 19 22:42:57 2022 +0000 Fixed setting `task_id` : - switch task-id from airflow job is to hash to "runid/stepname" - refactor xcom setting variables - added comments commit e2a1e50 Author: Valay Dave <[email protected]> Date: Sat Mar 19 17:51:59 2022 +0000 setting retry logic. commit a697b56 Author: Valay Dave <[email protected]> Date: Thu Mar 17 01:02:11 2022 +0000 Nit fix. commit 68f13be Author: Valay Dave <[email protected]> Date: Wed Mar 16 20:46:19 2022 +0000 Added @schedule support + readme commit 57bdde5 Author: Valay Dave <[email protected]> Date: Tue Mar 15 19:47:06 2022 +0000 Fixed setting run-id / task-id to labels in k8s - Fixed setting run-id has from cli macro - added hashing macro to ensure that jinja template set the correct run-id to k8s labels - commit 3d6c319 Author: Valay Dave <[email protected]> Date: Tue Mar 15 05:39:04 2022 +0000 Got linear workflows working on airflow. - Still not feature complete as lots of args are still unfilled / lots of unknows - minor tweek in eks to ensure airflow is k8s compatible. - passing state around via xcom-push - HACK : AWS keys are passed in a shady way. : Reverse this soon. commit db074b8 Author: Valay Dave <[email protected]> Date: Fri Mar 11 12:34:33 2022 -0800 Tweeks commit a9f0468 Author: Valay Dave <[email protected]> Date: Tue Mar 1 17:14:47 2022 -0800 some changes based on savin's comments. - Added changes to task datastore for different reason : (todo) Decouple these - Added comments to SFN for reference. - Airflow DAG is no longer dependent on metaflow commit f32d089 Author: Valay Dave <[email protected]> Date: Wed Feb 23 00:54:17 2022 -0800 First version of dynamic dag compiler. - Not completely finished code - Creates generic .py file a JSON that is parsed to create Airflow DAG. - Currently only boiler plate to make a linear dag but doesn't execute anything. - Unfinished code. commit d2def66 Author: Valay Dave <[email protected]> Date: Sat Feb 19 14:01:47 2022 -0800 more tweeks. commit b176311 Author: Valay Dave <[email protected]> Date: Thu Feb 17 09:04:29 2022 -0800 commit 0 - unfinished code. * Making version compatibility changes. - Minimum support version to 2.2.0 * Task-id macro related logic refactor - Done for better version support. * bug fix: param related task-id setting. * applied black * Reverting `decorators.py` to master
get_cards would not always respect a Task's ds-root leading to cases where a Task has cards but they cannot be accessed because of an invalid path.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot]
Can you help keep this open source service alive? 💖 Please sponsor : )