[pull] master from Netflix:master

.github/workflows/test.yml

@@ -25,7 +25,7 @@ jobs:
       fail-fast: false
       matrix:
         os: [ubuntu-20.04, macos-latest]
-        ver: ['3.5', '3.6', '3.7', '3.8', '3.9', '3.10', '3.11',]
+        ver: ['3.5', '3.6', '3.7', '3.8', '3.9', '3.10', '3.11', '3.12']
 
     steps:
     - uses: actions/checkout@ee0669bd1cc54295c223e0bb666b733df41de1c5 # v2.7.0

README.md

@@ -2,49 +2,68 @@
 
 # Metaflow
 
-Metaflow is a human-friendly Python/R library that helps scientists and engineers build and manage real-life data science projects. Metaflow was originally developed at Netflix to boost productivity of data scientists who work on a wide variety of projects from classical statistics to state-of-the-art deep learning.
+Metaflow is a human-friendly library that helps scientists and engineers build and manage real-life data science projects. Metaflow was [originally developed at Netflix](https://netflixtechblog.com/open-sourcing-metaflow-a-human-centric-framework-for-data-science-fa72e04a5d9) to boost productivity of data scientists who work on a wide variety of projects from classical statistics to state-of-the-art deep learning.
 
 For more information, see [Metaflow's website](https://metaflow.org) and [documentation](https://docs.metaflow.org).
 
-## Getting Started
+## From prototype to production (and back)
 
-Getting up and running with Metaflow is easy. 
+Metaflow provides a simple, friendly API that covers foundational needs of ML, AI, and data science projects:
+<img src="./docs/prototype-to-prod.png" width="800px">
 
-### Python
-Install metaflow from [pypi](https://pypi.org/project/metaflow/):
+1. [Rapid local prototyping](https://docs.metaflow.org/metaflow/basics), [support for notebooks](https://docs.metaflow.org/metaflow/visualizing-results), and [built-in experiment tracking and versioning](https://docs.metaflow.org/metaflow/client).
+2. [Horizontal and vertical scalability to the cloud](https://docs.metaflow.org/scaling/remote-tasks/introduction), utilizing both CPUs and GPUs, and [fast data access](https://docs.metaflow.org/scaling/data).
+3. [Managing dependencies](https://docs.metaflow.org/scaling/dependencies) and [one-click deployments to highly available production orchestrators](https://docs.metaflow.org/production/introduction).
+
+
+## Getting started
+
+Getting up and running is easy. If you don't know where to start, [Metaflow sandbox](https://outerbounds.com/sandbox) will have you running and exploring Metaflow in seconds.
+
+### Installing Metaflow in your Python environment
+
+To install Metaflow in your local environment, you can install from [PyPi](https://pypi.org/project/metaflow/):
 
 ```sh
 pip install metaflow
 ```
-
-and access tutorials by typing:
+Alternatively, you can also install from [conda-forge](https://anaconda.org/conda-forge/metaflow):
 
 ```sh
-metaflow tutorials pull
+conda install -c conda-forge metaflow
 ```
+If you are eager to try out Metaflow in practice, you can start with the [tutorial](https://docs.metaflow.org/getting-started/tutorials). After the tutorial, you can learn more about how Metaflow works [here](https://docs.metaflow.org/metaflow/basics).
 
-### R
+### Deploying infrastructure for Metaflow in your cloud
+<img src="./docs/multicloud.png" width="800px">
 
-Install Metaflow from [github](https://github.com/Netflix/metaflow/tree/master/R):
 
-```R
-devtools::install_github("Netflix/metaflow", subdir="R")
-metaflow::install()
-```
+While you can get started with Metaflow easily on your laptop, the main benefits of Metaflow lie in its ability to [scale out to external compute clusters](https://docs.metaflow.org/scaling/remote-tasks/introduction) 
+and to [deploy to production-grade workflow orchestrators](https://docs.metaflow.org/production/introduction). To benefit from these features, follow this [guide](https://outerbounds.com/engineering/welcome/) to 
+configure Metaflow and the infrastructure behind it appropriately.
 
-and access tutorials by typing:
+## [Resources](https://docs.metaflow.org/introduction/metaflow-resources)
 
-```R
-metaflow::pull_tutorials()
-```
+### [Slack Community](http://slack.outerbounds.co/)
+An active [community](http://slack.outerbounds.co/) of thousands of data scientists and ML engineers discussing the ins-and-outs of applied machine learning.
 
-## Get in Touch
-There are several ways to get in touch with us:
+### [Tutorials](https://outerbounds.com/docs/tutorials-index/)
+- [Introduction to Metaflow](https://outerbounds.com/docs/intro-tutorial-overview/)
+- [Natural Language Processing with Metaflow](https://outerbounds.com/docs/nlp-tutorial-overview/)
+- [Computer Vision with Metaflow](https://outerbounds.com/docs/cv-tutorial-overview/)
+- [Recommender Systems with Metaflow](https://outerbounds.com/docs/recsys-tutorial-overview/)
+- And more advanced content [here](https://outerbounds.com/docs/tutorials-index/)
 
-* Open an issue at: https://github.com/Netflix/metaflow 
-* Email us at: [email protected]
-* Chat with us on: http://chat.metaflow.org 
+### [Generative AI and LLM use cases](https://outerbounds.com/blog/?category=Foundation%20Models)
+- [Infrastructure Stack for Large Language Models](https://outerbounds.com/blog/llm-infrastructure-stack/)
+- [Parallelizing Stable Diffusion for Production Use Cases](https://outerbounds.com/blog/parallelizing-stable-diffusion-production-use-cases/)
+- [Whisper with Metaflow on Kubernetes](https://outerbounds.com/blog/whisper-kubernetes/)
+- [Training a Large Language Model With Metaflow, Featuring Dolly](https://outerbounds.com/blog/train-dolly-metaflow/)
 
-## Contributing
+## Get in touch
+There are several ways to get in touch with us:
+- [Slack Community](http://slack.outerbounds.co/)
+- [Github Issues](https://github.com/Netflix/metaflow/issues)
 
+## Contributing
 We welcome contributions to Metaflow. Please see our [contribution guide](https://docs.metaflow.org/introduction/contributing-to-metaflow) for more details.

metaflow/R.py

@@ -1,5 +1,6 @@
 import os
-import imp
+import sys
+from importlib import util as imp_util, machinery as imp_machinery
 from tempfile import NamedTemporaryFile
 
 from .util import to_bytes
@@ -70,6 +71,26 @@ def working_dir():
     return None
 
 
+def load_module_from_path(module_name: str, path: str):
+    """
+    Loads a module from a given path
+
+    Parameters
+    ----------
+    module_name: str
+        Name to assign for the loaded module. Usable for importing after loading.
+    path: str
+        path to the file to be loaded
+    """
+    loader = imp_machinery.SourceFileLoader(module_name, path)
+    spec = imp_util.spec_from_loader(loader.name, loader)
+    module = imp_util.module_from_spec(spec)
+    loader.exec_module(module)
+    # Required in order to be able to import the module by name later.
+    sys.modules[module_name] = module
+    return module
+
+
 def run(
     flow_script,
     r_functions,
@@ -100,7 +121,7 @@ def run(
     full_cmdline[0] = os.path.basename(full_cmdline[0])
     with NamedTemporaryFile(prefix="metaflowR.", delete=False) as tmp:
         tmp.write(to_bytes(flow_script))
-    module = imp.load_source("metaflowR", tmp.name)
+    module = load_module_from_path("metaflowR", tmp.name)
     flow = module.FLOW(use_cli=False)
 
     from . import exception