[FR] Consider detecting current running context is databricks notebook or not; if it is, install Feathr library

[xiaoyongzhu]

Willingness to contribute

No. I cannot contribute a bug fix at this time.

Feature Request Proposal

Some sample code:

! pip install databricks_cli

ctx = dbutils.notebook.entry_point.getDbutils().notebook().getContext()
token_value=ctx.apiToken().get()
workspace_instance_url=f"https://{ctx.tags().get('browserHostName').get()}"
cluster_id = ctx.tags().get('clusterId').get()

from databricks_cli.dbfs.api import DbfsApi
from databricks_cli.runs.api import RunsApi
from databricks_cli.dbfs.dbfs_path import DbfsPath
from databricks_cli.sdk.api_client import ApiClient
from databricks_cli.libraries.api import LibrariesApi
api_client = ApiClient(host=workspace_instance_url, token=token_value)

LibrariesApi(api_client=api_client).install_libraries(cluster_id, libraries = [{
      "maven": {
        "coordinates": "com.linkedin.feathr:feathr_2.12:0.9.0"
      }
    }])

res = LibrariesApi(api_client=api_client).cluster_status(cluster_id)
if 'feathr' in res['library_statuses'][0]['library']['maven']['coordinates']:
    print("feathr is registered successfully")

Motivation

What is the use case for this feature?

In this case, we don't have to start a new cluster everytime; we can reuse the existing cluster if necessary

Details

No response

What component(s) does this feature request affect?

• Python Client: This is the client users use to interact with most of our API. Mostly written in Python.
• Computation Engine: The computation engine that execute the actual feature join and generation work. Mostly in Scala and Spark.
• Feature Registry API: The frontend API layer supports SQL, Purview(Atlas) as storage. The API layer is in Python(FAST API)
• Feature Registry Web UI: The Web UI for feature registry. Written in React