Docling Operator

[dolfim-ibm]

Docling Operator

The Docling Operator distributes the docling-serve together with the docling-jobkit Kubeflow jobs.

Configuration options

Compute engine

docling-serve allows to be deployed with different compute engines:

• local process
• remote queue
• distributed

The operator will allow to choose which compute engine to use for the deployment.

Serving options

TBA, e.g. authentication considerations.

Components

The operator should configure the following

• docling-serve API Deployment and related Secret, ConfigMap, Service
• docling-kfp-job Data Science Pipeline for running the distributed Docling conversion. This is launched and inspected from docling-serve using the k8s api

[dolfim-ibm]

Which resource should we deploy for KFP?

When docling-serve schedules a KFP pipeline run we have two options:

1. Using client.create_run_from_pipeline_func() which will create a new Pipeline and install it
   • Pros:
     • No need to manage versions and experiments
   • Cons:
     • It will populate lots of content in the Data Science Application
2. Using client.run_pipeline()
   • Pros:
     • Only one instance installed
     • Management via Data Science Application
   • Cons:
     • Requires configuration of which pipeline, which version and which experiment to use in docling-serve

[acornett21]

@zanetworker That doesn't really answer my question, my question is about which piece of the stack is responsible for what.

[zanetworker]

Or are we suggesting that in the docling-operator CustomeResource, paramaters for the kubeflow CR are accepted, then if present the docling-operator creates the kubeflow CR? Hopefully my question is clear, as there are alot of moving parts in the question.

The assumption here Is that KubeFlow exists on the cluster. I.e., you just trigger this: https://github.com/bbrowning/docling-kfp-demo/blob/main/docling_convert_pipeline.py or via docling-job-kit when it has KFP.

If that is still not clear we can chat on slack or meet.

[dolfim-ibm]

Following the examples in https://github.com/redhat-ai-services/kubeflow-pipelines-examples/tree/main/pipelines, it seems a good practice to use the client.create_run_from_pipeline_func() approach.

An initial implementation is in docling-project/docling-serve#107. The CR schema would only need to provide an input for the KFP endpoint (e.g. DOCLING_SERVE_ENG_KFP_ENDPOINT="https://ds-pipeline-dspa-dol.dol-ibm-dev.svc.cluster.local:8888")

[acornett21]

@dolfim-ibm So the diagram is correct, and the operator doesn't have any interactions with KFP, anything KFP related is triggered from docling-serv?

[dolfim-ibm]

@dolfim-ibm So the diagram is correct, and the operator doesn't have any interactions with KFP, anything KFP related is triggered from docling-serv?

Correct. Also from the initial implementation there is nothing needed except for the KFP endpoint.

[dolfim-ibm]

Communication between docling-serve and kfp

For scheduling pipeline runs of the docling-jobkit kubeflow pipeline the docling-serve container requires a valid authentication for the Data Science Pipelines.

What is the best way to set it up?

1. The Docling Operator instruments a ServiceAccount and uses its token (via Secrets mount) to initialize the kubleflow client
2. The Docling Operator creates a ServiceAccount which is set in the Pod manifest as serviceAccountName: .... Can the kubleflow client use the ENV token directly?
3. The API is configured with the OpenShift internal OAuth, and the job is submitted on behalf of the user making the API call.

[dolfim-ibm]

Tested.

This is implemented and working here: https://github.com/docling-project/docling-serve/blob/feat-kfp-engine/docling_serve/engines/async_kfp/orchestrator.py#L49

[dolfim-ibm]

Which type of Deployment for the REST API?

• Option 1: Define the Deployment, Service, Ingress directly in the operator
• Option 2: Use a Knative manifest which encapsulates all the above.

[acornett21]

@zanetworker I was under the impression that OperatorHub.io was out of scope here and that the downstream community ie embedded OpenShift OperatorHub was in scope, can you please clarify, if the intent is both upstream and down stream community catalogs?

[zanetworker]

oh this might have been a miscomm as we didn't dive into "community", what's the diff that we are talking about here in terms of feasibility.

[acornett21]

There are two community repos and process:

• upstream is here, operator bundles for OLM are then put in a catalog on OperatorHub.io where a k8s cluster can install OLM and use this catalog.
• downstream is here, operator bundles are then put in a catalog that is avilable in the embedded OperatorHub within OpenShift clusters.

These repos have different processes for managing operator version releases overtime, so while the content (the bundle yamls) might be nearly identical, other configurations that live alongside the bundle vary from upstream/downstream, since the pipelines and end goals are different.

This means we need to know what the actual ask is, so we can target the appropriate place/process/etc and plan for upstream/downstream/both accordingly.

[dolfim-ibm]

@zanetworker I think we should target the downstream catalog, what do you think?

[zanetworker]

Yea, Lets prioritize downstream for now.

[vku-ibm]

How to deal with secrets?

The simplest example is user passing s3 credentials to the pipeline run with jobkit.
Local secrets on the cluster could be used but this might be polluting.

[dolfim-ibm]

For example, does the KFP DSL syntax allow to declare some fields as secrets? Such that they will be obfuscated from logs, etc

[bcrochet]

I see that kfp.client takes a number of different "secrets", but I don't see anything else in the DSL related to that.

[dolfim-ibm]

OAuth-Proxy

Following the approach of https://github.com/opendatahub-io/data-science-pipelines-operator we could add an option for adding the oauth-proxy sidecar and configure the external Route or Ingress accordingly.

[dolfim-ibm]

Example implementation in https://github.com/docling-project/docling-serve/blob/main/docs/deploy-examples/docling-serve-oauth.yaml

[dolfim-ibm]

spec proposal

All the fields are optional, i.e. they have default values.

The only exception is kfpEngine.endpoint which is required when engine=kfp.

apiVersion: docling.io/v1
kind: Docling
metadata:
  name: docling-deploy
  namespace: default
spec:

  apiServer:
    image: ""
    enableOauth: true
    resources: {}
    tolerations: []
    nodeSelector: {}
    
    uvicornProxyHeaders: true
    uvicornTimeoutKeepAlive: 60
    uvicornWorkers: 1

    extraConfigMap: {}  # to be used as EnvFromConfigMap

  engine:  # only one must be defined

    local:
      numWorkers: 2

    kfp:
      endpoint: https://NAME.NAMESPACE.svc.cluster.local:8888
      parallelism: 8
      batchSize: 20

      convertJob:
        image: ""
        resources: {}
        tolerations: []
        nodeSelector: {}
        extraConfigMap: {}  # to be used as EnvFromConfigMap

[skattoju]

Would there ever be a need for allocating a disk for storage ? I image models would be self contained in docling images and docling-serve will get docs and return results via the network so I guess not but thought I would confirm.

[dolfim-ibm]

Would there ever be a need for allocating a disk for storage ? I image models would be self contained in docling images and docling-serve will get docs and return results via the network so I guess not but thought I would confirm.

No need for a volume. The model weights are in the image or will later be mounted as a OCI read-only volume.

@bcrochet from your proposed payload, I don't think we need apiServer.artifactsPath since it will be set to the required value by default.

[acornett21]

@dolfim-ibm I'll remove apiServer.artifactsPath from the spec. Is there anything in the configuration that needs a value in the apiServer spec that is not already captured?

[dolfim-ibm]

If we enable the oauth proxy we might need to set a few more internal ENV setting (still
to be tested), but nothing the user should worry about in the spec schema

[acornett21]

I see oauth and a bool and the deployment reconciler, would just set what is needed for that sidecar/docling-serve containers.