Skip to content
This repository has been archived by the owner on Jul 21, 2024. It is now read-only.

HOWTO: Automate BlendNet without Addon

Jump to bottom
Rabit edited this page Sep 12, 2020 · 1 revision

From HOWTO: Setup provider: Your own render farm (local) you know how to setup BlendNet Manager & Agents and connect them, now it's time to find out how to feed the Manager with the blend file, it's dependencies and the task configuration to render frames without the Addon in a complete automated way.

The working example of such automation was prepared as CI for BlendNet project and the best way to understand it - dig into the logic of CircleCI config file and in the API test script.

Basic python clients

The Manager and Addon are using the same HTTPS RESTful interface that is implemented in 2 clients:

They use the Client, which implements the core functionality and all the common interfaces for both parties. So you can just import those modules in your project and use them to communicate with the Manager or the Agent.

CLI & API

The BlendNet API is json-based and quite easy to use, just follow it:

Request practices

Also you can skip the python and use just a regular console utils (like curl or wget) to communicate with the services. They are quite frendly and will tell you everything you need to know about the API:

$ curl --user 'None:None' --insecure https://localhost:8443/
{"success": false, "message": "Invalid request",
   "endpoints": [
      "/api/v1"
   ]
}
  • --user 'None:None' - gives BlendNet HTTP Basic Auth user:password parameters, by default it's 'None' for user and 'None' for password which is set during the Manager or the Agent configuration.
  • --insecure - means the certificate of the service will not be checked. Of course you can put --cacert ca.crt instead, if you properly configured the environment and the whole certificate structure to validate the certificate signature.
  • https://localhost:8443/ - is URL protocol, address and port. Also it could contain path like in the next command:

API possible requests

$ curl --user 'None:None' --insecure https://localhost:8443/api/v1
{"success": false, "message": "Invalid request", "endpoints": {"get": {"agent": {"*": {"log": {"?": "Returns the information about the task"}}}, "info": {"?": "Return information about the server system"}, "log": {"?": "Return the captured log"}, "resources": {"?": "Returns the available resources"}, "status": {"?": "Returns the current status of the server"}, "task": {"*": {"?": "Returns the information about the task", "details": {"?": "Return detailed execution information about the task"}, "file": {"**": {"?": "Returns the information about the task file"}, "?": "Returns the information about the task file list"}, "messages": {"?": "Return execution messages of the task"}, "status": {"result": {"*": {"?": "Streams the task result image for preview or render"}}, "?": "Return execution status information of the task"}, "run": {"?": "Mark task as ready to be executed"}, "stop": {"?": "Stop the task execution"}}, "?": "Returns list of existing tasks on the server"}}, "delete": {"agent": {"*": {"?": "Remove the custom agent from the pool"}}, "task": {"*": {"?": "Remove not running task from the task list"}}}, "put": {"agent": {"*": {"config": {"?": "Set the custom agent configuration as json (max 128KB)"}}}, "task": {"*": {"file": {"**": {"?": "Upload file required to execute task"}}, "config": {"?": "Set the configuration of task as json (max 512KB)"}}}}}}

Easier to read in yaml:

success: false
message: Invalid request
endpoints:
  get:
    agent:
      "*":
        log:
          "?": Returns the information about the task
      "?": List the available custom agents
    info:
      "?": Return information about the server system
    log:
      "?": Return the captured log
    status:
      "?": Returns the current status of the server
    task:
      "*":
        "?": Returns the information about the task
        details:
          "?": Return detailed execution information about the task
        file:
          "**":
            "?": Returns the information about the task file
          "?": Returns the information about the task file list
        messages:
          "?": Return execution messages of the task
        status:
          result:
            "*":
              "?": Streams the task result image for preview or render
          "?": Return execution status information of the task
        run:
          "?": Mark task as ready to be executed
        stop:
          "?": Stop the task execution
      "?": Returns list of existing tasks on the server
  delete:
    agent:
      "*":
        "?": Remove the custom agent from the pool
    task:
      "*":
        "?": Remove not running task from the task list
  put:
    agent:
      "*":
        config:
          "?": Set the custom agent configuration as json (max 512KB)
    task:
      "*":
        file:
          "**":
            "?": Upload file required to execute task
        config:
          "?": Set the configuration of task as json (max 512KB)

Getting the service status

Status is a simple and common request for both Manager and Agent:

$ curl --user 'None:None' --insecure https://localhost:8443/api/v1/status
{"success": true, "data": {"load": [4.4, 3.76, 3.4], "memory": {"MemTotal": 128808.98828125, "MemFree": 39758.55078125, "MemAvailable": 98616.375}, "disk": {"total": 78124.0, "available": 23295.44140625}, "running": ["testproject-2009020050-32-SQT"], "terminating": false}}

Error messages

It will also give you quite useful messages if you forgot to provide some parameters to the API call:

$ curl --user 'None:None' --insecure https://localhost:8443/api/v1/agent/notexisting-agent/log
{"success": false, "message": "Unable to find agent"}

But overall it's easier to check the processors to understand what is required:

Upload and run the task

And if you want to upload the blend file with dependencies to the Manager and run the frame render task:

$ for f in blendcache_*/*000323* tex/* test-project.blend; do
    curl -u 'None:None' --insecure \
      --header "X-Checksum-Sha1:$(sha1sum "${f}" | cut -d ' ' -f 1)" \
      --upload-file "${f}" "https://localhost:8443/api/v1/task/mytask-1/file/${f}"
  done
$ curl -u 'None:None' --insecure -X PUT \
    -d '{"samples": 1000, "project": "test-project.blend", "frame": 323}' \
    "https://localhost:8443/api/v1/task/mytask-1/config"
$ curl -u 'None:None' --insecure "https://localhost:8443/api/v1/task/mytask-1/run"

NOTE: It's mandatory to use only relative paths - so make sure all your deps are in the same working directory as the main project blend file.

Getting the task status

To check how it's going - you can request the status of a specific task:

$ curl -u 'None:None' --insecure "https://localhost:8443/api/v1/task/mytask-1/status
{"success": true, "message": "Got task status info", "data": {
  "name": "mytask-1",
  "create_time": 1599206103,
  "start_time": 1599206106,
  "end_time": 1599206175,
  "state": "COMPLETED",
  "done": 1.0,
  "project": "test-project.blend",
  "samples": 50,
  "seed": 686295367,
  "frame": 32,
  "samples_done": 50,
  "remaining": 0,
  "result": {
    "preview": "6770d2c7cd2145d05c5e473e82777367034ad817",
    "render": "02252f2b12cf8da6f452e2a34b7fe48d9172dc5c",
    "compose": "58cac7ffd28473184af611a09885592eb0344ae4"
  },
  "start_time_actual": 1599206107,
  "samples_per_workload": 10,
  "samples_acquired": 50,
  "workloads_taken": 5,
  "results_processing": false,
  "compose_filepath": "//out.png"
}}

Here you can see how much frames left in remaining, percentage (0.0-1.0) in done and state will help to understand it's current stage listed in enum here.

Receiving the task results

You can download the results pictures - just request the result streaming api and save the output. Result contains the next items:

  • preview is loose (DWAA) compressed single-layer EXR: 
    $ curl -u 'None:None' --insecure -o mytask-1-render.exr "https://localhost:8443/api/v1/task/mytask-1/status/result/preview
$ file mytask-1-preview.exr
mytask-1-preview.exr: OpenEXR image data, version 2, storage: scanline, compression: unknown, dataWindow: (0 0)-(1919 1079), displayWindow: (0 0)-(1919 1079), lineOrder: increasing y
  • render in multilayer looseless (ZIP) compressed EXR: 
    $ curl -u 'None:None' --insecure -o mytask-1-render.exr "https://localhost:8443/api/v1/task/mytask-1/status/result/render
$ file mytask-1-render.exr
mytask-1-render.exr: OpenEXR image data, version 2, storage: scanline, compression: zip, dataWindow: (0 0)-(1919 1079), displayWindow: (0 0)-(1919 1079), lineOrder: increasing y
  • composite saved in the user-defined output format: 
    $ curl -u 'None:None' --insecure -o mytask-1-composite.png "https://localhost:8443/api/v1/task/mytask-1/status/result/composite
$ file mytask-1-composite.png
mytask-1-composite.png: PNG image data, 1920 x 1080, 8-bit/color RGBA, non-interlaced

As you can see - you can do almost anything and automate your render farm as much as you need with using the BlendNet API.

Clone this wiki locally