v0.19.0

🚀 Added

🎥 Video processing in workflows 🤯

We’re excited to announce that, thanks to the contributions of @grzegorz-roboflow, our Workflows ecosystem now extends to video processing! Dive in and explore the new possibilities:

dwell_time_demo.mp4

New blocks:

• Time In Zone to analyse dwell time
• Line Counter to detect objects passing line
• Visualisation for zone and line counter

We've introduced minimal support for video processing in the Workflows UI, with plans to expand to more advanced features soon. To get started, you can create a Python script using the InferencePipeline, similar to the provided example.

Video source YT | Karol Majek

🔥 OWLv2 🤝 inference

Thanks to @probicheaux we have OWLv2 model in inference. OWLv2 was primarily trained to detect objects from text. The implementation in Inference currently only supports detecting objects from visual examples of that object.

You can use model in inference server - both CPU and GPU, as well as in Python package. Visit our 📖 docs to learn more.

Screen.Recording.2024-09-19.at.21.36.13.mov

👓 TROCR 🤝 inference

@stellasphere shipped TROCR model to expand OCR models offering in inference 🔥

You can use model in inference server - both CPU and GPU, as well as in Python package. Visit our 📖 docs to learn more.

🧑‍🎓 Workflows - endpoint to discover interface

Guessing the data format for Workflow inputs and outputs was a challange as for now, but thanks to @EmilyGavrilenko this is no longer the case. We offer two new endpoints (for workflows registered on the platform and for workflows submitted in payload). Details in #644.

🔔 Example response

{
    "inputs": {
        "image": ["image"],
        "model_id": ["roboflow_model_id"],
    },
    "outputs": {
        "detections": ["object_detection_prediction"],
         "crops": ["image"],
         "classification": {
            "inference_id": ["string"],
            "predictions": ["classification_prediction"],
         },
    },
    "typing_hints": {
        "image": "dict",
        "roboflow_model_id": "str",
        "object_detection_prediction": "dict",
        "string": "str",
        "classification_prediction": "dict",
    },
    "kinds_schemas": {
        "image": {},
        "object_detection_prediction": {"dict": "with OpenAPI 3.0 schema of result"},
        "classification_prediction": {"dict": "with OpenAPI 3.0 schema of result"}
    }
}

🌱 Changed

• Bring back transformers extras by @probicheaux in #639
• Move xformers to paligemma code by @probicheaux in #641
• Cache plan details by @grzegorz-roboflow in #636
• Bump next from 14.2.7 to 14.2.12 in /inference/landing by @dependabot in #649

🔧 Fixed

• Fixed bug with Workflows Execution Engine causing bug when conditional execution discards inputs of a step that changes dimensionality - see details in #645

♻️ Removed

• Remove unmaintained device management code by @robiscoding in #647

Full Changelog: v0.18.1...v0.19.0

v0.18.1

🔨 Fixed

New VLM as Classifier Workflows block had bug - multi-label classification results were generated with "class_name" instead of "class" field in prediction details: #637

🌱 Changed

• Increase timeout to 30 minutes for .github/workflows/test_package_install_inference_with_extras.yml by @grzegorz-roboflow in #635

Full Changelog: v0.18.0...v0.18.1

v0.18.0

🚀 Added

💪 New VLMs in Workflows

We've shipped blocks to integrate with Google Gemini and Anthropic Claude, but that's not everything! OpenAI block got updated. New "VLM Interface" of the block assumes that it can be prompted using pre-configured options and model output can be processed by set of formatter blocs to achieve desired end. It is now possible to:

• use classification prompting in VLM block and apply VLM as Classifier block to turn output string into classification result and process further using other blocks from ecosystem
• the same can be achieved for object-detection prompting and VLM as Detector block, which converts text produced by model into sv.Detections(...)

From now one, VLMs are much easier to integrate.

🧑‍🦱 USE CASE: PII protection when prompting VLM

Detect faces first, apply blur prediction visualisation and ask VLMs to tell what is the person eye colour - they won't be able to tell 🙃

👨‍🎨 USE CASE: VLM as object detection model

👓 USE CASE: VLM as secondary classifier

Turn VLM output into classification results and process using downstream blocks - here we ask Gemini to classify crops of dogs to tell what is dog's breed - then we extract top class as property.

🤯 Workflows previews in documentation 📖

Thanks to @joaomarcoscrs we can embed Workflows into documentation pages. Just take a look how amazing it is ❗

🌱 Changed

• E2E tests for workflows on hosted platform by @PawelPeczek-Roboflow in #622
• Allow model type "yolov8" without size by @SolomonLake in #627
• Fix GDAL issue in GHA by @PawelPeczek-Roboflow in #628
• Add support for DELETE in sqlite wrapper by @grzegorz-roboflow in #631
• Keep distinct exec sessions for inf pipeline usage by @grzegorz-roboflow in #632
• Insert usage payloads into redis and to sorted set atomically by @grzegorz-roboflow in #633
• remove height from workflow example by @capjamesg in #634

❗ BREAKING ❗ Batch[X] kinds removed from Workflows

What was changed and why?

In inference release 0.18.0 we decided to make drastic move to heal the ecosystem from the problem with ambiguous kinds names (Batch[X] vs X - see more here).

The change is breaking only for non-Roboflow Workflow plugins depending on imports from inference.core.workflows.execution_engine.entities.types module. To the best of our knowledge, there is no such plugin.

The change is not breaking in terms of running Workflows on Roboflow platform and on-prem given that external plugins were not used.

Migration guide

Migration should be relatively easy - in the code of a Workflow block, all instances of

from inference.core.workflows.execution_engine.entities.types import BATCH_OF_{{KIND_NAME}}

should be replaced with

from inference.core.workflows.execution_engine.entities.types import {{KIND_NAME}}

PR with changes as reference: #618

Full Changelog: v0.17.1...v0.18.0

v0.17.1

❗IMPORTANT ❗Security issue in opencv-python

This PR provides fix for the following security issue:

opencv-python versions before v4.8.1.78 bundled libwebp binaries in wheels that are vulnerable to GHSA-j7hp-h8jx-5ppr. opencv-python v4.8.1.78 upgrades the bundled libwebp binary to v1.3.2.

We advise all clients using inference to migrate, especially in production environments.

Full Changelog: v0.17.0...v0.17.1

v0.17.0

🚀 Added

💪 More Classical Computer Vision blocks in workflows

Good news for the fans of classical computer vision!
We heard you – and we’ve added a bunch of new blocks to enhance your workflows.

Basic operations on images

Workflow Definition	Preview

Camera focus check

Workflow Definition	Preview

🚀 Upgrade of CLIP Comparison and Roboflow Dataset Upload blocks

We’ve made it even more versatile. The new outputs allow seamless integration with many other blocks, enabling powerful workflows like:

detection → crop → CLIP classification (on crops) → detection class replacement

Get ready to streamline your processes with enhanced compatibility and new possibilities!

For Roboflow Dataset Upload @ v2 there is now possibility to sample percentage of data to upload and we changed the default sizes of saved images to be bigger.

❗ Do not worry! All your old Workflows using mentioned blocks are not affected with the change thanks to versioning 😄

💥 New version of 📖 Workflow docs 🔥

The Wait is Over – Our Workflows Documentation is Finally Here!

We’ve revamped and expanded the documentation to make your experience smoother. It’s now organized into three clear sections:

• General Overview: Perfect for getting you up and running quickly.
• Mid-Level User Guide: Gain a solid understanding of the ecosystem without diving too deep into the technical details.
• Detailed Developer Guide: Designed for contributors, packed with everything you need to develop within the ecosystem.

Check it out and let us know what you think of the new docs!

🌱 Changed

• Record resource details in each usage payload by @grzegorz-roboflow in #607
• json.dumps resource_details when adding to usage payload by @grzegorz-roboflow in #610
• enhancement in CLIP docs by @venkatram-dev in #599 - thanks for contribution 🏅
• Sam2 multi polygons by @probicheaux in #593
• Abstract sqlite3 wrapper in usage collector sqlite queue by @grzegorz-roboflow in #619

🔨 Fixed

• Dynamic Crop block was buggy in some contexts - see details: #604
• Bug in integration tests by @PawelPeczek-Roboflow in #600
• Bugfix sam2 logits cache and add test by @probicheaux in #606
• Fix bug with detections offset and cover problem with additional test by @PawelPeczek-Roboflow in #611
• only warn about version when its lower than latest release by @hansent in #609
• Add inference_ids to model blocks by @robiscoding in #615
• Fix Detection Offset Bug by @NickHerrig in #621
• Bump micromatch from 4.0.5 to 4.0.8 in /inference/landing by @dependabot in #617

🏅 New Contributors

• @reedajohns made their first contribution in #602
• @venkatram-dev made their first contribution in #599

Full Changelog: v0.16.3...v0.17.0

v0.16.3

🔨 Fixed

• Roboflow Dataset Upload block now can be dynamically configured with labelling batch prefixes passed as workflow inputs. Additionally predictions parameter is now optional - which means that you can upload images without predictions in your workflows - all thanks to @NickHerrig #586
• @chandlersupple fixed Perspective Correction Block in #589
• @robiscoding added missing inference_id for classification responses in #591
• @PacificDou in #592 introduced changes to inference server routing that were required to enhance Dedicated Deployment functionality
• Added limits for execution time of GHA in #596

🚀 Added

SAM2 extension

While making inference from SAM2 model you may request inference package and inference server to cache prompts and low-resolution masks from your inputs to be re-used later on upon request. You are given two parameters (both in SAM2 request payload and SegmentAnything2.segment_image(...) method:

• save_logits_to_cache
• load_logits_from_cache
  which decide how the functionality should work. Saving logits masks to cache will make it possible, to re-use them for consecutive inferences agains the same image. Enabling loading triggers search through cache intended to find the most similar prompt cached for this specific image to retrieve its mask. The mechanism is useful when the same image is segmented multiple times with slightly different sets of prompts - as injecting previous masks in that scenario may lead to better results:

Before	After

Please note that this feature is different than cache for image embeddings which speed consecutive requests with the same image up and if you don't wish the feature to be enabled, set DISABLE_SAM2_LOGITS_CACHE=True in your env.

🏅 @probicheaux and @tonylampada added the functionality in #582

Remaining changes

• @EmilyGavrilenko added Workflow block search metadata to improve UI experience in #588
• @grzegorz-roboflow added internal parameter for workflows request denoting preview in UI #595
• @grzegorz-roboflow improved usage tracking extending it to models in #601 and #548
• workflows equipped with new batch-oriented input - VideoFrameMetadata letting blocks to process videos statefully see #590, #597 more docs will come soon

Full Changelog: v0.16.2...v0.16.3

v0.16.2

🚀 Added

Segment Anything 2 in workflows 🥳

We prepared great amount of changes to workflows, could not really decide which update to start with, but at the end we found onboarding of SAM 2 model as most exciting.

Thanks to @hansent effort we have introduced SAM 2 workflow block.

You can use SAM2 standalone, or you can ground it's predictions with other detection models which is the true power of workflows. Thanks to grounding, you can generate instance segmentation masks for each bounding box predicted by your object detection model

❗ We do not support SAM2 at Roboflow Hosted Platform yet, but it is possible to use inference server start command to run local server supporting SAM2 model and connect it to workflows UI to run examples.

workflows 🤝 SAHI

We've added set of blocks that let people apply SAHI technique based on utilities provided by supervision.

We are going to work to simplify SAHI usage in workflows UI, but now you need to use three blocks to effectively apply technique:

Classical Computer Vision methods in workflows 🔥

We do not forget about old good friends - that's why we also added bunch of blocks with classical Computer Vision algorithms:

• Dominant Color block by @NickHerrig in #578
• SIFT, SIFT matching, classical pattern matching and others by @ryanjball in #581

🌱 Changed

• Added encoding='utf-8' to setup.py by @Bhavay-2001 in #556
• Move landing static assets to /static/ namespace by @iurisilvio in #577
• add exclusion rules for dedicated deployment authorizer by @PacificDou in #576
• Workflow private block properties by @EmilyGavrilenko in #579

🔨 Fixed

• Fix security issues with landing page by @PawelPeczek-Roboflow in #584
• Fixed a Regression in the Custom Metadata Block that was introduced in v0.16.0 - PR with fix by @chandlersupple (#573) - we kindly ask clients relying on the Custom Metadata Block, running their workflows on-prem to update inference or inference server
• Bug in workflows Execution Engine that was making it impossible to feed the same block with two identical selectors (fixed in #581)

❗ In release 0.16.0 we introduced bug impacting workflows and inference_sdk

The mistake was introduced in #565 and fixed in #585 (both by @PawelPeczek-Roboflow 😢 ) and was causing issues with order of results - regarding specific workflows blocks:

• blocks with Roboflow models, whenever used with batch input (for instance when workflow was run against multiple images, or Dynamic Crop was used) were mismatching order of predictions with respect to order of images
• the same was true for OpenAI block and GPT-4V block
• the problem was also introduced into inference_sdk, so whenever client was called with multiple images - results may have been missmatched

We advice all our clients to upgrade to new release and abandon usage inference=0.16.0

🏅 New Contributors

• @Bhavay-2001 made their first contribution in #556

Full Changelog: v0.16.0...v0.16.1

v0.16.0

❗ In release 0.16.0 we introduced bug impacting workflows and inference_sdk

The mistake was introduced in #565 and fixed in #585 (both by @PawelPeczek-Roboflow 😢 ) and was causing issues with order of results - regarding specific workflows blocks:

• blocks with Roboflow models, whenever used with batch input (for instance when workflow was run against multiple images, or Dynamic Crop was used) were mismatching order of predictions with respect to order of images
• the same was true for OpenAI block and GPT-4V block
• the problem was also introduced into inference_sdk, so whenever client was called with multiple images - results may have been missmatched

🚀 Added

Next bunch of updates for workflows 🥳

⚓ Versioning

From now on, both Execution Engine and workflows blocks are versioned to ensure greater stability across changes we make to improve ecosystem. Each workflow definition now declares version forcing the app to run against specific version of Execution Engine. If denoted version is 1.1.0, then workflow would require Execution Engine >=1.1.0,<2.0.0 and we gain ability to expose concurrently multiple major versions of EE in the library (doing our best to ensure that within a major version we only add features and support everything that was released earlier within the same major). On top of that:

• block manifest metadata field name now will be understood as name of blocks family with additional tag called version possible to be added; we propose the following naming conventions for block names: namespace/family_name@v1. Thanks to those changes anyone could maintain multiple versions of the same block (appending new implementation to their plugin) ensuring backwards compatibilities on breaking changes
• each block manifest class may optionally expose class method get_execution_engine_compatibility(...) which would be used while model loading to ensure that selected Execution Engine is capable to run specific block

✋ Example block manifest

class BlockManifest(WorkflowBlockManifest):
    model_config = ConfigDict(
        json_schema_extra={
            "name": "My Block",
            "version": "v1",
            ...
        }
    )
    type: Literal["my_namespace/mu_block@v1"]
    ...

    @classmethod
    def get_execution_engine_compatibility(cls) -> Optional[str]:
        return ">=1.0.0,<2.0.0"

🚨 ⚠️ BREAKING ⚠️ 🚨 Got rid of asyncio in Execution Engine

If you were tired of coroutines performing compute heavy tasks in workflows:

class MyBlock(WorkflowBlock):
    async def run():
        pass

we have great news. We've got rid of asyncio in favour of standard functions and methods which are much more intuitive in our setup. This change is obviously breaking all other steps, but worry not. Here is the example of what needs to be changed - usually you just need to remove async markers, but sometimes unfortunately pieces of asyncio code would need to be recreated.

class MyBlock(WorkflowBlock):
    def run():
        pass

Endpoint to expose workflow definition schema

Thanks to @EmilyGavrilenko (#550) UI would now be able to verify syntax errors in workflows definitions automatically.

Roboflow Dedicated Deployment is closer and closer 😃

Thanks to @PacificDou, inference server is getting ready to support new functionality which has a nickname Dedicated Deployment. Stay tuned to learn more details - we can tell that this is something worth waiting for. You may find some hints in the PR.

🔨 Fixed

🚨 ⚠️ BREAKING ⚠️ 🚨 HTTP client of inference server changes default behaviour

The default value for flag client_downsizing_disabled was changed from False to True in release 0.16.0! For clients using models with input size above 1024x1024, running models on hosted platform it should improve predictions quality (as previous default behaviour was causing that input was downsized and then artificially upsized on the server side with worse image quality). There may be some clients that would like to remain previous settings to potentially improve speed (when internet connection is a bottleneck and large images are submitted despite small model input size).

If you liked the previous behaviour more - simply:

from inference_sdk import InferenceHTTPClient, InferenceConfiguration

client = InferenceHTTPClient(
    "https://detect.roboflow.com",
    api_key="XXX",
).configure(InferenceConfiguration(
    client_downsizing_disabled=False,
))

setuptools were migrated to version above 70.0.0 to mitigate security issue

We've updated rf-clip package to support setuptools>70.0.0 and bumped the version on inference side.

🌱 Changed

• 📖 Add documentation for ONNXRUNTIME_EXECUTION_PROVIDERS by @grzegorz-roboflow in #562 - see here
• 📖 Update docs for easier quickstart by @komyg in #544
• 📖 Add Inference Windows CUDA documentation by @capjamesg in #502
• Add @capjamesg to CODEOWNERS by @capjamesg in #564
• Add persistent queue to usage collector by @grzegorz-roboflow in #568

🏅 New Contributors

• @komyg made their first contribution in #544

Full Changelog: v0.15.2...v0.16.0

v0.15.2

What's Changed

• Separate the LMM block into OpenAI and CogVLM by @EmilyGavrilenko in #549
• Do not log warning when usage payload is put back into the queue by @grzegorz-roboflow in #551
• Initialize usage_collector._async_lock only if async look can be obtained by @grzegorz-roboflow in #552
• Cache workflow specification for offline use by @sberan in #555
• Custom Metadata block for Model Monitoring users by @robiscoding in #553
• Allow zone defined as input parameter to be passed to perspective_correction by @grzegorz-roboflow in #559
• Increment num_errors only if pingback initialized by @iurisilvio in #527
• SAM2 by @hansent and @probicheaux in #557

Full Changelog: v0.15.1...v0.15.2

v0.15.1

What's Changed

• Refactor Visualization Workflow Block Inheritance by @yeldarby in #538
• Add florence2 aliases and bugfix TransformersModel by @probicheaux in #525
• Add BackgroundColorAnnotator block by @capjamesg in #542
• Bugfix: keypoint detection model block by @EmilyGavrilenko in #543

Full Changelog: v0.15.0...v0.15.1