Releases: roboflow/inference
v0.19.0
🚀 Added
🎥 Video processing in workflows
🤯
We’re excited to announce that, thanks to the contributions of @grzegorz-roboflow, our Workflows ecosystem now extends to video processing! Dive in and explore the new possibilities:
dwell_time_demo.mp4
New blocks:
- Time In Zone to analyse dwell time
- Line Counter to detect objects passing line
- Visualisation for zone and line counter
We've introduced minimal support for video processing in the Workflows UI, with plans to expand to more advanced features soon. To get started, you can create a Python script using the InferencePipeline, similar to the provided example.
Video source YT | Karol Majek
🔥 OWLv2 🤝 inference
Thanks to @probicheaux we have OWLv2 model in inference
. OWLv2 was primarily trained to detect objects from text. The implementation in Inference currently only supports detecting objects from visual examples of that object.
You can use model in inference
server - both CPU and GPU, as well as in Python package. Visit our 📖 docs to learn more.
Screen.Recording.2024-09-19.at.21.36.13.mov
👓 TROCR 🤝 inference
@stellasphere shipped TROCR model to expand OCR models offering in inference
🔥
You can use model in inference
server - both CPU and GPU, as well as in Python package. Visit our 📖 docs to learn more.
🧑🎓 Workflows - endpoint to discover interface
Guessing the data format for Workflow inputs and outputs was a challange as for now, but thanks to @EmilyGavrilenko this is no longer the case. We offer two new endpoints (for workflows registered on the platform and for workflows submitted in payload). Details in #644.
🔔 Example response
{
"inputs": {
"image": ["image"],
"model_id": ["roboflow_model_id"],
},
"outputs": {
"detections": ["object_detection_prediction"],
"crops": ["image"],
"classification": {
"inference_id": ["string"],
"predictions": ["classification_prediction"],
},
},
"typing_hints": {
"image": "dict",
"roboflow_model_id": "str",
"object_detection_prediction": "dict",
"string": "str",
"classification_prediction": "dict",
},
"kinds_schemas": {
"image": {},
"object_detection_prediction": {"dict": "with OpenAPI 3.0 schema of result"},
"classification_prediction": {"dict": "with OpenAPI 3.0 schema of result"}
}
}
🌱 Changed
- Bring back transformers extras by @probicheaux in #639
- Move xformers to paligemma code by @probicheaux in #641
- Cache plan details by @grzegorz-roboflow in #636
- Bump next from 14.2.7 to 14.2.12 in /inference/landing by @dependabot in #649
🔧 Fixed
- Fixed bug with Workflows Execution Engine causing bug when conditional execution discards inputs of a step that changes dimensionality - see details in #645
♻️ Removed
- Remove unmaintained device management code by @robiscoding in #647
Full Changelog: v0.18.1...v0.19.0
v0.18.1
🔨 Fixed
New VLM as Classifier Workflows
block had bug - multi-label classification results were generated with "class_name" instead of "class" field in prediction details: #637
🌱 Changed
- Increase timeout to 30 minutes for .github/workflows/test_package_install_inference_with_extras.yml by @grzegorz-roboflow in #635
Full Changelog: v0.18.0...v0.18.1
v0.18.0
🚀 Added
💪 New VLMs in Workflows
We've shipped blocks to integrate with Google Gemini and Anthropic Claude, but that's not everything! OpenAI block got updated. New "VLM Interface" of the block assumes that it can be prompted using pre-configured options and model output can be processed by set of formatter blocs to achieve desired end. It is now possible to:
- use
classification
prompting in VLM block and applyVLM as Classifier
block to turn output string into classification result and process further using other blocks from ecosystem - the same can be achieved for
object-detection
prompting andVLM as Detector
block, which converts text produced by model intosv.Detections(...)
From now one, VLMs are much easier to integrate.
🧑🦱 USE CASE: PII protection when prompting VLM
Detect faces first, apply blur prediction visualisation and ask VLMs to tell what is the person eye colour - they won't be able to tell 🙃
![]() |
![]() |
👨🎨 USE CASE: VLM as object detection model
![]() |
![]() |
👓 USE CASE: VLM as secondary classifier
Turn VLM output into classification results and process using downstream blocks - here we ask Gemini to classify crops of dogs to tell what is dog's breed - then we extract top class as property.
🤯 Workflows
previews in documentation 📖
Thanks to @joaomarcoscrs we can embed Workflows into documentation pages. Just take a look how amazing it is ❗
🌱 Changed
- E2E tests for
workflows
on hosted platform by @PawelPeczek-Roboflow in #622 - Allow model type "yolov8" without size by @SolomonLake in #627
- Fix
GDAL
issue in GHA by @PawelPeczek-Roboflow in #628 - Add support for DELETE in sqlite wrapper by @grzegorz-roboflow in #631
- Keep distinct exec sessions for inf pipeline usage by @grzegorz-roboflow in #632
- Insert usage payloads into redis and to sorted set atomically by @grzegorz-roboflow in #633
- remove height from workflow example by @capjamesg in #634
❗ BREAKING ❗ Batch[X]
kinds removed from Workflows
What was changed and why?
In inference
release 0.18.0
we decided to make drastic move to heal the ecosystem from the problem with ambiguous kinds names (Batch[X]
vs X
- see more here).
The change is breaking only for non-Roboflow Workflow plugins depending on imports from inference.core.workflows.execution_engine.entities.types
module. To the best of our knowledge, there is no such plugin.
The change is not breaking in terms of running Workflows on Roboflow platform and on-prem given that external plugins were not used.
Migration guide
Migration should be relatively easy - in the code of a Workflow block, all instances of
from inference.core.workflows.execution_engine.entities.types import BATCH_OF_{{KIND_NAME}}
should be replaced with
from inference.core.workflows.execution_engine.entities.types import {{KIND_NAME}}
PR with changes as reference: #618
Full Changelog: v0.17.1...v0.18.0
v0.17.1
❗IMPORTANT ❗Security issue in opencv-python
This PR provides fix for the following security issue:
opencv-python versions before v4.8.1.78 bundled libwebp binaries in wheels that are vulnerable to GHSA-j7hp-h8jx-5ppr. opencv-python v4.8.1.78 upgrades the bundled libwebp binary to v1.3.2.
We advise all clients using inference
to migrate, especially in production environments.
Full Changelog: v0.17.0...v0.17.1
v0.17.0
🚀 Added
💪 More Classical Computer Vision blocks in workflows
Good news for the fans of classical computer vision!
We heard you – and we’ve added a bunch of new blocks to enhance your workflows.
Basic operations on images
Workflow Definition | Preview |
---|---|
![]() |
![]() |
Camera focus check
Workflow Definition | Preview |
---|---|
![]() |
![]() |
🚀 Upgrade of CLIP Comparison
and Roboflow Dataset Upload
blocks
We’ve made it even more versatile. The new outputs allow seamless integration with many other blocks, enabling powerful workflows like:
detection → crop → CLIP classification (on crops) → detection class replacement
Get ready to streamline your processes with enhanced compatibility and new possibilities!
For Roboflow Dataset Upload @ v2
there is now possibility to sample percentage of data to upload and we changed the default sizes of saved images to be bigger.
❗ Do not worry! All your old Workflows using mentioned blocks are not affected with the change thanks to versioning 😄
💥 New version of 📖 Workflow docs 🔥
The Wait is Over – Our Workflows Documentation is Finally Here!
We’ve revamped and expanded the documentation to make your experience smoother. It’s now organized into three clear sections:
- General Overview: Perfect for getting you up and running quickly.
- Mid-Level User Guide: Gain a solid understanding of the ecosystem without diving too deep into the technical details.
- Detailed Developer Guide: Designed for contributors, packed with everything you need to develop within the ecosystem.
Check it out and let us know what you think of the new docs!
🌱 Changed
- Record resource details in each usage payload by @grzegorz-roboflow in #607
- json.dumps resource_details when adding to usage payload by @grzegorz-roboflow in #610
- enhancement in CLIP docs by @venkatram-dev in #599 - thanks for contribution 🏅
- Sam2 multi polygons by @probicheaux in #593
- Abstract sqlite3 wrapper in usage collector sqlite queue by @grzegorz-roboflow in #619
🔨 Fixed
- Dynamic Crop block was buggy in some contexts - see details: #604
- Bug in integration tests by @PawelPeczek-Roboflow in #600
- Bugfix sam2 logits cache and add test by @probicheaux in #606
- Fix bug with detections offset and cover problem with additional test by @PawelPeczek-Roboflow in #611
- only warn about version when its lower than latest release by @hansent in #609
- Add inference_ids to model blocks by @robiscoding in #615
- Fix Detection Offset Bug by @NickHerrig in #621
- Bump micromatch from 4.0.5 to 4.0.8 in /inference/landing by @dependabot in #617
🏅 New Contributors
- @reedajohns made their first contribution in #602
- @venkatram-dev made their first contribution in #599
Full Changelog: v0.16.3...v0.17.0
v0.16.3
🔨 Fixed
- Roboflow Dataset Upload block now can be dynamically configured with labelling batch prefixes passed as workflow inputs. Additionally
predictions
parameter is now optional - which means that you can upload images without predictions in your workflows - all thanks to @NickHerrig #586 - @chandlersupple fixed Perspective Correction Block in #589
- @robiscoding added missing
inference_id
for classification responses in #591 - @PacificDou in #592 introduced changes to
inference
server routing that were required to enhance Dedicated Deployment functionality - Added limits for execution time of GHA in #596
🚀 Added
SAM2 extension
While making inference from SAM2 model you may request inference
package and inference
server to cache prompts and low-resolution masks from your inputs to be re-used later on upon request. You are given two parameters (both in SAM2 request payload and SegmentAnything2.segment_image(...)
method:
save_logits_to_cache
load_logits_from_cache
which decide how the functionality should work. Saving logits masks to cache will make it possible, to re-use them for consecutive inferences agains the same image. Enabling loading triggers search through cache intended to find the most similar prompt cached for this specific image to retrieve its mask. The mechanism is useful when the same image is segmented multiple times with slightly different sets of prompts - as injecting previous masks in that scenario may lead to better results:
Before | After |
---|---|
![]() |
![]() |
Please note that this feature is different than cache for image embeddings which speed consecutive requests with the same image up and if you don't wish the feature to be enabled, set DISABLE_SAM2_LOGITS_CACHE=True
in your env.
🏅 @probicheaux and @tonylampada added the functionality in #582
Remaining changes
- @EmilyGavrilenko added Workflow block search metadata to improve UI experience in #588
- @grzegorz-roboflow added internal parameter for workflows request denoting preview in UI #595
- @grzegorz-roboflow improved usage tracking extending it to models in #601 and #548
workflows
equipped with new batch-oriented input -VideoFrameMetadata
letting blocks to process videos statefully see #590, #597 more docs will come soon
Full Changelog: v0.16.2...v0.16.3
v0.16.2
🚀 Added
Segment Anything 2 in workflows
🥳
We prepared great amount of changes to workflows
, could not really decide which update to start with, but at the end we found onboarding of SAM 2 model as most exciting.
Thanks to @hansent effort we have introduced SAM 2 workflow
block.
You can use SAM2 standalone, or you can ground it's predictions with other detection models which is the true power of workflows
. Thanks to grounding, you can generate instance segmentation masks for each bounding box predicted by your object detection model
❗ We do not support SAM2 at Roboflow Hosted Platform yet, but it is possible to use inference server start
command to run local server supporting SAM2 model and connect it to workflows
UI to run examples.
workflows
🤝 SAHI
We've added set of blocks that let people apply SAHI technique based on utilities provided by supervision
.
We are going to work to simplify SAHI usage in workflows
UI, but now you need to use three blocks to effectively apply technique:
Classical Computer Vision methods in workflows
🔥
We do not forget about old good friends - that's why we also added bunch of blocks with classical Computer Vision algorithms:
- Dominant Color block by @NickHerrig in #578
- SIFT, SIFT matching, classical pattern matching and others by @ryanjball in #581
🌱 Changed
- Added encoding='utf-8' to setup.py by @Bhavay-2001 in #556
- Move landing static assets to
/static/
namespace by @iurisilvio in #577 - add exclusion rules for dedicated deployment authorizer by @PacificDou in #576
- Workflow private block properties by @EmilyGavrilenko in #579
🔨 Fixed
- Fix security issues with landing page by @PawelPeczek-Roboflow in #584
- Fixed a Regression in the Custom Metadata Block that was introduced in
v0.16.0
- PR with fix by @chandlersupple (#573) - we kindly ask clients relying on the Custom Metadata Block, running their workflows on-prem to updateinference
orinference
server - Bug in
workflows
Execution Engine that was making it impossible to feed the same block with two identical selectors (fixed in #581)
❗ In release 0.16.0
we introduced bug impacting workflows
and inference_sdk
The mistake was introduced in #565 and fixed in #585 (both by @PawelPeczek-Roboflow 😢 ) and was causing issues with order of results - regarding specific workflows
blocks:
- blocks with Roboflow models, whenever used with batch input (for instance when workflow was run against multiple images, or Dynamic Crop was used) were mismatching order of predictions with respect to order of images
- the same was true for OpenAI block and GPT-4V block
- the problem was also introduced into
inference_sdk
, so whenever client was called with multiple images - results may have been missmatched
We advice all our clients to upgrade to new release and abandon usage inference=0.16.0
🏅 New Contributors
- @Bhavay-2001 made their first contribution in #556
Full Changelog: v0.16.0...v0.16.1
v0.16.0
❗ In release 0.16.0
we introduced bug impacting workflows
and inference_sdk
The mistake was introduced in #565 and fixed in #585 (both by @PawelPeczek-Roboflow 😢 ) and was causing issues with order of results - regarding specific workflows
blocks:
- blocks with Roboflow models, whenever used with batch input (for instance when workflow was run against multiple images, or Dynamic Crop was used) were mismatching order of predictions with respect to order of images
- the same was true for OpenAI block and GPT-4V block
- the problem was also introduced into
inference_sdk
, so whenever client was called with multiple images - results may have been missmatched
🚀 Added
Next bunch of updates for workflows
🥳
⚓ Versioning
From now on, both Execution Engine and workflows
blocks are versioned to ensure greater stability across changes we make to improve ecosystem. Each workflow definition now declares version
forcing the app to run against specific version of Execution Engine. If denoted version is 1.1.0
, then workflow would require Execution Engine >=1.1.0,<2.0.0
and we gain ability to expose concurrently multiple major versions of EE in the library (doing our best to ensure that within a major version we only add features and support everything that was released earlier within the same major). On top of that:
- block manifest metadata field
name
now will be understood as name of blocks family with additional tag calledversion
possible to be added; we propose the following naming conventions for block names:namespace/family_name@v1
. Thanks to those changes anyone could maintain multiple versions of the same block (appending new implementation to their plugin) ensuring backwards compatibilities on breaking changes - each block manifest class may optionally expose class method
get_execution_engine_compatibility(...)
which would be used while model loading to ensure that selected Execution Engine is capable to run specific block
✋ Example block manifest
class BlockManifest(WorkflowBlockManifest):
model_config = ConfigDict(
json_schema_extra={
"name": "My Block",
"version": "v1",
...
}
)
type: Literal["my_namespace/mu_block@v1"]
...
@classmethod
def get_execution_engine_compatibility(cls) -> Optional[str]:
return ">=1.0.0,<2.0.0"
🚨 ⚠️ BREAKING ⚠️ 🚨 Got rid of asyncio in Execution Engine
If you were tired of coroutines performing compute heavy tasks in workflows
:
class MyBlock(WorkflowBlock):
async def run():
pass
we have great news. We've got rid of asyncio in favour of standard functions and methods which are much more intuitive in our setup. This change is obviously breaking all other steps, but worry not. Here is the example of what needs to be changed - usually you just need to remove async
markers, but sometimes unfortunately pieces of asyncio code would need to be recreated.
class MyBlock(WorkflowBlock):
def run():
pass
Endpoint to expose workflow definition schema
Thanks to @EmilyGavrilenko (#550) UI would now be able to verify syntax errors in workflows definitions automatically.
Roboflow Dedicated Deployment is closer and closer 😃
Thanks to @PacificDou, inference
server is getting ready to support new functionality which has a nickname Dedicated Deployment. Stay tuned to learn more details - we can tell that this is something worth waiting for. You may find some hints in the PR.
🔨 Fixed
🚨 ⚠️ BREAKING ⚠️ 🚨 HTTP client of inference
server changes default behaviour
The default value for flag client_downsizing_disabled was changed from False
to True
in release 0.16.0! For clients using models with input size above 1024x1024, running models on hosted platform it should improve predictions quality (as previous default behaviour was causing that input was downsized and then artificially upsized on the server side with worse image quality). There may be some clients that would like to remain previous settings to potentially improve speed (when internet connection is a bottleneck and large images are submitted despite small model input size).
If you liked the previous behaviour more - simply:
from inference_sdk import InferenceHTTPClient, InferenceConfiguration
client = InferenceHTTPClient(
"https://detect.roboflow.com",
api_key="XXX",
).configure(InferenceConfiguration(
client_downsizing_disabled=False,
))
setuptools
were migrated to version above 70.0.0
to mitigate security issue
We've updated rf-clip
package to support setuptools>70.0.0
and bumped the version on inference
side.
🌱 Changed
- 📖 Add documentation for ONNXRUNTIME_EXECUTION_PROVIDERS by @grzegorz-roboflow in #562 - see here
- 📖 Update docs for easier quickstart by @komyg in #544
- 📖 Add Inference Windows CUDA documentation by @capjamesg in #502
- Add @capjamesg to CODEOWNERS by @capjamesg in #564
- Add persistent queue to usage collector by @grzegorz-roboflow in #568
🏅 New Contributors
Full Changelog: v0.15.2...v0.16.0
v0.15.2
What's Changed
- Separate the LMM block into OpenAI and CogVLM by @EmilyGavrilenko in #549
- Do not log warning when usage payload is put back into the queue by @grzegorz-roboflow in #551
- Initialize usage_collector._async_lock only if async look can be obtained by @grzegorz-roboflow in #552
- Cache workflow specification for offline use by @sberan in #555
- Custom Metadata block for Model Monitoring users by @robiscoding in #553
- Allow zone defined as input parameter to be passed to perspective_correction by @grzegorz-roboflow in #559
- Increment
num_errors
only ifpingback
initialized by @iurisilvio in #527 - SAM2 by @hansent and @probicheaux in #557
Full Changelog: v0.15.1...v0.15.2
v0.15.1
What's Changed
- Refactor Visualization Workflow Block Inheritance by @yeldarby in #538
- Add florence2 aliases and bugfix TransformersModel by @probicheaux in #525
- Add BackgroundColorAnnotator block by @capjamesg in #542
- Bugfix: keypoint detection model block by @EmilyGavrilenko in #543
Full Changelog: v0.15.0...v0.15.1