Better code sandbox lifecycle management #110
Conversation
vision_agent/tools/tools.py
Outdated
| @@ -523,8 +523,8 @@ def save_image(image: np.ndarray) -> str: | |||
|
|
|||
| def overlay_bounding_boxes( | |||
| image: np.ndarray, bboxes: List[Dict[str, Any]] | |||
| ) -> np.ndarray: | |||
| """'display_bounding_boxes' is a utility function that displays bounding boxes on | |||
| ) -> Image.Image: | |||
There was a problem hiding this comment.
I'm not sure about changing these to PIL images. Sometimes I ask it to visualize bounding boxes and then do something with that output, like mask or draw lines or whatever. While you can technically do this with PIL, I think it may be easier to do it with the numpy arrays/cv2. Need to test it out.
There was a problem hiding this comment.
We need PIL because the image needs to be visualized by the IPython display(), and display() doesn't understand numpy array.
There was a problem hiding this comment.
While you can technically do this with PIL, I think it may be easier to do it with the numpy arrays/cv2.
We don't have to do it with PIL. we could convert PIL to numpy array easily.
E.g. we could support both PIL and numpy array as the input image of overlay_* functions.
There was a problem hiding this comment.
With the upcoming video visualization feature, now i think it's better to run the display() function inside of every overlay_* functions because it will constraint the vizualization behavior without adding a verbose user prompt.
Then, I don't need PIL and I can change the output back to numpy array.
Motivation
Currently,
chat_with_workflow()shares a global sandbox, which causes concurrency issues: parallelchat_with_workflow()execution could convolute/mess up the python runtime.This PR changes it to creating a dedicated sandbox per
chat_with_workflow()call.Changes