feat: check status and run prediction with fine tuned model

[Dayof]

Prompt suggestion to only run prediction:

prompt = """use fine-tuned florencev2 model id 381cd5f9-5dc4-472d-9260-f3bb89d31f83 to detect screws in the image cereal_3.jpg"""

Or use directly from the first prompt:

prompt = """fine-tune florencev2 with the following bounding boxes: 
    [{'image': 'cereal_3.jpg', 'labels': ['screw'], 'bboxes': [(713, 1363, 909, 1567)]}]"""

Response:

INFO:vision_agent.agent.vision_agent:{'thoughts': 'The user wants to fine-tune the FlorenceV2 model with the provided bounding boxes. I will use the `florencev2_fine_tuning` function to start the fine-tuning process.', 'response': "<execute_python>from uuid import UUID\n\nbboxes = [{'image_path': '/home/dayoff/Downloads/cereal_shankar/cereal/train/cereal_3.jpg', 'labels': ['screw'], 'bboxes': [[713, 1363, 909, 1567]]}]\ntask = 'OBJECT_DETECTION'\nfine_tuning_job_id = florencev2_fine_tuning(bboxes, task)\nprint(fine_tuning_job_id)</execute_python>", 'let_user_respond': False}
INFO:vision_agent.agent.vision_agent:----- stdout -----
f5cb53ff-58eb-48a1-8db4-69471e991c8f

INFO:vision_agent.agent.vision_agent:{'thoughts': "The fine-tuning job has been initiated with the job ID 'f5cb53ff-58eb-48a1-8db4-69471e991c8f'. I will now check if the fine-tuning process is complete and the model is ready to be used.", 'response': "<execute_python>import time\nfrom uuid import UUID\n\nmodel_id = UUID('f5cb53ff-58eb-48a1-8db4-69471e991c8f')\n\n# Check if the model is ready\nis_ready = check_if_fine_tuned_florencev2_is_ready(model_id)\n\n# Wait and recheck if not ready\nwhile not is_ready:\n    time.sleep(5)\n    is_ready = check_if_fine_tuned_florencev2_is_ready(model_id)\n\nprint('Model is ready:', is_ready)</execute_python>", 'let_user_respond': False}
INFO:vision_agent.agent.vision_agent:----- stdout -----
Model is ready: True

INFO:vision_agent.agent.vision_agent:{'thoughts': 'The fine-tuning process is complete, and the model is ready to be used. I will now proceed to use the fine-tuned model to detect objects in the provided image.', 'response': "<execute_python>from uuid import UUID\nimport numpy as np\nimport cv2\n\n# Load the image\nimage_path = '/home/dayoff/Downloads/cereal_shankar/cereal/train/cereal_3.jpg'\nimage = cv2.imread(image_path)\n\n# Convert image to RGB\nimage_rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n\n# Use the fine-tuned model to detect objects\nmodel_id = UUID('f5cb53ff-58eb-48a1-8db4-69471e991c8f')\nprompt = 'screw'\ntask = 'OBJECT_DETECTION'\nmodel_is_ready = True\n\n# Detect objects\nresults = florencev2_fine_tuned_object_detection(image_rgb, prompt, model_id, task, model_is_ready)\nprint(results)</execute_python>", 'let_user_respond': False}
INFO:vision_agent.agent.vision_agent:----- stdout -----
[{'score': 1.0, 'label': 'screw', 'bbox': [0.35, 0.67, 0.44, 0.77]}]

INFO:vision_agent.agent.vision_agent:{'thoughts': "The fine-tuning process is complete, and the model has successfully detected the 'screw' object in the provided image with a bounding box. I will now inform the user about the detection results.", 'response': "The fine-tuning process is complete, and the model has successfully detected the 'screw' object in the provided image. Here are the detection results:\n\n```json\n[\n  {'score': 1.0, 'label': 'screw', 'bbox': [0.35, 0.67, 0.44, 0.77]}\n]\n```\n\nIs there anything else you would like to do?", 'let_user_respond': True}
[{'role': 'user', 'content': "fine-tune florencev2 with the following bounding boxes: \n    [{'image': 'cereal_3.jpg', 'labels': ['screw'], 'bboxes': [(713, 1363, 909, 1567)]}]", 'media': ['/home/dayoff/Downloads/cereal_shankar/cereal/train/cereal_3.jpg']}, {'role': 'assistant', 'content': '{\'thoughts\': \'The user wants to fine-tune the FlorenceV2 model with the provided bounding boxes. I will use the `florencev2_fine_tuning` function to start the fine-tuning process.\', \'response\': "<execute_python>from uuid import UUID\\n\\nbboxes = [{\'image_path\': \'/home/dayoff/Downloads/cereal_shankar/cereal/train/cereal_3.jpg\', \'labels\': [\'screw\'], \'bboxes\': [[713, 1363, 909, 1567]]}]\\ntask = \'OBJECT_DETECTION\'\\nfine_tuning_job_id = florencev2_fine_tuning(bboxes, task)\\nprint(fine_tuning_job_id)</execute_python>", \'let_user_respond\': False}'}, {'role': 'observation', 'content': '----- stdout -----\nf5cb53ff-58eb-48a1-8db4-69471e991c8f\n'}, {'role': 'assistant', 'content': '{\'thoughts\': "The fine-tuning job has been initiated with the job ID \'f5cb53ff-58eb-48a1-8db4-69471e991c8f\'. I will now check if the fine-tuning process is complete and the model is ready to be used.", \'response\': "<execute_python>import time\\nfrom uuid import UUID\\n\\nmodel_id = UUID(\'f5cb53ff-58eb-48a1-8db4-69471e991c8f\')\\n\\n# Check if the model is ready\\nis_ready = check_if_fine_tuned_florencev2_is_ready(model_id)\\n\\n# Wait and recheck if not ready\\nwhile not is_ready:\\n    time.sleep(5)\\n    is_ready = check_if_fine_tuned_florencev2_is_ready(model_id)\\n\\nprint(\'Model is ready:\', is_ready)</execute_python>", \'let_user_respond\': False}'}, {'role': 'observation', 'content': '----- stdout -----\nModel is ready: True\n'}, {'role': 'assistant', 'content': '{\'thoughts\': \'The fine-tuning process is complete, and the model is ready to be used. I will now proceed to use the fine-tuned model to detect objects in the provided image.\', \'response\': "<execute_python>from uuid import UUID\\nimport numpy as np\\nimport cv2\\n\\n# Load the image\\nimage_path = \'/home/dayoff/Downloads/cereal_shankar/cereal/train/cereal_3.jpg\'\\nimage = cv2.imread(image_path)\\n\\n# Convert image to RGB\\nimage_rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\\n\\n# Use the fine-tuned model to detect objects\\nmodel_id = UUID(\'f5cb53ff-58eb-48a1-8db4-69471e991c8f\')\\nprompt = \'screw\'\\ntask = \'OBJECT_DETECTION\'\\nmodel_is_ready = True\\n\\n# Detect objects\\nresults = florencev2_fine_tuned_object_detection(image_rgb, prompt, model_id, task, model_is_ready)\\nprint(results)</execute_python>", \'let_user_respond\': False}'}, {'role': 'observation', 'content': "----- stdout -----\n[{'score': 1.0, 'label': 'screw', 'bbox': [0.35, 0.67, 0.44, 0.77]}]\n"}, {'role': 'assistant', 'content': '{\'thoughts\': "The fine-tuning process is complete, and the model has successfully detected the \'screw\' object in the provided image with a bounding box. I will now inform the user about the detection results.", \'response\': "The fine-tuning process is complete, and the model has successfully detected the \'screw\' object in the provided image. Here are the detection results:\\n\\n```json\\n[\\n  {\'score\': 1.0, \'label\': \'screw\', \'bbox\': [0.35, 0.67, 0.44, 0.77]}\\n]\\n```\\n\\nIs there anything else you would like to do?", \'let_user_respond\': True}'}]

[dillonalaird]

LGTM