"Executor" agent isn't installing missing modules - Magentic One

[Kumala3]

What happened?

Hi, I've recently noticed that when I asked Magentic One to scrape the dynamic content on the certain page where the context is being loaded dynamically by scrolling down. The Coder agent has written the code that uses Selenium and I didn't have it installed before running the script, so when Executor agent tried to execute the code, it failed due to "missing modules" such as selenium and webdriver-manager based on the .
I got into a loop where the AI-Agent tried to install the necessary modules, but it constantly failed and therefore attempts of executing code were pointless.

What did you expect to happen?

I expected Executor to install dependencies that were identified as missing into an active virtual environment to fulfill my query.

How can we reproduce it (as minimally and precisely as possible)?

To reproduce the issue, I am talking about, please follow the steps below:

1. Run the example with human-in-the-loop mode: python examples/example.py --logs_dir ./logs --hil_mode
2. Use the next prompt: Your task is to scrape all the submissions card titles on this page: [[3D Chess Design Contest](https://www.tripo3d.ai/event/3d-chess-contest#submissions)](https://www.tripo3d.ai/event/3d-chess-contest#submissions). Note: the new submissions are loaded dynamically, so you need to first scroll up to down of the page and only then scrape the data or do it dynamically, up to you (decide what’s easier and more efficient).You need to provide me with a full list of scraped submissions card titles.
3. Observe the Coder agent writes the code in Selenium
4. Watch the Executor tries to execute the code that uses external modules and returns errors if any are missing

AutoGen version

0.2.38

Which package was this bug in

Magentic One

Model used

gpt-4o

Python version

3.12

Operating system

Windows 11

Any additional info you think would be helpful for fixing this bug

I have attached the full logs copied from the terminal, so you can understand the situation better and perhaps help me resolve the issue with AI-Agent isn't installing modules. Failed to install missing modules logs file

[husseinmozannar]

I have noticed this issue as well quite a few times as well, it is because the code passed to the Executor is not in a shell script to install and so it fails. The few times I see it succeeding is when the Orchestrator fixes the script to install to the Executor.

It is something we need to fix in Magentic-One. A temporary fix I use is to pre-install packages to in my local environment or on start-up and have it run.

Potentially this could be addressed by fixing the system prompt of the Coder or with a smarter Executor agent.

[bgeesaman]

In my use case, the coder was writing a script that needed the requests module to work, and maybe 1 in 4 times, the coder was able to self install it into the ephemeral docker container for execution. I tracked down the fact that the default image is python:3-slim and where to override that with a container of my own that had a bunch of python packages. It's in examples/example.py on line 39, you can pass an image parameter. In this case, my local image is tagged executor:latest.

async def main(logs_dir: str, hil_mode: bool, save_screenshots: bool) -> None:
    # Create the runtime.
    runtime = SingleThreadedAgentRuntime()

    # Create an appropriate client
    client = create_completion_client_from_env(model="gpt-4o")

    async with DockerCommandLineCodeExecutor(image="executor:latest",work_dir=logs_dir) as code_executor:
        # Register agents.
        await Coder.register(runtime, "Coder", lambda: Coder(model_client=client))
        coder = AgentProxy(AgentId("Coder", "default"), runtime)

My Dockerfile (not the most efficient Dockerfile structure, but hey, it was a test, and it worked first shot)

FROM python:3-slim

RUN pip install --upgrade pip
RUN pip install openai pillow aiohttp typing-extensions pydantic types-aiofiles grpcio protobuf
RUN pip install numpy pandas matplotlib seaborn scikit-learn requests urllib3 nltk pytest

[Kumala3]

In my use case, the coder was writing a script that needed the requests module to work, and maybe 1 in 4 times, the coder was able to self install it into the ephemeral docker container for execution. I tracked down the fact that the default image is python:3-slim and where to override that with a container of my own that had a bunch of python packages. It's in examples/example.py on line 39, you can pass an image parameter. In this case, my local image is tagged executor:latest.

async def main(logs_dir: str, hil_mode: bool, save_screenshots: bool) -> None:
    # Create the runtime.
    runtime = SingleThreadedAgentRuntime()

    # Create an appropriate client
    client = create_completion_client_from_env(model="gpt-4o")

    async with DockerCommandLineCodeExecutor(image="executor:latest",work_dir=logs_dir) as code_executor:
        # Register agents.
        await Coder.register(runtime, "Coder", lambda: Coder(model_client=client))
        coder = AgentProxy(AgentId("Coder", "default"), runtime)

My Dockerfile (not the most efficient Dockerfile structure, but hey, it was a test, and it worked first shot)

FROM python:3-slim

RUN pip install --upgrade pip
RUN pip install openai pillow aiohttp typing-extensions pydantic types-aiofiles grpcio protobuf
RUN pip install numpy pandas matplotlib seaborn scikit-learn requests urllib3 nltk pytest

@husseinmozannar @bgeesaman
Thank you so much for clarification. I think I understand it now better. So, I can tell the Executor to execute the code the Coder has written in my specified docker environment? If I'd like to create a custom Docker image, where should I create it and then I can pass it's name as param to a main func in examples/example.py?

[bgeesaman]

I created that Dockerfile in a separate directory and built the image locally named executor:latest with the command

docker build -t executor:latest .

(note the trailing period to mean the build context is the current directory) in that directory where the Dockerfile was present.