[Q&A] Change of behaviour: where is the simulator saving its output?

[virginiafdez]

Python version (python3 -V)

3.10

NVFlare version (python3 -m pip list | grep "nvflare")

2.5.2

NVFlare branch (if running examples, please use the branch that corresponds to the NVFlare version, git branch)

2.5

Operating system

Ubuntu 22.04

Have you successfully run any of the following examples?

• hello-numpy-sag with simulator
• hello-pt with simulator
• hello-numpy-sag with POC
• hello-pt with POC

Please describe your question

When upgrading from nvflare==2.2.5 to nvflare==2.5.2, and running a mock version of a job we are currently deploying in production.
One of the last executors deletes the job directory for safety reasons.
This used to work fine in nvflare==2.2.5, but currently it's erroring out looking for the simulator_job folder within the client.
Has any of the wrapping-up functions changed between these versions that would make it incorrect to delete this job folder?
This is the error call stack:

2025-01-24 14:40:25,631 - ClientTaskWorker - INFO - ClientTaskWorker started to run
2025-01-24 14:40:25,691 - CoreCell - INFO - site1.simulate_job: created backbone external connector to tcp://localhost:34039
2025-01-24 14:40:25,691 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connector [CH00001 ACTIVE tcp://localhost:34039] is starting
2025-01-24 14:40:25,692 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00002 127.0.0.1:36026 => 127.0.0.1:34039] is created: PID: 1365337
2025-01-24 14:40:25,692 - nvflare.fuel.f3.sfm.conn_manager - INFO - Connection [CN00017 127.0.0.1:34039 <= 127.0.0.1:36026] is created: PID: 1364696
2025-01-24 14:40:25,793 - ClientTaskWorker - ERROR - do_one_task execute exception: FileNotFoundError: config_file_names 
SIMULATOR/site1/simulate_job/app_site1/config/config_fed_client.json does not exist or not a file
2025-01-24 14:40:25,795 - root - ERROR - Traceback (most recent call last):
  File "/venv2/lib/python3.10/site-packages/nvflare/private/fed/app/simulator/simulator_worker.py", line 92, in do_one_task
    self.create_client_runner(client)
  File "/venv2/lib/python3.10/site-packages/nvflare/private/fed/app/simulator/simulator_worker.py", line 75, in create_client_runner
    client_app_runner.client_runner = client_app_runner.create_client_runner(
  File "/venv2/lib/python3.10/site-packages/nvflare/private/fed/client/client_app_runner.py", line 87, in create_client_runner
    conf = ClientJsonConfigurator(
  File "/venv2/lib/python3.10/site-packages/nvflare/private/fed/client/client_json_config.py", line 76, in __init__
    FedJsonConfigurator.__init__(
  File "/venv2/lib/python3.10/site-packages/nvflare/private/fed_json_config.py", line 54, in __init__
    JsonConfigurator.__init__(
  File "/venv2/lib/python3.10/site-packages/nvflare/private/json_configer.py", line 71, in __init__
    raise FileNotFoundError(f"config_file_names {f} does not exist or not a file")
FileNotFoundError: config_file_names 
SIMULATOR/site1/simulate_job/app_site1/config/config_fed_client.json does not exist or not a file

No doubt about the origin of the error, it's simply looking for a folder that has been deleted, but I just wonder whether the versions have changed which makes the error appear now and not before.

[virginiafdez]

Update:

What was being run before was:

cwd = os.getcwd()
job_dir = os.path.join(cwd, fl_ctx.get_job_id())

If job_dir exists, then it's deleted.
When running the app on the simulator using the previous nvflare (2.2.5) version, this folder, which turned out to be:

[SIMULATE_FODLER]/simulate_job/app_site1/simulate_job

Did not exist. Therefore I imagine the directory stayed in its place and no error happened.

With the new version, 2.5.2, this folder has changed to

[SIMULATE_FOLDER]/site1/simulate_job

It does exist and therefore gets deleted.

Does this mean that nvflare has changed the folder structure with the simulator, or in general? Was it a bug that os.path.join(cwd, fl_ctx.get_job_id()) did not exist?

[YuanTingHsieh]

@virginiafdez thank you for raising the issue.

Yes, there is a behavior change regarding to the simulator workspace saving.
In 2.5.2 and forward, we make the simulator workspace consistent with the POC and the real-world deployment workspace.
The POC and real-world case remains the same as follows:

So each site will have:

siteX/
    job1/
        app_siteX/
            custom/
            config/
        meta.json
    job2/
        app_siteX
    local/
        log.config
    startup

[virginiafdez]

Perfect! Thanks for the confirmation!