[Q&A] Logic Error: no engine in fl_ctx

[jingyunc]

Python version (python3 -V)

3.8

NVFlare version (python3 -m pip list | grep "nvflare")

2.4.0rc3

NVFlare branch (if running examples, please use the branch that corresponds to the NVFlare version, git branch)

No response

Operating system

Ubuntu 20.04

Have you successfully run any of the following examples?

• hello-numpy-sag with simulator
• hello-pt with simulator
• hello-numpy-sag with POC
• hello-pt with POC

Please describe your question

I added a few processing functions to the Learner in Brats18 example (examples/advanced/brats18/pt/learners/supervised_monai_brats_learner.py). During the simulation, after the processing functions got called I got following error:

2023-11-15 21:23:10 xxx ModifiedLearner[1384236] ERROR [identity=site-2, run=simulate_job]: Logic Error: no engine in fl_ctx: run_num: <class 'str'> identity_name: <class 'str'> client_name: <class 'str'> fl_token: <class 'str'> current_job_id: <class 'str'> Current_Receiver: <class 'str'> Current_Sender: <class 'str'> event_data: <class 'nvflare.apis.shareable.Shareable'> event_scope: <class 'str'> event_origin: <class 'str'>

How may I fix this, maybe by setting up the engine property somewhere for the added functions? Thanks.

[YuanTingHsieh]

@jingyunc what are you trying to get from or do with engine?

[jingyunc]

I made several customizations to the Swarm Learning example (nvflare/app_common/ccwf/swarm_client_ctl.py), including extra validation before aggregation, and replacing the original cifar_model_learner with supervised_monai_brats_learner, so that it runs on Brats data. However my customized functions did not involve reading or writing to the engine.

The "no engine" error showed after my customized functions were executed. However, it did not stop the workflow, and my functions seemed to run through fine. I wonder if there is a place I should "set up the engine" when I added those customized functions into the Controller and Learner.

[YuanTingHsieh]

If possible can you share your code or repo somewhere so it might be easier to find the issue

[jingyunc]

I made following changes to the Swarm Learning example:

1. Modify the learner_executor to add a new task (https://github.com/NVIDIA/NVFlare/blob/main/nvflare/app_common/executors/learner_executor.py#L90):

     ...
        elif task_name == "my_task":
            return self.my_task(shareable, fl_ctx, abort_signal)
        else:
            self.log_error(fl_ctx, f"Could not handle task: {task_name}")
            return make_reply(ReturnCode.TASK_UNKNOWN)

    def my_task(self, shareable: Shareable, fl_ctx: FLContext, abort_signal: Signal) -> Shareable:
            return self.learner.my_task(shareable, fl_ctx, abort_signal)
    ...

2. Replace the cifar10_model_learner with supervised_monai_brats_learner, in order to run on Brats data.

3. Modify the supervised_monai_brats_learner to add definition of my_task (https://github.com/NVIDIA/NVFlare/blob/main/examples/advanced/brats18/pt/learners/supervised_monai_brats_learner.py#L228):

   def my_task(self, shareable: Shareable, fl_ctx: FLContext, abort_signal: Signal):
         # my processing steps ...
         self.log_info(fl_ctx, f"My task is done.")

4. Insert my task in the swarm_client_ctl.py (https://github.com/NVIDIA/NVFlare/blob/main/nvflare/app_common/ccwf/swarm_client_ctl.py#L161):

...
       self.log_info(fl_ctx, f"Start aggregation for round {self.for_round}")
        peer_ctx = fl_ctx.get_peer_context()
        assert isinstance(peer_ctx, FLContext)
        models = self.aggregator.models # this is a new property I created
        abort_signal = Signal()
        self.executor.learn_executor.execute("my_task", models, peer_ctx, abort_signal)
        self.log_info(fl_ctx, f"Back to gatherer")
...

During the simulation, the aforementioned "Logic Error: no engine in fl_ctx" showed on console between the log messages "My task is done" and "Back to gatherer". So I suppose the "fl_ctx" in the error was referring to the peer_ctx in step 4. And the missing engine was probably because it was a peer_context?

[YuanTingHsieh]

@jingyunc can you share what you want to do with your custom task?
Is your custom task part of training? or it is part of submission of model?
What are you trying to do?

To your question, your should be using fl_ctx directly NOT peer_ctx

[jingyunc]

Thanks @YuanTingHsieh. My lab at Columbia Irving Medical is developing a decentralized learning method for cancer study. In general, our custom task involves randomly sending local model from one site to the other, and co-train both (sender's and receiver's) models on the receiver site. Our method requires each site to act as both trainer and aggregator, which is similar to the Swarm Learning example. Thanks to the previous Q&A from you and Sean, I managed to insert my task codes into the Swarm Learning codes, but still had issue simulating >4 sites (as described in post #2159).

I'm happy to give more details on our custom task. It's probably easier to explain on Zoom with share screen. Do you have time for a short Zoom, perhaps sometime next week?

[jingyunc]

Confirming the "Logic Error" was fixed after using fl_ctx instead of peer_ctx in step 4, i.e.: self.executor.learn_executor.execute("my_task", models, fl_ctx, abort_signal)

I then moved the fl_ctx.get_peer_context() to step 3, where the peer_ctx properties were actually needed, and the task still worked.

[YuanTingHsieh]

Hi @jingyunc thanks for getting back to us.

Can you send an email to my email address: [email protected]?

Thanks

[jingyunc]

With pleasure!