-
Notifications
You must be signed in to change notification settings - Fork 725
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
getting a parallel run error when submitting a run in MACOS #2915
Comments
Hi @rajib76 , does your code run outside of if __name__ == '__main__':
pf.run() |
tried as below, still getting the error
Starting prompt flow service... The above exception was the direct cause of the following exception: Traceback (most recent call last): Process finished with exit code 1 |
Hi @rajib76 , seems you installed
|
Had the same problem on Mac OS using interactive Python in vscode. Added the if |
Yes, that's because we used multi-processing to execute flow and it's required to use |
Hi, we're sending this friendly reminder because we haven't heard back from you in 30 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 7 days of this comment, the issue will be automatically closed. Thank you! |
You can view the traces from local: http://localhost:54796/v1.0/ui/traces/?#run=groundedness_variant_0_20240420_165405_648733
[2024-04-20 16:54:05,790][promptflow._sdk._orchestrator.run_submitter][INFO] - Submitting run groundedness_variant_0_20240420_165405_648733, log path: /Users/joyeed/.promptflow/.runs/groundedness_variant_0_20240420_165405_648733/logs.txt
[2024-04-20 16:54:09,345][promptflow._sdk._orchestrator.run_submitter][WARNING] - Run groundedness_variant_0_20240420_165405_648733 failed when executing in executor with exception Unexpected error occurred while executing the batch run. Error: (RuntimeError)
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
Traceback (most recent call last):
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/promptflow/batch/_batch_engine.py", line 254, in run
return async_run_allowing_running_loop(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/promptflow/_utils/async_utils.py", line 40, in async_run_allowing_running_loop
return asyncio.run(async_func(*args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/nest_asyncio.py", line 30, in run
return loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/nest_asyncio.py", line 98, in run_until_complete
return f.result()
^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/futures.py", line 203, in result
raise self._exception.with_traceback(self._exception_tb)
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/tasks.py", line 277, in __step
result = coro.send(None)
^^^^^^^^^^^^^^^
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/promptflow/batch/_batch_engine.py", line 406, in _exec_in_task
return task.result()
^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/futures.py", line 203, in result
raise self._exception.with_traceback(self._exception_tb)
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/tasks.py", line 277, in __step
result = coro.send(None)
^^^^^^^^^^^^^^^
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/promptflow/batch/_batch_engine.py", line 465, in _exec
results, is_timeout = await self._executor_proxy._exec_batch(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/promptflow/_proxy/_python_executor_proxy.py", line 110, in _exec_batch
with LineExecutionProcessPool(
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/promptflow/executor/_line_execution_process_pool.py", line 146, in enter
self.start()
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/promptflow/executor/_line_execution_process_pool.py", line 158, in start
manager = Manager()
^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/context.py", line 57, in Manager
m.start()
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/managers.py", line 563, in start
self._process.start()
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/context.py", line 288, in _Popen
return Popen(process_obj)
^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 32, in init
super().init(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/popen_fork.py", line 19, in init
self._launch(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 42, in _launch
prep_data = spawn.get_preparation_data(process_obj._name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/spawn.py", line 164, in get_preparation_data
_check_not_importing_main()
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/spawn.py", line 140, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "", line 1, in
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/spawn.py", line 122, in spawn_main
exitcode = _main(fd, parent_sentinel)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/spawn.py", line 131, in _main
prepare(preparation_data)
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/spawn.py", line 246, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/spawn.py", line 297, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 291, in run_path
File "", line 98, in _run_module_code
File "", line 88, in _run_code
File "/Users/joyeed/promptflow/promptflow/groundedness/flowrun.py", line 16, in
result = pf.runs.create_or_update(run,stream=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/promptflow/_sdk/_telemetry/activity.py", line 265, in wrapper
return f(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/promptflow/_sdk/operations/_run_operations.py", line 126, in create_or_update
created_run = RunSubmitter(client=self._client).submit(run=run, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/promptflow/_sdk/_orchestrator/run_submitter.py", line 42, in submit
self._run_bulk(run=run, stream=stream, **kwargs)
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/promptflow/_sdk/_orchestrator/run_submitter.py", line 105, in _run_bulk
self._submit_bulk_run(flow=flow, run=run, local_storage=local_storage)
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/promptflow/_sdk/_orchestrator/run_submitter.py", line 198, in _submit_bulk_run
raise e
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/promptflow/_sdk/_orchestrator/run_submitter.py", line 164, in _submit_bulk_run
batch_result = batch_engine.run(
^^^^^^^^^^^^^^^^^
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/promptflow/batch/_batch_engine.py", line 277, in run
raise unexpected_error from e
promptflow._core._errors.UnexpectedError: Unexpected error occurred while executing the batch run. Error: (RuntimeError)
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
[2024-04-20 16:54:10,850][promptflow._sdk._orchestrator.run_submitter][WARNING] - Run groundedness_variant_0_20240420_165401_806540 failed when executing in executor with exception Unexpected error occurred while executing the batch run. Error: (EOFError) ..
Traceback (most recent call last):
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/promptflow/batch/_batch_engine.py", line 254, in run
return async_run_allowing_running_loop(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/promptflow/_utils/async_utils.py", line 40, in async_run_allowing_running_loop
return asyncio.run(async_func(*args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/nest_asyncio.py", line 30, in run
return loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/nest_asyncio.py", line 98, in run_until_complete
return f.result()
^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/futures.py", line 203, in result
raise self._exception.with_traceback(self._exception_tb)
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/tasks.py", line 277, in __step
result = coro.send(None)
^^^^^^^^^^^^^^^
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/promptflow/batch/_batch_engine.py", line 406, in _exec_in_task
return task.result()
^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/futures.py", line 203, in result
raise self._exception.with_traceback(self._exception_tb)
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/tasks.py", line 277, in __step
result = coro.send(None)
^^^^^^^^^^^^^^^
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/promptflow/batch/_batch_engine.py", line 465, in _exec
results, is_timeout = await self._executor_proxy._exec_batch(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/promptflow/_proxy/_python_executor_proxy.py", line 110, in _exec_batch
with LineExecutionProcessPool(
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/promptflow/executor/_line_execution_process_pool.py", line 146, in enter
self.start()
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/promptflow/executor/_line_execution_process_pool.py", line 158, in start
manager = Manager()
^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/context.py", line 57, in Manager
m.start()
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/managers.py", line 567, in start
self._address = reader.recv()
^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/connection.py", line 430, in _recv_bytes
buf = self._recv(4)
^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/connection.py", line 399, in _recv
raise EOFError
EOFError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/joyeed/promptflow/promptflow/groundedness/flowrun.py", line 16, in
result = pf.runs.create_or_update(run,stream=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/promptflow/_sdk/_telemetry/activity.py", line 265, in wrapper
return f(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/promptflow/_sdk/operations/_run_operations.py", line 126, in create_or_update
created_run = RunSubmitter(client=self._client).submit(run=run, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/promptflow/_sdk/_orchestrator/run_submitter.py", line 42, in submit
self._run_bulk(run=run, stream=stream, **kwargs)
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/promptflow/_sdk/_orchestrator/run_submitter.py", line 105, in _run_bulk
self._submit_bulk_run(flow=flow, run=run, local_storage=local_storage)
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/promptflow/_sdk/_orchestrator/run_submitter.py", line 198, in _submit_bulk_run
raise e
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/promptflow/_sdk/_orchestrator/run_submitter.py", line 164, in _submit_bulk_run
batch_result = batch_engine.run(
^^^^^^^^^^^^^^^^^
File "/Users/joyeed/promptflow/promptflow/venv/lib/python3.11/site-packages/promptflow/batch/_batch_engine.py", line 277, in run
raise unexpected_error from e
promptflow._core._errors.UnexpectedError: Unexpected error occurred while executing the batch run. Error: (EOFError) .
The text was updated successfully, but these errors were encountered: