Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Tenacity errors which are happening in CHECK_COMMAND_LINE_STATUS #243

Open
sivkovic opened this issue Aug 17, 2021 · 0 comments
Open

Comments

@sivkovic
Copy link
Collaborator

Traceback:

Traceback (most recent call last):
  File "/juno/work/ci/prod_ridgeback/venv/lib/python3.7/site-packages/celery/app/trace.py", line 412, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/juno/work/ci/prod_ridgeback/venv/lib/python3.7/site-packages/celery/app/trace.py", line 704, in __protected_call__
    return self.run(*args, **kwargs)
  File "/srv/services/ridgeback/code/ridgeback/orchestrator/tasks.py", line 143, in command_processor
    check_status_of_command_line_jobs(job)
  File "/srv/services/ridgeback/code/ridgeback/orchestrator/tasks.py", line 381, in check_status_of_command_line_jobs
    command_line_status = submiter.get_commandline_status(track_cache_str)
  File "/srv/services/ridgeback/code/ridgeback/submitter/toil_submitter/toil_jobsubmitter.py", line 81, in get_commandline_status
    toil_track_obj.check_status()
  File "/srv/services/ridgeback/code/ridgeback/submitter/toil_submitter/toil_track_utils.py", line 654, in check_status
    job_store, root_job = _resume_job_store(self.job_store_path, self.total_attempts)
  File "/juno/work/ci/prod_ridgeback/venv/lib/python3.7/site-packages/tenacity/__init__.py", line 339, in wrapped_f
    return self(f, *args, **kw)
  File "/juno/work/ci/prod_ridgeback/venv/lib/python3.7/site-packages/tenacity/__init__.py", line 430, in __call__
    do = self.iter(retry_state=retry_state)
  File "/juno/work/ci/prod_ridgeback/venv/lib/python3.7/site-packages/tenacity/__init__.py", line 379, in iter
    six.raise_from(retry_exc, fut.exception())
  File "<string>", line 3, in raise_from
tenacity.RetryError: <Future at 0x7fe9d6ac96d8 state=finished raised KeyError>
[2021-08-17 11:43:35,153: ERROR/MainProcess] Task handler raised error: <MaybeEncodingError: Error sending result: '"(1, <ExceptionInfo: RetryError('<Future at 0x7fe9d6ac96d8 state=finished raised KeyError>')>, None)"'. Reason: ''PicklingError("Can\'t pickle <class \'tenacity.RetryError\'>: it\'s not the same object as tenacity.RetryError")''.>
Traceback (most recent call last):
  File "/juno/work/ci/prod_ridgeback/venv/lib/python3.7/site-packages/billiard/pool.py", line 366, in workloop
    put((READY, (job, i, result, inqW_fd)))
  File "/juno/work/ci/prod_ridgeback/venv/lib/python3.7/site-packages/billiard/queues.py", line 366, in put
    self.send_payload(ForkingPickler.dumps(obj))
  File "/juno/work/ci/prod_ridgeback/venv/lib/python3.7/site-packages/billiard/reduction.py", line 56, in dumps
    cls(buf, protocol).dump(obj)
billiard.pool.MaybeEncodingError: Error sending result: '"(1, <ExceptionInfo: RetryError('<Future at 0x7fe9d6ac96d8 state=finished raised KeyError>')>, None)"'. Reason: ''PicklingError("Can\'t pickle <class \'tenacity.RetryError\'>: it\'s not the same object as tenacity.RetryError")''.```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant