feat(cmd): Implemented a CLI for task management #4609
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This patch implements the whole support ecosystem for server-side background tasks, in order to help lessen the load (and blocking) of API handlers in the web-server for long-running operations. A **Task** is represented by two things in strict co-existence: a lightweight, `pickle`-able implementation in the server's code (a subclass of `AbstractTask`) and a corresponding `BackgroundTask` database entity, which resides in the "configuration" database (shared across all products). A Task is created by API request handlers and then the user is instructed to retain the `TaskToken`: the task's unique identifier. Following, the server will dispatch execution of the object into a background worker process, and keep status synchronisation via the database. Even in a service cluster deployment, load balancing will not interfere with users' ability to query a task's status. While normal users can only query the status of a single task (which is usually automatically done by client code, and not the user manually executing something); product administrators, and especially server administrators have the ability to query an arbitrary set of tasks using the potential filters, with a dedicated API function (`getTasks()`) for this purpose. Tasks can be cancelled only by `SUPERUSER`s, at which point a special binary flag is set in the status record. However, to prevent complicating inter-process communication, cancellation is supposed to be implemented by `AbstractTask` subclasses in a co-operative way. The execution of tasks in a process and a `Task`'s ability to "communicate" with its execution environment is achieved through the new `TaskManager` instance, which is created for every process of a server's deployment. Unfortunately, tasks can die gracelessly if the server is terminated (either internally, or even externally). For this reason, the `DROPPED` status will indicate that the server has terminated prior to, or during a task's execution, and it was unable to produce results. The server was refactored significantly around the handling of subprocesses in order to support various server shutdown scenarios. Servers will start `background_worker_processes` number of task handling subprocesses, which are distinct from the already existing "API handling" subprocesses. By default, if unconfigured, `background_worker_processes` is equal to `worker_processes` (the number of API processes to spawn), which is equal to `$(nproc)` (CPU count in the system). This patch includes a `TestingDummyTask` demonstrative subclass of `AbstractTask` which counts up to an input number of seconds, and each second it gracefully checks whether it is being killed. The corresponding testing API endpoint, `createDummyTask()` can specify whether the task should simulate a failing status. This endpoint can only be used from, but is used extensively, the unit testing of the project. This patch does not include "nice" or "ergonomic" facilities for admins to manage the tasks, and so far, only the server-side of the corresponding API calls are supported.
The test files contain hard-coded sleep() operations. These are lowered.
This patch extends `CodeChecker cmd` with a new sub-command,
`serverside-tasks`, which lets users and administrators deal with
querying the status of running server-side tasks.
By default, the CLI queries the information of the task(s) specified by
their token(s) in the `--token` argument from the server using
`getTaskInfo(token)`, and shows this information in either verbose
"plain text" (available if precisely **one** task was specified), "table"
or JSON formats.
In addition to `--token`, it also supports 19 more parameters, each of
which correspond to a filter option in the `TaskFilter` API type.
If any filters in addition to `--token` is specified, it will exercise
`getTasks(filter)` instead.
This mode is only available to administrators.
The resulting, more detailed information structs are printed in "table"
or JSON formats.
Apart from querying the current status, two additional flags are
available, irrespective of which query method is used to obtain a list
of "matching tasks":
* `--kill` will call `cancelTask(token)` for each task.
* `--await` will block execution until the specified task(s) terminate
(in one way or another).
`--await` is implemented by calling the new **`await_task_termination`**
library function, which is implemented with the goal of being reusable
by other clients later.
bruntib
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Important
⛔ Blocked by #4317.
This is patch 2 of the Asynchronous Store Protocol (#3672).
This patch extends
CodeChecker cmdwith a new sub-command,serverside-tasks, which lets users and administrators deal with querying the status of running server-side tasks.By default, the CLI queries the information of the task(s) specified by their token(s) in the
--tokenargument from the server usinggetTaskInfo(token), and shows this information in either verbose "plain text" (available if precisely one task was specified), "table" or JSON formats.In addition to
--token, it also supports 19 more parameters, each of which correspond to a filter option in theTaskFilterAPI type. If any filters in addition to--tokenis specified, it will exercisegetTasks(filter)instead. This mode is only available to administrators. The resulting more detailed information structs are printed in "table" or JSON formats.Apart from querying the current status, two additional flags are available, irrespective of which query method is used to obtain a list of "matching tasks" after filtering:
--killwill callcancelTask(token)for each task.--awaitwill block execution until the specified task(s) terminate (in one way or another).--awaitis implemented by calling the newawait_task_terminationlibrary function, which is implemented with the goal of being reusable by other clients later.Example outputs
Query that results in using the filters by default shows
tableformatTip
Notice the showing of
MachineID in the output, as this is a query run by an administrator.User can modify the output format via the usual
--outputparameterDefault behaviour when searching for a specific task (identified by token)
Tip
Comments are formatted as if they were e-mails or chat messages for better visual separation.
Caution
This output is only meant to be consumed by humans!