Skip to content
Merged
Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
6cc5c84
Add start:dev command
sbliven Oct 14, 2024
47d2467
Reformat backend/configuration.md
sbliven Oct 14, 2024
caf1894
Add jobconfig documentation
sbliven Oct 14, 2024
23ac019
Convert json job configuration to yaml
sbliven Oct 29, 2024
0da65eb
Update job documentation
sbliven Nov 3, 2024
1a9dc3f
document rabbitmq
despadam Jan 8, 2025
a3c2844
document job groups
despadam Jan 8, 2025
94c3df1
document job auth
despadam Jan 8, 2025
05baec0
document job tests - wip
despadam Jan 9, 2025
31acd6f
document job tests done
despadam Jan 9, 2025
941a2f4
fixes job test
despadam Jan 10, 2025
3d8b8dd
fix variable names in jobs tests
despadam Jan 10, 2025
82b9ac7
refactored job tests update
sofyalaski Jan 14, 2025
7d765bc
Update Development/v4.x/backend/testing/jobs.md
despadam Jan 16, 2025
32d163a
Merge pull request #1 from sofyalaski/refactor-job-tests
despadam Jan 16, 2025
5303c4e
document email and url actions
despadam Jan 24, 2025
cac610b
rename statusUpdate to update
despadam Feb 28, 2025
76a27cd
add test for anonymous user
despadam Mar 14, 2025
61da2ac
Merge pull request #2 from sbliven/fix-1758
despadam Mar 14, 2025
cb0f333
changes in backend #1774
sofyalaski Mar 20, 2025
c851a34
Merge branch 'master' into jobconfig
despadam Mar 20, 2025
68ee700
add a new last test
sofyalaski Mar 20, 2025
7f857d3
Merge branch 'jobconfig' into jobs-read-permission
despadam Mar 20, 2025
893cca8
Merge pull request #4 from sofyalaski/jobs-read-permission
despadam Mar 20, 2025
2c9322d
changes from release-jobs
sofyalaski Mar 21, 2025
2d17400
Update Development/v4.x/backend/testing/jobs.md
sofyalaski Mar 21, 2025
989780d
add changes after review
sofyalaski Mar 21, 2025
3afc578
changes after review
sofyalaski Mar 21, 2025
61b8fe3
Merge pull request #5 from sofyalaski/release-jobs-tests
despadam Mar 21, 2025
4a373e4
fix typos
despadam Mar 26, 2025
8003a17
fix typos in jobs tests
despadam Mar 26, 2025
624b421
fix last typos
despadam Mar 26, 2025
ab89e1d
add a note that admin can do any job
sofyalaski Mar 28, 2025
1516972
Merge pull request #6 from sofyalaski/admin-jobs
despadam Mar 28, 2025
a302b0d
fix config
despadam Mar 28, 2025
cee0ae6
finish config for jobs and emails
despadam Mar 28, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,7 @@ docs
*.pdf
.idea
cat

# Website output
_site
.jekyll-cache
2 changes: 1 addition & 1 deletion Development/v4.x/Data_Model.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ The data model and its API endpoints are described [here](https://scicatproject.

### Job

When a user wants to archive, retrieve or publish a dataset, SciCat must interact with the facility's storage and archiving system. This is done by creating a `Job` to track the interaction. The exact actions taken for each Job are often customized for each facility. A common action would be to forward the new Job entry to a Message Broker, e.g. RabbitMQ, from where it can be picked up by any program willing to react to this Job. For example, at PSI the RabbitMQ queue is emptied by a Node-RED process, which reads the Job information from the RabbitMQ queue. Alternatively, you can use other messaging solutions, such as Apache Kafka. In this way the (site specific) logic to handle the Jobs is kept outside the core of the SciCat system, giving a greater degree of flexibility. The external systems should ideally respond with status updates to the Job model, when there job is finished. As the job progresses, the system can make PUT/PATCH API calls back that update the `Job Status Message` and, when necessary, also the individual status of the involved datasets by updating the DatasetLifecycle information.
When a user wants to archive, retrieve or publish a dataset, SciCat must interact with the facility's storage and archiving system. This is done by creating a `Job` to track the interaction. The exact actions taken for each Job are often customized for each facility. A common action would be to forward the new Job entry to a Message Broker, e.g. RabbitMQ, from where it can be picked up by any program willing to react to this Job. For example, at PSI the RabbitMQ queue is emptied by a Node-RED process, which reads the Job information from the RabbitMQ queue. Alternatively, you can use other messaging solutions, such as Apache Kafka. In this way the (site specific) logic to handle the Jobs is kept outside the core of the SciCat system, giving a greater degree of flexibility. The external systems should ideally respond with updates to the Job model, when the job is finished. As the job progresses, the system can make PUT/PATCH API calls back that update the `Job Status Message` and, when necessary, also the individual status of the involved datasets by updating the DatasetLifecycle information.

The data model and its API endpoints are described [here](https://scicatproject.github.io/api/#operation/Job.create).

Expand Down
49 changes: 24 additions & 25 deletions Development/v4.x/backend/authorization/authorization_jobs.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,19 +6,18 @@ The authorization for jobs is consistently different from all the other endpoint
### Endpoint Authorization
- JobCreate
- JobRead
- JobStatusUpdate
- JobUpdate
- JobDelete

### (Data) Instance Authorization
- JobCreateConfiguration (The jobs create section of the configuration dictats if the user can create the job)
- JobCreateConfiguration (The jobs create section of the configuration dictates if the user can create the job)
- JobCreateOwner (Users with this privileges can create jobs only for themselves)
- JobCreateAny (Users with this privileges can create jobs for any of the users that are defined in the create section of the job configuration)
- JobReadAccess
- JobReadAny
- JobStatusUpdateConfiguration (The jobs update section in configuration dictates if the user can update the status of the job)
- JobStatusUpdateOwner (Users with this privileges can update the status of jobs belonging to themselves)
- JobStatusUpdateAny (Users with this privileges can update the status of any job)
- JobDeleteAny
- JobUpdateConfiguration (The jobs update section in configuration dictates if the user can update the job)
- JobUpdateOwner (Users with this privileges can update jobs belonging to themselves)
- JobUpdateAny (Users with this privileges can update any job)

#### Priority
```mermaid
Expand All @@ -27,19 +26,19 @@ graph LR;
JobCreateConfiguration-->JobCreateAny;
JobRead-->JobReadAccess;
JobReadAccess-->JobReadAny;
JobStatusUpdate-->JobStatusUpdateConfiguration;
JobStatusUpdateConfiguration-->JobStatusUpdateAny;
JobDelete-->JobDeleteAny;
JobUpdate-->JobUpdateConfiguration;
JobUpdateConfiguration-->JobUpdateAny;
JobDelete;
```

#### Authorization table
| HTTP method | Endpoint | Endpoint Authentication | Anonymous | Authenticated | Create Jobs Groups | Update Jobs Groups | Admin Groups | Delete Groups | Notes |
| -------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- |
| POST | Jobs | _JobCreate_ | _JobCreateConfiguration_ | _JobCreateConfiguration_ | Any<br>_JobsCreateOwner_ | __no__ | Any<br>_JobsCreateAny_ | __no__ | |
| GET | Jobs | _JobReadMany_ | __no__ | Has Access<br>_JobReadAccess_ | Has Access<br>_JobReadAccess_ | __no__ | Any<br>_JobReadAny_ | __no__ | |
| GET | Jobs/_jid_ | _JobReadOne_ | __no__ | Has Access<br>_JobReadAccess_ | Has Access<br>_JobReadAccess_ | __no__ | Any<br>_JobReadAny_ | __no__ | |
| POST | Jobs/statusUpdate | _JobStatusUpdate_ | __no__ | _JobStatusUpdateConfiguration_ | __no__ | Owner<br>_JobStatusUpdateOwner_ | Any<br>_JobStatusUpdateAny_ | __no__ | |
| DELETE | Jobs/_jid_ | _JobDelete_ | __no__ | __no__ | __no__ | __no__ | __no__ | _JobDeleteAny_ | |
| HTTP method | Endpoint | Endpoint Authentication | Anonymous | Authenticated | Create Jobs Groups | Update Jobs Groups | Admin Groups | Delete Groups |
| -------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- |
| POST | Jobs | _JobCreate_ | _JobCreateConfiguration_ | _JobCreateConfiguration_ | Any<br>_JobsCreateOwner_ | __no__ | Any<br>_JobsCreateAny_ | __no__ |
| GET | Jobs | _JobReadMany_ | __no__ | Has Access<br>_JobReadAccess_ | Has Access<br>_JobReadAccess_ | __no__ | Any<br>_JobReadAny_ | __no__ |
| GET | Jobs/_jid_ | _JobReadOne_ | __no__ | Has Access<br>_JobReadAccess_ | Has Access<br>_JobReadAccess_ | __no__ | Any<br>_JobReadAny_ | __no__ |
| PATCH | Jobs/_jid_ | _JobUpdate_ | __no__ | _JobUpdateConfiguration_ | __no__ | Owner<br>_JobUpdateOwner_ | Any<br>_JobUpdateAny_ | __no__ |
| DELETE | Jobs/_jid_ | _JobDelete_ | __no__ | __no__ | __no__ | __no__ | __no__ | __no__ |

#### Job Create Authorization Table
The _JobCreateConfiguration_ authorization permissions are configured directly in the __*create*__ section of the job configuration.
Expand All @@ -57,23 +56,23 @@ Any positive match will results in the user acquiring _JobCreate_ endpoint autho

__IMPORTANT__: use option _#all_ carefully, as it allows anybody to create a new job. It is mostly use for debuging and testing

#### Job Status Update Authorization Table
The _JobStatusUpdateConfiguration_ authorization permissions are configured directly in the __*update*__ section of the job configuration.
Any positive match will results in the user acquiring _JobStatusUpdate_ endpoint authorization apply to the jobs endpoint POST:Jobs/statusUpdate
#### Job Update Authorization Table
The _JobUpdateConfiguration_ authorization permissions are configured directly in the __*update*__ section of the job configuration.
Any positive match will results in the user acquiring _JobUpdate_ endpoint authorization apply to the jobs endpoint PATCH:Jobs/id

| Job Status Update Authorization | Endpoint Authentication Translation | Endpoint Authentication Description | Instance Authentication Translation | Instance Authentication Description |
| Job Update Authorization | Endpoint Authentication Translation | Endpoint Authentication Description | Instance Authentication Translation | Instance Authentication Description |
| --- | --- | --- | --- | --- |
| _#all_ | _#all_ | any user can access this endpoint, both anonymous and authenticated | _#all_ | Any user can update the status of this job instance |
| _#all_ | _#all_ | any user can access this endpoint, both anonymous and authenticated | _#all_ | Any user can update this job instance |
| _#jobOwnerUser_ | _#user_ | authenticated user can access the endpoint | _#jobOwnerUser_ | only the user that is listed in field _ownerUser_ can perform the update |
| _#jobOwnerGroup_ | _#user_ | authenticated user can access the endpoint | _#jobOwnerGroup_ | any user that belongs to the group listed in field _ownerGroup_ can perform the update |
| __*@GROUP*__ | __*GROUP*__ | only users that belongs to the specified group can access the endpoint | __*GROUP*__ | the job status can be updated only by users who belong to the group specified |
| __*USER*__ | __*USER*__ | only the specified user can access the endpoint | __*USER*__ | the job status can be updated only by the user indicated |
| __*@GROUP*__ | __*GROUP*__ | only users that belongs to the specified group can access the endpoint | __*GROUP*__ | the job can be updated only by users who belong to the group specified |
| __*USER*__ | __*USER*__ | only the specified user can access the endpoint | __*USER*__ | the job can be updated only by the user indicated |

__IMPORTANT__: use option _#all_ carefully, as it allows anybody to update the status of the job. It is mostly use for debuging and testing
__IMPORTANT__: use option _#all_ carefully, as it allows anybody to update the job. It is mostly use for debuging and testing

#### Job Authorization priority
The endpoint authorization is the most permissive authorization across all the jobs defined.
The priority between job create and status update authorization is as follow:
The priority between job create and update authorization is as follow:

```mermaid
graph LR;
Expand Down
Loading