Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cancel_jobs.sql and delete_datasets.sql stored procedures #269

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 27 additions & 1 deletion stored_procedures/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,38 @@ CALL bqutil.procedure.GetNextIds(10, next_ids);

## Stored Procedures

* [GetNextIds](#GetNextIds)
* [cancel_jobs](#cancel_jobscancel_jobssql)

* [delete_datasets](#delete_datasetsproject_id-string-dataset_like_filter-stringdelete_datasetssql)

* [GetNextIds](#getnextidsid_count-int64-out-next_ids-arrayint64get_next_idsql)

## Documentation

### [cancel_jobs(project_id STRING, job_id_like_filter STRING)](cancel_jobs.sql)

Cancels all running or pending jobs which match the input argument, job_id_like_filter.

```sql
# Cancel jobs which match a job_id prefix
CALL bqutil.procedure.cancel_jobs("YOUR_PROJECT_ID", "job_id_prefix%");
# Cancel jobs which match a job_id suffix
CALL bqutil.procedure.cancel_jobs("YOUR_PROJECT_ID", "%job_id_suffix");
```

### [delete_datasets(project_id STRING, dataset_like_filter STRING)](delete_datasets.sql)

Deletes all BigQuery datasets which match the input argument, dataset_like_filter.
```sql
# Delete datasets which match a dataset_id prefix
CALL bqutil.procedure.delete_datasets("YOUR_PROJECT_ID", "dataset_id_prefix%");
# Delete datasets which match a dataset_id suffix
CALL bqutil.procedure.delete_datasets("YOUR_PROJECT_ID", "%dataset_id_suffix");
```

### [GetNextIds(id_count INT64, OUT next_ids ARRAY<INT64>)](get_next_id.sql)
Generates next ids and inserts them into a sample table. This implementation prevents against race condition.

```sql
BEGIN
DECLARE next_ids ARRAY<INT64> DEFAULT [];
Expand Down
47 changes: 47 additions & 0 deletions stored_procedures/cancel_jobs.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
/*
* Copyright 2021 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/


CREATE OR REPLACE PROCEDURE bqutil.procedure.cancel_jobs (project_id STRING, job_id_like_filter STRING)
BEGIN
DECLARE job_ids ARRAY<STRING>;
DECLARE i INT64;

EXECUTE IMMEDIATE
FORMAT("""
(SELECT
ARRAY_AGG(job_id)
FROM
`%s`.`region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
WHERE
state IN ("RUNNING","PENDING")
AND job_id LIKE "%s"
AND job_id <> @@script.job_id);
""", project_id, job_id_like_filter)
INTO job_ids;

SET i = ARRAY_LENGTH(job_ids);
WHILE i > 0 DO
BEGIN
CALL BQ.JOBS.CANCEL(project_id || '.' || job_ids[ORDINAL(i)]);
SET i = i - 1;
EXCEPTION WHEN ERROR THEN
SET i = i - 1;
SELECT @@error.message;
CONTINUE;
END;
END WHILE;
END;
44 changes: 44 additions & 0 deletions stored_procedures/delete_datasets.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
/*
* Copyright 2021 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

CREATE OR REPLACE PROCEDURE bqutil.procedure.delete_datasets (project_id STRING, dataset_like_filter STRING)
BEGIN
DECLARE datasets ARRAY<STRING>;
DECLARE i INT64 DEFAULT 0;

EXECUTE IMMEDIATE
FORMAT("""
(SELECT
ARRAY_AGG(SCHEMA_NAME)
FROM region-us.INFORMATION_SCHEMA.SCHEMATA
WHERE SCHEMA_NAME LIKE "%s")
""", dataset_like_filter)
INTO datasets;

SET i = ARRAY_LENGTH(datasets);
WHILE i > 0 DO
BEGIN
EXECUTE IMMEDIATE
FORMAT("DROP SCHEMA IF EXISTS `%s`.%s CASCADE",
project_id, datasets[ORDINAL(i)]);
SET i = i - 1;
EXCEPTION WHEN ERROR THEN
SET i = i - 1;
SELECT @@error.message;
CONTINUE;
END;
END WHILE;
END;