Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BugFix] br_cgu_cartao_pagamento #922

Merged
merged 3 commits into from
Jan 16, 2025
Merged

[BugFix] br_cgu_cartao_pagamento #922

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
71264b5
fix pipeline cartao pagamento
tricktx Jan 16, 2025
6af302e
fix pipeline cartao pagamento register now
tricktx Jan 16, 2025
b7a9667
register flow definitive
tricktx Jan 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions pipelines/datasets/br_cgu_cartao_pagamento/schedules.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
from pipelines.constants import constants
from pipelines.utils.crawler_cgu.constants import constants as constants_cgu


every_day_microdados_governo_federal = Schedule(
clocks=[
CronClock(
Expand Down
1 change: 1 addition & 0 deletions pipelines/utils/crawler_cgu/constants.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,3 +175,4 @@ class constants(Enum): # pylint: disable=c0103
"READ": "_TermoAditivo.csv",
},
}

4 changes: 3 additions & 1 deletion pipelines/utils/crawler_cgu/flows.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@
) as flow_cgu_cartao_pagamento:

dataset_id = Parameter("dataset_id", default='br_cgu_cartao_pagamento', required=True)
table_id = Parameter("table_id", default ="microdados_governo_federal", required=True)
table_id = Parameter("table_id", required=True)
####
# Relative_month = 1 means that the data will be downloaded for the current month
####
Expand Down Expand Up @@ -317,3 +317,5 @@
)
flow_cgu_licitacao_contrato.storage = GCS(constants.GCS_FLOWS_BUCKET.value)
flow_cgu_licitacao_contrato.run_config = KubernetesRun(image=constants.DOCKER_IMAGE.value)


5 changes: 3 additions & 2 deletions pipelines/utils/crawler_cgu/tasks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,8 @@ def partition_data(table_id: str, dataset_id : str) -> str:
log("---------------------------- Read data ----------------------------")
df = read_csv(dataset_id = dataset_id, table_id = table_id)
log(df.head())
if dataset_id == "br_cgu_cartao_pagamento:":
log(" ---------------------------- Partiting data -----------------------")
if dataset_id == "br_cgu_cartao_pagamento":
log("---------------------------- Partiting data -----------------------")
to_partitions(
data = df,
partition_columns=['ANO_EXTRATO', 'MES_EXTRATO'],
Expand Down Expand Up @@ -77,6 +77,7 @@ def partition_data(table_id: str, dataset_id : str) -> str:
log("---------------------------- Data partitioned ----------------------")
return constants.TABELA_SERVIDORES.value[table_id]['OUTPUT']


@task
def get_current_date_and_download_file(table_id : str,
dataset_id : str,
Expand Down
1 change: 1 addition & 0 deletions pipelines/utils/crawler_cgu/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
"""
General purpose functions for the br_cgu_cartao_pagamento project
"""

import datetime
from arrow import get
from dateutil.relativedelta import relativedelta
Expand Down
Loading