Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Store opensanctions data based on run IDs #839

Closed
wants to merge 168 commits into from
Closed

Store opensanctions data based on run IDs #839

wants to merge 168 commits into from

Conversation

pudo
Copy link
Member

@pudo pudo commented May 10, 2024

No description provided.

@pudo pudo marked this pull request as draft May 10, 2024 18:25
SimonThordal
SimonThordal reviewed May 13, 2024
View reviewed changes
Copy link
Contributor

@SimonThordal SimonThordal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!
So this stores the history of a dataset in runs/{dataset}/meta.json where each run is stored as {"items": ["12042024-ABCD", "13042024-AFEB",...]}, correct?

zavod/zavod/runs.py Outdated Show resolved Hide resolved
zavod/zavod/runs.py Outdated Show resolved Hide resolved
@pudo
Copy link
Member Author

pudo commented May 13, 2024

Yeah it might be too simple but so far I haven't yet figured out how - it just keeps a list of history items and also puts it into a run folder. So if the items in the index file is limited to 100 and you want to go back 120, you just fetch the latest history, and then history file inside of the run folder for items[0] (the oldest run)....

pudo and others added 23 commits May 14, 2024 17:51
@pudo
remove invalid retry
13b8a2b
@pudo
address simon's feedback
5233a8c
@pudo
actually store statements.pack and entities.ftm.json in run folder
d63ef48
@pudo
rename function
b040446
@pudo
adapt for new nomenklatura version
6628a05
@pudo
redis-based delta exporter
e1e3ca4
@pudo
fix opcode
9fd7365
@pudo
up nk
caba63c
@pudo
fix sqlite handling in tests
31c6c55
@pudo
rename runs to artifacts
c1ac8b0
@pudo
ignore weird google tooling warnings
2926136
@pudo
try to make sense of the artifacts logic
542b813
@pudo
continue introducing artifact/resource separation
2433af5
@pudo
clean up redundant functionality, adapt tests
f2c3c52
@jbothma @pudo
Rough idea for dangling references
fb54e39
@jbothma @pudo
Rough idea of self-reference
0156d31
@jbothma @pudo
Test dataset
33e76d8
@jbothma @pudo
Failing test
e1bda79
@jbothma @pudo
dangling passes
465b3f6
@jbothma @pudo
Passing tests
d18534a
@jbothma @pudo
Tidy
bfc5e3e
@jbothma @pudo
Check topicless targets tittelatingly using trivia
9feb1bd
@jbothma @pudo
Move assertions to verify step
fbfdc8f
jbothma and others added 28 commits May 24, 2024 16:48
@jbothma @pudo
New nigerian sanctions page
e8ec31d
@jbothma @pudo
Handle organisations listed as individuals
c9864ab
@jbothma @pudo
Use name mappings we have
69fd1df
@jbothma @pudo
Tidy
e1ddddc
@jbothma @pudo
lint
143639c
@jbothma @pudo
Add enricher for the Kazakh company register
13bb90f
@jbothma @pudo
Fixes
68dd51d
@jbothma @pudo
Update query to find moved warnings, more consistent IDs
8482d6c
@jbothma @pudo
Awkward
bf108f6
@jbothma @pudo
Note exception
f2a3b89
@jbothma @pudo
New position
82e1518
@jbothma @pudo
Exclude new non-pep positions
596e529
@bgmello @pudo
adding Luxembourg administrative sanctions
55dd7ed
@bgmello @pudo
adding new line at the end of YAML
40d7dd3
@bgmello @jbothma @pudo
Apply suggestions from code review
92d7574 
Co-authored-by: JD Bothma <[email protected]>
@bgmello @pudo
Passing list of names in name property of entity
4e36b60
@bgmello @pudo
Improving date parser
c1edb52
@bgmello @pudo
Changing spacing to four spaces in publisher description
433ac81
@bgmello @pudo
Improving name parser by using REGEX
5e6bbe1
@bgmello @pudo
Using only listing page to crawl data
a8e845e
@bgmello @pudo
Changing topic to reg.warn
aa1bcbd
@bgmello @pudo
Improving lookup by checking names on PDFs
2648244
@jbothma @pudo
Fix topic and sanction, add detail URL
9a78b56
@jbothma @pudo
Fix bare except and handle missing date
6a673ed
@jbothma @pudo
Ruffen some more
a5032fb
@pudo
Remove runs code that is now in NK
a0619f5
@pudo
try and make the delta exporter pass type checks
49c26cb
@pudo
wat
353a600
@pudo pudo closed this May 24, 2024
@pudo pudo deleted the pudo/run-id branch May 24, 2024 15:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@SimonThordal SimonThordal SimonThordal left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@pudo @SimonThordal @jbothma @dchaplinsky @bgmello