Fix #1001: [Lake][ETL] Implement incremental ETL pipeline

[idiom-bytes]

Pull Request Description

By the way of cherry-picking all changes found in PR #1000, this PR incorporates all updates required to implement the incremental ETL pipeline

What happened?
branch 685 was deleted... I did not realize this PR was a branch from there, so the original PR was also closed....this PR rescues all the work and prepares all changes for merging.

• Reviewed new DuckDB/CRUD functions

[calina-c]

I can't tell which comments are new and which old. I trust that you resolved only comments that you addressed. I would also recommend a second look from another dev on the team, just to be sure. Otherwise LGTM.

[idiom-bytes]

I can't tell which comments are new and which old. I trust that you resolved only comments that you addressed. I would also recommend a second look from another dev on the team, just to be sure. Otherwise LGTM.

I have unresolved the conversation on all items where I didn't implement any change.

Why? either the comments:

• do not apply, i.e. I am using row_count() and move_data_from_table_to_table()
• seemed subjective enough where I strongly disagree, i.e. use new terminology that's not used elsewhere (like source and destination, instead of from and to that's used everywhere)
• should not implement due to architectural or engineering related considerations: i.e. move core logic and code to other files, or change component logic in a way that doesn't make sense relative to it's function

@KatunaNorbert @kdetry can you please review, test, or provide any feedback/response?
i'll address everything applicable and leave the rest for a final chat/sign off.
cheers!

[KatunaNorbert]

Tested is and I might found an issue. The trueval field is NULL for all the rows inside the bronze_pdr_predictions table

[idiom-bytes]

You did find an issue. Nice work.

truevalue had never been implemented from the subgraph fetch, so it was missing from payout.

it's now being fetched from subgraph -> saved to csv -> processed in ETL -> updated in bronze table

I have updated the ETL tests to validate that it's working

[trizin]

I noticed the mermaid diagram added. Not sure if this is in the scope of the PR but here's how the table structure could be improved:

• pair, timeframe, source: duplicated data, should be derived from contract address
• remove PDR_PREDICTIONS and PDR_PAYOUTS, just a single table BRONZE_PDR_PREDICTIONS should be enough
• slot_id is literally {contract}-{slot}, it's redundant.

[idiom-bytes]

Answering trizin's comments in here

slot_id is literally {contract}-{slot}, it's redundant
I have a PR to work update slot-related queries, i can implement it there -> #1466

slot_id will be required for certain joins. Having this pre-processed (string split) and then available for joins (string equal vs. string like operation) will save computation time when it's needed most.

pair, timeframe, source: duplicated data, should be derived from contract address
100% agree, I've wanted to do this for a while. i'm working towards this and added this to the ETL & Analytics backlog.

remove PDR_PREDICTIONS and PDR_PAYOUTS, just a single table BRONZE_PDR_PREDICTIONS should be enough

This is the goal (to only have bronze), this would also reduce DB size. The challenge is that we need to build all the bronze tables before removing all raw_data from DB.

Now that we have Incremental ETL, this should be possible.

I'm proposing we deprecate raw tables once we have more bronze tables in place, and start working towards silver (aggregate) tables

Challenge:
If raw tables aren't in the DB, then any ETL work is going to be slow (scanning + reading CSVs into memory for processing)
If raw tables are in DB, then DuckDB is very fast and we can do queries/aggregates/summaries efficiently.

because we have discussed all the feedback and there were no more comments

[idiom-bytes]

We had multiple reviewers, discussed feedback, and agreed on how to move forward across the board. Merging.