{WIP} Use SQLite as the file format for CDC streaming. #715
Conversation
| " and startpos <= $1 " | ||
| " and (endpos is null or $2 <= endpos) " |
There was a problem hiding this comment.
@dimitri I'm wondering how comparison works here? AFAIK, SQlite stores pg_lsn type as just a blob and we don't seem to provide comparison operator either. Probably because Sqlite uses memcmp to compare blobs and it works correctly on hex string?
There was a problem hiding this comment.
That's the gist of it yeah, LSN as text in hexadecimal still allows using the usual operators to compare values... (I believe?)
| if (privateContext->timeline == 0) | ||
| { | ||
| log_error("BUG: ld_store_open_replaydb: timeline is zero"); | ||
| return false; | ||
| } | ||
|
|
||
| sformat(replayDB->dbfile, MAXPGPATH, "%s/%08d-%08X-%08X.db", | ||
| specs->paths.dir, | ||
| privateContext->timeline, | ||
| LSN_FORMAT_ARGS(privateContext->startpos)); |
There was a problem hiding this comment.
IIUC, it is one sqlite database per timeline? If that is the case, I afraid the sqlite database file would grow larger. Currently, we have a script in place to cleanup old files(.json,.sql) which are already applied into the target.
There was a problem hiding this comment.
The idea is to re-introduce file rotation later. It just seems to much to solve at once, so I'm not sure yet if I'll be able to introduce it within the PR or if it will have to wait. The file rotation will not split within a transaction this time, though, by design.
There was a problem hiding this comment.
Makes sense. Thanks @dimitri .
dimitri
force-pushed
the
feature/use-sqlite-for-cdc
branch
2 times, most recently
from
e35a012
to
b290f30
Compare
dimitri
force-pushed
the
feature/use-sqlite-for-cdc
branch
3 times, most recently
from
daea3f5
to
9d46a32
Compare
dimitri
force-pushed
the
feature/use-sqlite-for-cdc
branch
from
9d46a32
to
b83612d
Compare
dimitri
force-pushed
the
feature/use-sqlite-for-cdc
branch
from
b83612d
to
eadf844
Compare
dimitri
force-pushed
the
feature/use-sqlite-for-cdc
branch
2 times, most recently
from
6dd703b
to
bdb7593
Compare
dimitri
force-pushed
the
feature/use-sqlite-for-cdc
branch
2 times, most recently
from
9520000
to
cf3e7e4
Compare
Add new options to `pgcopydb stream sentinel get` allowing to fetch a single LSN value, avoiding the need to use `awk`, `jq`, or something equivalent to then parse a single entry in the command output.
In unit tests where we use both an inject and a test services, we need the inject service to access the same pgcopydb work directory as the test service. For that we use an external docker volume. The previous implementation of that idea was found to be wrong, this is fixing it by properly using docker APIs for shared "external" volumes.
dimitri
force-pushed
the
feature/use-sqlite-for-cdc
branch
from
cf3e7e4
to
5e3dfe2
Compare
This is a Work-In-Progress refactoring. The only reason a PR already exists is to allow me to focus on other bits and pieces and get back to that work later. Feedback is welcome of course, discussion is opened.