how to make ceph radosgw as storage? #37
Comments
|
yea. ceph radosgw by default is using Path-Style Addressing. and see if it works? |
|
thanks for your help. URL_STYLE worked to some extend. pg_mooncake did not request domain # psql -U postgres -h 127.0.0.1
psql (16.4)
Type "help" for help.
postgres=# CREATE EXTENSION pg_mooncake;
CREATE EXTENSION
postgres=# SELECT mooncake.create_secret('test', 'S3', 'J99FFEUCNV428W9DXS5M', '0nGNMIACzFOpRgD1XuqbmtOhv7V3mQWIPT71eAOt', '{"ENDPOINT":"192.168.2.201", "USE_SSL": 0}');
create_secret
---------------
(1 row)
postgres=# update mooncake.secrets set duckdb_query='CREATE SECRET "duckdb_secret_test" (TYPE S3, KEY_ID ''J99FFEUCNV428W9DXS5M'', SECRET ''0nGNMIACzFOpRgD1XuqbmtOhv7V3mQWIPT71eAOt'', ENDPOINT ''192.168.2.201'', USE_SSL FALSE, URL_STYLE ''PATH'')';
UPDATE 1
postgres=# SELECT duckdb_query FROM mooncake.secrets;
duckdb_query
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CREATE SECRET "duckdb_secret_test" (TYPE S3, KEY_ID 'J99FFEUCNV428W9DXS5M', SECRET '0nGNMIACzFOpRgD1XuqbmtOhv7V3mQWIPT71eAOt', ENDPOINT '192.168.2.201', USE_SSL FALSE, URL_STYLE 'PATH')
(1 row)
postgres=# SET mooncake.default_bucket = 's3://test';
SET
postgres=# CREATE TABLE pg_mooncake_test(id BIGINT, log TEXT, ts TIMESTAMP) USING columnstore;
CREATE TABLE
postgres=# INSERT INTO pg_mooncake_test VALUES (1, 'test', '2024-01-01 08:00:00');
ERROR: (PGDuckDB/ExecuteQuery) Invalid Error: Unexpected response while initializing S3 multipart upload
postgres=#
ceph radosgw log, |
|
urrr. seems we are hitting some duckdb issue. try this: |
|
great! it worked. though CREATE TABLE and INSERT were slow on s3 storage, say 10 seconds, they completed with no errors. # psql -U postgres -h 127.0.0.1
psql (16.4)
Type "help" for help.
postgres=# \timing
Timing is on.
postgres=# CREATE EXTENSION pg_mooncake;
CREATE EXTENSION
Time: 48.550 ms
postgres=# SELECT mooncake.create_secret('test', 'S3', 'J99FFEUCNV428W9DXS5M', '0nGNMIACzFOpRgD1XuqbmtOhv7V3mQWIPT71eAOt', '{"ENDPOINT":"192.168.2.201", "USE_SSL": 0}');
create_secret
---------------
(1 row)
Time: 7.540 ms
postgres=# update mooncake.secrets set duckdb_query='CREATE SECRET "duckdb_secret_test" (TYPE S3, KEY_ID ''J99FFEUCNV428W9DXS5M'', SECRET ''0nGNMIACzFOpRgD1XuqbmtOhv7V3mQWIPT71eAOt'', ENDPOINT ''192.168.2.201/test'', USE_SSL FALSE, URL_STYLE ''PATH'')';
UPDATE 1
Time: 6.783 ms
postgres=# SET mooncake.default_bucket = 's3://test';
SET
Time: 0.337 ms
postgres=# CREATE TABLE pg_mooncake_test(id BIGINT, log TEXT, ts TIMESTAMP) USING columnstore;
CREATE TABLE
Time: 9731.790 ms (00:09.732)
postgres=# INSERT INTO pg_mooncake_test VALUES (1, 'test', '2024-01-01 08:00:00');
INSERT 0 1
Time: 11142.336 ms (00:11.142)
postgres=# SELECT * FROM pg_mooncake_test;
id | log | ts
----+------+---------------------
1 | test | 2024-01-01 08:00:00
(1 row)
Time: 3.287 ms
postgres=# s3 bucket was like, # s3cmd ls s3://test/mooncake_postgres_pg_mooncake_test_16419/
DIR s3://test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/
2024-11-20 02:15 435 s3://test/mooncake_postgres_pg_mooncake_test_16419/d6eef5aa-8c9c-4285-b532-836865a9e934.parquetlocal cache, # ls -l /var/lib/pgsql/data/mooncake_local_cache/
total 4
-rw------- 1 postgres postgres 435 Nov 20 10:15 d6eef5aa-8c9c-4285-b532-836865a9e934.parquetceph side, |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What happens?
i tried to use ceph radosgw as an s3 storage for pg_mooncake but failed. did i do something wrong?
To Reproduce
first, i built pg_mooncake from latest git with this fix, for postgresql 16.4 on RHEL8-like system.
the ceph 14.2.22 rgw was configured on http://192.168.2.201 and the bucket name is 'test'. i tried,
CREATE TABLE succeeded, while INSERT INTO failed with,
add host name "test.192.168.2.201" to /etc/hosts was not a workaround, since ceph expected something like
http://192.168.2.201/<bucket-name>/<path>/, but pg_mooncake senthttp://<bucket-name>.192.168.2.201/<path>/instead. it seams nginx reverse proxy may fix this but i did not succeeded on it.why pg_mooncake talked with rgw well during CREATE TABLE but not INSERT INTO?
ceph rgw log,
the bucket was like (the content of the json file is log of CREATE TABLE),
OS:
RHEL 8
pg_mooncake Version:
f1818d4
Postgres Version:
16.4
Are you using pg_mooncake Docker, Neon, or the extension standalone?
pg_mooncake extension standalone
The text was updated successfully, but these errors were encountered: