Skip to content

[feat](fs)Refactor Catalog FS Access to Use Unified StorageProperties #52084

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 12 commits into
base: master
Choose a base branch
from
Open

[feat](fs)Refactor Catalog FS Access to Use Unified StorageProperties #52084

wants to merge 12 commits into from
+791 −913

Conversation

CalvinKirs
Copy link
Member

What problem does this PR solve?

Refactor Catalog FS Access to Use Unified StorageProperties

Description:

This PR refactors the file system access logic within the Catalog framework to adopt the new StorageProperties abstraction. Many internal catalogs—such as HMS—rely on the internal file system for file access from the Frontend (FE).

This change ensures that all file system operations initiated within the Catalog layer now use the unified StorageProperties mechanism, which provides consistent configuration handling and supports pluggable backends like HDFS, S3, OSS, COS, etc.

Changes:

Integrated StorageProperties into Catalog-related file system logic.

Replaced legacy FS property resolution with StorageProperties throughout Catalog modules.

Ensured backward compatibility with existing Catalogs using internal FS (e.g., HMS).

Refactored test cases to assert FS normalization behavior line-by-line instead of using a static map.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@CalvinKirs
init
f69bd95
@CalvinKirs
fix
8640605
@CalvinKirs
Merge remote-tracking branch 'refs/remotes/upstream/master' into mast…
03e6b9c 
…er-hms-new-fs
@CalvinKirs
Merge remote-tracking branch 'refs/remotes/upstream/master' into mast…
9c6e774 
…er-hms-new-fs

# Conflicts:
#	fe/fe-core/src/main/java/org/apache/doris/fs/FileSystemCache.java
@CalvinKirs
[Feat](fs)Refactor Catalog FS Access to Use Unified StorageProperties
76c93c7 
Description:
This PR refactors the file system access logic within the Catalog framework to adopt the new StorageProperties abstraction. Many internal catalogs—such as HMS—rely on the internal file system for file access from the Frontend (FE).

This change ensures that all file system operations initiated within the Catalog layer now use the unified StorageProperties mechanism, which provides consistent configuration handling and supports pluggable backends like HDFS, S3, OSS, COS, etc.

Changes:
Integrated StorageProperties into Catalog-related file system logic.

Replaced legacy FS property resolution with StorageProperties throughout Catalog modules.

Ensured backward compatibility with existing Catalogs using internal FS (e.g., HMS).

Refactored test cases to assert FS normalization behavior line-by-line instead of using a static map.
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@CalvinKirs
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33941 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 39362eae11cb84c10f837e2d230dd4aed2973a39, data reload: false

------ Round 1 ----------------------------------
q1	17591	5161	5078	5078
q2	1940	286	197	197
q3	10288	1317	708	708
q4	10225	1007	528	528
q5	7519	2314	2369	2314
q6	188	163	134	134
q7	910	752	615	615
q8	9300	1334	1127	1127
q9	6879	5092	5064	5064
q10	6930	2415	1969	1969
q11	486	296	278	278
q12	343	342	220	220
q13	17797	3710	3098	3098
q14	243	231	226	226
q15	549	481	482	481
q16	432	429	374	374
q17	626	873	392	392
q18	7860	7029	6989	6989
q19	1818	963	554	554
q20	342	347	233	233
q21	3743	3240	2393	2393
q22	1076	991	969	969
Total cold run time: 107085 ms
Total hot run time: 33941 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5286	5121	5086	5086
q2	246	335	222	222
q3	2180	2675	2294	2294
q4	1378	1806	1348	1348
q5	4192	4123	4309	4123
q6	211	177	143	143
q7	2026	1958	1777	1777
q8	2627	2575	2492	2492
q9	7225	7018	7061	7018
q10	3107	3297	2847	2847
q11	602	514	492	492
q12	683	787	629	629
q13	3557	3887	3304	3304
q14	294	297	287	287
q15	512	481	471	471
q16	453	500	448	448
q17	1186	1592	1398	1398
q18	7536	7198	6925	6925
q19	801	790	950	790
q20	1925	1973	1872	1872
q21	4724	4426	4301	4301
q22	1054	1025	1009	1009
Total cold run time: 51805 ms
Total hot run time: 49276 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186295 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 39362eae11cb84c10f837e2d230dd4aed2973a39, data reload: false

query1	987	397	395	395
query2	6571	1784	1821	1784
query3	6739	223	233	223
query4	25962	23718	23092	23092
query5	4314	603	479	479
query6	295	207	204	204
query7	4631	484	278	278
query8	261	217	210	210
query9	8646	2651	2655	2651
query10	503	328	272	272
query11	15485	15126	15028	15028
query12	169	110	113	110
query13	1660	558	427	427
query14	9378	6180	6202	6180
query15	193	188	161	161
query16	7263	644	467	467
query17	1172	741	568	568
query18	1986	403	299	299
query19	215	190	159	159
query20	124	114	114	114
query21	213	123	105	105
query22	4048	4082	4020	4020
query23	34473	33220	33247	33220
query24	8460	2367	2392	2367
query25	530	458	403	403
query26	1084	277	150	150
query27	2734	512	342	342
query28	4323	2111	2098	2098
query29	697	546	437	437
query30	281	217	197	197
query31	926	812	772	772
query32	71	65	65	65
query33	542	354	338	338
query34	775	867	528	528
query35	790	817	746	746
query36	940	950	887	887
query37	110	99	78	78
query38	4183	4058	4055	4055
query39	1478	1454	1406	1406
query40	218	115	109	109
query41	62	61	60	60
query42	128	107	111	107
query43	490	491	452	452
query44	1294	817	808	808
query45	191	178	172	172
query46	837	1027	624	624
query47	1789	1806	1698	1698
query48	383	414	318	318
query49	723	500	401	401
query50	624	715	417	417
query51	4135	4128	4097	4097
query52	108	105	99	99
query53	234	252	186	186
query54	579	576	526	526
query55	86	83	83	83
query56	320	297	300	297
query57	1193	1211	1139	1139
query58	267	264	252	252
query59	2487	2627	2521	2521
query60	334	317	311	311
query61	127	128	124	124
query62	801	710	654	654
query63	227	189	189	189
query64	3902	1096	782	782
query65	4292	4156	4189	4156
query66	1096	435	362	362
query67	15808	15639	15557	15557
query68	7909	893	540	540
query69	491	323	280	280
query70	1172	1116	1060	1060
query71	507	319	307	307
query72	5726	4684	4892	4684
query73	706	637	354	354
query74	8968	9117	8913	8913
query75	3880	3228	2717	2717
query76	3547	1195	763	763
query77	781	383	288	288
query78	10133	10187	9346	9346
query79	2387	842	583	583
query80	617	529	493	493
query81	477	258	220	220
query82	427	132	99	99
query83	286	263	236	236
query84	305	102	88	88
query85	795	350	307	307
query86	336	309	281	281
query87	4431	4413	4332	4332
query88	3423	2281	2288	2281
query89	391	311	285	285
query90	1926	205	206	205
query91	145	142	113	113
query92	78	66	56	56
query93	1823	961	591	591
query94	675	409	310	310
query95	378	300	293	293
query96	493	577	292	292
query97	2756	2785	2639	2639
query98	230	207	208	207
query99	1449	1419	1316	1316
Total cold run time: 273595 ms
Total hot run time: 186295 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.53 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 39362eae11cb84c10f837e2d230dd4aed2973a39, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.04	0.03
query3	0.24	0.07	0.07
query4	1.61	0.10	0.10
query5	0.44	0.42	0.40
query6	1.16	0.66	0.68
query7	0.03	0.01	0.02
query8	0.05	0.03	0.04
query9	0.57	0.52	0.52
query10	0.56	0.57	0.56
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.63	0.61	0.61
query14	0.80	0.83	0.82
query15	0.91	0.88	0.88
query16	0.39	0.39	0.39
query17	1.08	1.09	1.04
query18	0.23	0.21	0.21
query19	1.98	1.88	1.92
query20	0.02	0.01	0.01
query21	15.41	0.90	0.56
query22	0.76	1.20	0.88
query23	14.70	1.40	0.64
query24	7.11	2.08	0.51
query25	0.46	0.21	0.14
query26	0.66	0.16	0.14
query27	0.05	0.06	0.05
query28	9.62	0.90	0.43
query29	12.63	3.93	3.29
query30	0.25	0.08	0.06
query31	2.84	0.59	0.41
query32	3.23	0.55	0.47
query33	3.01	3.11	3.13
query34	16.08	5.32	4.76
query35	4.80	4.78	4.80
query36	0.70	0.51	0.49
query37	0.09	0.07	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.18	0.15	0.13
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 103.92 s
Total hot run time: 29.53 s

@CalvinKirs
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33931 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 46ec65a392150623762a73cab044e65d096f2c94, data reload: false

------ Round 1 ----------------------------------
q1	17594	5217	5057	5057
q2	1946	310	197	197
q3	10257	1324	732	732
q4	10210	1008	535	535
q5	7528	2351	2321	2321
q6	183	162	131	131
q7	894	726	617	617
q8	9322	1306	1058	1058
q9	6797	5060	5069	5060
q10	6875	2379	1946	1946
q11	498	293	271	271
q12	345	339	222	222
q13	17779	3688	3097	3097
q14	229	227	223	223
q15	569	474	486	474
q16	431	425	374	374
q17	628	869	382	382
q18	7588	7306	7152	7152
q19	1794	947	542	542
q20	336	337	225	225
q21	3873	3247	2364	2364
q22	1038	1025	951	951
Total cold run time: 106714 ms
Total hot run time: 33931 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5183	5073	5106	5073
q2	261	317	223	223
q3	2141	2643	2308	2308
q4	1346	1798	1389	1389
q5	4223	4097	4438	4097
q6	219	167	130	130
q7	1988	1872	1771	1771
q8	2567	2644	2504	2504
q9	7102	7084	7107	7084
q10	3065	3250	2886	2886
q11	575	548	487	487
q12	667	774	643	643
q13	3506	4048	3272	3272
q14	277	300	300	300
q15	548	462	479	462
q16	447	476	448	448
q17	1141	1527	1345	1345
q18	7364	7217	6946	6946
q19	787	796	791	791
q20	1906	2003	1828	1828
q21	4842	4476	4372	4372
q22	1054	980	992	980
Total cold run time: 51209 ms
Total hot run time: 49339 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185691 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 46ec65a392150623762a73cab044e65d096f2c94, data reload: false

query1	981	395	388	388
query2	6567	1805	1770	1770
query3	6745	221	224	221
query4	26451	23893	23283	23283
query5	4428	646	471	471
query6	323	224	200	200
query7	4620	509	284	284
query8	266	231	208	208
query9	8653	2613	2634	2613
query10	460	328	273	273
query11	15500	15125	15033	15033
query12	168	111	120	111
query13	1663	547	420	420
query14	9342	6059	6155	6059
query15	202	202	177	177
query16	7403	636	447	447
query17	1198	748	619	619
query18	2034	416	309	309
query19	198	217	161	161
query20	124	115	117	115
query21	216	126	113	113
query22	4091	4148	4181	4148
query23	33900	33045	33137	33045
query24	8477	2394	2385	2385
query25	542	458	441	441
query26	1232	272	148	148
query27	2748	510	349	349
query28	4316	2112	2085	2085
query29	716	545	456	456
query30	289	218	199	199
query31	938	835	763	763
query32	70	68	62	62
query33	556	365	306	306
query34	793	861	513	513
query35	791	816	741	741
query36	955	982	890	890
query37	120	96	74	74
query38	4091	4220	4024	4024
query39	1558	1429	1396	1396
query40	217	117	108	108
query41	63	60	65	60
query42	125	107	110	107
query43	467	499	470	470
query44	1310	822	823	822
query45	180	176	166	166
query46	841	1027	631	631
query47	1767	1796	1713	1713
query48	381	436	308	308
query49	748	489	386	386
query50	654	686	397	397
query51	4200	4113	4217	4113
query52	109	106	99	99
query53	224	258	181	181
query54	576	577	498	498
query55	88	82	90	82
query56	292	298	280	280
query57	1195	1184	1113	1113
query58	280	268	254	254
query59	2607	2638	2583	2583
query60	325	326	327	326
query61	148	120	124	120
query62	823	742	663	663
query63	223	193	183	183
query64	4273	998	669	669
query65	4295	4155	4165	4155
query66	1119	408	318	318
query67	15548	15603	15399	15399
query68	8538	888	536	536
query69	463	302	263	263
query70	1229	1095	1063	1063
query71	466	343	294	294
query72	5336	4701	4663	4663
query73	712	574	356	356
query74	9085	9036	8673	8673
query75	3875	3235	2713	2713
query76	3648	1200	771	771
query77	788	376	283	283
query78	10019	10249	9401	9401
query79	1921	826	572	572
query80	601	517	456	456
query81	489	259	219	219
query82	432	127	103	103
query83	257	252	245	245
query84	247	106	88	88
query85	819	350	308	308
query86	337	306	286	286
query87	4401	4466	4373	4373
query88	3383	2271	2255	2255
query89	385	315	295	295
query90	1923	208	213	208
query91	140	140	112	112
query92	72	62	63	62
query93	1152	948	599	599
query94	655	396	308	308
query95	373	290	294	290
query96	494	570	282	282
query97	2713	2780	2611	2611
query98	236	218	206	206
query99	1439	1412	1293	1293
Total cold run time: 273431 ms
Total hot run time: 185691 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.22 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 46ec65a392150623762a73cab044e65d096f2c94, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.03
query3	0.24	0.08	0.06
query4	1.62	0.10	0.11
query5	0.44	0.40	0.40
query6	1.19	0.68	0.66
query7	0.03	0.02	0.01
query8	0.05	0.04	0.03
query9	0.58	0.51	0.52
query10	0.57	0.59	0.56
query11	0.14	0.11	0.11
query12	0.15	0.12	0.11
query13	0.63	0.61	0.61
query14	0.80	0.82	0.83
query15	0.90	0.85	0.89
query16	0.39	0.38	0.39
query17	1.02	1.10	1.08
query18	0.22	0.21	0.20
query19	1.94	1.86	1.84
query20	0.01	0.01	0.02
query21	15.43	0.88	0.54
query22	0.76	1.19	0.60
query23	15.00	1.36	0.63
query24	7.91	0.76	0.65
query25	0.46	0.21	0.07
query26	0.68	0.18	0.15
query27	0.06	0.05	0.04
query28	8.40	0.92	0.45
query29	12.53	3.91	3.27
query30	0.26	0.09	0.07
query31	2.83	0.59	0.40
query32	3.25	0.56	0.47
query33	3.01	3.10	3.12
query34	16.11	5.36	4.77
query35	4.82	4.84	4.80
query36	0.68	0.50	0.49
query37	0.09	0.06	0.07
query38	0.06	0.05	0.03
query39	0.03	0.02	0.02
query40	0.17	0.14	0.14
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.72 s
Total hot run time: 29.22 s

@CalvinKirs
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34153 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 42cddc10cdc86a98a24b4d341f109d53f1a8c910, data reload: false

------ Round 1 ----------------------------------
q1	17588	5209	5100	5100
q2	1927	284	194	194
q3	10283	1249	693	693
q4	10220	1008	543	543
q5	7479	2364	2347	2347
q6	207	169	134	134
q7	889	721	596	596
q8	9314	1265	1124	1124
q9	6764	5080	5015	5015
q10	6964	2384	1977	1977
q11	483	308	287	287
q12	343	344	216	216
q13	17785	3660	3115	3115
q14	223	253	222	222
q15	588	480	483	480
q16	423	422	369	369
q17	621	859	379	379
q18	7433	7187	7241	7187
q19	1565	943	540	540
q20	335	331	227	227
q21	3966	3229	2410	2410
q22	1081	1020	998	998
Total cold run time: 106481 ms
Total hot run time: 34153 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5258	5044	5275	5044
q2	247	325	218	218
q3	2149	2623	2310	2310
q4	1350	1767	1323	1323
q5	4212	4107	4361	4107
q6	216	171	130	130
q7	1988	1931	1778	1778
q8	2610	2508	2490	2490
q9	7165	7111	7128	7111
q10	3047	3246	2931	2931
q11	584	503	492	492
q12	672	755	592	592
q13	3612	3938	3207	3207
q14	293	298	270	270
q15	529	473	482	473
q16	439	489	439	439
q17	1171	1598	1340	1340
q18	7590	7069	7173	7069
q19	774	779	952	779
q20	1937	1983	1808	1808
q21	4808	4387	4319	4319
q22	1076	1039	1011	1011
Total cold run time: 51727 ms
Total hot run time: 49241 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186120 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 42cddc10cdc86a98a24b4d341f109d53f1a8c910, data reload: false

query1	990	383	397	383
query2	6527	1815	1822	1815
query3	6740	220	222	220
query4	26176	23291	23336	23291
query5	4381	604	468	468
query6	319	228	222	222
query7	4636	483	287	287
query8	275	220	208	208
query9	8639	2657	2681	2657
query10	469	343	265	265
query11	15424	15020	14812	14812
query12	159	113	105	105
query13	1662	521	419	419
query14	9328	6307	6335	6307
query15	204	196	178	178
query16	7522	634	462	462
query17	1202	716	594	594
query18	2064	416	311	311
query19	214	205	160	160
query20	121	115	115	115
query21	220	127	111	111
query22	4029	4200	4008	4008
query23	34004	33016	33092	33016
query24	8455	2421	2385	2385
query25	536	468	429	429
query26	1225	273	165	165
query27	2734	511	343	343
query28	4287	2113	2092	2092
query29	757	547	438	438
query30	284	216	202	202
query31	919	829	759	759
query32	77	67	67	67
query33	551	363	336	336
query34	802	854	535	535
query35	785	822	745	745
query36	938	968	890	890
query37	104	99	93	93
query38	4141	4075	4098	4075
query39	1466	1425	1379	1379
query40	221	117	106	106
query41	65	60	60	60
query42	127	114	110	110
query43	513	486	492	486
query44	1338	844	846	844
query45	175	177	170	170
query46	837	1022	649	649
query47	1734	1767	1699	1699
query48	375	416	327	327
query49	774	480	398	398
query50	629	686	405	405
query51	4154	4165	4143	4143
query52	112	115	100	100
query53	227	259	191	191
query54	586	574	511	511
query55	89	87	86	86
query56	295	303	295	295
query57	1178	1188	1125	1125
query58	264	265	264	264
query59	2540	2607	2462	2462
query60	341	338	291	291
query61	125	128	123	123
query62	787	702	675	675
query63	224	186	189	186
query64	4291	988	677	677
query65	4274	4196	4176	4176
query66	1096	421	321	321
query67	15817	15483	15257	15257
query68	8006	940	542	542
query69	482	302	278	278
query70	1187	1097	1131	1097
query71	478	340	300	300
query72	5827	4792	4807	4792
query73	720	645	361	361
query74	8960	9170	8974	8974
query75	3911	3193	2704	2704
query76	3740	1180	755	755
query77	790	386	300	300
query78	10137	10253	9358	9358
query79	2199	832	582	582
query80	590	562	455	455
query81	494	269	223	223
query82	460	125	102	102
query83	249	257	239	239
query84	255	120	85	85
query85	788	352	319	319
query86	392	299	283	283
query87	4427	4373	4305	4305
query88	3738	2297	2273	2273
query89	381	324	284	284
query90	1912	214	208	208
query91	144	143	121	121
query92	82	60	57	57
query93	1688	979	589	589
query94	670	405	348	348
query95	375	294	285	285
query96	495	570	280	280
query97	2661	2733	2646	2646
query98	240	214	206	206
query99	1327	1392	1285	1285
Total cold run time: 274359 ms
Total hot run time: 186120 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.91 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 42cddc10cdc86a98a24b4d341f109d53f1a8c910, data reload: false

query1	0.03	0.04	0.02
query2	0.07	0.04	0.03
query3	0.25	0.07	0.07
query4	1.60	0.10	0.10
query5	0.43	0.42	0.40
query6	1.18	0.66	0.65
query7	0.03	0.01	0.02
query8	0.04	0.04	0.03
query9	0.59	0.52	0.53
query10	0.56	0.59	0.56
query11	0.16	0.10	0.11
query12	0.14	0.12	0.12
query13	0.64	0.61	0.61
query14	0.81	0.83	0.83
query15	0.91	0.85	0.88
query16	0.40	0.39	0.39
query17	1.07	1.10	1.05
query18	0.22	0.21	0.21
query19	1.93	1.80	1.89
query20	0.02	0.01	0.01
query21	15.39	0.91	0.56
query22	0.77	1.15	0.67
query23	14.96	1.38	0.64
query24	6.88	1.71	0.37
query25	0.47	0.10	0.21
query26	0.61	0.16	0.15
query27	0.07	0.06	0.05
query28	9.12	0.93	0.45
query29	12.52	3.99	3.26
query30	0.26	0.11	0.06
query31	2.82	0.60	0.40
query32	3.22	0.55	0.46
query33	3.06	3.04	3.14
query34	16.09	5.42	4.75
query35	4.82	4.80	4.78
query36	0.68	0.51	0.49
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.02	0.02	0.02
query40	0.17	0.13	0.14
query41	0.08	0.02	0.03
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.32 s
Total hot run time: 28.91 s

@CalvinKirs
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33776 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f9a37776c26af532b9860159bc613ee6592475b4, data reload: false

------ Round 1 ----------------------------------
q1	17598	5211	5059	5059
q2	1950	297	189	189
q3	10271	1287	712	712
q4	10235	999	532	532
q5	7561	2400	2318	2318
q6	182	163	129	129
q7	924	753	600	600
q8	9298	1276	1091	1091
q9	6813	5094	5153	5094
q10	6879	2367	1963	1963
q11	494	295	288	288
q12	338	346	210	210
q13	17786	3662	3023	3023
q14	226	231	227	227
q15	574	479	473	473
q16	426	427	375	375
q17	617	846	378	378
q18	7542	7257	6997	6997
q19	1685	944	542	542
q20	322	328	237	237
q21	3657	3489	2360	2360
q22	1059	1022	979	979
Total cold run time: 106437 ms
Total hot run time: 33776 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5106	5066	5082	5066
q2	235	324	220	220
q3	2134	2658	2244	2244
q4	1330	1770	1326	1326
q5	4231	4098	4377	4098
q6	215	177	125	125
q7	2069	1944	1777	1777
q8	2598	2477	2506	2477
q9	7312	7209	7143	7143
q10	3036	3213	2822	2822
q11	579	516	488	488
q12	689	747	610	610
q13	3507	3862	3192	3192
q14	291	313	261	261
q15	517	469	468	468
q16	441	504	437	437
q17	1127	1585	1328	1328
q18	7474	7031	7100	7031
q19	780	756	763	756
q20	1920	2029	1808	1808
q21	4680	4387	4199	4199
q22	1073	1039	989	989
Total cold run time: 51344 ms
Total hot run time: 48865 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184935 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f9a37776c26af532b9860159bc613ee6592475b4, data reload: false

query1	976	396	400	396
query2	6554	1766	1804	1766
query3	6746	223	221	221
query4	26222	23513	22820	22820
query5	4341	619	450	450
query6	297	199	193	193
query7	4616	500	285	285
query8	260	213	209	209
query9	8605	2639	2640	2639
query10	486	339	264	264
query11	15305	15041	14701	14701
query12	153	107	104	104
query13	1666	548	404	404
query14	8703	6181	6246	6181
query15	210	198	191	191
query16	7343	637	454	454
query17	1212	726	590	590
query18	1999	408	315	315
query19	196	199	176	176
query20	135	119	117	117
query21	213	126	112	112
query22	4119	4200	4126	4126
query23	34011	33059	33045	33045
query24	8503	2388	2444	2388
query25	549	435	391	391
query26	1226	272	154	154
query27	2761	503	339	339
query28	4285	2102	2087	2087
query29	777	545	427	427
query30	283	220	188	188
query31	893	821	761	761
query32	72	65	108	65
query33	554	371	305	305
query34	789	843	519	519
query35	823	819	723	723
query36	944	1000	863	863
query37	112	100	78	78
query38	4087	4092	4032	4032
query39	1502	1422	1404	1404
query40	220	124	132	124
query41	69	57	60	57
query42	128	110	108	108
query43	495	489	471	471
query44	1346	823	824	823
query45	191	181	174	174
query46	833	1015	642	642
query47	1735	1780	1712	1712
query48	387	418	316	316
query49	755	490	386	386
query50	637	662	404	404
query51	4113	4308	4059	4059
query52	110	108	104	104
query53	234	259	191	191
query54	582	575	513	513
query55	85	90	90	90
query56	319	311	297	297
query57	1182	1181	1106	1106
query58	266	260	253	253
query59	2476	2621	2475	2475
query60	337	325	304	304
query61	129	124	121	121
query62	807	735	677	677
query63	221	185	187	185
query64	4333	992	718	718
query65	4246	4214	4118	4118
query66	1149	412	319	319
query67	15682	15518	15309	15309
query68	8230	904	576	576
query69	480	322	268	268
query70	1218	1080	1047	1047
query71	455	321	308	308
query72	5541	4681	4869	4681
query73	694	618	359	359
query74	9038	9109	8857	8857
query75	3817	3176	2667	2667
query76	3641	1180	756	756
query77	786	371	295	295
query78	9904	10413	9366	9366
query79	1548	836	597	597
query80	594	524	512	512
query81	469	256	227	227
query82	408	134	100	100
query83	246	265	235	235
query84	242	114	82	82
query85	797	358	316	316
query86	337	305	279	279
query87	4383	4436	4314	4314
query88	2967	2302	2277	2277
query89	379	321	350	321
query90	1942	220	221	220
query91	141	161	108	108
query92	78	63	59	59
query93	1090	962	595	595
query94	676	404	275	275
query95	376	297	284	284
query96	498	570	278	278
query97	2744	2747	2667	2667
query98	235	205	210	205
query99	1559	1396	1265	1265
Total cold run time: 271108 ms
Total hot run time: 184935 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.05 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f9a37776c26af532b9860159bc613ee6592475b4, data reload: false

query1	0.03	0.03	0.02
query2	0.07	0.03	0.04
query3	0.24	0.08	0.07
query4	1.60	0.11	0.11
query5	0.42	0.42	0.43
query6	1.16	0.67	0.68
query7	0.03	0.01	0.02
query8	0.05	0.03	0.03
query9	0.57	0.52	0.51
query10	0.56	0.57	0.56
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.63	0.62	0.60
query14	0.80	0.80	0.84
query15	0.89	0.87	0.88
query16	0.39	0.40	0.40
query17	1.08	1.05	1.07
query18	0.22	0.20	0.21
query19	2.00	1.92	1.90
query20	0.02	0.01	0.01
query21	15.50	0.90	0.54
query22	0.75	1.23	0.60
query23	14.94	1.37	0.66
query24	7.58	0.70	0.34
query25	0.50	0.23	0.05
query26	0.55	0.16	0.14
query27	0.06	0.05	0.05
query28	8.68	0.93	0.46
query29	12.57	3.90	3.28
query30	0.26	0.09	0.07
query31	2.82	0.58	0.39
query32	3.24	0.56	0.48
query33	3.02	3.08	3.18
query34	16.01	5.40	4.77
query35	4.80	4.82	4.84
query36	0.68	0.51	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.02
query40	0.17	0.14	0.15
query41	0.08	0.02	0.02
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 103.53 s
Total hot run time: 29.05 s

@CalvinKirs
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34732 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0f9a25250809c166ba2f70472cbc4b01ee150bad, data reload: false

------ Round 1 ----------------------------------
q1	17601	5183	5038	5038
q2	1959	292	193	193
q3	10270	1296	703	703
q4	10228	1012	513	513
q5	7556	2341	2352	2341
q6	183	169	134	134
q7	875	741	610	610
q8	9331	1301	1085	1085
q9	6788	5116	5177	5116
q10	6919	2371	1962	1962
q11	499	296	280	280
q12	348	352	216	216
q13	17781	3720	3136	3136
q14	246	228	218	218
q15	578	474	472	472
q16	432	429	379	379
q17	612	884	383	383
q18	7726	7243	7216	7216
q19	1539	954	549	549
q20	345	358	239	239
q21	4091	3209	2977	2977
q22	1059	1016	972	972
Total cold run time: 106966 ms
Total hot run time: 34732 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5188	5138	5082	5082
q2	251	314	226	226
q3	2199	2656	2310	2310
q4	1327	1818	1384	1384
q5	4234	4197	4366	4197
q6	205	171	127	127
q7	2031	1881	1779	1779
q8	2604	2586	2446	2446
q9	7193	7134	7119	7119
q10	3072	3239	2794	2794
q11	605	514	489	489
q12	645	755	606	606
q13	3519	3955	3317	3317
q14	299	312	287	287
q15	536	494	466	466
q16	432	495	442	442
q17	1154	1577	1330	1330
q18	7413	7224	7164	7164
q19	788	783	791	783
q20	1986	1990	1849	1849
q21	4862	4379	4336	4336
q22	1073	1032	1031	1031
Total cold run time: 51616 ms
Total hot run time: 49564 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186159 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0f9a25250809c166ba2f70472cbc4b01ee150bad, data reload: false

query1	996	387	454	387
query2	6549	1819	1864	1819
query3	6753	228	223	223
query4	26280	23269	23743	23269
query5	4831	633	467	467
query6	322	215	210	210
query7	4639	496	287	287
query8	274	237	213	213
query9	8643	2641	2670	2641
query10	499	343	318	318
query11	15693	15053	15047	15047
query12	176	113	118	113
query13	1665	561	431	431
query14	9610	6260	6294	6260
query15	203	207	185	185
query16	7344	631	479	479
query17	1161	720	576	576
query18	2102	405	311	311
query19	199	208	169	169
query20	124	127	118	118
query21	218	125	117	117
query22	4013	4257	4019	4019
query23	33923	33153	33316	33153
query24	7970	2385	2399	2385
query25	536	455	405	405
query26	730	273	153	153
query27	2685	507	346	346
query28	4166	2128	2092	2092
query29	670	568	425	425
query30	283	219	190	190
query31	927	832	752	752
query32	74	72	65	65
query33	514	367	305	305
query34	808	867	530	530
query35	787	824	727	727
query36	945	984	903	903
query37	121	99	75	75
query38	4132	4082	4067	4067
query39	1498	1416	1407	1407
query40	212	121	108	108
query41	61	68	72	68
query42	140	109	109	109
query43	516	501	476	476
query44	1304	849	818	818
query45	185	172	174	172
query46	850	1026	634	634
query47	1755	1814	1700	1700
query48	395	426	315	315
query49	696	468	396	396
query50	656	687	409	409
query51	4034	4198	4216	4198
query52	113	113	105	105
query53	229	264	183	183
query54	574	574	503	503
query55	97	83	88	83
query56	313	301	290	290
query57	1196	1200	1124	1124
query58	276	262	267	262
query59	2477	2639	2538	2538
query60	324	327	316	316
query61	154	125	125	125
query62	800	722	669	669
query63	220	188	192	188
query64	2872	992	673	673
query65	4323	4195	4207	4195
query66	931	412	318	318
query67	15634	15608	15426	15426
query68	8089	852	537	537
query69	474	320	274	274
query70	1272	1111	1074	1074
query71	471	334	290	290
query72	5355	4763	4784	4763
query73	724	614	353	353
query74	8956	8854	8688	8688
query75	3853	3209	2733	2733
query76	3610	1200	763	763
query77	785	394	284	284
query78	10113	10376	9244	9244
query79	1876	819	592	592
query80	587	512	446	446
query81	491	255	223	223
query82	199	128	104	104
query83	263	261	241	241
query84	253	111	88	88
query85	774	386	318	318
query86	379	307	299	299
query87	4447	4442	4263	4263
query88	3402	2284	2270	2270
query89	386	315	284	284
query90	2012	211	211	211
query91	143	148	111	111
query92	78	61	57	57
query93	1673	964	597	597
query94	643	423	291	291
query95	375	300	290	290
query96	498	569	283	283
query97	2705	2765	2657	2657
query98	241	264	198	198
query99	1339	1418	1293	1293
Total cold run time: 270866 ms
Total hot run time: 186159 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.61 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0f9a25250809c166ba2f70472cbc4b01ee150bad, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.24	0.07	0.07
query4	1.60	0.10	0.11
query5	0.44	0.44	0.42
query6	1.15	0.66	0.66
query7	0.03	0.02	0.02
query8	0.04	0.03	0.04
query9	0.58	0.51	0.52
query10	0.57	0.58	0.55
query11	0.16	0.11	0.12
query12	0.16	0.12	0.12
query13	0.65	0.60	0.61
query14	0.82	0.81	0.83
query15	0.90	0.89	0.88
query16	0.39	0.39	0.41
query17	1.07	1.06	1.04
query18	0.22	0.21	0.21
query19	1.94	1.88	1.83
query20	0.01	0.01	0.01
query21	15.43	0.91	0.54
query22	0.76	1.28	0.65
query23	14.79	1.41	0.64
query24	7.08	1.25	0.89
query25	0.47	0.27	0.06
query26	0.56	0.17	0.14
query27	0.06	0.05	0.05
query28	9.45	0.90	0.46
query29	12.58	3.97	3.27
query30	0.25	0.09	0.06
query31	2.83	0.59	0.40
query32	3.23	0.54	0.46
query33	3.03	3.08	3.09
query34	15.91	5.44	4.82
query35	4.84	4.86	4.86
query36	0.69	0.51	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.18	0.14	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.55 s
Total hot run time: 29.61 s

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 57.96% (142/245) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@CalvinKirs @hello-stephen @doris-robot