Skip to content

[enhance](nereids)rewrite aggregate to limit when all group by key is uniform and not null, and there is no aggregate functions (#46223) #52091

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: branch-3.1
Choose a base branch
from
Open

[enhance](nereids)rewrite aggregate to limit when all group by key is uniform and not null, and there is no aggregate functions (#46223) #52091

wants to merge 2 commits into from
+84 −8

Conversation

feiniaofeiafei
Copy link
Contributor

cherry-pick from #46223

@feiniaofeiafei
[enhance](nereids)rewrite aggregate to limit when all group by key is…
a7a19f8 
… uniform and not null, and there is no aggregate functions (apache#46223)

if c is uniform and aggregate can be eliminated, we could do transform as below
select 1 c1 from test group by c; -> select 1 c1 from test limit 1;
@Thearas
Copy link
Contributor

Thearas commented Jun 20, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@feiniaofeiafei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39817 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a7a19f8e7e049ec7c8c58957f0f1a3a6a81c4105, data reload: false

------ Round 1 ----------------------------------
q1	17876	6905	6629	6629
q2	2056	199	165	165
q3	10539	1136	1141	1136
q4	10215	751	729	729
q5	7712	2883	2853	2853
q6	217	132	135	132
q7	968	614	615	614
q8	9382	1945	2028	1945
q9	6593	6403	6427	6403
q10	6993	2285	2328	2285
q11	461	262	266	262
q12	396	218	211	211
q13	17761	2969	3005	2969
q14	240	205	219	205
q15	509	470	470	470
q16	446	374	394	374
q17	973	673	613	613
q18	7272	6621	6528	6528
q19	1322	1003	950	950
q20	500	204	206	204
q21	3946	3192	3164	3164
q22	1112	993	976	976
Total cold run time: 107489 ms
Total hot run time: 39817 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6614	6520	6585	6520
q2	327	235	231	231
q3	2961	2744	2841	2744
q4	2018	1883	1775	1775
q5	5671	5727	5671	5671
q6	211	126	128	126
q7	2210	1818	1780	1780
q8	3309	3465	3477	3465
q9	8876	8844	8839	8839
q10	3570	3518	3560	3518
q11	603	493	490	490
q12	834	608	615	608
q13	15298	3114	3190	3114
q14	312	278	262	262
q15	510	482	479	479
q16	489	441	468	441
q17	1892	1636	1575	1575
q18	8095	7675	7668	7668
q19	1675	1626	1466	1466
q20	2113	1897	1846	1846
q21	5206	5042	4935	4935
q22	1063	1007	997	997
Total cold run time: 73857 ms
Total hot run time: 58550 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190463 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a7a19f8e7e049ec7c8c58957f0f1a3a6a81c4105, data reload: false

query1	962	384	363	363
query2	6543	2034	1937	1937
query3	6708	233	230	230
query4	33748	23637	23945	23637
query5	4314	505	467	467
query6	267	177	177	177
query7	4611	302	308	302
query8	272	224	212	212
query9	9686	2564	2565	2564
query10	483	258	257	257
query11	18576	15278	15216	15216
query12	149	104	105	104
query13	1641	431	411	411
query14	9081	7168	7324	7168
query15	208	173	182	173
query16	7903	434	451	434
query17	1633	579	569	569
query18	2034	310	315	310
query19	220	165	162	162
query20	118	109	107	107
query21	203	103	104	103
query22	4312	4153	4032	4032
query23	34341	33676	33318	33318
query24	11568	2824	2953	2824
query25	706	408	410	408
query26	1477	172	174	172
query27	2993	353	346	346
query28	7622	2089	2094	2089
query29	986	440	442	440
query30	321	162	168	162
query31	1016	814	812	812
query32	99	59	63	59
query33	794	314	317	314
query34	911	509	527	509
query35	859	713	691	691
query36	1076	941	924	924
query37	132	69	72	69
query38	3913	3825	3811	3811
query39	1501	1462	1442	1442
query40	293	107	105	105
query41	50	51	49	49
query42	118	104	100	100
query43	519	496	495	495
query44	1220	805	786	786
query45	182	172	170	170
query46	1139	736	709	709
query47	1912	1811	1833	1811
query48	442	339	348	339
query49	1210	420	409	409
query50	847	421	423	421
query51	7334	7198	7121	7121
query52	108	94	91	91
query53	264	185	189	185
query54	1281	482	471	471
query55	85	81	88	81
query56	285	258	257	257
query57	1292	1144	1176	1144
query58	261	226	224	224
query59	3149	2936	2877	2877
query60	291	273	259	259
query61	115	111	110	110
query62	875	686	668	668
query63	222	185	187	185
query64	5329	633	626	626
query65	3293	3195	3189	3189
query66	1450	311	336	311
query67	16095	15786	15331	15331
query68	4520	589	588	588
query69	451	273	266	266
query70	1159	1070	1087	1070
query71	409	249	249	249
query72	6490	3951	4070	3951
query73	757	350	366	350
query74	10249	9176	9300	9176
query75	3466	2650	2659	2650
query76	2955	1105	1117	1105
query77	443	261	277	261
query78	10463	9641	9576	9576
query79	1512	589	593	589
query80	1050	441	431	431
query81	522	220	218	218
query82	962	90	86	86
query83	223	145	142	142
query84	243	79	80	79
query85	1286	329	298	298
query86	358	306	296	296
query87	4507	4304	4244	4244
query88	3586	2401	2376	2376
query89	408	292	288	288
query90	1910	190	189	189
query91	135	110	111	110
query92	61	53	50	50
query93	1082	551	545	545
query94	913	305	279	279
query95	367	259	258	258
query96	601	290	280	280
query97	3305	3226	3140	3140
query98	211	204	189	189
query99	1497	1285	1310	1285
Total cold run time: 299749 ms
Total hot run time: 190463 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.8 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a7a19f8e7e049ec7c8c58957f0f1a3a6a81c4105, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.63	0.11	0.10
query5	0.50	0.51	0.50
query6	1.13	0.72	0.72
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.56	0.51	0.50
query10	0.54	0.56	0.59
query11	0.15	0.11	0.11
query12	0.14	0.12	0.11
query13	0.61	0.60	0.60
query14	0.77	0.81	0.79
query15	0.85	0.83	0.82
query16	0.37	0.38	0.38
query17	1.04	1.04	1.00
query18	0.23	0.21	0.21
query19	1.88	1.79	1.81
query20	0.02	0.01	0.01
query21	15.43	0.59	0.59
query22	2.29	1.62	1.52
query23	17.05	1.05	0.89
query24	2.93	0.92	0.82
query25	0.25	0.12	0.11
query26	0.26	0.14	0.14
query27	0.05	0.04	0.05
query28	10.87	0.54	0.51
query29	12.53	3.23	3.24
query30	0.24	0.06	0.06
query31	2.85	0.39	0.39
query32	3.24	0.46	0.46
query33	2.98	2.99	3.00
query34	17.07	4.41	4.58
query35	4.51	4.50	4.52
query36	0.67	0.49	0.48
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.03	0.02
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 104.51 s
Total hot run time: 29.8 s

@feiniaofeiafei
Copy link
Contributor Author

run buildall

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morrySnow morrySnow Awaiting requested review from morrySnow morrySnow is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@feiniaofeiafei @Thearas @doris-robot