Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](local shuffle) Set serial execution for schema scan operator #47498

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

Gabriel39
Copy link
Contributor

@Gabriel39 Gabriel39 commented Feb 5, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

select * from information_schema.workload_group_resource_usage; got duplicate rows if enable_local_shuffle is true. This is caused by a serial schema scan operator which is not set correctly.

A regression test case could not be used because the workload_group_resource_usage will not get a stable result.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Feb 5, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add regression test for this case

@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32319 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3cf8787f999dbf490eae8b2c9597940db1a2683b, data reload: false

------ Round 1 ----------------------------------
q1	17578	5567	5353	5353
q2	2048	315	173	173
q3	10415	1246	747	747
q4	10213	951	528	528
q5	7549	2373	2121	2121
q6	185	163	132	132
q7	892	754	612	612
q8	9247	1310	1157	1157
q9	5159	4943	4914	4914
q10	7132	2388	1954	1954
q11	492	268	262	262
q12	350	366	234	234
q13	17765	3710	3135	3135
q14	228	223	210	210
q15	499	475	475	475
q16	640	617	582	582
q17	559	860	333	333
q18	6797	6477	6404	6404
q19	1689	952	543	543
q20	315	312	191	191
q21	2783	2135	1947	1947
q22	361	334	312	312
Total cold run time: 102896 ms
Total hot run time: 32319 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5494	5479	5471	5471
q2	242	327	237	237
q3	2270	2679	2313	2313
q4	1399	1851	1382	1382
q5	4302	4722	4627	4627
q6	167	160	129	129
q7	2007	1978	1842	1842
q8	2626	2761	2744	2744
q9	7313	7247	7229	7229
q10	3041	3339	2809	2809
q11	579	524	507	507
q12	656	755	577	577
q13	3404	3983	3278	3278
q14	276	302	273	273
q15	509	469	464	464
q16	642	700	635	635
q17	1198	1737	1258	1258
q18	7717	7402	7402	7402
q19	772	1064	1056	1056
q20	2058	2046	1903	1903
q21	5794	5517	4999	4999
q22	620	617	589	589
Total cold run time: 53086 ms
Total hot run time: 51724 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192129 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3cf8787f999dbf490eae8b2c9597940db1a2683b, data reload: false

query1	1312	954	955	954
query2	6202	2039	2027	2027
query3	11085	4668	4535	4535
query4	61541	29861	23045	23045
query5	5495	616	468	468
query6	431	226	182	182
query7	5530	499	298	298
query8	324	233	231	231
query9	8529	2650	2619	2619
query10	429	299	259	259
query11	17751	15160	15439	15160
query12	160	113	108	108
query13	1444	527	432	432
query14	11340	6930	7092	6930
query15	212	205	191	191
query16	7289	653	484	484
query17	1138	738	546	546
query18	1851	438	306	306
query19	200	185	154	154
query20	114	106	106	106
query21	209	123	104	104
query22	4493	4454	4550	4454
query23	34353	33471	33510	33471
query24	5689	2349	2334	2334
query25	469	472	410	410
query26	634	276	160	160
query27	1767	487	342	342
query28	4264	2501	2444	2444
query29	538	551	418	418
query30	222	192	159	159
query31	905	859	821	821
query32	74	59	58	58
query33	451	353	315	315
query34	720	876	523	523
query35	848	831	762	762
query36	1043	1059	985	985
query37	127	107	81	81
query38	4312	4334	4267	4267
query39	1483	1456	1447	1447
query40	213	154	103	103
query41	54	51	48	48
query42	124	97	99	97
query43	509	526	514	514
query44	1344	855	848	848
query45	192	178	169	169
query46	874	1051	649	649
query47	1891	1871	1822	1822
query48	411	418	317	317
query49	690	500	432	432
query50	644	656	406	406
query51	4365	4338	4226	4226
query52	117	114	96	96
query53	250	258	191	191
query54	497	500	467	467
query55	83	80	80	80
query56	271	273	273	273
query57	1219	1192	1148	1148
query58	244	241	241	241
query59	3189	3173	3041	3041
query60	295	275	271	271
query61	127	122	119	119
query62	745	733	667	667
query63	229	201	193	193
query64	1265	1013	765	765
query65	3254	3228	3152	3152
query66	714	397	300	300
query67	15810	15649	15313	15313
query68	2789	834	558	558
query69	425	291	258	258
query70	1215	1152	1134	1134
query71	345	297	258	258
query72	5030	3808	4008	3808
query73	637	747	359	359
query74	9757	9115	8916	8916
query75	3151	3164	2690	2690
query76	1940	1139	754	754
query77	337	348	288	288
query78	10082	10183	9386	9386
query79	1269	865	585	585
query80	869	531	448	448
query81	517	271	242	242
query82	1210	153	124	124
query83	280	177	155	155
query84	289	93	81	81
query85	852	361	309	309
query86	359	316	285	285
query87	4478	4685	4547	4547
query88	3705	2200	2154	2154
query89	399	326	294	294
query90	1671	193	194	193
query91	138	136	107	107
query92	62	56	54	54
query93	1110	877	531	531
query94	569	409	295	295
query95	336	273	265	265
query96	487	607	278	278
query97	2786	2911	2779	2779
query98	218	200	204	200
query99	1278	1405	1256	1256
Total cold run time: 303188 ms
Total hot run time: 192129 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.05 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3cf8787f999dbf490eae8b2c9597940db1a2683b, data reload: false

query1	0.04	0.05	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.61	0.10	0.10
query5	0.43	0.41	0.42
query6	1.13	0.66	0.66
query7	0.02	0.01	0.02
query8	0.04	0.03	0.04
query9	0.57	0.52	0.49
query10	0.58	0.57	0.55
query11	0.15	0.10	0.10
query12	0.15	0.12	0.11
query13	0.60	0.62	0.60
query14	2.71	2.75	2.87
query15	0.90	0.83	0.82
query16	0.38	0.37	0.36
query17	1.06	1.05	1.05
query18	0.25	0.21	0.21
query19	1.87	2.03	1.86
query20	0.02	0.02	0.02
query21	15.36	0.99	0.59
query22	0.75	0.76	0.63
query23	15.35	1.44	0.56
query24	3.09	1.18	1.70
query25	0.17	0.18	0.16
query26	0.22	0.14	0.14
query27	0.05	0.06	0.06
query28	14.28	0.98	0.43
query29	12.58	3.91	3.20
query30	0.25	0.10	0.07
query31	2.81	0.60	0.39
query32	3.24	0.55	0.45
query33	3.02	3.07	3.03
query34	16.78	5.16	4.54
query35	4.55	4.57	4.64
query36	0.66	0.50	0.49
query37	0.10	0.06	0.06
query38	0.06	0.05	0.03
query39	0.03	0.02	0.03
query40	0.17	0.13	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.48 s
Total hot run time: 31.05 s

@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32637 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c2e506b4f2a79f21d322a4e4924acbdb12aab32e, data reload: false

------ Round 1 ----------------------------------
q1	17575	5486	5371	5371
q2	2047	308	166	166
q3	10414	1282	748	748
q4	10213	960	548	548
q5	7529	2418	2204	2204
q6	193	172	133	133
q7	906	779	607	607
q8	9245	1373	1237	1237
q9	5384	4948	4914	4914
q10	6861	2313	1893	1893
q11	469	276	258	258
q12	339	359	215	215
q13	17774	3663	3132	3132
q14	240	228	211	211
q15	512	475	470	470
q16	635	601	592	592
q17	568	863	331	331
q18	7068	6499	6577	6499
q19	1210	961	528	528
q20	326	338	201	201
q21	2892	2328	2056	2056
q22	377	337	323	323
Total cold run time: 102777 ms
Total hot run time: 32637 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5466	5431	5475	5431
q2	243	326	235	235
q3	2303	2680	2337	2337
q4	1435	1842	1448	1448
q5	4418	4779	4655	4655
q6	168	155	126	126
q7	2046	1977	1818	1818
q8	2686	2829	2722	2722
q9	7313	7185	7209	7185
q10	2994	3266	2873	2873
q11	593	518	514	514
q12	681	810	630	630
q13	3521	3963	3272	3272
q14	288	309	258	258
q15	519	486	472	472
q16	637	683	624	624
q17	1213	1728	1282	1282
q18	7675	7600	7269	7269
q19	786	1185	1043	1043
q20	2002	2042	1871	1871
q21	5792	5229	5138	5138
q22	567	575	562	562
Total cold run time: 53346 ms
Total hot run time: 51765 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183892 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c2e506b4f2a79f21d322a4e4924acbdb12aab32e, data reload: false

query1	983	391	374	374
query2	6529	2032	1989	1989
query3	6799	228	213	213
query4	33522	23353	22982	22982
query5	4404	620	462	462
query6	285	202	171	171
query7	4612	487	299	299
query8	290	236	216	216
query9	9584	2664	2648	2648
query10	491	309	251	251
query11	17835	15196	14906	14906
query12	161	109	113	109
query13	1662	532	402	402
query14	10956	6836	6621	6621
query15	224	194	193	193
query16	7879	608	451	451
query17	1605	720	564	564
query18	1965	392	328	328
query19	216	180	150	150
query20	117	113	112	112
query21	204	117	109	109
query22	4168	4136	4312	4136
query23	34302	32888	33084	32888
query24	6917	2207	2203	2203
query25	474	448	389	389
query26	1204	268	159	159
query27	2056	466	328	328
query28	5381	2444	2436	2436
query29	692	551	418	418
query30	232	197	163	163
query31	977	876	802	802
query32	77	61	58	58
query33	512	343	317	317
query34	738	854	514	514
query35	793	818	728	728
query36	979	1046	968	968
query37	121	102	79	79
query38	4138	4122	4058	4058
query39	1452	1593	1439	1439
query40	202	111	101	101
query41	53	53	54	53
query42	122	113	106	106
query43	516	529	484	484
query44	1312	807	821	807
query45	182	174	164	164
query46	855	1050	655	655
query47	1775	1808	1756	1756
query48	379	399	306	306
query49	770	483	403	403
query50	633	652	386	386
query51	4174	4144	4157	4144
query52	108	111	100	100
query53	228	254	187	187
query54	479	485	411	411
query55	85	76	82	76
query56	259	252	242	242
query57	1197	1159	1083	1083
query58	251	229	238	229
query59	2938	3113	2936	2936
query60	294	279	278	278
query61	117	114	118	114
query62	774	732	668	668
query63	226	191	192	191
query64	4330	1010	655	655
query65	3237	3157	3210	3157
query66	1156	407	312	312
query67	15582	15305	15181	15181
query68	2329	815	559	559
query69	402	293	260	260
query70	1203	1137	1134	1134
query71	333	285	266	266
query72	5724	3843	3857	3843
query73	634	744	359	359
query74	9698	9072	8708	8708
query75	3196	3168	2678	2678
query76	2163	1146	780	780
query77	343	360	267	267
query78	10179	10001	9344	9344
query79	1033	867	588	588
query80	1361	568	438	438
query81	540	282	236	236
query82	360	158	122	122
query83	270	185	152	152
query84	236	86	77	77
query85	771	351	301	301
query86	406	318	295	295
query87	4420	4606	4296	4296
query88	4029	2175	2125	2125
query89	379	327	299	299
query90	1853	192	193	192
query91	131	137	111	111
query92	57	57	56	56
query93	1001	850	535	535
query94	671	386	290	290
query95	334	261	264	261
query96	488	663	277	277
query97	2801	2838	2713	2713
query98	218	201	201	201
query99	1287	1365	1311	1311
Total cold run time: 276620 ms
Total hot run time: 183892 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.92 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c2e506b4f2a79f21d322a4e4924acbdb12aab32e, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.06	0.07
query4	1.63	0.11	0.11
query5	0.44	0.43	0.40
query6	1.16	0.67	0.65
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.50	0.49
query10	0.55	0.56	0.55
query11	0.14	0.10	0.10
query12	0.14	0.12	0.12
query13	0.62	0.61	0.59
query14	2.73	2.86	2.85
query15	0.88	0.82	0.83
query16	0.38	0.38	0.38
query17	1.06	1.02	1.04
query18	0.23	0.21	0.21
query19	1.98	1.91	1.98
query20	0.02	0.01	0.01
query21	15.36	0.99	0.58
query22	0.75	0.74	0.61
query23	15.43	1.49	0.59
query24	3.02	1.16	1.42
query25	0.17	0.17	0.13
query26	0.33	0.15	0.14
query27	0.04	0.05	0.07
query28	13.92	0.99	0.44
query29	12.59	3.89	3.24
query30	0.25	0.09	0.06
query31	2.82	0.59	0.39
query32	3.24	0.55	0.47
query33	3.06	3.01	3.10
query34	16.62	5.07	4.46
query35	4.50	4.47	4.44
query36	0.65	0.49	0.47
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.18	0.14	0.11
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.21 s
Total hot run time: 30.92 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.09% (11003/26142)
Line Coverage: 32.35% (92850/287032)
Region Coverage: 31.50% (47614/151144)
Branch Coverage: 27.52% (24091/87538)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c2e506b4f2a79f21d322a4e4924acbdb12aab32e_c2e506b4f2a79f21d322a4e4924acbdb12aab32e/report/index.html

@Gabriel39
Copy link
Contributor Author

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 5, 2025
Copy link
Contributor

github-actions bot commented Feb 5, 2025

PR approved by at least one committer and no changes requested.

Copy link
Contributor

github-actions bot commented Feb 5, 2025

PR approved by anyone and no changes requested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.x dev/3.0.x p0_w reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants