[SLP] Relax store chain user limit to 2 - #215700
Open
akadutta wants to merge 1 commit into
Open
Conversation
akadutta
marked this pull request as ready for review
August 12, 2026 00:10
|
@llvm/pr-subscribers-backend-amdgpu Author: Akash Dutta (akadutta) ChangesThe store-chain early reject treated operands with two outside users as unprofitable, but two such users can still be part of the same profitable SLP tree—for example carry compare plus a dependent add on the next limb. That caused profitable store-chain vectorization to be skipped before the cost model could evaluate it. Relaxing the threshold to require three or more outside users before early rejection restores those cases and fixes related performance regressions without removing the heuristic for heavily reused operands. Full diff: https://github.com/llvm/llvm-project/pull/215700.diff 2 Files Affected:
diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
index a3350cf3f3e52..7eec02574302e 100644
--- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
+++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
@@ -28679,13 +28679,15 @@ SLPVectorizerPass::vectorizeStoreChainImpl(ArrayRef<Value *> Chain, BoUpSLP &R,
// only merges the stores, while the scalars remain live for the other users
// and all the lanes are gathered back. A single outside use may still be a
// part of the larger vectorizable graph, same for the values, fed by the
- // loads, where the vector loads may pay off the gathering.
+ // loads, where the vector loads may pay off the gathering. Likewise two
+ // outside uses (e.g. carry compare and next-limb add) may still be part of
+ // the same tree; require 3+ outside users before early-rejecting here.
if (S && S.getOpcode() != Instruction::Load &&
all_of(ValOps.getArrayRef(), [&](Value *V) {
return none_of(cast<Instruction>(V)->operand_values(),
IsaPred<LoadInst>) &&
count_if(V->users(),
- [&](User *U) { return !Stores.contains(U); }) > 1;
+ [&](User *U) { return !Stores.contains(U); }) > 2;
})) {
Size = 1;
return false;
diff --git a/llvm/test/Transforms/SLPVectorizer/AMDGPU/store-chain-two-external-users.ll b/llvm/test/Transforms/SLPVectorizer/AMDGPU/store-chain-two-external-users.ll
new file mode 100644
index 0000000000000..8db7283bbdc53
--- /dev/null
+++ b/llvm/test/Transforms/SLPVectorizer/AMDGPU/store-chain-two-external-users.ll
@@ -0,0 +1,67 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
+; RUN: opt -S -passes=slp-vectorizer -slp-threshold=-100 -mtriple=amdgpu9.0a-amd-amdhsa < %s | FileCheck %s
+
+; Store-chain operands may have two users outside the store chain and still be
+; worth considering. This mirrors carry-propagating counter updates,
+; where the updated limbs remain live for scalar carry/use
+; chains after the state stores. The old "> 1 outside user" early return rejected
+; this before the cost model; with a "> 2" threshold the chain reaches costing.
+
+declare void @use(i32)
+
+define void @store_chain_two_external_users(ptr %dst, i32 %x0, i32 %x1, i32 %x2, i32 %x3, i32 %y0, i32 %y1, i32 %y2, i32 %y3) {
+; CHECK-LABEL: define void @store_chain_two_external_users(
+; CHECK-SAME: ptr [[DST:%.*]], i32 [[X0:%.*]], i32 [[X1:%.*]], i32 [[X2:%.*]], i32 [[X3:%.*]], i32 [[Y0:%.*]], i32 [[Y1:%.*]], i32 [[Y2:%.*]], i32 [[Y3:%.*]]) {
+; CHECK-NEXT: [[ENTRY:.*:]]
+; CHECK-NEXT: [[DST2:%.*]] = getelementptr inbounds i32, ptr [[DST]], i64 2
+; CHECK-NEXT: [[TMP0:%.*]] = insertelement <2 x i32> poison, i32 [[X0]], i64 0
+; CHECK-NEXT: [[TMP1:%.*]] = insertelement <2 x i32> [[TMP0]], i32 [[X1]], i64 1
+; CHECK-NEXT: [[TMP2:%.*]] = insertelement <2 x i32> poison, i32 [[Y0]], i64 0
+; CHECK-NEXT: [[TMP3:%.*]] = insertelement <2 x i32> [[TMP2]], i32 [[Y1]], i64 1
+; CHECK-NEXT: [[V0:%.*]] = add <2 x i32> [[TMP1]], [[TMP3]]
+; CHECK-NEXT: [[TMP5:%.*]] = insertelement <2 x i32> poison, i32 [[X2]], i64 0
+; CHECK-NEXT: [[TMP6:%.*]] = insertelement <2 x i32> [[TMP5]], i32 [[X3]], i64 1
+; CHECK-NEXT: [[TMP7:%.*]] = insertelement <2 x i32> poison, i32 [[Y2]], i64 0
+; CHECK-NEXT: [[TMP8:%.*]] = insertelement <2 x i32> [[TMP7]], i32 [[Y3]], i64 1
+; CHECK-NEXT: [[V1:%.*]] = add <2 x i32> [[TMP6]], [[TMP8]]
+; CHECK-NEXT: store <2 x i32> [[V0]], ptr [[DST]], align 4
+; CHECK-NEXT: store <2 x i32> [[V1]], ptr [[DST2]], align 4
+; CHECK-NEXT: [[TMP10:%.*]] = extractelement <2 x i32> [[V0]], i64 0
+; CHECK-NEXT: call void @use(i32 [[TMP10]])
+; CHECK-NEXT: [[TMP11:%.*]] = extractelement <2 x i32> [[V0]], i64 1
+; CHECK-NEXT: call void @use(i32 [[TMP11]])
+; CHECK-NEXT: [[TMP12:%.*]] = extractelement <2 x i32> [[V1]], i64 0
+; CHECK-NEXT: call void @use(i32 [[TMP12]])
+; CHECK-NEXT: [[TMP13:%.*]] = extractelement <2 x i32> [[V1]], i64 1
+; CHECK-NEXT: call void @use(i32 [[TMP13]])
+; CHECK-NEXT: call void @use(i32 [[TMP10]])
+; CHECK-NEXT: call void @use(i32 [[TMP11]])
+; CHECK-NEXT: call void @use(i32 [[TMP12]])
+; CHECK-NEXT: call void @use(i32 [[TMP13]])
+; CHECK-NEXT: ret void
+;
+entry:
+ %dst1 = getelementptr inbounds i32, ptr %dst, i64 1
+ %dst2 = getelementptr inbounds i32, ptr %dst, i64 2
+ %dst3 = getelementptr inbounds i32, ptr %dst, i64 3
+
+ %a0 = add i32 %x0, %y0
+ %a1 = add i32 %x1, %y1
+ %a2 = add i32 %x2, %y2
+ %a3 = add i32 %x3, %y3
+
+ store i32 %a0, ptr %dst, align 4
+ store i32 %a1, ptr %dst1, align 4
+ store i32 %a2, ptr %dst2, align 4
+ store i32 %a3, ptr %dst3, align 4
+
+ call void @use(i32 %a0)
+ call void @use(i32 %a1)
+ call void @use(i32 %a2)
+ call void @use(i32 %a3)
+ call void @use(i32 %a0)
+ call void @use(i32 %a1)
+ call void @use(i32 %a2)
+ call void @use(i32 %a3)
+ ret void
+}
|
|
@llvm/pr-subscribers-vectorizers Author: Akash Dutta (akadutta) ChangesThe store-chain early reject treated operands with two outside users as unprofitable, but two such users can still be part of the same profitable SLP tree—for example carry compare plus a dependent add on the next limb. That caused profitable store-chain vectorization to be skipped before the cost model could evaluate it. Relaxing the threshold to require three or more outside users before early rejection restores those cases and fixes related performance regressions without removing the heuristic for heavily reused operands. Full diff: https://github.com/llvm/llvm-project/pull/215700.diff 2 Files Affected:
diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
index a3350cf3f3e52..7eec02574302e 100644
--- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
+++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
@@ -28679,13 +28679,15 @@ SLPVectorizerPass::vectorizeStoreChainImpl(ArrayRef<Value *> Chain, BoUpSLP &R,
// only merges the stores, while the scalars remain live for the other users
// and all the lanes are gathered back. A single outside use may still be a
// part of the larger vectorizable graph, same for the values, fed by the
- // loads, where the vector loads may pay off the gathering.
+ // loads, where the vector loads may pay off the gathering. Likewise two
+ // outside uses (e.g. carry compare and next-limb add) may still be part of
+ // the same tree; require 3+ outside users before early-rejecting here.
if (S && S.getOpcode() != Instruction::Load &&
all_of(ValOps.getArrayRef(), [&](Value *V) {
return none_of(cast<Instruction>(V)->operand_values(),
IsaPred<LoadInst>) &&
count_if(V->users(),
- [&](User *U) { return !Stores.contains(U); }) > 1;
+ [&](User *U) { return !Stores.contains(U); }) > 2;
})) {
Size = 1;
return false;
diff --git a/llvm/test/Transforms/SLPVectorizer/AMDGPU/store-chain-two-external-users.ll b/llvm/test/Transforms/SLPVectorizer/AMDGPU/store-chain-two-external-users.ll
new file mode 100644
index 0000000000000..8db7283bbdc53
--- /dev/null
+++ b/llvm/test/Transforms/SLPVectorizer/AMDGPU/store-chain-two-external-users.ll
@@ -0,0 +1,67 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
+; RUN: opt -S -passes=slp-vectorizer -slp-threshold=-100 -mtriple=amdgpu9.0a-amd-amdhsa < %s | FileCheck %s
+
+; Store-chain operands may have two users outside the store chain and still be
+; worth considering. This mirrors carry-propagating counter updates,
+; where the updated limbs remain live for scalar carry/use
+; chains after the state stores. The old "> 1 outside user" early return rejected
+; this before the cost model; with a "> 2" threshold the chain reaches costing.
+
+declare void @use(i32)
+
+define void @store_chain_two_external_users(ptr %dst, i32 %x0, i32 %x1, i32 %x2, i32 %x3, i32 %y0, i32 %y1, i32 %y2, i32 %y3) {
+; CHECK-LABEL: define void @store_chain_two_external_users(
+; CHECK-SAME: ptr [[DST:%.*]], i32 [[X0:%.*]], i32 [[X1:%.*]], i32 [[X2:%.*]], i32 [[X3:%.*]], i32 [[Y0:%.*]], i32 [[Y1:%.*]], i32 [[Y2:%.*]], i32 [[Y3:%.*]]) {
+; CHECK-NEXT: [[ENTRY:.*:]]
+; CHECK-NEXT: [[DST2:%.*]] = getelementptr inbounds i32, ptr [[DST]], i64 2
+; CHECK-NEXT: [[TMP0:%.*]] = insertelement <2 x i32> poison, i32 [[X0]], i64 0
+; CHECK-NEXT: [[TMP1:%.*]] = insertelement <2 x i32> [[TMP0]], i32 [[X1]], i64 1
+; CHECK-NEXT: [[TMP2:%.*]] = insertelement <2 x i32> poison, i32 [[Y0]], i64 0
+; CHECK-NEXT: [[TMP3:%.*]] = insertelement <2 x i32> [[TMP2]], i32 [[Y1]], i64 1
+; CHECK-NEXT: [[V0:%.*]] = add <2 x i32> [[TMP1]], [[TMP3]]
+; CHECK-NEXT: [[TMP5:%.*]] = insertelement <2 x i32> poison, i32 [[X2]], i64 0
+; CHECK-NEXT: [[TMP6:%.*]] = insertelement <2 x i32> [[TMP5]], i32 [[X3]], i64 1
+; CHECK-NEXT: [[TMP7:%.*]] = insertelement <2 x i32> poison, i32 [[Y2]], i64 0
+; CHECK-NEXT: [[TMP8:%.*]] = insertelement <2 x i32> [[TMP7]], i32 [[Y3]], i64 1
+; CHECK-NEXT: [[V1:%.*]] = add <2 x i32> [[TMP6]], [[TMP8]]
+; CHECK-NEXT: store <2 x i32> [[V0]], ptr [[DST]], align 4
+; CHECK-NEXT: store <2 x i32> [[V1]], ptr [[DST2]], align 4
+; CHECK-NEXT: [[TMP10:%.*]] = extractelement <2 x i32> [[V0]], i64 0
+; CHECK-NEXT: call void @use(i32 [[TMP10]])
+; CHECK-NEXT: [[TMP11:%.*]] = extractelement <2 x i32> [[V0]], i64 1
+; CHECK-NEXT: call void @use(i32 [[TMP11]])
+; CHECK-NEXT: [[TMP12:%.*]] = extractelement <2 x i32> [[V1]], i64 0
+; CHECK-NEXT: call void @use(i32 [[TMP12]])
+; CHECK-NEXT: [[TMP13:%.*]] = extractelement <2 x i32> [[V1]], i64 1
+; CHECK-NEXT: call void @use(i32 [[TMP13]])
+; CHECK-NEXT: call void @use(i32 [[TMP10]])
+; CHECK-NEXT: call void @use(i32 [[TMP11]])
+; CHECK-NEXT: call void @use(i32 [[TMP12]])
+; CHECK-NEXT: call void @use(i32 [[TMP13]])
+; CHECK-NEXT: ret void
+;
+entry:
+ %dst1 = getelementptr inbounds i32, ptr %dst, i64 1
+ %dst2 = getelementptr inbounds i32, ptr %dst, i64 2
+ %dst3 = getelementptr inbounds i32, ptr %dst, i64 3
+
+ %a0 = add i32 %x0, %y0
+ %a1 = add i32 %x1, %y1
+ %a2 = add i32 %x2, %y2
+ %a3 = add i32 %x3, %y3
+
+ store i32 %a0, ptr %dst, align 4
+ store i32 %a1, ptr %dst1, align 4
+ store i32 %a2, ptr %dst2, align 4
+ store i32 %a3, ptr %dst3, align 4
+
+ call void @use(i32 %a0)
+ call void @use(i32 %a1)
+ call void @use(i32 %a2)
+ call void @use(i32 %a3)
+ call void @use(i32 %a0)
+ call void @use(i32 %a1)
+ call void @use(i32 %a2)
+ call void @use(i32 %a3)
+ ret void
+}
|
|
@llvm/pr-subscribers-llvm-transforms Author: Akash Dutta (akadutta) ChangesThe store-chain early reject treated operands with two outside users as unprofitable, but two such users can still be part of the same profitable SLP tree—for example carry compare plus a dependent add on the next limb. That caused profitable store-chain vectorization to be skipped before the cost model could evaluate it. Relaxing the threshold to require three or more outside users before early rejection restores those cases and fixes related performance regressions without removing the heuristic for heavily reused operands. Full diff: https://github.com/llvm/llvm-project/pull/215700.diff 2 Files Affected:
diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
index a3350cf3f3e52..7eec02574302e 100644
--- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
+++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
@@ -28679,13 +28679,15 @@ SLPVectorizerPass::vectorizeStoreChainImpl(ArrayRef<Value *> Chain, BoUpSLP &R,
// only merges the stores, while the scalars remain live for the other users
// and all the lanes are gathered back. A single outside use may still be a
// part of the larger vectorizable graph, same for the values, fed by the
- // loads, where the vector loads may pay off the gathering.
+ // loads, where the vector loads may pay off the gathering. Likewise two
+ // outside uses (e.g. carry compare and next-limb add) may still be part of
+ // the same tree; require 3+ outside users before early-rejecting here.
if (S && S.getOpcode() != Instruction::Load &&
all_of(ValOps.getArrayRef(), [&](Value *V) {
return none_of(cast<Instruction>(V)->operand_values(),
IsaPred<LoadInst>) &&
count_if(V->users(),
- [&](User *U) { return !Stores.contains(U); }) > 1;
+ [&](User *U) { return !Stores.contains(U); }) > 2;
})) {
Size = 1;
return false;
diff --git a/llvm/test/Transforms/SLPVectorizer/AMDGPU/store-chain-two-external-users.ll b/llvm/test/Transforms/SLPVectorizer/AMDGPU/store-chain-two-external-users.ll
new file mode 100644
index 0000000000000..8db7283bbdc53
--- /dev/null
+++ b/llvm/test/Transforms/SLPVectorizer/AMDGPU/store-chain-two-external-users.ll
@@ -0,0 +1,67 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
+; RUN: opt -S -passes=slp-vectorizer -slp-threshold=-100 -mtriple=amdgpu9.0a-amd-amdhsa < %s | FileCheck %s
+
+; Store-chain operands may have two users outside the store chain and still be
+; worth considering. This mirrors carry-propagating counter updates,
+; where the updated limbs remain live for scalar carry/use
+; chains after the state stores. The old "> 1 outside user" early return rejected
+; this before the cost model; with a "> 2" threshold the chain reaches costing.
+
+declare void @use(i32)
+
+define void @store_chain_two_external_users(ptr %dst, i32 %x0, i32 %x1, i32 %x2, i32 %x3, i32 %y0, i32 %y1, i32 %y2, i32 %y3) {
+; CHECK-LABEL: define void @store_chain_two_external_users(
+; CHECK-SAME: ptr [[DST:%.*]], i32 [[X0:%.*]], i32 [[X1:%.*]], i32 [[X2:%.*]], i32 [[X3:%.*]], i32 [[Y0:%.*]], i32 [[Y1:%.*]], i32 [[Y2:%.*]], i32 [[Y3:%.*]]) {
+; CHECK-NEXT: [[ENTRY:.*:]]
+; CHECK-NEXT: [[DST2:%.*]] = getelementptr inbounds i32, ptr [[DST]], i64 2
+; CHECK-NEXT: [[TMP0:%.*]] = insertelement <2 x i32> poison, i32 [[X0]], i64 0
+; CHECK-NEXT: [[TMP1:%.*]] = insertelement <2 x i32> [[TMP0]], i32 [[X1]], i64 1
+; CHECK-NEXT: [[TMP2:%.*]] = insertelement <2 x i32> poison, i32 [[Y0]], i64 0
+; CHECK-NEXT: [[TMP3:%.*]] = insertelement <2 x i32> [[TMP2]], i32 [[Y1]], i64 1
+; CHECK-NEXT: [[V0:%.*]] = add <2 x i32> [[TMP1]], [[TMP3]]
+; CHECK-NEXT: [[TMP5:%.*]] = insertelement <2 x i32> poison, i32 [[X2]], i64 0
+; CHECK-NEXT: [[TMP6:%.*]] = insertelement <2 x i32> [[TMP5]], i32 [[X3]], i64 1
+; CHECK-NEXT: [[TMP7:%.*]] = insertelement <2 x i32> poison, i32 [[Y2]], i64 0
+; CHECK-NEXT: [[TMP8:%.*]] = insertelement <2 x i32> [[TMP7]], i32 [[Y3]], i64 1
+; CHECK-NEXT: [[V1:%.*]] = add <2 x i32> [[TMP6]], [[TMP8]]
+; CHECK-NEXT: store <2 x i32> [[V0]], ptr [[DST]], align 4
+; CHECK-NEXT: store <2 x i32> [[V1]], ptr [[DST2]], align 4
+; CHECK-NEXT: [[TMP10:%.*]] = extractelement <2 x i32> [[V0]], i64 0
+; CHECK-NEXT: call void @use(i32 [[TMP10]])
+; CHECK-NEXT: [[TMP11:%.*]] = extractelement <2 x i32> [[V0]], i64 1
+; CHECK-NEXT: call void @use(i32 [[TMP11]])
+; CHECK-NEXT: [[TMP12:%.*]] = extractelement <2 x i32> [[V1]], i64 0
+; CHECK-NEXT: call void @use(i32 [[TMP12]])
+; CHECK-NEXT: [[TMP13:%.*]] = extractelement <2 x i32> [[V1]], i64 1
+; CHECK-NEXT: call void @use(i32 [[TMP13]])
+; CHECK-NEXT: call void @use(i32 [[TMP10]])
+; CHECK-NEXT: call void @use(i32 [[TMP11]])
+; CHECK-NEXT: call void @use(i32 [[TMP12]])
+; CHECK-NEXT: call void @use(i32 [[TMP13]])
+; CHECK-NEXT: ret void
+;
+entry:
+ %dst1 = getelementptr inbounds i32, ptr %dst, i64 1
+ %dst2 = getelementptr inbounds i32, ptr %dst, i64 2
+ %dst3 = getelementptr inbounds i32, ptr %dst, i64 3
+
+ %a0 = add i32 %x0, %y0
+ %a1 = add i32 %x1, %y1
+ %a2 = add i32 %x2, %y2
+ %a3 = add i32 %x3, %y3
+
+ store i32 %a0, ptr %dst, align 4
+ store i32 %a1, ptr %dst1, align 4
+ store i32 %a2, ptr %dst2, align 4
+ store i32 %a3, ptr %dst3, align 4
+
+ call void @use(i32 %a0)
+ call void @use(i32 %a1)
+ call void @use(i32 %a2)
+ call void @use(i32 %a3)
+ call void @use(i32 %a0)
+ call void @use(i32 %a1)
+ call void @use(i32 %a2)
+ call void @use(i32 %a3)
+ ret void
+}
|
Member
|
Check compile time effect, it was a check to save compile time blow |
Contributor
Author
I did not see any noticeable difference in compile time for the cases I tested. Do you have a benchmark/app in mind that would be a good candidate? |
Member
|
Contributor
Author
I tested CTMark locally. There's no significant difference in wall time. All differences are less than 0.2 seconds. In terms of instructions, O3 increases by 0.05% for this patch. The remaining configs are very similar. |
Contributor
Author
|
@alexey-bataev ping |
Member
Contributor
Author
Member
Yep, as expected. Would be good to try to reduce compile time effect |
Contributor
Author
Just to clarify, is 0.05% compile time hit unacceptable? I'm seeing upwards of 8% performance hit on several kernels if the store chain limit is kept at 1. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The store-chain early reject treated operands with two outside users as unprofitable, but two such users can still be part of the same profitable SLP tree—for example carry compare plus a dependent add on the next limb. That caused profitable store-chain vectorization to be skipped before the cost model could evaluate it. Relaxing the threshold to require three or more outside users before early rejection restores those cases and fixes related performance regressions without removing the heuristic for heavily reused operands.