[X86][SchedModel] Use ZMM zero-idiom class for 512-bit VPSUB/VPCMPGT on Znver4 - #215109
Open
nazar-pc wants to merge 1 commit into
Open
[X86][SchedModel] Use ZMM zero-idiom class for 512-bit VPSUB/VPCMPGT on Znver4#215109nazar-pc wants to merge 1 commit into
nazar-pc wants to merge 1 commit into
Conversation
…on Znver4
X86ScheduleZnver4.td defines Zn4WriteVZeroIdiomLogic{X,Y,Z} and
Zn4WriteVZeroIdiomALU{X,Y,Z}, each paired with an InstRW listing the
instructions of that width. Every pair matches except the last: the
InstRW for the 512-bit VPSUB and VPCMPGT forms names
Zn4WriteVZeroIdiomALUY, leaving Zn4WriteVZeroIdiomALUZ unused and giving
those instructions WriteVecALUY.
WriteVecALUY occupies a vector pipe for one cycle and WriteVecALUZ for
two, which is how the model accounts for Zen 4 splitting a 512-bit
operation into two 256-bit halves. VPSUB{B,W,D,Q}Zrr and
VPCMPGT{B,W,D,Q}Zrr therefore had half their port occupancy: llvm-mca
reported an inverse throughput of 0.25 for vpsubd on zmm, matching the
ymm form and half of the neighbouring vpaddd on zmm.
The zero-idiom arm of the variant is unchanged, so
vpsubd %zmm0, %zmm0, %zmm0 is still recognised as a dependency break.
|
Hello @nazar-pc 👋 Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.
Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description. Frequently asked questionsHow do I add reviewers? This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically. You can also add reviewers manually using the Reviewers section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using What if there are no comments? If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers. Are any special GitHub settings required to contribute to LLVM? We only require contributors to have a public email address associated with their GitHub commits, see this section of LLVM Developer Policy for details. If you have questions, feel free to leave a comment on this PR, or ask on LLVM Discord or LLVM Discourse. Thank you, |
|
@llvm/pr-subscribers-backend-x86 Author: Nazar Mokrynskyi (nazar-pc) ChangesUp front: I don't work on LLVM and I don't know the scheduling models. I was chasing a performance difference between hand-written assembly and C intrinsics in BLAKE3 on a Zen 4 box, used an LLM (Claude) to help me dig through The issue I was chasing here specifically was that compiling BLAKE3 for The problem
The last one ( That matters because Evidence
Before the change a 512-bit The zero-idiom arm of the variant is not modified, so the dependency-breaking case still works — Testing
Patch is 29.56 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/215109.diff 3 Files Affected:
diff --git a/llvm/lib/Target/X86/X86ScheduleZnver4.td b/llvm/lib/Target/X86/X86ScheduleZnver4.td
index ac4d31de8dbfe..86e1d8858a35e 100644
--- a/llvm/lib/Target/X86/X86ScheduleZnver4.td
+++ b/llvm/lib/Target/X86/X86ScheduleZnver4.td
@@ -9,7 +9,7 @@
// This file defines the machine model for Znver4 to support instruction
// scheduling and other instruction cost heuristics.
// Based on:
-// * AMD Software Optimization Guide for the AMD Family 19h (Zen4)
+// * AMD Software Optimization Guide for the AMD Family 19h (Zen4)
// Microarchitecture
// https://www.amd.com/system/files/TechDocs/57647.zip
//===----------------------------------------------------------------------===//
@@ -1550,7 +1550,7 @@ def Zn4WriteVFIXUPIMMPDZrr_VRANGESDrr : SchedWriteRes<[Zn4FPFMisc01]> {
let NumMicroOps = 1;
}
def : InstRW<[Zn4WriteVFIXUPIMMPDZrr_VRANGESDrr], (instregex
- "VFIXUPIMM(S|P)(S|D)(Z|Z128|Z256?)rrik", "VFIXUPIMM(S|P)(S|D)(Z?|Z128?|Z256?)rrikz",
+ "VFIXUPIMM(S|P)(S|D)(Z|Z128|Z256?)rrik", "VFIXUPIMM(S|P)(S|D)(Z?|Z128?|Z256?)rrikz",
"VFIXUPIMM(S|P)(S|D)(Z128|Z256?)rri", "VRANGE(S|P)(S|D)(Z?|Z128?|Z256?)rri(b?)",
"VRANGE(S|P)(S|D)(Z|Z128|Z256?)rri(b?)k","VRANGE(S|P)(S|D)(Z?|Z128?|Z256?)rri(b?)kz"
)>;
@@ -1824,20 +1824,20 @@ def Zn4VecALUZSlow: SchedWriteRes<[Zn4FPFMisc01]> {
let ReleaseAtCycles = [2];
let NumMicroOps = 1;
}
-def : InstRW<[Zn4VecALUZSlow], (instrs
- VPABSBZ128rr, VPABSBZ128rrk, VPABSBZ128rrkz, VPABSDZ128rr,
- VPABSDZ128rrk, VPABSDZ128rrkz, VPABSQZ128rr, VPABSQZ128rrk,
- VPABSQZ128rrkz, VPABSWZ128rr, VPABSWZ128rrk, VPABSWZ128rrkz,
- VPADDSBZ128rr, VPADDSBZ128rrk, VPADDSBZ128rrkz, VPADDSWZ128rr,
- VPADDSWZ128rrk, VPADDSWZ128rrkz,VPADDUSBZ128rr, VPADDUSBZ128rrk,
- VPADDUSBZ128rrkz, VPADDUSWZ128rr, VPADDUSWZ128rrk, VPADDUSWZ128rrkz,
- VPAVGBZ128rr, VPAVGBZ128rrk, VPAVGBZ128rrkz, VPAVGWZ128rr,
- VPAVGWZ128rrk, VPAVGWZ128rrkz, VPOPCNTBZ128rr, VPOPCNTBZ128rrk,
- VPOPCNTBZ128rrkz, VPOPCNTDZ128rr, VPOPCNTDZ128rrk, VPOPCNTDZ128rrkz,
- VPOPCNTQZ128rr, VPOPCNTQZ128rrk,VPOPCNTQZ128rrkz, VPOPCNTWZ128rr,
- VPOPCNTWZ128rrk, VPOPCNTWZ128rrkz,VPSUBSBZ128rr, VPSUBSBZ128rrk,
- VPSUBSBZ128rrkz, VPSUBSWZ128rr, VPSUBSWZ128rrk, VPSUBSWZ128rrkz,
- VPSUBUSBZ128rr, VPSUBUSBZ128rrk, VPSUBUSBZ128rrkz,VPSUBUSWZ128rr,
+def : InstRW<[Zn4VecALUZSlow], (instrs
+ VPABSBZ128rr, VPABSBZ128rrk, VPABSBZ128rrkz, VPABSDZ128rr,
+ VPABSDZ128rrk, VPABSDZ128rrkz, VPABSQZ128rr, VPABSQZ128rrk,
+ VPABSQZ128rrkz, VPABSWZ128rr, VPABSWZ128rrk, VPABSWZ128rrkz,
+ VPADDSBZ128rr, VPADDSBZ128rrk, VPADDSBZ128rrkz, VPADDSWZ128rr,
+ VPADDSWZ128rrk, VPADDSWZ128rrkz,VPADDUSBZ128rr, VPADDUSBZ128rrk,
+ VPADDUSBZ128rrkz, VPADDUSWZ128rr, VPADDUSWZ128rrk, VPADDUSWZ128rrkz,
+ VPAVGBZ128rr, VPAVGBZ128rrk, VPAVGBZ128rrkz, VPAVGWZ128rr,
+ VPAVGWZ128rrk, VPAVGWZ128rrkz, VPOPCNTBZ128rr, VPOPCNTBZ128rrk,
+ VPOPCNTBZ128rrkz, VPOPCNTDZ128rr, VPOPCNTDZ128rrk, VPOPCNTDZ128rrkz,
+ VPOPCNTQZ128rr, VPOPCNTQZ128rrk,VPOPCNTQZ128rrkz, VPOPCNTWZ128rr,
+ VPOPCNTWZ128rrk, VPOPCNTWZ128rrkz,VPSUBSBZ128rr, VPSUBSBZ128rrk,
+ VPSUBSBZ128rrkz, VPSUBSWZ128rr, VPSUBSWZ128rrk, VPSUBSWZ128rrkz,
+ VPSUBUSBZ128rr, VPSUBUSBZ128rrk, VPSUBUSBZ128rrkz,VPSUBUSWZ128rr,
VPSUBUSWZ128rrk, VPSUBUSWZ128rrkz
)>;
@@ -1952,7 +1952,7 @@ def Zn4WriteVZeroIdiomALUZ : SchedWriteVariant<[
SchedVar<MCSchedPredicate<ZeroIdiomPredicate>, [Zn4WriteZeroLatency]>,
SchedVar<NoSchedPred, [WriteVecALUZ]>
]>;
-def : InstRW<[Zn4WriteVZeroIdiomALUY],
+def : InstRW<[Zn4WriteVZeroIdiomALUZ],
(instrs VPSUBBZrr, VPSUBWZrr, VPSUBDZrr, VPSUBQZrr,
VPCMPGTBZrr, VPCMPGTWZrr, VPCMPGTDZrr, VPCMPGTQZrr)>;
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512.s
index 14b8e5f36c666..f1712aca08311 100644
--- a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512.s
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512.s
@@ -1702,13 +1702,13 @@ vunpcklps (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
# CHECK-NEXT: 1 1 0.50 vpcmpeqq %zmm0, %zmm1, %k2 {%k3}
# CHECK-NEXT: 1 8 0.50 * vpcmpeqq (%rax), %zmm1, %k2 {%k3}
# CHECK-NEXT: 1 8 0.50 * vpcmpeqq (%rax){1to8}, %zmm1, %k2 {%k3}
-# CHECK-NEXT: 1 1 0.25 vpcmpgtd %zmm0, %zmm1, %k2
+# CHECK-NEXT: 1 1 0.50 vpcmpgtd %zmm0, %zmm1, %k2
# CHECK-NEXT: 1 8 0.50 * vpcmpgtd (%rax), %zmm1, %k2
# CHECK-NEXT: 1 8 0.50 * vpcmpgtd (%rax){1to16}, %zmm1, %k2
# CHECK-NEXT: 1 1 0.50 vpcmpgtd %zmm0, %zmm1, %k2 {%k3}
# CHECK-NEXT: 1 8 0.50 * vpcmpgtd (%rax), %zmm1, %k2 {%k3}
# CHECK-NEXT: 1 8 0.50 * vpcmpgtd (%rax){1to16}, %zmm1, %k2 {%k3}
-# CHECK-NEXT: 1 1 0.25 vpcmpgtq %zmm0, %zmm1, %k2
+# CHECK-NEXT: 1 1 0.50 vpcmpgtq %zmm0, %zmm1, %k2
# CHECK-NEXT: 1 8 0.50 * vpcmpgtq (%rax), %zmm1, %k2
# CHECK-NEXT: 1 8 0.50 * vpcmpgtq (%rax){1to8}, %zmm1, %k2
# CHECK-NEXT: 1 1 0.50 vpcmpgtq %zmm0, %zmm1, %k2 {%k3}
@@ -1983,7 +1983,7 @@ vunpcklps (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
# CHECK-NEXT: 1 1 1.00 vpshufd $0, %zmm16, %zmm19 {%k1} {z}
# CHECK-NEXT: 1 8 1.00 * vpshufd $0, (%rax), %zmm19 {%k1} {z}
# CHECK-NEXT: 1 8 1.00 * vpshufd $0, (%rax){1to16}, %zmm19 {%k1} {z}
-# CHECK-NEXT: 1 1 0.25 vpsubd %zmm16, %zmm17, %zmm19
+# CHECK-NEXT: 1 1 0.50 vpsubd %zmm16, %zmm17, %zmm19
# CHECK-NEXT: 1 8 0.50 * vpsubd (%rax), %zmm17, %zmm19
# CHECK-NEXT: 1 8 0.50 * vpsubd (%rax){1to16}, %zmm17, %zmm19
# CHECK-NEXT: 1 1 0.50 vpsubd %zmm16, %zmm17, %zmm19 {%k1}
@@ -1992,7 +1992,7 @@ vunpcklps (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
# CHECK-NEXT: 1 1 0.50 vpsubd %zmm16, %zmm17, %zmm19 {%k1} {z}
# CHECK-NEXT: 1 8 0.50 * vpsubd (%rax), %zmm17, %zmm19 {%k1} {z}
# CHECK-NEXT: 1 8 0.50 * vpsubd (%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
-# CHECK-NEXT: 1 1 0.25 vpsubq %zmm16, %zmm17, %zmm19
+# CHECK-NEXT: 1 1 0.50 vpsubq %zmm16, %zmm17, %zmm19
# CHECK-NEXT: 1 8 0.50 * vpsubq (%rax), %zmm17, %zmm19
# CHECK-NEXT: 1 8 0.50 * vpsubq (%rax){1to8}, %zmm17, %zmm19
# CHECK-NEXT: 1 1 0.50 vpsubq %zmm16, %zmm17, %zmm19 {%k1}
@@ -2107,7 +2107,7 @@ vunpcklps (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
# CHECK-NEXT: 1 22 5.00 * vsqrtss (%rax), %xmm17, %xmm19 {%k1}
# CHECK-NEXT: 1 15 5.00 vsqrtss %xmm16, %xmm17, %xmm19 {%k1} {z}
# CHECK-NEXT: 1 22 5.00 * vsqrtss (%rax), %xmm17, %xmm19 {%k1} {z}
-# CHECK-NEXT: 1 1 0.25 vpsubd %zmm16, %zmm17, %zmm19
+# CHECK-NEXT: 1 1 0.50 vpsubd %zmm16, %zmm17, %zmm19
# CHECK-NEXT: 1 8 0.50 * vpsubd (%rax), %zmm17, %zmm19
# CHECK-NEXT: 1 8 0.50 * vpsubd (%rax){1to16}, %zmm17, %zmm19
# CHECK-NEXT: 1 1 0.50 vpsubd %zmm16, %zmm17, %zmm19 {%k1}
@@ -2116,7 +2116,7 @@ vunpcklps (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
# CHECK-NEXT: 1 1 0.50 vpsubd %zmm16, %zmm17, %zmm19 {%k1} {z}
# CHECK-NEXT: 1 8 0.50 * vpsubd (%rax), %zmm17, %zmm19 {%k1} {z}
# CHECK-NEXT: 1 8 0.50 * vpsubd (%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
-# CHECK-NEXT: 1 1 0.25 vpsubq %zmm16, %zmm17, %zmm19
+# CHECK-NEXT: 1 1 0.50 vpsubq %zmm16, %zmm17, %zmm19
# CHECK-NEXT: 1 8 0.50 * vpsubq (%rax), %zmm17, %zmm19
# CHECK-NEXT: 1 8 0.50 * vpsubq (%rax){1to8}, %zmm17, %zmm19
# CHECK-NEXT: 1 1 0.50 vpsubq %zmm16, %zmm17, %zmm19 {%k1}
@@ -2233,7 +2233,7 @@ vunpcklps (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
# CHECK: Resource pressure per iteration:
# CHECK-NEXT: [0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12.0] [12.1] [13] [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1]
-# CHECK-NEXT: 5.33 5.33 5.33 - - - - - 219.50 1119.00 676.50 351.00 312.50 312.50 17.00 215.67 215.67 215.67 204.67 204.67 204.67 16.50 16.50
+# CHECK-NEXT: 5.33 5.33 5.33 - - - - - 221.00 1120.50 678.00 352.50 312.50 312.50 17.00 215.67 215.67 215.67 204.67 204.67 204.67 16.50 16.50
# CHECK: Resource pressure by instruction:
# CHECK-NEXT: [0] [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12.0] [12.1] [13] [14.0] [14.1] [14.2] [15.0] [15.1] [15.2] [16.0] [16.1] Instructions:
@@ -2759,13 +2759,13 @@ vunpcklps (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 - - - - - - - - - - - vpcmpeqq %zmm0, %zmm1, %k2 {%k3}
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpcmpeqq (%rax), %zmm1, %k2 {%k3}
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpcmpeqq (%rax){1to8}, %zmm1, %k2 {%k3}
-# CHECK-NEXT: - - - - - - - - 0.25 0.25 0.25 0.25 - - - - - - - - - - - vpcmpgtd %zmm0, %zmm1, %k2
+# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 - - - - - - - - - - - vpcmpgtd %zmm0, %zmm1, %k2
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpcmpgtd (%rax), %zmm1, %k2
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpcmpgtd (%rax){1to16}, %zmm1, %k2
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 - - - - - - - - - - - vpcmpgtd %zmm0, %zmm1, %k2 {%k3}
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpcmpgtd (%rax), %zmm1, %k2 {%k3}
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpcmpgtd (%rax){1to16}, %zmm1, %k2 {%k3}
-# CHECK-NEXT: - - - - - - - - 0.25 0.25 0.25 0.25 - - - - - - - - - - - vpcmpgtq %zmm0, %zmm1, %k2
+# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 - - - - - - - - - - - vpcmpgtq %zmm0, %zmm1, %k2
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpcmpgtq (%rax), %zmm1, %k2
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpcmpgtq (%rax){1to8}, %zmm1, %k2
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 - - - - - - - - - - - vpcmpgtq %zmm0, %zmm1, %k2 {%k3}
@@ -3040,7 +3040,7 @@ vunpcklps (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
# CHECK-NEXT: - - - - - - - - - 1.00 1.00 - - - - - - - - - - - - vpshufd $0, %zmm16, %zmm19 {%k1} {z}
# CHECK-NEXT: - - - - - - - - - 1.00 1.00 - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpshufd $0, (%rax), %zmm19 {%k1} {z}
# CHECK-NEXT: - - - - - - - - - 1.00 1.00 - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpshufd $0, (%rax){1to16}, %zmm19 {%k1} {z}
-# CHECK-NEXT: - - - - - - - - 0.25 0.25 0.25 0.25 - - - - - - - - - - - vpsubd %zmm16, %zmm17, %zmm19
+# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 - - - - - - - - - - - vpsubd %zmm16, %zmm17, %zmm19
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpsubd (%rax), %zmm17, %zmm19
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpsubd (%rax){1to16}, %zmm17, %zmm19
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 - - - - - - - - - - - vpsubd %zmm16, %zmm17, %zmm19 {%k1}
@@ -3049,7 +3049,7 @@ vunpcklps (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 - - - - - - - - - - - vpsubd %zmm16, %zmm17, %zmm19 {%k1} {z}
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpsubd (%rax), %zmm17, %zmm19 {%k1} {z}
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpsubd (%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
-# CHECK-NEXT: - - - - - - - - 0.25 0.25 0.25 0.25 - - - - - - - - - - - vpsubq %zmm16, %zmm17, %zmm19
+# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 - - - - - - - - - - - vpsubq %zmm16, %zmm17, %zmm19
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpsubq (%rax), %zmm17, %zmm19
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpsubq (%rax){1to8}, %zmm17, %zmm19
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 - - - - - - - - - - - vpsubq %zmm16, %zmm17, %zmm19 {%k1}
@@ -3164,7 +3164,7 @@ vunpcklps (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
# CHECK-NEXT: - - - - - - - - - 5.00 - - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vsqrtss (%rax), %xmm17, %xmm19 {%k1}
# CHECK-NEXT: - - - - - - - - - 5.00 - - - - - - - - - - - - - vsqrtss %xmm16, %xmm17, %xmm19 {%k1} {z}
# CHECK-NEXT: - - - - - - - - - 5.00 - - 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vsqrtss (%rax), %xmm17, %xmm19 {%k1} {z}
-# CHECK-NEXT: - - - - - - - - 0.25 0.25 0.25 0.25 - - - - - - - - - - - vpsubd %zmm16, %zmm17, %zmm19
+# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 - - - - - - - - - - - vpsubd %zmm16, %zmm17, %zmm19
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpsubd (%rax), %zmm17, %zmm19
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpsubd (%rax){1to16}, %zmm17, %zmm19
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 - - - - - - - - - - - vpsubd %zmm16, %zmm17, %zmm19 {%k1}
@@ -3173,7 +3173,7 @@ vunpcklps (%rax){1to16}, %zmm17, %zmm19 {z}{k1}
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 - - - - - - - - - - - vpsubd %zmm16, %zmm17, %zmm19 {%k1} {z}
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpsubd (%rax), %zmm17, %zmm19 {%k1} {z}
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpsubd (%rax){1to16}, %zmm17, %zmm19 {%k1} {z}
-# CHECK-NEXT: - - - - - - - - 0.25 0.25 0.25 0.25 - - - - - - - - - - - vpsubq %zmm16, %zmm17, %zmm19
+# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 - - - - - - - - - - - vpsubq %zmm16, %zmm17, %zmm19
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpsubq (%rax), %zmm17, %zmm19
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 0.50 0.50 - 0.33 0.33 0.33 0.33 0.33 0.33 - - vpsubq (%rax){1to8}, %zmm17, %zmm19
# CHECK-NEXT: - - - - - - - - 0.50 0.50 0.50 0.50 - - - - - - - - - - - vpsubq %zmm16, %zmm17, %zmm19 {%k1}
diff --git a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512bw.s b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512bw.s
index ffdfe3fe98955..a18c074ff3075 100644
--- a/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512bw.s
+++ b/llvm/test/tools/llvm-mca/X86/Znver4/resources-avx512bw.s
@@ -802,11 +802,11 @@ vpunpcklwd (%rax), %zmm17, %zmm19 {z}{k1}
# CHECK-NEXT: 1 8 0.50 * vpcmpeqw (%rax), %zmm1, %k2
# CHECK-NEXT: 1 1 0.50 ...
[truncated]
|
Andarwinux
added
backend:X86 Scheduler Models
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Up front: I don't work on LLVM and I don't know the scheduling models. I was chasing a performance difference between hand-written assembly and C intrinsics in BLAKE3 on a Zen 4 box, used an LLM (Claude) to help me dig through
X86ScheduleZnver4.td, and it turned up what looks like a copy-paste slip.The issue I was chasing here specifically was that compiling BLAKE3 for
znver4is quite bad for performance: generic tuning results in 8512 MB/s vs 8285 MB/s forznver4and handwritten assembly is even higher at 8734 MB/s.This is why I was even looking in this direction to begin with.
The problem
X86ScheduleZnver4.tddeclares zero-idiom scheduling variants in pairs — aSchedWriteVariantfor each vector width, immediately followed by anInstRWlisting the instructions of that width. There are six such pairs. Five of them reference their own width:The last one (
X86ScheduleZnver4.td:1951-1957) names the Y variant, soZn4WriteVZeroIdiomALUZis defined and then never referenced anywhere in the tree, and the sixteen 512-bit instructions it was meant to cover fall back toWriteVecALUY.That matters because
WriteVecALUYholds a vector pipe for one cycle andWriteVecALUZfor two — the doubling is how the model accounts for Zen 4 splitting a 512-bit operation into two 256-bit halves. SoVPSUB{B,W,D,Q}ZrrandVPCMPGT{B,W,D,Q}Zrrwere being given half their port occupancy.Evidence
llvm-mca -mcpu=znver4 -mattr=+avx512f,+avx512bw -instruction-tables, inverse throughput column:vpadddon zmm (for comparison)vpsubd %zmm1, %zmm2, %zmm0vpcmpgtd %zmm1, %zmm2, %k0vpsubd %ymm1, %ymm2, %ymm0Before the change a 512-bit
vpsubdcost exactly what the 256-bit form costs, and half what the neighbouring 512-bitvpadddcosts. After it, the ZMM forms agree withvpadddand the YMM forms are untouched.The zero-idiom arm of the variant is not modified, so the dependency-breaking case still works —
vpsubd %zmm0, %zmm0, %zmm0still reports latency 0.Testing
llvm/test/tools/llvm-mcapasses in full (1785 tests).resources-avx512.sandresources-avx512bw.sregenerated withupdate_mca_test_checks.py. The only differences are the sixteen affected instructions moving from 0.25 to 0.50.