Skip to content

[ISel] Improve clmul fallback implementation #204802

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
folkertdev wants to merge 3 commits into
base: main
Choose a base branch
from
Open

[ISel] Improve clmul fallback implementation #204802

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
55d1ed5
generalize clmul multiplication with holes approach
folkertdev Jun 19, 2026
0ffdac3
update tests
folkertdev Jun 19, 2026
95caab0
update more tests
folkertdev Jun 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 16 additions & 6 deletions llvm/include/llvm/CodeGen/BasicTTIImpl.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3100,12 +3100,22 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
InstructionCost MulCost =
thisT()->getArithmeticInstrCost(Instruction::Mul, RetTy, CostKind);

// When the multiplication with holes approach is used, that emits 16
// MULs, 8 + 4 ANDs, 12 XORs and 3 ORs.
if (BW >= 32 && BW <= 64 &&
TLI->isOperationLegalOrCustom(ISD::MUL,
TLI->getValueType(DL, RetTy))) {
return 16 * MulCost + 12 * AndCost + 12 * XorCost + 3 * OrCost;
// When the multiplication with holes approach is used, it splits the
// operands into S phases (the smallest stride with ceil(BW/S) <= 2^S) and
// emits S*S MULs, 3*S ANDs, S*(S-1) XORs and S-1 ORs.
//
// * BW <= 8 uses S = 2
// * BW <= 24 uses S = 3
// * BW <= 64 uses S = 4
// * BW <= 160 uses S = 5
// * BW <= 384 uses S = 6
unsigned S = 1;
while (S < 32 && divideCeil(BW, S) > (1u << S))
++S;
if (S * S < BW && TLI->isOperationLegalOrCustom(
ISD::MUL, TLI->getValueType(DL, RetTy))) {
return S * S * MulCost + 3 * S * AndCost + S * (S - 1) * XorCost +
(S - 1) * OrCost;
}

InstructionCost PerBitCostMul = AndCost + MulCost + XorCost;
Expand Down
87 changes: 72 additions & 15 deletions llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8903,37 +8903,94 @@ SDValue TargetLowering::expandCLMUL(SDNode *Node, SelectionDAG &DAG) const {
// do occur, they wind up in a "hole" and are subsequently masked out of the
// result.
//
// A hole of 3 bits is optimal for 32-bit and 64-bit inputs. 128-bit
// integers need a larger hole, and for smaller integers the fallback below
// is more efficient.
// https://www.bearssl.org/constanttime.html#ghash-for-gcm describes this
// approach.

// Stride S handles operands up to S·2^S bits using S² multiplies.
//
// * BW <= 8 uses S = 2 (holes of 1 bit)
// * BW <= 24 uses S = 3 (holes of 2 bits)
// * BW <= 64 uses S = 4 (holes of 3 bits)
// * BW <= 160 uses S = 5 (holes of 4 bits)
// * BW <= 384 uses S = 6 (holes of 5 bits)
//
// We distribute the BW bits over S phases:
//
// phase 0 keeps bits: 0, S, 2S, ...
// phase 1 keeps bits: 1, S + 1, 2S + 1, ...
// ...
//
// Each phase has up to n = ceil(BW / S) bits set, and the holes are S-1
// bits wide.
Comment on lines +8917 to +8924

@folkertdev folkertdev Jun 19, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lmk if this makes sense, it's kind of tricky to explain.

//
// Take BW = 4, S = 2, n = 2. The worst case is a fully populated phase (all
// non-hole bits are set to 1) multiplied by itself, 0b0101 * 0b0101. Each
// set bit of one operand shifts a copy of the other, and we add the copies:
//
// col: 4 3 2 1 0
// 0b0101 << 0: 0 0 1 0 1
// 0b0101 << 2: 1 0 1 0 0
// ----------------- +
// count: 1 0 2 0 1
//
// Counting the number of one-bits in each column gives a triangle: the
// counts climb 1, 2, ..., n and back down (here 1, 2, 1 across the data
// columns). So a column holds at most n one-bits, and that maximum n is
// reached in only one column: the peak. Every other column holds at most n
// - 1 one-bits.
//
// A stack of one-bits in a column turns into carries: column 2 above really
// stores the value 1 + 1 = 2 = n. A column spans S bits, its kept bit
// plus S-1 hole bits, and the count is written from the kept bit upward,
// so any count <= 2^S - 1 stays within the column and never interferes with
// the next data bit S positions up. Every non-peak column holds at most n -
// 1, so they all fit as soon as n - 1 <= 2^S - 1.
//
// That leaves only the peak column. Because both operands set all data
// bits, the triangle peaks at the top of the word at the highest data bit
// still inside BW. Here the count reaches exactly n = 2^S and overflows.
// But its carry lands at bit n*S >= BW, off the top, where it (and the
// whole descending half of the triangle) is truncated.
//
// Hence the holes suffice exactly when n = ceil(BW / S) <= 2^S, i.e. BW <=
// S*2^S.
//
// Based on bmul64 in bearssl and bmul in the rust polyval crate.
if (BW >= 32 && BW <= 64 &&
// Here we find the smallest S that satisfies this inequality.
unsigned S = 1;
while (S < 32 && divideCeil(BW, S) > (1u << S))
++S;

// Continue to use the naive fallback below if it seems cheaper. We compare
// the number of multiplications here (S * S) versus the number of
// iterations there (BW). The naive fallback is still used for i1, i3, i4
// and i9, and when multiplication isn't available.
if (S * S < BW &&
isOperationLegalOrCustom(ISD::MUL, getTypeToTransformTo(Ctx, VT))) {

// Set every fourth bit of each nibble, equivalent to 0b00010001...0001.
APInt MaskVal = APInt::getSplat(BW, APInt(4, 0b0001));
// Set a bit every S positions, e.g. for S = 4 this is equivalent to
// 0b...00010001...0001.
APInt MaskVal = APInt::getSplat(BW, APInt(S, 1));

// Create versions of X and Y that keep only the I-th bit of
// each nibble.
SDValue M[4], Xp[4], Yp[4];
for (unsigned I = 0; I < 4; ++I) {
// Create versions of X and Y that keep only the I-th bit of each S-bit
// slice.
SmallVector<SDValue, 4> M(S), Xp(S), Yp(S);
for (unsigned I = 0; I < S; ++I) {
M[I] = DAG.getConstant(MaskVal.shl(I), DL, VT);
Xp[I] = DAG.getNode(ISD::AND, DL, VT, X, M[I]);
Yp[I] = DAG.getNode(ISD::AND, DL, VT, Y, M[I]);
}

// Codegens these expressions (16 multiplications):
// Codegens these expressions (S*S multiplications), e.g. for S=4:
//
// z0 = (x0 * y0) ^ (x1 * y3) ^ (x2 * y2) ^ (x3 * y1);
// z1 = (x0 * y1) ^ (x1 * y0) ^ (x2 * y3) ^ (x3 * y2);
// z2 = (x0 * y2) ^ (x1 * y1) ^ (x2 * y0) ^ (x3 * y3);
// z3 = (x0 * y3) ^ (x1 * y2) ^ (x2 * y1) ^ (x3 * y0);
SDValue Res = DAG.getConstant(0, DL, VT);
for (unsigned I = 0; I < 4; ++I) {
for (unsigned I = 0; I < S; ++I) {
SDValue Zi = DAG.getConstant(0, DL, VT);
for (unsigned J = 0; J < 4; ++J) {
unsigned K = (I + 4 - J) % 4;
for (unsigned J = 0; J < S; ++J) {
unsigned K = (I + S - J) % S;
SDValue P = DAG.getNode(ISD::MUL, DL, VT, Xp[J], Yp[K]);
Zi = DAG.getNode(ISD::XOR, DL, VT, Zi, P);
}
Expand Down
8 changes: 4 additions & 4 deletions llvm/test/Analysis/CostModel/AArch64/clmul-fixed.ll
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ define void @clmul_fixed() {
; NOAES-NEXT: Cost Model: Found an estimated cost of 43 for instruction: %c1 = call <1 x i64> @llvm.clmul.v1i64(<1 x i64> poison, <1 x i64> poison)
; NOAES-NEXT: Cost Model: Found an estimated cost of 43 for instruction: %c32 = call <4 x i32> @llvm.clmul.v4i32(<4 x i32> poison, <4 x i32> poison)
; NOAES-NEXT: Cost Model: Found an estimated cost of 43 for instruction: %c2 = call <2 x i32> @llvm.clmul.v2i32(<2 x i32> poison, <2 x i32> poison)
; NOAES-NEXT: Cost Model: Found an estimated cost of 48 for instruction: %c16 = call <4 x i16> @llvm.clmul.v4i16(<4 x i16> poison, <4 x i16> poison)
; NOAES-NEXT: Cost Model: Found an estimated cost of 48 for instruction: %c16x8 = call <8 x i16> @llvm.clmul.v8i16(<8 x i16> poison, <8 x i16> poison)
; NOAES-NEXT: Cost Model: Found an estimated cost of 26 for instruction: %c16 = call <4 x i16> @llvm.clmul.v4i16(<4 x i16> poison, <4 x i16> poison)
; NOAES-NEXT: Cost Model: Found an estimated cost of 26 for instruction: %c16x8 = call <8 x i16> @llvm.clmul.v8i16(<8 x i16> poison, <8 x i16> poison)
; NOAES-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; AES-LABEL: 'clmul_fixed'
Expand All @@ -21,8 +21,8 @@ define void @clmul_fixed() {
; AES-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %c1 = call <1 x i64> @llvm.clmul.v1i64(<1 x i64> poison, <1 x i64> poison)
; AES-NEXT: Cost Model: Found an estimated cost of 43 for instruction: %c32 = call <4 x i32> @llvm.clmul.v4i32(<4 x i32> poison, <4 x i32> poison)
; AES-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %c2 = call <2 x i32> @llvm.clmul.v2i32(<2 x i32> poison, <2 x i32> poison)
; AES-NEXT: Cost Model: Found an estimated cost of 48 for instruction: %c16 = call <4 x i16> @llvm.clmul.v4i16(<4 x i16> poison, <4 x i16> poison)
; AES-NEXT: Cost Model: Found an estimated cost of 48 for instruction: %c16x8 = call <8 x i16> @llvm.clmul.v8i16(<8 x i16> poison, <8 x i16> poison)
; AES-NEXT: Cost Model: Found an estimated cost of 26 for instruction: %c16 = call <4 x i16> @llvm.clmul.v4i16(<4 x i16> poison, <4 x i16> poison)
; AES-NEXT: Cost Model: Found an estimated cost of 26 for instruction: %c16x8 = call <8 x i16> @llvm.clmul.v8i16(<8 x i16> poison, <8 x i16> poison)
; AES-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
%c8 = call <16 x i8> @llvm.clmul.v16i8(<16 x i8> poison, <16 x i8> poison)
Expand Down
8 changes: 4 additions & 4 deletions llvm/test/Analysis/CostModel/AArch64/clmul-scalable.ll
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,15 @@

define void @clmul_scalable() {
; SVE-LABEL: 'clmul_scalable'
; SVE-NEXT: Cost Model: Found an estimated cost of 24 for instruction: %c8 = call <vscale x 16 x i8> @llvm.clmul.nxv16i8(<vscale x 16 x i8> poison, <vscale x 16 x i8> poison)
; SVE-NEXT: Cost Model: Found an estimated cost of 48 for instruction: %c16 = call <vscale x 8 x i16> @llvm.clmul.nxv8i16(<vscale x 8 x i16> poison, <vscale x 8 x i16> poison)
; SVE-NEXT: Cost Model: Found an estimated cost of 13 for instruction: %c8 = call <vscale x 16 x i8> @llvm.clmul.nxv16i8(<vscale x 16 x i8> poison, <vscale x 16 x i8> poison)
; SVE-NEXT: Cost Model: Found an estimated cost of 26 for instruction: %c16 = call <vscale x 8 x i16> @llvm.clmul.nxv8i16(<vscale x 8 x i16> poison, <vscale x 8 x i16> poison)
; SVE-NEXT: Cost Model: Found an estimated cost of 43 for instruction: %c32 = call <vscale x 4 x i32> @llvm.clmul.nxv4i32(<vscale x 4 x i32> poison, <vscale x 4 x i32> poison)
; SVE-NEXT: Cost Model: Found an estimated cost of 192 for instruction: %c64 = call <vscale x 2 x i64> @llvm.clmul.nxv2i64(<vscale x 2 x i64> poison, <vscale x 2 x i64> poison)
; SVE-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; SVE-AES-LABEL: 'clmul_scalable'
; SVE-AES-NEXT: Cost Model: Found an estimated cost of 24 for instruction: %c8 = call <vscale x 16 x i8> @llvm.clmul.nxv16i8(<vscale x 16 x i8> poison, <vscale x 16 x i8> poison)
; SVE-AES-NEXT: Cost Model: Found an estimated cost of 48 for instruction: %c16 = call <vscale x 8 x i16> @llvm.clmul.nxv8i16(<vscale x 8 x i16> poison, <vscale x 8 x i16> poison)
; SVE-AES-NEXT: Cost Model: Found an estimated cost of 13 for instruction: %c8 = call <vscale x 16 x i8> @llvm.clmul.nxv16i8(<vscale x 16 x i8> poison, <vscale x 16 x i8> poison)
; SVE-AES-NEXT: Cost Model: Found an estimated cost of 26 for instruction: %c16 = call <vscale x 8 x i16> @llvm.clmul.nxv8i16(<vscale x 8 x i16> poison, <vscale x 8 x i16> poison)
; SVE-AES-NEXT: Cost Model: Found an estimated cost of 43 for instruction: %c32 = call <vscale x 4 x i32> @llvm.clmul.nxv4i32(<vscale x 4 x i32> poison, <vscale x 4 x i32> poison)
; SVE-AES-NEXT: Cost Model: Found an estimated cost of 3 for instruction: %c64 = call <vscale x 2 x i64> @llvm.clmul.nxv2i64(<vscale x 2 x i64> poison, <vscale x 2 x i64> poison)
; SVE-AES-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
Expand Down
12 changes: 6 additions & 6 deletions llvm/test/Analysis/CostModel/X86/clmul.ll
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@ define void @clmul(i128 %a128, i128 %b128, i64 %a64, i64 %b64, i32 %a32, i32 %b3
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 768 for instruction: %call_i128 = call i128 @llvm.clmul.i128(i128 %a128, i128 %b128)
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 59 for instruction: %call_i64 = call i64 @llvm.clmul.i64(i64 %a64, i64 %b64)
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 43 for instruction: %call_i32 = call i32 @llvm.clmul.i32(i32 %a32, i32 %b32)
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 48 for instruction: %call_i16 = call i16 @llvm.clmul.i16(i16 %a16, i16 %b16)
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 24 for instruction: %call_i8 = call i8 @llvm.clmul.i8(i8 %a8, i8 %b8)
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 35 for instruction: %call_i16 = call i16 @llvm.clmul.i16(i16 %a16, i16 %b16)
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 21 for instruction: %call_i8 = call i8 @llvm.clmul.i8(i8 %a8, i8 %b8)
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
%call_i128 = call i128 @llvm.clmul.i128(i128 %a128, i128 %b128)
Expand All @@ -33,16 +33,16 @@ define void @clmul_128(<1 x i128> %a128, <1 x i128> %b128, <2 x i64> %a64, <2 x
; PCLMUL-NEXT: Cost Model: Found an estimated cost of 4 for instruction: %call_i128 = call <1 x i128> @llvm.clmul.v1i128(<1 x i128> %a128, <1 x i128> %b128)
; PCLMUL-NEXT: Cost Model: Found an estimated cost of 2 for instruction: %call_i64 = call <2 x i64> @llvm.clmul.v2i64(<2 x i64> %a64, <2 x i64> %b64)
; PCLMUL-NEXT: Cost Model: Found an estimated cost of 2 for instruction: %call_i32 = call <4 x i32> @llvm.clmul.v4i32(<4 x i32> %a32, <4 x i32> %b32)
; PCLMUL-NEXT: Cost Model: Found an estimated cost of 48 for instruction: %call_i16 = call <8 x i16> @llvm.clmul.v8i16(<8 x i16> %a16, <8 x i16> %b16)
; PCLMUL-NEXT: Cost Model: Found an estimated cost of 40 for instruction: %call_i8 = call <16 x i8> @llvm.clmul.v16i8(<16 x i8> %a8, <16 x i8> %b8)
; PCLMUL-NEXT: Cost Model: Found an estimated cost of 26 for instruction: %call_i16 = call <8 x i16> @llvm.clmul.v8i16(<8 x i16> %a16, <8 x i16> %b16)
; PCLMUL-NEXT: Cost Model: Found an estimated cost of 33 for instruction: %call_i8 = call <16 x i8> @llvm.clmul.v16i8(<16 x i8> %a8, <16 x i8> %b8)
; PCLMUL-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
; NO-PCLMUL-LABEL: 'clmul_128'
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 768 for instruction: %call_i128 = call <1 x i128> @llvm.clmul.v1i128(<1 x i128> %a128, <1 x i128> %b128)
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 139 for instruction: %call_i64 = call <2 x i64> @llvm.clmul.v2i64(<2 x i64> %a64, <2 x i64> %b64)
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 123 for instruction: %call_i32 = call <4 x i32> @llvm.clmul.v4i32(<4 x i32> %a32, <4 x i32> %b32)
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 48 for instruction: %call_i16 = call <8 x i16> @llvm.clmul.v8i16(<8 x i16> %a16, <8 x i16> %b16)
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 40 for instruction: %call_i8 = call <16 x i8> @llvm.clmul.v16i8(<16 x i8> %a8, <16 x i8> %b8)
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 26 for instruction: %call_i16 = call <8 x i16> @llvm.clmul.v8i16(<8 x i16> %a16, <8 x i16> %b16)
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 33 for instruction: %call_i8 = call <16 x i8> @llvm.clmul.v16i8(<16 x i8> %a8, <16 x i8> %b8)
; NO-PCLMUL-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
;
%call_i128 = call <1 x i128> @llvm.clmul.v1i128(<1 x i128> %a128, <1 x i128> %b128)
Expand Down
Loading