Skip to content

[profcheck] Fix profile metadata in IntegerDivision/ExpandIRinsts #173114

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jinhuang1102 wants to merge 1 commit into
base: main
Choose a base branch
from
+161 −70
Open

[profcheck] Fix profile metadata in IntegerDivision/ExpandIRinsts #173114

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 70 additions & 5 deletions llvm/lib/Transforms/Utils/IntegerDivision.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,22 @@
#include "llvm/Transforms/Utils/IntegerDivision.h"
#include "llvm/IR/Function.h"
#include "llvm/IR/IRBuilder.h"
#include "llvm/IR/Instruction.h"
#include "llvm/IR/Instructions.h"
#include "llvm/IR/Intrinsics.h"
#include "llvm/IR/MDBuilder.h"
#include "llvm/IR/ProfDataUtils.h"
#include "llvm/Support/Casting.h"
#include "llvm/Support/CommandLine.h"

using namespace llvm;

#define DEBUG_TYPE "integer-division"

namespace llvm {
extern cl::opt<bool> ProfcheckDisableMetadataFixes;
}

/// Generate code to compute the remainder of two signed integers. Returns the
/// remainder, which will have the sign of the dividend. Builder's insert point
/// should be pointing where the caller wants code generated, e.g. at the srem
Expand Down Expand Up @@ -235,11 +244,53 @@ static Value *generateUnsignedDivisionCode(Value *Dividend, Value *Divisor,
Value *Tmp1 = Builder.CreateCall(CTLZ, {Dividend, True});
Value *SR = Builder.CreateSub(Tmp0, Tmp1);
Value *Ret0_4 = Builder.CreateICmpUGT(SR, MSB);

// Add 'unlikely' branch weights to the select instruction ('Ret0') generated
// by 'CreateLogicalOr'. We assume the 'true' path, where either the divisor
// or dividend is zero ('Ret0_3'), is less likely than the 'false' path,
// which corresponds to the |divisor| > |dividend| ('Ret0_4').
Copy link
Contributor

@boomanaiden154 boomanaiden154 Dec 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would reword this to talk about the actual case we're marking unlikely rather than to mention all of the implementation details.

Something like the following:

We mark the case where either the divisor or the dividend is equal to zero as unlikely.

Value *Ret0 = Builder.CreateLogicalOr(Ret0_3, Ret0_4);
if (!ProfcheckDisableMetadataFixes) {
if (Instruction *Ret0SelectI = dyn_cast<Instruction>(Ret0)) {
Ret0SelectI->setMetadata(
LLVMContext::MD_prof,
MDBuilder(Ret0SelectI->getContext()).createUnlikelyBranchWeights());
}
}
Value *RetDividend = Builder.CreateICmpEQ(SR, MSB);
Value *RetVal = Builder.CreateSelect(Ret0, Zero, Dividend);
Value *EarlyRet = Builder.CreateLogicalOr(Ret0, RetDividend);
Builder.CreateCondBr(EarlyRet, End, BB1);

// This select instruction (RetVal) immediately follows the Ret0 check. Since
// Ret0_4 determines if the divisor is greater then the dividend so the
// unknown (50/50) branch weights are fairly accurate for RetVal.
Copy link
Contributor

@boomanaiden154 boomanaiden154 Dec 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be reworded too.

I'm also suspect of this logic. You're basically making the assumption that both the dividend and the divisor are being randomly sampled from the same distribution, which I would not expect to be the case. I would expect the common case to be that the dividend is greater than the divisor.

Value *RetVal;
if (!ProfcheckDisableMetadataFixes) {
RetVal = Builder.CreateSelectWithUnknownProfile(Ret0, Zero, Dividend,
DEBUG_TYPE);
} else {
RetVal = Builder.CreateSelect(Ret0, Zero, Dividend);
}
// This select instruction (EarlyRet) is used to check another edge case, and
// it share the same branch weights as RetVal so the unknown (50/50)branch
// weights are also accurate for EarlyRet.
Value *EarlyRet;
if (!ProfcheckDisableMetadataFixes) {
EarlyRet = Builder.CreateSelectWithUnknownProfile(
Ret0, ConstantInt::getAllOnesValue(Ret0->getType()), RetDividend,
DEBUG_TYPE);
} else {
EarlyRet = Builder.CreateLogicalOr(Ret0, RetDividend);
}

// The condition of this branch is based on `EarlyRet`. `EarlyRet` is true
// only for special cases like dividend or divisor being zero, or the divisor
// being greater than the dividend. Thus, the branch to `End` is unlikely,
// and we expect to more frequently enter `BB1`.
Instruction *ConBrSpecialCases = Builder.CreateCondBr(EarlyRet, End, BB1);
if (!ProfcheckDisableMetadataFixes) {
ConBrSpecialCases->setMetadata(LLVMContext::MD_prof,
MDBuilder(ConBrSpecialCases->getContext())
.createUnlikelyBranchWeights());
}

// ; bb1: ; preds = %special-cases
// ; %sr_1 = add i32 %sr, 1
Expand All @@ -252,7 +303,15 @@ static Value *generateUnsignedDivisionCode(Value *Dividend, Value *Divisor,
Value *Tmp2 = Builder.CreateSub(MSB, SR);
Value *Q = Builder.CreateShl(Dividend, Tmp2);
Value *SkipLoop = Builder.CreateICmpEQ(SR_1, Zero);
Builder.CreateCondBr(SkipLoop, LoopExit, Preheader);
// This branch is highly unlikely. SR_1 (SR + 1) is expected to be in [1, 129]
Copy link
Contributor

@boomanaiden154 boomanaiden154 Dec 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This numbers are not necessarily right. This function should be able to handle arbitrary bit widths, but this number looks specific to the 32-bit case used for the example IR.

// because the case where |divisor| > |dividend| (which would make SR
// negative) has alredy been intercepted in the previous SpecialCases BB.
Instruction *ConBrBB1 = Builder.CreateCondBr(SkipLoop, LoopExit, Preheader);
if (!ProfcheckDisableMetadataFixes) {
ConBrBB1->setMetadata(LLVMContext::MD_prof,
MDBuilder(ConBrBB1->getContext())
.createUnlikelyBranchWeights());
}

// ; preheader: ; preds = %bb1
// ; %tmp3 = lshr i32 %dividend, %sr_1
Expand Down Expand Up @@ -298,7 +357,13 @@ static Value *generateUnsignedDivisionCode(Value *Dividend, Value *Divisor,
Value *R = Builder.CreateSub(Tmp7, Tmp11);
Value *SR_2 = Builder.CreateAdd(SR_3, NegOne);
Value *Tmp12 = Builder.CreateICmpEQ(SR_2, Zero);
Builder.CreateCondBr(Tmp12, LoopExit, DoWhile);
// The loop implements the core bit-by-bit binary long division algorithm.
// Since each iteration generates one bit of the quotient until covering all
// significant bits so we would label it as unknown (50/50) branch weights.
Copy link
Contributor

@boomanaiden154 boomanaiden154 Dec 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This logic doesn't really follow for me. The back edge would be significantly more likely to be taken than 50/50 odds, assuming that the typical bit width used is >2 which I assume would be the case?

Instruction *ConBrDoWhile = Builder.CreateCondBr(Tmp12, LoopExit, DoWhile);
if (!ProfcheckDisableMetadataFixes) {
setExplicitlyUnknownBranchWeightsIfProfiled(*ConBrDoWhile, DEBUG_TYPE, F);
}

// ; loop-exit: ; preds = %do-while, %bb1
// ; %carry_2 = phi i32 [ 0, %bb1 ], [ %carry, %do-while ]
Expand Down
20 changes: 13 additions & 7 deletions llvm/test/Transforms/ExpandIRInsts/X86/sdiv129.ll
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,9 @@
; RUN: opt -S -mtriple=x86_64-- -expand-ir-insts -expand-div-rem-bits 128 < %s | FileCheck %s
; RUN: opt -S -mtriple=x86_64-- -passes='require<libcall-lowering-info>,expand-ir-insts' -expand-div-rem-bits 128 < %s | FileCheck %s

define void @sdiv129(ptr %ptr, ptr %out) nounwind {
define void @sdiv129(ptr %ptr, ptr %out) nounwind !prof !0 {
; CHECK-LABEL: @sdiv129(
; CHECK: !prof [[PROF_0:![0-9]+]] {
; CHECK-NEXT: _udiv-special-cases:
; CHECK-NEXT: [[A:%.*]] = load i129, ptr [[PTR:%.*]], align 16
; CHECK-NEXT: [[TMP0:%.*]] = freeze i129 [[A]]
Expand All @@ -24,11 +25,11 @@ define void @sdiv129(ptr %ptr, ptr %out) nounwind {
; CHECK-NEXT: [[TMP15:%.*]] = call i129 @llvm.ctlz.i129(i129 [[TMP10]], i1 true)
; CHECK-NEXT: [[TMP16:%.*]] = sub i129 [[TMP14]], [[TMP15]]
; CHECK-NEXT: [[TMP17:%.*]] = icmp ugt i129 [[TMP16]], 128
; CHECK-NEXT: [[TMP18:%.*]] = select i1 [[TMP13]], i1 true, i1 [[TMP17]]
; CHECK-NEXT: [[TMP18:%.*]] = select i1 [[TMP13]], i1 true, i1 [[TMP17]], !prof [[PROF_1:![0-9]+]]
; CHECK-NEXT: [[TMP19:%.*]] = icmp eq i129 [[TMP16]], 128
; CHECK-NEXT: [[TMP20:%.*]] = select i1 [[TMP18]], i129 0, i129 [[TMP10]]
; CHECK-NEXT: [[TMP21:%.*]] = select i1 [[TMP18]], i1 true, i1 [[TMP19]]
; CHECK-NEXT: br i1 [[TMP21]], label [[UDIV_END:%.*]], label [[UDIV_BB1:%.*]]
; CHECK-NEXT: [[TMP20:%.*]] = select i1 [[TMP18]], i129 0, i129 [[TMP10]], !prof [[PROF_2:![0-9]+]]
; CHECK-NEXT: [[TMP21:%.*]] = select i1 [[TMP18]], i1 true, i1 [[TMP19]], !prof [[PROF_2:![0-9]+]]
; CHECK-NEXT: br i1 [[TMP21]], label [[UDIV_END:%.*]], label [[UDIV_BB1:%.*]], !prof [[PROF_1:![0-9]+]]
; CHECK: udiv-loop-exit:
; CHECK-NEXT: [[TMP22:%.*]] = phi i129 [ 0, [[UDIV_BB1]] ], [ [[TMP37:%.*]], [[UDIV_DO_WHILE:%.*]] ]
; CHECK-NEXT: [[TMP23:%.*]] = phi i129 [ [[TMP46:%.*]], [[UDIV_BB1]] ], [ [[TMP34:%.*]], [[UDIV_DO_WHILE]] ]
Expand All @@ -52,7 +53,7 @@ define void @sdiv129(ptr %ptr, ptr %out) nounwind {
; CHECK-NEXT: [[TMP39]] = sub i129 [[TMP32]], [[TMP38]]
; CHECK-NEXT: [[TMP40]] = add i129 [[TMP27]], -1
; CHECK-NEXT: [[TMP41:%.*]] = icmp eq i129 [[TMP40]], 0
; CHECK-NEXT: br i1 [[TMP41]], label [[UDIV_LOOP_EXIT:%.*]], label [[UDIV_DO_WHILE]]
; CHECK-NEXT: br i1 [[TMP41]], label [[UDIV_LOOP_EXIT:%.*]], label [[UDIV_DO_WHILE]], !prof [[PROF_2:![0-9]+]]
; CHECK: udiv-preheader:
; CHECK-NEXT: [[TMP42]] = lshr i129 [[TMP10]], [[TMP44]]
; CHECK-NEXT: [[TMP43]] = add i129 [[TMP9]], -1
Expand All @@ -62,7 +63,7 @@ define void @sdiv129(ptr %ptr, ptr %out) nounwind {
; CHECK-NEXT: [[TMP45:%.*]] = sub i129 128, [[TMP16]]
; CHECK-NEXT: [[TMP46]] = shl i129 [[TMP10]], [[TMP45]]
; CHECK-NEXT: [[TMP47:%.*]] = icmp eq i129 [[TMP44]], 0
; CHECK-NEXT: br i1 [[TMP47]], label [[UDIV_LOOP_EXIT]], label [[UDIV_PREHEADER]]
; CHECK-NEXT: br i1 [[TMP47]], label [[UDIV_LOOP_EXIT]], label [[UDIV_PREHEADER]], !prof [[PROF_1:![0-9]+]]
; CHECK: udiv-end:
; CHECK-NEXT: [[TMP48:%.*]] = phi i129 [ [[TMP25]], [[UDIV_LOOP_EXIT]] ], [ [[TMP20]], [[_UDIV_SPECIAL_CASES:%.*]] ]
; CHECK-NEXT: [[TMP49:%.*]] = xor i129 [[TMP48]], [[TMP8]]
Expand All @@ -75,3 +76,8 @@ define void @sdiv129(ptr %ptr, ptr %out) nounwind {
store i129 %res, ptr %out
ret void
}

!0 = !{!"function_entry_count", i64 1000}
; CHECK: [[PROF_0]] = !{!"function_entry_count", i64 1000}
; CHECK: [[PROF_1]] = !{!"branch_weights", i32 1, i32 1048575}
; CHECK: [[PROF_2]] = !{!"unknown", !"integer-division"}
20 changes: 13 additions & 7 deletions llvm/test/Transforms/ExpandIRInsts/X86/srem129.ll
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,9 @@
; RUN: opt -S -mtriple=x86_64-- -expand-ir-insts -expand-div-rem-bits 128 < %s | FileCheck %s
; RUN: opt -S -mtriple=x86_64-- -passes='require<libcall-lowering-info>,expand-ir-insts' -expand-div-rem-bits 128 < %s | FileCheck %s

define void @test(ptr %ptr, ptr %out) nounwind {
define void @test(ptr %ptr, ptr %out) nounwind !prof !0 {
; CHECK-LABEL: @test(
; CHECK: !prof [[PROF_0:![0-9]+]] {
; CHECK-NEXT: _udiv-special-cases:
; CHECK-NEXT: [[A:%.*]] = load i129, ptr [[PTR:%.*]], align 16
; CHECK-NEXT: [[TMP0:%.*]] = freeze i129 [[A]]
Expand All @@ -25,11 +26,11 @@ define void @test(ptr %ptr, ptr %out) nounwind {
; CHECK-NEXT: [[TMP16:%.*]] = call i129 @llvm.ctlz.i129(i129 [[TMP11]], i1 true)
; CHECK-NEXT: [[TMP17:%.*]] = sub i129 [[TMP15]], [[TMP16]]
; CHECK-NEXT: [[TMP18:%.*]] = icmp ugt i129 [[TMP17]], 128
; CHECK-NEXT: [[TMP19:%.*]] = select i1 [[TMP14]], i1 true, i1 [[TMP18]]
; CHECK-NEXT: [[TMP19:%.*]] = select i1 [[TMP14]], i1 true, i1 [[TMP18]], !prof [[PROF_1:![0-9]+]]
; CHECK-NEXT: [[TMP20:%.*]] = icmp eq i129 [[TMP17]], 128
; CHECK-NEXT: [[TMP21:%.*]] = select i1 [[TMP19]], i129 0, i129 [[TMP11]]
; CHECK-NEXT: [[TMP22:%.*]] = select i1 [[TMP19]], i1 true, i1 [[TMP20]]
; CHECK-NEXT: br i1 [[TMP22]], label [[UDIV_END:%.*]], label [[UDIV_BB1:%.*]]
; CHECK-NEXT: [[TMP21:%.*]] = select i1 [[TMP19]], i129 0, i129 [[TMP11]], !prof [[PROF_2:![0-9]+]]
; CHECK-NEXT: [[TMP22:%.*]] = select i1 [[TMP19]], i1 true, i1 [[TMP20]], !prof [[PROF_2:![0-9]+]]
; CHECK-NEXT: br i1 [[TMP22]], label [[UDIV_END:%.*]], label [[UDIV_BB1:%.*]], !prof [[PROF_1:![0-9]+]]
; CHECK: udiv-loop-exit:
; CHECK-NEXT: [[TMP23:%.*]] = phi i129 [ 0, [[UDIV_BB1]] ], [ [[TMP38:%.*]], [[UDIV_DO_WHILE:%.*]] ]
; CHECK-NEXT: [[TMP24:%.*]] = phi i129 [ [[TMP47:%.*]], [[UDIV_BB1]] ], [ [[TMP35:%.*]], [[UDIV_DO_WHILE]] ]
Expand All @@ -53,7 +54,7 @@ define void @test(ptr %ptr, ptr %out) nounwind {
; CHECK-NEXT: [[TMP40]] = sub i129 [[TMP33]], [[TMP39]]
; CHECK-NEXT: [[TMP41]] = add i129 [[TMP28]], -1
; CHECK-NEXT: [[TMP42:%.*]] = icmp eq i129 [[TMP41]], 0
; CHECK-NEXT: br i1 [[TMP42]], label [[UDIV_LOOP_EXIT:%.*]], label [[UDIV_DO_WHILE]]
; CHECK-NEXT: br i1 [[TMP42]], label [[UDIV_LOOP_EXIT:%.*]], label [[UDIV_DO_WHILE]], !prof [[PROF_2:![0-9]+]]
; CHECK: udiv-preheader:
; CHECK-NEXT: [[TMP43]] = lshr i129 [[TMP11]], [[TMP45]]
; CHECK-NEXT: [[TMP44]] = add i129 [[TMP10]], -1
Expand All @@ -63,7 +64,7 @@ define void @test(ptr %ptr, ptr %out) nounwind {
; CHECK-NEXT: [[TMP46:%.*]] = sub i129 128, [[TMP17]]
; CHECK-NEXT: [[TMP47]] = shl i129 [[TMP11]], [[TMP46]]
; CHECK-NEXT: [[TMP48:%.*]] = icmp eq i129 [[TMP45]], 0
; CHECK-NEXT: br i1 [[TMP48]], label [[UDIV_LOOP_EXIT]], label [[UDIV_PREHEADER]]
; CHECK-NEXT: br i1 [[TMP48]], label [[UDIV_LOOP_EXIT]], label [[UDIV_PREHEADER]], !prof [[PROF_1:![0-9]+]]
; CHECK: udiv-end:
; CHECK-NEXT: [[TMP49:%.*]] = phi i129 [ [[TMP26]], [[UDIV_LOOP_EXIT]] ], [ [[TMP21]], [[_UDIV_SPECIAL_CASES:%.*]] ]
; CHECK-NEXT: [[TMP50:%.*]] = mul i129 [[TMP9]], [[TMP49]]
Expand All @@ -78,3 +79,8 @@ define void @test(ptr %ptr, ptr %out) nounwind {
store i129 %res, ptr %out
ret void
}

!0 = !{!"function_entry_count", i64 1000}
; CHECK: [[PROF_0]] = !{!"function_entry_count", i64 1000}
; CHECK: [[PROF_1]] = !{!"branch_weights", i32 1, i32 1048575}
; CHECK: [[PROF_2]] = !{!"unknown", !"integer-division"}
20 changes: 13 additions & 7 deletions llvm/test/Transforms/ExpandIRInsts/X86/udiv129.ll
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,9 @@
; RUN: opt -S -mtriple=x86_64-- -expand-ir-insts -expand-div-rem-bits 128 < %s | FileCheck %s
; RUN: opt -S -mtriple=x86_64-- -passes='require<libcall-lowering-info>,expand-ir-insts' -expand-div-rem-bits 128 < %s | FileCheck %s

define void @test(ptr %ptr, ptr %out) nounwind {
define void @test(ptr %ptr, ptr %out) nounwind !prof !0 {
; CHECK-LABEL: @test(
; CHECK: !prof [[PROF_0:![0-9]+]] {
; CHECK-NEXT: _udiv-special-cases:
; CHECK-NEXT: [[A:%.*]] = load i129, ptr [[PTR:%.*]], align 16
; CHECK-NEXT: [[TMP0:%.*]] = freeze i129 3
Expand All @@ -15,11 +16,11 @@ define void @test(ptr %ptr, ptr %out) nounwind {
; CHECK-NEXT: [[TMP6:%.*]] = call i129 @llvm.ctlz.i129(i129 [[TMP1]], i1 true)
; CHECK-NEXT: [[TMP7:%.*]] = sub i129 [[TMP5]], [[TMP6]]
; CHECK-NEXT: [[TMP8:%.*]] = icmp ugt i129 [[TMP7]], 128
; CHECK-NEXT: [[TMP9:%.*]] = select i1 [[TMP4]], i1 true, i1 [[TMP8]]
; CHECK-NEXT: [[TMP9:%.*]] = select i1 [[TMP4]], i1 true, i1 [[TMP8]], !prof [[PROF_1:![0-9]+]]
; CHECK-NEXT: [[TMP10:%.*]] = icmp eq i129 [[TMP7]], 128
; CHECK-NEXT: [[TMP11:%.*]] = select i1 [[TMP9]], i129 0, i129 [[TMP1]]
; CHECK-NEXT: [[TMP12:%.*]] = select i1 [[TMP9]], i1 true, i1 [[TMP10]]
; CHECK-NEXT: br i1 [[TMP12]], label [[UDIV_END:%.*]], label [[UDIV_BB1:%.*]]
; CHECK-NEXT: [[TMP11:%.*]] = select i1 [[TMP9]], i129 0, i129 [[TMP1]], !prof [[PROF_2:![0-9]+]]
; CHECK-NEXT: [[TMP12:%.*]] = select i1 [[TMP9]], i1 true, i1 [[TMP10]], !prof [[PROF_2:![0-9]+]]
; CHECK-NEXT: br i1 [[TMP12]], label [[UDIV_END:%.*]], label [[UDIV_BB1:%.*]], !prof [[PROF_1:![0-9]+]]
; CHECK: udiv-loop-exit:
; CHECK-NEXT: [[TMP13:%.*]] = phi i129 [ 0, [[UDIV_BB1]] ], [ [[TMP28:%.*]], [[UDIV_DO_WHILE:%.*]] ]
; CHECK-NEXT: [[TMP14:%.*]] = phi i129 [ [[TMP37:%.*]], [[UDIV_BB1]] ], [ [[TMP25:%.*]], [[UDIV_DO_WHILE]] ]
Expand All @@ -43,7 +44,7 @@ define void @test(ptr %ptr, ptr %out) nounwind {
; CHECK-NEXT: [[TMP30]] = sub i129 [[TMP23]], [[TMP29]]
; CHECK-NEXT: [[TMP31]] = add i129 [[TMP18]], -1
; CHECK-NEXT: [[TMP32:%.*]] = icmp eq i129 [[TMP31]], 0
; CHECK-NEXT: br i1 [[TMP32]], label [[UDIV_LOOP_EXIT:%.*]], label [[UDIV_DO_WHILE]]
; CHECK-NEXT: br i1 [[TMP32]], label [[UDIV_LOOP_EXIT:%.*]], label [[UDIV_DO_WHILE]], !prof [[PROF_2:![0-9]+]]
; CHECK: udiv-preheader:
; CHECK-NEXT: [[TMP33]] = lshr i129 [[TMP1]], [[TMP35]]
; CHECK-NEXT: [[TMP34]] = add i129 [[TMP0]], -1
Expand All @@ -53,7 +54,7 @@ define void @test(ptr %ptr, ptr %out) nounwind {
; CHECK-NEXT: [[TMP36:%.*]] = sub i129 128, [[TMP7]]
; CHECK-NEXT: [[TMP37]] = shl i129 [[TMP1]], [[TMP36]]
; CHECK-NEXT: [[TMP38:%.*]] = icmp eq i129 [[TMP35]], 0
; CHECK-NEXT: br i1 [[TMP38]], label [[UDIV_LOOP_EXIT]], label [[UDIV_PREHEADER]]
; CHECK-NEXT: br i1 [[TMP38]], label [[UDIV_LOOP_EXIT]], label [[UDIV_PREHEADER]], !prof [[PROF_1:![0-9]+]]
; CHECK: udiv-end:
; CHECK-NEXT: [[TMP39:%.*]] = phi i129 [ [[TMP16]], [[UDIV_LOOP_EXIT]] ], [ [[TMP11]], [[_UDIV_SPECIAL_CASES:%.*]] ]
; CHECK-NEXT: store i129 [[TMP39]], ptr [[OUT:%.*]], align 16
Expand All @@ -64,3 +65,8 @@ define void @test(ptr %ptr, ptr %out) nounwind {
store i129 %res, ptr %out
ret void
}

!0 = !{!"function_entry_count", i64 1000}
; CHECK: [[PROF_0]] = !{!"function_entry_count", i64 1000}
; CHECK: [[PROF_1]] = !{!"branch_weights", i32 1, i32 1048575}
; CHECK: [[PROF_2]] = !{!"unknown", !"integer-division"}
20 changes: 13 additions & 7 deletions llvm/test/Transforms/ExpandIRInsts/X86/urem129.ll
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,9 @@
; RUN: opt -S -mtriple=x86_64-- -expand-ir-insts -expand-div-rem-bits 128 < %s | FileCheck %s
; RUN: opt -S -mtriple=x86_64-- -passes='require<libcall-lowering-info>,expand-ir-insts' -expand-div-rem-bits 128 < %s | FileCheck %s

define void @test(ptr %ptr, ptr %out) nounwind {
define void @test(ptr %ptr, ptr %out) nounwind !prof !0 {
; CHECK-LABEL: @test(
; CHECK: !prof [[PROF_0:![0-9]+]] {
; CHECK-NEXT: _udiv-special-cases:
; CHECK-NEXT: [[A:%.*]] = load i129, ptr [[PTR:%.*]], align 16
; CHECK-NEXT: [[TMP0:%.*]] = freeze i129 [[A]]
Expand All @@ -17,11 +18,11 @@ define void @test(ptr %ptr, ptr %out) nounwind {
; CHECK-NEXT: [[TMP8:%.*]] = call i129 @llvm.ctlz.i129(i129 [[TMP3]], i1 true)
; CHECK-NEXT: [[TMP9:%.*]] = sub i129 [[TMP7]], [[TMP8]]
; CHECK-NEXT: [[TMP10:%.*]] = icmp ugt i129 [[TMP9]], 128
; CHECK-NEXT: [[TMP11:%.*]] = select i1 [[TMP6]], i1 true, i1 [[TMP10]]
; CHECK-NEXT: [[TMP11:%.*]] = select i1 [[TMP6]], i1 true, i1 [[TMP10]], !prof [[PROF_1:![0-9]+]]
; CHECK-NEXT: [[TMP12:%.*]] = icmp eq i129 [[TMP9]], 128
; CHECK-NEXT: [[TMP13:%.*]] = select i1 [[TMP11]], i129 0, i129 [[TMP3]]
; CHECK-NEXT: [[TMP14:%.*]] = select i1 [[TMP11]], i1 true, i1 [[TMP12]]
; CHECK-NEXT: br i1 [[TMP14]], label [[UDIV_END:%.*]], label [[UDIV_BB1:%.*]]
; CHECK-NEXT: [[TMP13:%.*]] = select i1 [[TMP11]], i129 0, i129 [[TMP3]], !prof [[PROF_2:![0-9]+]]
; CHECK-NEXT: [[TMP14:%.*]] = select i1 [[TMP11]], i1 true, i1 [[TMP12]], !prof [[PROF_2:![0-9]+]]
; CHECK-NEXT: br i1 [[TMP14]], label [[UDIV_END:%.*]], label [[UDIV_BB1:%.*]], !prof [[PROF_1:![0-9]+]]
; CHECK: udiv-loop-exit:
; CHECK-NEXT: [[TMP15:%.*]] = phi i129 [ 0, [[UDIV_BB1]] ], [ [[TMP30:%.*]], [[UDIV_DO_WHILE:%.*]] ]
; CHECK-NEXT: [[TMP16:%.*]] = phi i129 [ [[TMP39:%.*]], [[UDIV_BB1]] ], [ [[TMP27:%.*]], [[UDIV_DO_WHILE]] ]
Expand All @@ -45,7 +46,7 @@ define void @test(ptr %ptr, ptr %out) nounwind {
; CHECK-NEXT: [[TMP32]] = sub i129 [[TMP25]], [[TMP31]]
; CHECK-NEXT: [[TMP33]] = add i129 [[TMP20]], -1
; CHECK-NEXT: [[TMP34:%.*]] = icmp eq i129 [[TMP33]], 0
; CHECK-NEXT: br i1 [[TMP34]], label [[UDIV_LOOP_EXIT:%.*]], label [[UDIV_DO_WHILE]]
; CHECK-NEXT: br i1 [[TMP34]], label [[UDIV_LOOP_EXIT:%.*]], label [[UDIV_DO_WHILE]], !prof [[PROF_2:![0-9]+]]
; CHECK: udiv-preheader:
; CHECK-NEXT: [[TMP35]] = lshr i129 [[TMP3]], [[TMP37]]
; CHECK-NEXT: [[TMP36]] = add i129 [[TMP2]], -1
Expand All @@ -55,7 +56,7 @@ define void @test(ptr %ptr, ptr %out) nounwind {
; CHECK-NEXT: [[TMP38:%.*]] = sub i129 128, [[TMP9]]
; CHECK-NEXT: [[TMP39]] = shl i129 [[TMP3]], [[TMP38]]
; CHECK-NEXT: [[TMP40:%.*]] = icmp eq i129 [[TMP37]], 0
; CHECK-NEXT: br i1 [[TMP40]], label [[UDIV_LOOP_EXIT]], label [[UDIV_PREHEADER]]
; CHECK-NEXT: br i1 [[TMP40]], label [[UDIV_LOOP_EXIT]], label [[UDIV_PREHEADER]], !prof [[PROF_1:![0-9]+]]
; CHECK: udiv-end:
; CHECK-NEXT: [[TMP41:%.*]] = phi i129 [ [[TMP18]], [[UDIV_LOOP_EXIT]] ], [ [[TMP13]], [[_UDIV_SPECIAL_CASES:%.*]] ]
; CHECK-NEXT: [[TMP42:%.*]] = mul i129 [[TMP1]], [[TMP41]]
Expand All @@ -68,3 +69,8 @@ define void @test(ptr %ptr, ptr %out) nounwind {
store i129 %res, ptr %out
ret void
}

!0 = !{!"function_entry_count", i64 1000}
; CHECK: [[PROF_0]] = !{!"function_entry_count", i64 1000}
; CHECK: [[PROF_1]] = !{!"branch_weights", i32 1, i32 1048575}
; CHECK: [[PROF_2]] = !{!"unknown", !"integer-division"}
Copy link
Contributor

@alanzhao1 alanzhao1 Dec 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: missing a newline here

Loading
Loading