Skip to content

ast-exporter: Export some macro arg expansions #1792

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Rua wants to merge 7 commits into
base: master
Choose a base branch
from
Open

ast-exporter: Export some macro arg expansions #1792

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
3e22a2b
ast-exporter: Add `macro_fns.c` snapshot test
Rua May 14, 2026
7ef8fa5
ast-exporter: Skip over parameter expansions in `VisitExpr`
Rua May 13, 2026
05f86c9
ast-exporter: Move `getMacroInfo` closer to the code that calls it
Rua May 14, 2026
1a27263
ast-exporter: Split off `getIdentifierInfo`
Rua May 14, 2026
9b797bf
ast-exporter: Rename "expansion" to "invocation"
Rua May 14, 2026
e470201
ast-exporter: Add parameter information to macro invocations
Rua May 14, 2026
025413b
ast-exporter: Export macro parameter information to Rust
Rua May 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
240 changes: 152 additions & 88 deletions c2rust-ast-exporter/src/AstExporter.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -711,7 +711,7 @@ class TypeEncoder final : public TypeVisitor<TypeEncoder> {
class TranslateASTVisitor final
: public RecursiveASTVisitor<TranslateASTVisitor> {

struct MacroExpansionInfo {
struct MacroDeclInfo {
StringRef Name;
};

Expand All @@ -723,13 +723,20 @@ class TranslateASTVisitor final
// Mapping from SourceManager FileID to index in files
DenseMap<FileID, size_t> file_id_mapping;
std::set<std::pair<void *, ASTEntryTag>> exportedTags;
std::unordered_map<MacroInfo*, MacroExpansionInfo> macros;
std::unordered_map<MacroInfo*, MacroDeclInfo> macros;

struct MacroInvocationInfo {
uint32_t key;
MacroInfo* macro;
StringRef parameter;
};

// This stores a raw encoding of the macro call site SourceLocation, since
// SourceLocation isn't hashable.
std::unordered_set<unsigned> macroCallSites;
SmallVector<MacroInfo*, 1> curMacroExpansionStack;
StringRef curMacroExpansionSource;
// TODO: Can this be made to use `unordered_set` for speed?
std::set<std::pair<unsigned, const StringRef>> macroCallSites;

@ahomescu ahomescu Jul 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do these StringRefs point into? Is there a potential lifetime issue here?

@Rua Rua Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They point to the Clang AST in the same way that macros does, which also holds a StringRef. I assume that both live as long as the AST is kept alive?

SmallVector<MacroInvocationInfo, 1> curMacroInvocationStack;
StringRef curMacroInvocationSource;

// Returns true when a new entry is added to exportedTags
bool markForExport(void *ptr, ASTEntryTag tag) {
Expand Down Expand Up @@ -769,7 +776,7 @@ class TranslateASTVisitor final
// Template required because Decl and Stmt don't share a common base class
void encode_entry_raw(void *ast, ASTEntryTag tag, SourceRange loc,
const QualType ty, bool rvalue,
bool isVaList, bool encodeMacroExpansions,
bool isVaList, bool encodeMacroInvocations,

@ahomescu ahomescu Jul 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you explain why the rename from "expansions" to "invocations"? I'm guessing there's a subtle difference but I'm not sure what it is.

@Rua Rua Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my understanding, an invocation is the place where the macro name is written, the "call". And the expansion is the act or result of replacing the macro name with its output tokens. Most of the logic for macros works "backwards": the fully expanded macro, in the form of a C expression, is the start point, and c2rust is working to find the original macro invocation(s)/call(s) that produced it. So I felt that naming that "expansion" was confusing and not very descriptive.

Maybe there are better things to name these though?

const std::vector<void *> &childIds,
std::function<void(CborEncoder *)> extra) {
if (!markForExport(ast, tag))
Expand Down Expand Up @@ -808,21 +815,33 @@ class TranslateASTVisitor final
// 9 - Is Rvalue (only for expressions)
cbor_encode_boolean(&local, rvalue);

// 10 - Macro expansion stack, starting with initial macro call and ending
// 10 - Macro invocation stack, starting with initial macro call and ending
// with the innermost replacement.
cbor_encoder_create_array(&local, &childEnc,
encodeMacroExpansions ? curMacroExpansionStack.size() : 0);
if (encodeMacroExpansions) {
for (auto I = curMacroExpansionStack.rbegin(), E = curMacroExpansionStack.rend();
encodeMacroInvocations ? curMacroInvocationStack.size() : 0);
if (encodeMacroInvocations) {
for (auto I = curMacroInvocationStack.rbegin(), E = curMacroInvocationStack.rend();
I != E; ++I) {
cbor_encode_uint(&childEnc, uintptr_t(*I));
CborEncoder expansionEnc;
cbor_encoder_create_array(&childEnc, &expansionEnc, 3);

cbor_encode_uint(&expansionEnc, I->key);
cbor_encode_uint(&expansionEnc, uintptr_t(I->macro));

if (!I->parameter.empty()) {
cbor_encode_string(&expansionEnc, I->parameter.str());
} else {
cbor_encode_null(&expansionEnc);
}

cbor_encoder_close_container(&childEnc, &expansionEnc);
}
}
cbor_encoder_close_container(&local, &childEnc);

// 11 - Macro expansion source string, if applicable.
if (!curMacroExpansionSource.empty()) {
cbor_encode_string(&local, curMacroExpansionSource.str());
// 11 - Macro invocation source string, if applicable.
if (!curMacroInvocationSource.empty()) {
cbor_encode_string(&local, curMacroInvocationSource.str());
} else {
cbor_encode_null(&local);
}
Expand All @@ -846,7 +865,7 @@ class TranslateASTVisitor final
std::function<void(CborEncoder *)> extra = [](CborEncoder *) {}) {
auto ty = ast->getType();
auto isVaList = false;
auto encodeMacroExpansions = true;
auto encodeMacroInvocations = true;
#if CLANG_VERSION_MAJOR < 13
bool isRValue = ast->isRValue();
#else
Expand All @@ -860,7 +879,7 @@ class TranslateASTVisitor final
auto span = ast->getSourceRange();
expandSpanToFinalChar(span, Context);
encode_entry_raw(ast, tag, span, ty, isRValue, isVaList,
encodeMacroExpansions, childIds, extra);
encodeMacroInvocations, childIds, extra);
typeEncoder.VisitQualTypeOf(ty, ast);
}

Expand All @@ -870,23 +889,23 @@ class TranslateASTVisitor final
QualType s = QualType(static_cast<clang::Type *>(nullptr), 0);
auto rvalue = false;
auto isVaList = false;
auto encodeMacroExpansions = false;
auto encodeMacroInvocations = false;
auto span = ast->getSourceRange();
expandSpanToFinalChar(span, Context);
encode_entry_raw(ast, tag, span, s, rvalue, isVaList,
encodeMacroExpansions, childIds, extra);
encodeMacroInvocations, childIds, extra);
}

void encode_entry(
Decl *ast, ASTEntryTag tag, const std::vector<void *> &childIds,
const QualType T,
std::function<void(CborEncoder *)> extra = [](CborEncoder *) {}) {
auto rvalue = false;
auto encodeMacroExpansions = false;
auto encodeMacroInvocations = false;
auto span = ast->getSourceRange();
expandSpanToFinalChar(span, Context);
encode_entry_raw(ast, tag, span, T, rvalue,
isVaList(ast, T), encodeMacroExpansions, childIds, extra);
isVaList(ast, T), encodeMacroInvocations, childIds, extra);
}

/// Explicitly override the source location of this decl for cases where the
Expand All @@ -897,54 +916,21 @@ class TranslateASTVisitor final
const std::vector<void *> &childIds, const QualType T,
std::function<void(CborEncoder *)> extra = [](CborEncoder *) {}) {
auto rvalue = false;
auto encodeMacroExpansions = false;
auto encodeMacroInvocations = false;
expandSpanToFinalChar(loc, Context);
encode_entry_raw(ast, tag, loc, T, rvalue,
isVaList(ast, T), encodeMacroExpansions, childIds, extra);
}

MacroInfo* getMacroInfo(SourceLocation loc, StringRef &name) const {
auto &Mgr = Context->getSourceManager();
Token Result;
if (!Lexer::getRawToken(Mgr.getSpellingLoc(loc), Result,
Mgr, Context->getLangOpts(), false)) {
if (Result.is(tok::raw_identifier)) {
PP.LookUpIdentifierInfo(Result);
}
IdentifierInfo *IdentifierInfo = Result.getIdentifierInfo();
if (IdentifierInfo && IdentifierInfo->hadMacroDefinition()) {
std::pair<FileID, unsigned int> DecLoc =
Mgr.getDecomposedExpansionLoc(loc);
// Get the definition just before the searched location
// so that a macro referenced in a '#undef MACRO' can
// still be found.
SourceLocation BeforeSearchedLocation =
Mgr.getMacroArgExpandedLocation(
Mgr.getLocForStartOfFile(DecLoc.first)
.getLocWithOffset(DecLoc.second - 1));
MacroDefinition MacroDef = PP.getMacroDefinitionAtLoc(
IdentifierInfo, BeforeSearchedLocation);
MacroInfo *MacroInf = MacroDef.getMacroInfo();
if (MacroInf) {
LLVM_DEBUG(dbgs() << IdentifierInfo->getName() << "\n");
LLVM_DEBUG(MacroInf->dump());
LLVM_DEBUG(dbgs() << "\n");
name = IdentifierInfo->getName();
return MacroInf;
}
}
}
return nullptr;
isVaList(ast, T), encodeMacroInvocations, childIds, extra);
}

bool VisitMacro(StringRef name, SourceLocation loc, MacroInfo *mac, Expr *E) {
bool VisitMacro(StringRef name, StringRef parameter, SourceLocation loc, MacroInfo *mac, Expr *E) {
// TODO: handle builtin macros
if (mac->isBuiltinMacro())
return false;
// If this isn't the first time we've seen this macro call site, we
// shouldn't associate this expression with the macro as it is a subexpr
// of a previously seen expression.
if (!macroCallSites.insert(loc.getRawEncoding()).second)
std::pair<unsigned, StringRef> key { loc.getRawEncoding(), parameter };
if (!macroCallSites.insert(key).second)
return false;
auto &info = macros[mac];
if (info.Name.empty())
Expand Down Expand Up @@ -1007,11 +993,11 @@ class TranslateASTVisitor final

void encodeMacros() {
// Sort macros by source location
std::vector<std::pair<MacroInfo *, MacroExpansionInfo>> macro_vec(
std::vector<std::pair<MacroInfo *, MacroDeclInfo>> macro_vec(
macros.begin(), macros.end());
std::sort(macro_vec.begin(), macro_vec.end(),
[](const std::pair<MacroInfo *, MacroExpansionInfo> &a,
const std::pair<MacroInfo *, MacroExpansionInfo> &b) {
[](const std::pair<MacroInfo *, MacroDeclInfo> &a,
const std::pair<MacroInfo *, MacroDeclInfo> &b) {
return a.first->getDefinitionLoc() <
b.first->getDefinitionLoc();
});
Expand Down Expand Up @@ -1403,8 +1389,8 @@ class TranslateASTVisitor final
//

bool VisitExpr(Expr *E) {
curMacroExpansionStack.clear();
curMacroExpansionSource = StringRef();
curMacroInvocationStack.clear();
curMacroInvocationSource = StringRef();

// We only translate constant macro objects to Rust consts, so this
// expression must be constant.
Expand Down Expand Up @@ -1438,39 +1424,73 @@ class TranslateASTVisitor final
return true;
}

curMacroExpansionSource =
Lexer::getSourceText(ExpansionStack[0], Mgr, Context->getLangOpts());
curMacroInvocationSource =
Lexer::getSourceText(ExpansionStack[0].range, Mgr, Context->getLangOpts());

for (auto &elem : ExpansionStack) {
auto loc = elem.range.getBegin();
StringRef parameter;

if (elem.isParameter) {
auto IdentifierInfo = getIdentifierInfo(loc);

if (!IdentifierInfo) {
return true;
}

parameter = IdentifierInfo->getName();
loc = Mgr.getImmediateExpansionRange(loc).getBegin();
}

auto IdentifierInfo = getIdentifierInfo(loc);

if (!IdentifierInfo) {
return true;
}

for (auto &ExpansionRange : ExpansionStack) {
StringRef name;
MacroInfo *mac = getMacroInfo(ExpansionRange.getBegin(), name);
auto MacroInfo = getMacroInfo(loc, IdentifierInfo);

if (!mac || mac->getNumTokens() == 0) {
if (!MacroInfo || MacroInfo->getNumTokens() == 0) {
return true;
}

if (VisitMacro(name, ExpansionRange.getBegin(), mac, E)) {
curMacroExpansionStack.push_back(mac);
if (VisitMacro(IdentifierInfo->getName(), parameter, loc, MacroInfo, E)) {
MacroInvocationInfo info = { loc.getRawEncoding(), MacroInfo, parameter };
curMacroInvocationStack.push_back(info);
}
}

return true;
}

std::vector<CharSourceRange> getMacroExpansionStack(SourceRange Range) const {
struct ExpansionRange {
CharSourceRange range;
bool isParameter;

bool operator== (const ExpansionRange &rhs) const {
return this->range.getAsRange() == rhs.range.getAsRange()
&& this->range.isTokenRange() == rhs.range.isTokenRange()
&& this->isParameter == rhs.isParameter;
}
};

std::vector<ExpansionRange> getMacroExpansionStack(SourceRange Range) const {
auto &Mgr = Context->getSourceManager();
auto Begin = Range.getBegin();
auto End = Range.getEnd();
std::vector<CharSourceRange> ExpansionStack;
std::vector<ExpansionRange> ExpansionStack;

do {
if (!isAtStartOfImmediateMacroExpansion(Begin)) {
break;
}

auto ExpansionRange = Mgr.getImmediateExpansionRange(Begin);
ExpansionStack.push_back(ExpansionRange);
Begin = ExpansionRange.getBegin();
ExpansionRange elem = {
Mgr.getImmediateExpansionRange(Begin),
Mgr.isMacroArgExpansion(Begin)
};
Begin = elem.range.getBegin();
ExpansionStack.push_back(elem);
} while (Begin.isMacroID());

// Find the point at which `Begin` and `End` converge on the same expansion range.
Expand All @@ -1482,16 +1502,12 @@ class TranslateASTVisitor final
break;
}

auto ExpansionRange = Mgr.getImmediateExpansionRange(End);
ConvergencePoint = std::find_if(
ExpansionStack.begin(),
ExpansionStack.end(),
[ExpansionRange](auto &R) {
return R.getAsRange() == ExpansionRange.getAsRange() &&
R.isTokenRange() == ExpansionRange.isTokenRange();
}
);
End = ExpansionRange.getEnd();
ExpansionRange elem = {
Mgr.getImmediateExpansionRange(End),
Mgr.isMacroArgExpansion(End)
};
End = elem.range.getEnd();
ConvergencePoint = std::find(ExpansionStack.begin(), ExpansionStack.end(), elem);
} while (End.isMacroID() && ConvergencePoint == ExpansionStack.end());

// Remove all elements before the convergence point.
Expand All @@ -1501,8 +1517,8 @@ class TranslateASTVisitor final
auto EraseAfter = std::find_if(
ExpansionStack.begin(),
ExpansionStack.end(),
[this](auto &R) {
auto End = R.getEnd();
[this](auto &elem) {
auto End = elem.range.getEnd();
return End.isMacroID() && !isAtEndOfImmediateMacroExpansion(End);
}
);
Expand All @@ -1527,6 +1543,54 @@ class TranslateASTVisitor final
return Mgr.isAtEndOfImmediateMacroExpansion(loc.getLocWithOffset(len));
}

IdentifierInfo *getIdentifierInfo(SourceLocation loc) const {
auto &Mgr = Context->getSourceManager();
Token Result;

if (!Lexer::getRawToken(
Mgr.getSpellingLoc(loc),
Result,
Mgr,
Context->getLangOpts(),
false
)) {
if (Result.is(tok::raw_identifier)) {
PP.LookUpIdentifierInfo(Result);
}

return Result.getIdentifierInfo();
}

return nullptr;
}

MacroInfo* getMacroInfo(SourceLocation loc, IdentifierInfo *IdentifierInfo) const {
if (!IdentifierInfo->hadMacroDefinition()) {
return nullptr;
}

auto &Mgr = Context->getSourceManager();
std::pair<FileID, unsigned int> DecLoc = Mgr.getDecomposedExpansionLoc(loc);
// Get the definition just before the searched location
// so that a macro referenced in a '#undef MACRO' can
// still be found.
SourceLocation BeforeSearchedLocation = Mgr.getMacroArgExpandedLocation(
Mgr.getLocForStartOfFile(DecLoc.first).getLocWithOffset(DecLoc.second - 1));
MacroDefinition MacroDef = PP.getMacroDefinitionAtLoc(
IdentifierInfo, BeforeSearchedLocation);
MacroInfo *MacroInf = MacroDef.getMacroInfo();

if (!MacroInf) {
return nullptr;
}

LLVM_DEBUG(dbgs() << IdentifierInfo->getName() << "\n");
LLVM_DEBUG(MacroInf->dump());
LLVM_DEBUG(dbgs() << "\n");

return MacroInf;
}

bool VisitVAArgExpr(VAArgExpr *E) {
std::vector<void *> childIds{E->getSubExpr()};
encode_entry(E, TagVAArgExpr, childIds);
Expand Down
Loading
Loading