Skip to content

Reduce the amount of code we generate from macros#6187

Open
alex wants to merge 6 commits into
PyO3:mainfrom
alex:slim-codegen
Open

Reduce the amount of code we generate from macros#6187
alex wants to merge 6 commits into
PyO3:mainfrom
alex:slim-codegen

Conversation

@alex

@alex alex commented Jul 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Headline results for pyca/cryptography:

Metric (median) stock 0.29.0 patched Δ
Macro-expanded size (-Zunpretty=expanded, lines) 84,975 67,069 −21.1%
…of which pyo3-generated (excl. ~25.7k user lines) ~59,300 ~41,300 −30%
Debug touch-rebuild of leaf, incremental (7 runs) 2.84 s 2.53 s −11.0%
Debug rebuild of leaf, non-incremental (4 runs) 10.31 s 9.59 s −6.9%
rustc total CPU for leaf (-Ztime-passes, non-incr.) 10.30 s 9.41 s −8.6%
Release rebuild of leaf, non-incremental (3 runs) 18.85 s 18.45 s −2.1%

Origin of these wins:

Category stock patched Δ
760 __pymethod_* wrappers 27,704 16,949 −39%
133 PyClassImpl impls 10,694 8,519 −20%
96 PyMethods items arrays 6,986 6,644 −5%
77 __pyfunction_* wrappers 4,808 2,595 −46%
146 PyTypeInfo impls 2,032 1,500 −26%
108 IntoPyObject impls 1,973 1,433 −27%

(Work performed with Claude)

@alex alex added the CI-skip-changelog Skip checking changelog entry label Jul 5, 2026
davidhewitt
davidhewitt reviewed Jul 5, 2026
View reviewed changes

@davidhewitt davidhewitt left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, generally always keen to improve codegen and I'm not surprised there were low-hanging fruit here. A few thoughts scattered below.

Comment thread pyo3-macros-backend/src/pyclass.rs Outdated
Comment thread pyo3-macros-backend/src/method.rs
Comment thread src/impl_/wrap.rs
Comment thread src/pyclass/create_type_object.rs Outdated
Comment thread tests/ui/invalid_pyclass_generic.rs Outdated
@mejrs

mejrs commented Jul 5, 2026

Copy link
Copy Markdown
Member

You probably need some type annotations to fix the never type fallback errors.

@codspeed-hq

codspeed-hq Bot commented Jul 5, 2026 •
edited
Loading

Copy link
Copy Markdown

Merging this PR will degrade performance by 24.75%

❌ 1 regressed benchmark
✅ 139 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
❌ into_u128_u64_max 1.5 µs 2 µs -24.75%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

Comparing alex:slim-codegen (3b2f821) with main (9e6697d)

Open in CodSpeed

claude and others added 6 commits July 5, 2026 13:22
@claude @alex
Reduce macro-generated code size
f2a7dba 
Slims the code generated by #[pyclass], #[pymethods], and #[pyfunction]
without changing runtime behavior:

- Fuse return-value conversion into single converter calls
  (wrap_into_ptr / wrap_into_pyobject)
- Fuse required-argument unwrap+extract (extract_required_argument,
  from_py_with_required) and TryFrom<&Bound<T>>-style receiver
  extraction (extract_receiver, extract_receiver_trusted)
- Fold the receiver cast into the trusted self-extraction unsafe block
- Declare argument holders with a single tuple pattern and drop the
  vestigial per-argument Probe import
- Emit PyClassImpl consts only when they differ from new trait defaults
- Replace per-class compile-time doc concatenation (DOC) with DOC_PIECES
  assembled once at type-object creation
- Outline type_object_raw into pyclass_type_object_raw helper
- Share an empty PyClassItems static; skip empty impl blocks and
  assertion consts; drop the HasAutomaticFromPyObject deprecation probe
- Shorten trampoline shim expansion; compact FunctionDescription::new
@alex @claude
Fix UI test failures from slimmed codegen
6d95dd5 
Three codegen bugs plus stale UI test snapshots:

- The fused `unsafe` extraction blocks (extract_required_argument,
  from_py_with_required, extract_receiver{,_trusted}) were emitted
  inside quote_spanned! with user-code spans, so the `unsafe` token
  was attributed to user code and rejected by #![forbid(unsafe_code)].
  Build the unsafe wrapper with call-site spans and interpolate the
  spanned call into it, as the previous codegen did.

- `fn into_pyobject(...) -> PyResult<Self::Output>` is ambiguous for
  enums with a variant named `Output`; spell out the concrete return
  type `PyResult<Bound<'py, Self>>` instead.

- Re-bless the UI test snapshots for the renamed extraction/conversion
  helpers, and update inline annotations:
  - invalid_pymethod_receiver now uses a local receiver type, because
    the fused receiver extraction produces a `From` candidate list for
    `i32` which varies with enabled features (jiff impls under `full`)
  - missing_intopy no longer emits the `map_err` error (the old
    ok_wrap + map_result_into_ptr codegen is gone)
  - invalid_pyclass_generic gains never_type_fallback diagnostics from
    the unsafe fused helpers in already-erroneous code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@alex @claude
Remove dead map_into_ptr/map_into_pyobject converters
9f1b471 
Nothing in the macro backend emits `map_into_ptr` / `map_into_pyobject`
since return-value conversion was fused into `wrap_into_ptr` /
`wrap_into_pyobject` (`quotes::map_result_into_ptr` is gone), so drop:

- EmptyTupleConverter<PyResult<()>>::{map_into_ptr, map_into_pyobject}
- IntoPyObjectConverter<Result<T, E>>::{map_into_ptr, map_into_pyobject}
- UnknownReturnType::{map_into_ptr, map_into_pyobject} (the
  unreachable! diagnostic anchors for the removed call shape)

The last reference was the example expansion in the `PyClassGuardMut`
docs, which still showed the pre-slimming codegen; update it to the
current expansion.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@alex @claude
Restore compile-time class doc concatenation
0968e9c 
Undo the move of class doc assembly from compile time to type-object
creation time: `PyClassImpl::DOC` is back as a required const rendered
at compile time via `impl_::concat`, `PyTypeBuilder::type_doc` takes
the `&'static CStr` again, and the `type_doc_storage` buffer is gone.

`RAW_DOC` keeps its `c""` default (part of the emit-only-non-default
consts slimming), which composes the same way through
`PyClassDocGenerator`.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@alex @claude
Make fused required-argument extraction safe
d0e7920 
`extract_required_argument` and `from_py_with_required` are now safe
functions which branch on the `Option` internally, with an
`unreachable!()` arm for `None` (the macro-generated caller always
provides `Some` for required arguments, previously expressed as an
unsafe contract using `unreachable_unchecked`).

This fixes noisy `never_type_fallback_flowing_into_unsafe` diagnostics
in already-erroneous user code: when a surrounding error (e.g. an
ambiguous method) leaves the extraction's type parameter unconstrained,
never-type fallback kicks in, and fallback flowing into an *unsafe*
call trips that deny-by-default rust-2024-compatibility lint. With a
safe function the fallback is harmless and the lint does not fire, so
the extra errors disappear from `invalid_pyclass_generic`.

This also removes the `unsafe` block from generated argument
extraction entirely (no more call-site span juggling for
`#![forbid(unsafe_code)]`), and with it the now-unused
`unwrap_required_argument` / `unwrap_required_argument_bound` helpers.

Cost: one always-predicted branch per required argument compared to
the `unreachable_unchecked` version.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@alex @claude
Re-bless UI tests after rebase onto main
3b2f821 
The removal of the `FromPyObject` blanket implementation (PyO3#6188)
changed the extraction diagnostics; these snapshots are the combined
output of that change and the slimmed codegen on this branch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@alex

alex commented Jul 20, 2026

Copy link
Copy Markdown
Member Author

@davidhewitt any blockers to landing this?

davidhewitt
davidhewitt approved these changes Jul 24, 2026
View reviewed changes

@davidhewitt davidhewitt left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the delay, no blockers other than me being busy over the past couple of weeks. Thanks for the cleanups!

@davidhewitt
davidhewitt added this pull request to the merge queue Jul 24, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@davidhewitt davidhewitt davidhewitt approved these changes

Assignees

No one assigned

Labels

CI-build-full CI-skip-changelog Skip checking changelog entry

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@alex @mejrs @davidhewitt @claude