Repository navigation
Evaluate using Profile-Guided Optimization (PGO) and Post-Link Optimization (PLO) #349
Description
Describe the Bug
Hi!
Another optimization idea, in addition to the already completed #214
Recently I checked Profile-Guided Optimization (PGO) improvements on multiple projects (including many LSP servers like Clangd and Rust Analyzer) - the results are available in the awesome-pgo repo. Since PGO has helped in many similar to pyrefly cases, I think it would be a good idea to try optimizing pyrefly by applying PGO to it.
I can suggest the following things to do:
- We need to perform PGO benchmarks on
pyrefly. If it shows improvements - add a note about possible improvements to the documentation. Providing an easier way (e.g. a build option) to build scripts to buildpyreflywith PGO can be useful for the end-users too. - It will be a good idea to integrate building with PGO into your CI pipelines. In this case, users will get already PGO-optimized binaries from the website
Here you can find different materials about PGO: benchmarks for multiple apps, examples of how PGO is already integrated with various projects, PGO support in multiple Rust compilers, and some PGO-related pieces of advice. For Rust-based applications, I recommend starting with cargo-pgo.
After PGO, I suggest evaluating the Post-Link Optimization (with tools like LLVM BOLT) - it can give more aggressive optimizations even after PGO. However, starting with regular PGO will be easier to do. Since LLVM BOLT is mainly developed by Facebook/Meta - probably you are a bit more motivated to give it a try ;)
Thank you.
Sandbox Link
N/A
(Only applicable for extension issues) IDE Information
N/A
Activity
ndmitchell commentedon May 29, 2025
Thanks for the suggestions - we'll take a look in a bit, when the CI pipeline isn't changing as much. I'd expect 5-15% improvement from these changes.
zamazan4ik commentedon May 23, 2026
After a quick discussion on Reddit under the Pyrefly post, I decided to run PGO benchmarks for the project (especially since there is a comment that PGO doesn't work well on the project). So here are my findings.
Test environment:
92413ec83a599d50cd2aa1dfd8e94a22ba8a56a9commin from themainbranchI haven't disabled Turbo Boost (tbh, I just remembered it after finishing the benches - but it doesn't influence the results much) and haven't used CPU pinning via
taskset -c <CPU_NUM>since I don't know the optimal number of cores/threads. I didn't to make the test single-threaded either: even if it can improve the CPU scheduler noise, it can influence measuring multithreaded cases. Anyway, if we can easily reproduce the tests with additional more noise-proven config - if we decide to do so.For PGO routines, cargo-pgo was used:
cargo build --release --manifest-path pyrefly/Cargo.toml -p pyreflycargo pgo optimize build -- --release --manifest-path pyrefly/Cargo.toml -p pyreflycargo pgo build -- --release --manifest-path pyrefly/Cargo.toml -p pyreflyAs a training workload and evaluation benchmark, I've used official benches from the repository. As for now, I've used only
typecheckerbenches - no LSP benchmarks yet. Training was done with thepython3 typecheck_benchmark.py -c pyrefly_instrumented -w 0 -r 1command, comparing non-PGO vs PGOpyreflyversions was done withpython3 typecheck_benchmark.py -c pyrefly_release pyrefly_optimized(just hacked a little bit the scripts to run proper custom Pyrefly binaries). I think such a training set is large enough to be a good training and validation set.Here are the results (last lines of the benchmark script):
According to the benchmark, I definitely see CPU performance improvements by applying PGO to the project. I think integrating PGO into build pipelines (as it was done for kinda similar project from other domains like Rust Analyzer) would be a valuable option for Pyrefly users.
I think we can perform more tests (like applying BOLT as an additional step after PGO) a bit later (but no need to close the task!).
Attached raw benchmark data files for history and raw processing (if needed):
benchmark_2026-05-23_release_and_optimized.json
benchmark_2026-05-23_instrumented.json
samwgoldman commentedon May 24, 2026
Very cool, thank you for trying it out! I think this confirms that I was holding it wrong in my attempt. I will try to replicate your results.
samwgoldman commentedon Nov 7, 2026
Done in c7a8e3a