Return the windows-2022 leg on the source factory's scenario axis - #16
Merged
Merged
Conversation
…DO.python/05 item 2) The windows row comes back to the matrix (63e8336's descope), now buildable: tamatebako/python's platform-scenario axis ships the msys2/ucrt64 patch series as tfs-python-<base>-src-windows-msys.tar.gz assets (tamatebako/python#14). - .github/matrix.json: re-add the windows-2022 row, exactly as descoped. The windows-gated legs in _build-platform.yml (msys2 provisioning, build, symbol provenance, boot smoke) and publish.yml's windows job need no edits — they light up on the row's legs. - SourceFetcher: scenario-aware assets (the ruby factory's scenario_asset_names convention, minus ruby's msys two-pass split — CPython needs one tree per scenario). fetch/tarball_sha256 take an optional platform; a mingw host names the -windows-msys asset, POSIX keeps the unsuffixed back-compat asset. Builder threads @platform. - compute_matrix.rb: jit lines skip windows (upstream's JIT target whitelist admits MSVC windows only — the musl skip's sibling), and a windows leg exists only when the pinned source_release's SHA256SUMS carries the line's windows-msys asset — missing asset = loud skip, not a red leg. The SHA256SUMS read is lazy and scoped to a windows env row surviving the filters (POSIX triggers stay fetch-free), and an unreadable pin surfaces as a named exit-64 error, never a stack trace. - contract.yml: source_release v0.2.0 -> v0.3.0 (the release carrying the scenario axis). EXPECTED RED until tamatebako/python#14 merges and v0.3.0 is cut: the pinned SHA256SUMS 404s, and lint.yml's unfiltered compute_matrix fails named. Once v0.3.0 exists this PR goes green with no further edits (the #2 expected-red precedent). Verified locally: rspec 14/0 (new source_fetcher_spec covers the scenario grammar over a file:// mirror); check_contract green; POSIX matrices byte-identical to main; PLATFORM_FILTER=windows against a staged SHA256SUMS yields exactly the 3.14.7 leg with loud skips for the other five catalog lines, and the all-missing case yields an empty matrix with six loud skips; ruby -c sweep clean.
v0.3.1 is the v0.3.0 catalog plus the MSYS2-MINGW-packages 0007 port (tamatebako/python#16): PC/errmap.h is checked-in source and upstream's Makefile grants -I$(srcdir)/PC to MSVC only; the windows-msys scenario tarball now compiles Objects/exceptions.o through. The version catalog is unchanged.
v0.3.2 is the v0.3.1 catalog plus the MSYS2-MINGW-packages sweep for the static ucrt64 shape (tamatebako/python#17): 18 more patches ported — the observed round-2 walls (0054 -DVPATH on the sysmodule.o rule, 0001 --with-nt-threads default + the _POSIX_THREADS undefs) plus the series audit's compile/link/module-reach set (0005 mingw configure defaults — the alarm/dev-ptmx probe pins that walled the native ucrt64 smoke leg, 0008 nt module naming, 0016 frozenmain exe-link, 0017+0041 socketmodule + inet_pton probes, 0036 _signal -lws2_32, 0037 winconsoleio in the _io bootstrap line, 0045 ucrt _timezone spellings, 0056 -municode for _bootstrap_python, 0059 ssl/hashlib -l tails, 0060 winsock netdb probes, 0063 deterministic EXEEXT, 0075 POSIX-only modules n/a, 0078 -Wno-error=incompatible-pointer-types for gcc>=14, 0097 mmapmodule SEH, 0100 _PTHREAD_NAME_MAXLEN) and the syslibs deviation extended with 0010's -lversion -lshlwapi -lpathcch tail. The version catalog is unchanged.
…nery) v0.3.3 is the v0.3.2 catalog plus the MSYS2-MINGW-packages shared-build port (tamatebako/python#18): 0009/0010 (declspec exports + the enable_shared mingw configure arms — LDLIBRARY= libpython3.14.dll.a / DLLLIBRARY=libpython3.14.dll), the dynload_win coherence cluster (0011 PYD_PLATFORM_TAG, 0012 dynload_win + PC/dl_nt.o, 0013 SHLIB_SUFFIX=.pyd, 0014 mingw SOABI/EXT_SUFFIX, 0015 MS_DLL_ID), the two interpreter boot walls the PR's own smoke leg forced (0018: winreg/msvcrt/_winapi as bootstrap-static modules — _bootstrap_external imports winreg at top level on win32; 0034: sysmodule's winver/dllhandle guard MS_COREDLL -> defined(MS_WINDOWS) && defined(Py_ENABLE_SHARED)), 0066 (the _Py_CheckPython3 declaration conflict), and 0039 (-municode exe link + the win resources + pythonw.exe). Observable rename on windows-msys: extension modules are .cp314-mingw_x86_64_ucrt_gnu.pyd. The version catalog is unchanged.
…s answer) The v0.3.3 source release carries the MSYS2-series machinery (0009/0010 shared build, 0011-0015, 0018, 0034, 0039, 0066); the first trp leg against it (run 34745803168) failed exactly as predicted on the static shape: the stdlib extensions link as loadable PE modules (array.cp314-mingw_x86_64_ucrt_gnu.pyd, Makefile:3624) and PE modules cannot carry undefined symbols — the link died on _Py_Dealloc & co because --disable-shared gives them no libpython to resolve against. On PE the answer is structural, not a patch: build --enable-shared and ship the DLL. - configure: --enable-shared on windows-msys (POSIX keeps --disable-shared, the shipped single-exe shape). - substitute_makefiles: the exe link recipe substitution is now platform-shaped. The 0039 patch (win_resources_pythonw_msys) rewrites the rule to `...$(LINKFORSHARED) -municode -o $@ Programs/python.o ... $(SYSLIBS) python_exe.o` — the old POSIX regex never matched that line, so the prereq side rewrote python.o -> tebako_python.o while the recipe kept linking Programs/python.o: the driver link was SILENTLY lost (the fs TU compiled but never linked). The msys substitution strips -municode (the fs TU defines plain main; the mingw CRT demands wmain otherwise) and keeps python_exe.o (the version/icon resource). verify_substitutions now also requires the rewritten recipe line (-o $@ Programs/tebako_python.o) — the silent-loss class fails closed. - abi: EXT_SUFFIX regex accepts the mingw spelling (cp314-mingw_x86_64_ucrt_gnu.pyd) alongside the POSIX cpython-* one. - make: on msys pre-build the DLL target SERIALLY before the -j make. Upstream's `$(DLLLIBRARY) libpython$(LDVERSION).dll.a` two-target rule (-Wl,--out-implib=$@) is parallel-unsafe: observed locally, the interleaved import-library write replaced the 39 MB libpython3.14.dll with the 1.2 MB import stub. The serial pre-build materializes both files with the correct expansion; the parallel make then finds the rule up-to-date and never re-runs it. - finalize stages libpython<X.Y>.dll beside the runtime package as <package>.dll (unique per leg — two same-ABI legs share the PE name and would collide in the merged release workspace); a missing DLL is a named error (exit 130), never a silent static regression. The image layout card declares runtime_dll (additive, schema_minor 3, spec 22 §2.1); PythonVersion#msys_dll_name stays the name's single owner. tools/boot_smoke materializes the PE-named copy next to the in-leg exe before any scenario boots it (the OS loader resolves it before any driver code runs). - upload_release: expected_facets requires <package>.dll on windows — a leg that failed to stage it is an incomplete release, not an optional facet. (The dll manifest facet, install_as included, was already modeled; the comments claiming "nothing ships it today" are corrected.) - The mingw support set never reaches the user: LIBS="-static-libgcc -l:libwinpthread.a" static-links libgcc/libwinpthread INTO the runtime's PE modules (ruby factory's proven Mlibs::MSYS_DLL_LIBRARIES recipe; LIBS lands in the link tail, after the objects, so the archives resolve). Verified against a local cross build of the v0.3.3 tree: libpython3.14.dll (39,320,791 bytes) imports system DLLs only (KERNEL32/ADVAPI32/VERSION/WS2_32/bcrypt/api-ms-win-*) — libgcc_s_seh-1.dll and libwinpthread-1.dll are gone; the exe imports system + libpython3.14.dll only. A bare windows machine installs nothing (audience rule 1). - README's windows-boundary section describes the shared shape. Gates: rspec 14/0; check_contract.rb green; ruby -c sweep green. The recipe substitutions were cross-checked by driving the real exe_recipe_substitution against the real v0.3.3 Makefile.pre shapes (msys probe tree + POSIX smoke trees): each pair matches its own shape exactly once; the msys result line verified char-exact. trp#16 stays draft — no merge without the owner's go-ahead.
v0.3.4 (tamatebako/python#19) ports MSYS2 0003 (sys._is_mingw flags) + 0020 (mingw builds take the posix sysconfig path) — the answer to the first --enable-shared leg's pybuilddir.txt wall (run 34748169561: KeyError 'LIBPL'; upstream's nt path never merges the Makefile-parsed build_time_vars). 0034's hunk 3 returns to MSYS2-verbatim context. Windows-msys asset verified on the release (sha256 d5628491ee94b21dce0c92ba6913342cd68c3f99e7a569428ab16a4278803052).
ronaldtse
changed the title
v0.3.5 (tamatebako/python#20) ports MSYS2 0061 + 0073 (_multiprocessing: the win32 enable condition, then -lws2_32), 0074 (select: _MSC_VER → MS_WIN32 guards + -lws2_32), and 0076 (_ctypes: -lole32 -loleaut32 -luuid) — the answer to the shared-extension link wall the v0.3.4 leg hit (run 34750005177: undefined __WSAFDIsSet / __imp_select / __imp_WSAGetLastError at the select.pyd and _multiprocessing.pyd links; PE resolves every symbol at link time and upstream's PY_STDLIB_MOD stanzas carry no per-module libs — MSVC gets them via PCbuild). Windows-msys asset verified on the release (sha256 18f20a10922e55da3ce2213769cccf2db4a732b8898e2c128173051ed7b1f8b3; the POSIX assets are byte-identical with v0.3.4's).
v0.3.6 (tamatebako/python#21) ports MSYS2 0062 (mmap builds on win32 — its absence killed ensurepip in `make install` on the v0.3.5 leg, run 34753278108: pip's vendored cachecontrol imports mmap), 0067 (PYTHONLEGACYWINDOWSDLLLOADING — the bpo-36085 DEFAULT_DIRS search never consults PATH, so the build-tree check removed _decimal/_zstd/_testinternalcapi whose dep DLLs live in <ucrt64>/bin), and 0064 (have_dynamic_loading for the dlopen-gated test modules); 0076 returns to MSYS2-verbatim context. Windows-msys asset verified on the release (sha256 20ec675d03bb3612e4c69026fa723d97cdd323a2faf38a22bee32105236f1c23; the POSIX assets are byte-identical with v0.3.5's).
CPython's install rules spell every destination $(DESTDIR)$(dir) with no separator; with the baked drive-qualified prefix (A:/t) that concatenates into one junk component (stage + "A:/t/..." — the failing recipes are mostly `-`-prefixed, so make exited green over the wreckage and staged_prefix_tree's "no A/t tree" was the first truth, exit 105 on run 34756006093). The msys install now overrides prefix to the POSIX spelling /A/t — every install dir var derives unexpanded from $(prefix) (autoconf convention), so the override flows to every rule — and skips ensurepip: pip's --root rebase drive-strips the built python's BUILD-TREE prefix, never the staged one, so its pip landed under <stage>/<build-tree path>. place_pip stages pip explicitly instead: the bundled wheel's own installer run into the staged lib/pythonX.Y/site-packages (the series' 0026 site.py branch puts the runtime's site-packages there — verified against the applied tree), mirroring ensurepip's -c invocation but with --target deciding placement. The paths ride argv so the msys runtime translates them for the native exe (a path embedded in the -c payload crosses verbatim), and --no-compile mirrors ensurepip: .pyc records written here would spell the stage paths — dead links in the mounted image. POSIX legs keep the old recipe exactly (ensurepip still rides the install). Recorded follow-up: whether ensurepip's --root placement ever landed pip in the POSIX staged trees either (the build-tree prefix rebase exists there too) — checked separately, not in this commit.
msys stat() transparency lets foo.exe satisfy `[ -s foo ]`, so the step's extensionless-or-.exe dance always kept the extensionless spelling; the native nm/objdump receive the argument verbatim, get ENOENT, and the script's discarded stderr read back as a stripped exe (five failed provenance checks on run 34758424181 — the exe itself was fine: the same build's place_pip had just run it). The name carries .exe on this leg by construction (Platform#exe_suffix) — spell it out, the ruby factory's ruby.exe spelling. The script now also prints nm's own diagnostic when the symbol-table capture comes back empty, so a silent nm failure never reads as "stripped" again.
grep -q exits on its first match; under pipefail the still-writing printf then dies of SIGPIPE and the pipeline reads 141 — a false "not found". POSIX legs never noticed: their objdump honors --disassemble=main tightly (a screenful, under the pipe buffer). A binutils PE objdump's span runs from the symbol to the next data island — 4,700 lines on the windows leg — so grep matched early, printf was killed, and run 34759975017 failed the forwarding check while its own evidence printout showed main calling tebako_driver_boot at 140da1789. Same discipline as the nm side already documented in this script: capture first, then test.
…us (round-9) Run 34760958198's windows leg failed the bare scenario on its POSIX assertions: the child printed 'unreached' and exited 0, with only the driver warning and getpath's "Could not find platform independent libraries <prefix>" on stderr. Mechanism, established against the 3.14.7 sources: the encodings import at startup is unconditional (pylifecycle.c _PyUnicode_InitEncodings -> unicodeobject.c _PyCodec_InitRegistry, fatal on failure), so a child that ran user code necessarily FOUND a stdlib. The channel is upstream: Modules/getpath.py's winreg section (os_name == 'nt' and use_environment and winreg) appends the HKCU/HKLM SOFTWARE\Python\PythonCore\<ver>\PythonPath registry values to sys.path — and the GHA runner has a registered host python. Bare boot on windows is therefore environment-dependent by upstream design: exit status and the encodings fatal are not assertable there. The msys bare scenario now pins the deterministic surface instead: the driver's bare-boot warning (ours, every platform) plus getpath's platstdlib warning (the prefix fell back to the baked-in mount root, which never exists on the runner). POSIX keeps the strict trio (non-zero exit + driver warning + encodings fatal) — no registry fallback exists there.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Return the windows-2022 leg on the source factory's scenario axis
Draft — do not merge. Expected red until the source release exists. Stacked on tamatebako/python#14 (the platform-scenario axis + the 3.14 msys2/ucrt64 patch series). This PR bumps
source_releaseto v0.3.0, which does not exist yet: lint.yml's unfilteredcompute_matrix.rbfails named (exit 64, "cannot read the pinned source release's SHA256SUMS: 404 …") until the owner merges tamatebako/python#14 and cuts tamatebako/python v0.3.0 — then this PR goes green with no further edits (the #2 expected-red precedent).What changes
.github/matrix.json— the windows-2022 row returns, byte-identical to the one 63e8336 descoped. The windows-gated steps in_build-platform.yml(msys2/ucrt64 provisioning, build, symbol provenance, boot smoke) and publish.yml's windows job were kept when the row left — they light up on their own; no workflow edits here.SourceFetcher— scenario-aware assets, the ruby factory'sscenario_asset_namesconvention minus ruby's msys two-pass split (CPython needs one tree per scenario):scenario_asset_name(version, platform)namestfs-python-<v>-src-windows-msys.tar.gzon a mingw host, the unsuffixed back-compat asset elsewhere;fetch/tarball_sha256takeplatform:;Builderthreads its@platform.scripts/compute_matrix.rb— two loud skips:jit × windows: upstream's JIT target whitelist (Tools/jit/_targets.py) admits MSVC windows targets only; the ucrt64x86_64-w64-mingw32triple is rejected — thejit × linux-muslskip's sibling.windows × <line without the asset>: a windows leg exists only when the pinnedsource_release's SHA256SUMS carries the line's-windows-msysasset. 3.14.7 ships one from v0.3.0; 3.11/3.12/3.13 follow in later source PRs. A missing asset is a loud note + skip, never a red leg. The SHA256SUMS read is lazy and scoped to a windows env row surviving the filters — POSIX-only triggers stay fetch-free (POSIX matrices verified byte-identical to main).contract.yml—source_release: v0.2.0 → v0.3.0.AGENTS.md— the descope note now documents the scenario-gated return.Why the windows legs can't fail closed forever
Without the SHA256SUMS gate, every catalog line's windows leg would 404 on its scenario asset the moment the row returns. With it, the matrix emits exactly the buildable windows legs (today: 3.14.7 only) and
--format pythonsderives the publish gate's expectation from the legs — a line never lands on the release expectation without a leg (the v0.1.3 musl-incompleteness fix's semantics, extended to the scenario axis).Verification (this host; the windows leg itself is CI's oracle)
rspec: 14 examples, 0 failures — newspec/source_fetcher_spec.rbcovers the scenario grammar end-to-end over a stagedfile://mirror (no network): POSIX/unsuffixed, mingw/windows-msys, sha parity, and the named error when a line's scenario asset is absent.scripts/check_contract.rb: green.PLATFORM_FILTER=linux-gnu|linux-musl|macosmatrices byte-identical to main (12/8/12 legs) — no fetch on POSIX triggers.PLATFORM_FILTER=windowsagainst a staged.build/downloads/v0.3.0/SHA256SUMS: exactly one leg (3.14.7 on windows-2022,host_id=windows-ucrt64,link_unit_pid=x86_64-windows-gnu), loud skip notes for 3.11.16/3.12.14/3.13.15 (missing asset) and both jit lines (MSVC-only whitelist);--format pythons=["3.14.7"]. With no windows assets staged: empty matrix, six loud notes, exit 0.compute_matrix: cannot read the pinned source release's SHA256SUMS: 404 …, exit 64 — no stack trace.ruby -csweep (lint.yml's exact file set): clean.Merge order (owner)
Follow-ups (not this PR): boot_smoke windows scenarios beyond the bare/named-error trio, and the packed-mn windows flip.