3.14/msys: port MSYS2 0003 + 0020 (the sysconfig posix-mode pair) - #19
Merged
Merged
Conversation
tebako-runtime-python's first --enable-shared leg (run 34748169561) died at the pybuilddir.txt rule: ./python.exe -E -S -m sysconfig --generate-posix-vars Written build/lib.win32-3.14\_sysconfigdata__win32_.py KeyError: 'LIBPL' make: *** [Makefile:1179: pybuilddir.txt] Error 1 Root cause: upstream's os.name == 'nt' path (_init_non_posix) never merges the Makefile-parsed build_time_vars — the posix half (_init_posix) is the only one that carries the Makefile variable set (LIBPL, BINDIR, ...) into get_config_vars(), and _generate_posix_vars' post-write step indexes install_vars['LIBPL']. MSYS2's answer is 0020 "sysconfig: treat MINGW builds as POSIX builds" — every nt special case gains `and not sys._is_mingw`, the two posix checks gain `or sys._is_mingw` — riding on 0003's sys._is_mingw flag, which this series previously skipped for lack of a consumer (the flags are defined on every platform, 0 off-mingw). - sysmodule_mingw_flags_msys.patch: MSYS2 0003 verbatim. - sysconfig_posix_builds_msys.patch: MSYS2 0020 verbatim (beyond the LIBPL wall it also moves get_config_h_filename off PC/pyconfig.h — the MSVC config the nt branch was silently parsing — onto the build root's autotools pyconfig.h, and gives the mingw interpreter the posix install schemes that match the factory's image layout). - sysmodule_coredll_msys.patch: hunk 3 returns to its MSYS2-verbatim (0003-carrying) context — the diff body is byte-identical with upstream's 0034; the header note records the history. Verified: tools/apply + tools/lint (4/4 catalog versions) apply-clean; the applied tree diffs from the v0.3.3 tree in exactly two files (Python/sysmodule.c +36 = 0003; Lib/sysconfig/__init__.py 14 lines = 0020); the failing function itself — the patched _generate_posix_vars — driven under os.name='nt' + sys._is_mingw=1 against the real cross-configured Makefile/pyconfig.h completes with LIBPL/BINDIR/srcdir/EXT_SUFFIX all resolving and pybuilddir.txt written. rspec 103/0; validate_manifests OK.
ronaldtse
marked this pull request as ready for review
September 13, 2026 09:30
ronaldtse
pushed a commit
to tamatebako/tebako-runtime-python
that referenced
this pull request
Sep 13, 2026
v0.3.4 (tamatebako/python#19) ports MSYS2 0003 (sys._is_mingw flags) + 0020 (mingw builds take the posix sysconfig path) — the answer to the first --enable-shared leg's pybuilddir.txt wall (run 34748169561: KeyError 'LIBPL'; upstream's nt path never merges the Makefile-parsed build_time_vars). 0034's hunk 3 returns to MSYS2-verbatim context. Windows-msys asset verified on the release (sha256 d5628491ee94b21dce0c92ba6913342cd68c3f99e7a569428ab16a4278803052).
ronaldtse
added a commit
to tamatebako/tebako-runtime-python
that referenced
this pull request
Sep 13, 2026
* Return the windows-2022 leg on the source factory's scenario axis (TODO.python/05 item 2) The windows row comes back to the matrix (63e8336's descope), now buildable: tamatebako/python's platform-scenario axis ships the msys2/ucrt64 patch series as tfs-python-<base>-src-windows-msys.tar.gz assets (tamatebako/python#14). - .github/matrix.json: re-add the windows-2022 row, exactly as descoped. The windows-gated legs in _build-platform.yml (msys2 provisioning, build, symbol provenance, boot smoke) and publish.yml's windows job need no edits — they light up on the row's legs. - SourceFetcher: scenario-aware assets (the ruby factory's scenario_asset_names convention, minus ruby's msys two-pass split — CPython needs one tree per scenario). fetch/tarball_sha256 take an optional platform; a mingw host names the -windows-msys asset, POSIX keeps the unsuffixed back-compat asset. Builder threads @platform. - compute_matrix.rb: jit lines skip windows (upstream's JIT target whitelist admits MSVC windows only — the musl skip's sibling), and a windows leg exists only when the pinned source_release's SHA256SUMS carries the line's windows-msys asset — missing asset = loud skip, not a red leg. The SHA256SUMS read is lazy and scoped to a windows env row surviving the filters (POSIX triggers stay fetch-free), and an unreadable pin surfaces as a named exit-64 error, never a stack trace. - contract.yml: source_release v0.2.0 -> v0.3.0 (the release carrying the scenario axis). EXPECTED RED until tamatebako/python#14 merges and v0.3.0 is cut: the pinned SHA256SUMS 404s, and lint.yml's unfiltered compute_matrix fails named. Once v0.3.0 exists this PR goes green with no further edits (the #2 expected-red precedent). Verified locally: rspec 14/0 (new source_fetcher_spec covers the scenario grammar over a file:// mirror); check_contract green; POSIX matrices byte-identical to main; PLATFORM_FILTER=windows against a staged SHA256SUMS yields exactly the 3.14.7 leg with loud skips for the other five catalog lines, and the all-missing case yields an empty matrix with six loud skips; ruby -c sweep clean. * contract: consume source release v0.3.1 (the errmap.h msys include fix) v0.3.1 is the v0.3.0 catalog plus the MSYS2-MINGW-packages 0007 port (tamatebako/python#16): PC/errmap.h is checked-in source and upstream's Makefile grants -I$(srcdir)/PC to MSVC only; the windows-msys scenario tarball now compiles Objects/exceptions.o through. The version catalog is unchanged. * contract: consume source release v0.3.2 (the msys compile-wall sweep) v0.3.2 is the v0.3.1 catalog plus the MSYS2-MINGW-packages sweep for the static ucrt64 shape (tamatebako/python#17): 18 more patches ported — the observed round-2 walls (0054 -DVPATH on the sysmodule.o rule, 0001 --with-nt-threads default + the _POSIX_THREADS undefs) plus the series audit's compile/link/module-reach set (0005 mingw configure defaults — the alarm/dev-ptmx probe pins that walled the native ucrt64 smoke leg, 0008 nt module naming, 0016 frozenmain exe-link, 0017+0041 socketmodule + inet_pton probes, 0036 _signal -lws2_32, 0037 winconsoleio in the _io bootstrap line, 0045 ucrt _timezone spellings, 0056 -municode for _bootstrap_python, 0059 ssl/hashlib -l tails, 0060 winsock netdb probes, 0063 deterministic EXEEXT, 0075 POSIX-only modules n/a, 0078 -Wno-error=incompatible-pointer-types for gcc>=14, 0097 mmapmodule SEH, 0100 _PTHREAD_NAME_MAXLEN) and the syslibs deviation extended with 0010's -lversion -lshlwapi -lpathcch tail. The version catalog is unchanged. * contract: consume source release v0.3.3 (the mingw shared-build machinery) v0.3.3 is the v0.3.2 catalog plus the MSYS2-MINGW-packages shared-build port (tamatebako/python#18): 0009/0010 (declspec exports + the enable_shared mingw configure arms — LDLIBRARY= libpython3.14.dll.a / DLLLIBRARY=libpython3.14.dll), the dynload_win coherence cluster (0011 PYD_PLATFORM_TAG, 0012 dynload_win + PC/dl_nt.o, 0013 SHLIB_SUFFIX=.pyd, 0014 mingw SOABI/EXT_SUFFIX, 0015 MS_DLL_ID), the two interpreter boot walls the PR's own smoke leg forced (0018: winreg/msvcrt/_winapi as bootstrap-static modules — _bootstrap_external imports winreg at top level on win32; 0034: sysmodule's winver/dllhandle guard MS_COREDLL -> defined(MS_WINDOWS) && defined(Py_ENABLE_SHARED)), 0066 (the _Py_CheckPython3 declaration conflict), and 0039 (-municode exe link + the win resources + pythonw.exe). Observable rename on windows-msys: extension modules are .cp314-mingw_x86_64_ucrt_gnu.pyd. The version catalog is unchanged. * windows-msys: build --enable-shared + ship the runtime DLL (issue 40's answer) The v0.3.3 source release carries the MSYS2-series machinery (0009/0010 shared build, 0011-0015, 0018, 0034, 0039, 0066); the first trp leg against it (run 34745803168) failed exactly as predicted on the static shape: the stdlib extensions link as loadable PE modules (array.cp314-mingw_x86_64_ucrt_gnu.pyd, Makefile:3624) and PE modules cannot carry undefined symbols — the link died on _Py_Dealloc & co because --disable-shared gives them no libpython to resolve against. On PE the answer is structural, not a patch: build --enable-shared and ship the DLL. - configure: --enable-shared on windows-msys (POSIX keeps --disable-shared, the shipped single-exe shape). - substitute_makefiles: the exe link recipe substitution is now platform-shaped. The 0039 patch (win_resources_pythonw_msys) rewrites the rule to `...$(LINKFORSHARED) -municode -o $@ Programs/python.o ... $(SYSLIBS) python_exe.o` — the old POSIX regex never matched that line, so the prereq side rewrote python.o -> tebako_python.o while the recipe kept linking Programs/python.o: the driver link was SILENTLY lost (the fs TU compiled but never linked). The msys substitution strips -municode (the fs TU defines plain main; the mingw CRT demands wmain otherwise) and keeps python_exe.o (the version/icon resource). verify_substitutions now also requires the rewritten recipe line (-o $@ Programs/tebako_python.o) — the silent-loss class fails closed. - abi: EXT_SUFFIX regex accepts the mingw spelling (cp314-mingw_x86_64_ucrt_gnu.pyd) alongside the POSIX cpython-* one. - make: on msys pre-build the DLL target SERIALLY before the -j make. Upstream's `$(DLLLIBRARY) libpython$(LDVERSION).dll.a` two-target rule (-Wl,--out-implib=$@) is parallel-unsafe: observed locally, the interleaved import-library write replaced the 39 MB libpython3.14.dll with the 1.2 MB import stub. The serial pre-build materializes both files with the correct expansion; the parallel make then finds the rule up-to-date and never re-runs it. - finalize stages libpython<X.Y>.dll beside the runtime package as <package>.dll (unique per leg — two same-ABI legs share the PE name and would collide in the merged release workspace); a missing DLL is a named error (exit 130), never a silent static regression. The image layout card declares runtime_dll (additive, schema_minor 3, spec 22 §2.1); PythonVersion#msys_dll_name stays the name's single owner. tools/boot_smoke materializes the PE-named copy next to the in-leg exe before any scenario boots it (the OS loader resolves it before any driver code runs). - upload_release: expected_facets requires <package>.dll on windows — a leg that failed to stage it is an incomplete release, not an optional facet. (The dll manifest facet, install_as included, was already modeled; the comments claiming "nothing ships it today" are corrected.) - The mingw support set never reaches the user: LIBS="-static-libgcc -l:libwinpthread.a" static-links libgcc/libwinpthread INTO the runtime's PE modules (ruby factory's proven Mlibs::MSYS_DLL_LIBRARIES recipe; LIBS lands in the link tail, after the objects, so the archives resolve). Verified against a local cross build of the v0.3.3 tree: libpython3.14.dll (39,320,791 bytes) imports system DLLs only (KERNEL32/ADVAPI32/VERSION/WS2_32/bcrypt/api-ms-win-*) — libgcc_s_seh-1.dll and libwinpthread-1.dll are gone; the exe imports system + libpython3.14.dll only. A bare windows machine installs nothing (audience rule 1). - README's windows-boundary section describes the shared shape. Gates: rspec 14/0; check_contract.rb green; ruby -c sweep green. The recipe substitutions were cross-checked by driving the real exe_recipe_substitution against the real v0.3.3 Makefile.pre shapes (msys probe tree + POSIX smoke trees): each pair matches its own shape exactly once; the msys result line verified char-exact. trp#16 stays draft — no merge without the owner's go-ahead. * contract: consume source release v0.3.4 (the sysconfig posix-mode pair) v0.3.4 (tamatebako/python#19) ports MSYS2 0003 (sys._is_mingw flags) + 0020 (mingw builds take the posix sysconfig path) — the answer to the first --enable-shared leg's pybuilddir.txt wall (run 34748169561: KeyError 'LIBPL'; upstream's nt path never merges the Makefile-parsed build_time_vars). 0034's hunk 3 returns to MSYS2-verbatim context. Windows-msys asset verified on the release (sha256 d5628491ee94b21dce0c92ba6913342cd68c3f99e7a569428ab16a4278803052). * contract: consume source release v0.3.5 (the module-libs cluster) v0.3.5 (tamatebako/python#20) ports MSYS2 0061 + 0073 (_multiprocessing: the win32 enable condition, then -lws2_32), 0074 (select: _MSC_VER → MS_WIN32 guards + -lws2_32), and 0076 (_ctypes: -lole32 -loleaut32 -luuid) — the answer to the shared-extension link wall the v0.3.4 leg hit (run 34750005177: undefined __WSAFDIsSet / __imp_select / __imp_WSAGetLastError at the select.pyd and _multiprocessing.pyd links; PE resolves every symbol at link time and upstream's PY_STDLIB_MOD stanzas carry no per-module libs — MSVC gets them via PCbuild). Windows-msys asset verified on the release (sha256 18f20a10922e55da3ce2213769cccf2db4a732b8898e2c128173051ed7b1f8b3; the POSIX assets are byte-identical with v0.3.4's). * contract: consume source release v0.3.6 (the mmap + legacy-DLL trio) v0.3.6 (tamatebako/python#21) ports MSYS2 0062 (mmap builds on win32 — its absence killed ensurepip in `make install` on the v0.3.5 leg, run 34753278108: pip's vendored cachecontrol imports mmap), 0067 (PYTHONLEGACYWINDOWSDLLLOADING — the bpo-36085 DEFAULT_DIRS search never consults PATH, so the build-tree check removed _decimal/_zstd/_testinternalcapi whose dep DLLs live in <ucrt64>/bin), and 0064 (have_dynamic_loading for the dlopen-gated test modules); 0076 returns to MSYS2-verbatim context. Windows-msys asset verified on the release (sha256 20ec675d03bb3612e4c69026fa723d97cdd323a2faf38a22bee32105236f1c23; the POSIX assets are byte-identical with v0.3.5's). * windows-msys: install with prefix=/A/t + stage pip explicitly (round-6) CPython's install rules spell every destination $(DESTDIR)$(dir) with no separator; with the baked drive-qualified prefix (A:/t) that concatenates into one junk component (stage + "A:/t/..." — the failing recipes are mostly `-`-prefixed, so make exited green over the wreckage and staged_prefix_tree's "no A/t tree" was the first truth, exit 105 on run 34756006093). The msys install now overrides prefix to the POSIX spelling /A/t — every install dir var derives unexpanded from $(prefix) (autoconf convention), so the override flows to every rule — and skips ensurepip: pip's --root rebase drive-strips the built python's BUILD-TREE prefix, never the staged one, so its pip landed under <stage>/<build-tree path>. place_pip stages pip explicitly instead: the bundled wheel's own installer run into the staged lib/pythonX.Y/site-packages (the series' 0026 site.py branch puts the runtime's site-packages there — verified against the applied tree), mirroring ensurepip's -c invocation but with --target deciding placement. The paths ride argv so the msys runtime translates them for the native exe (a path embedded in the -c payload crosses verbatim), and --no-compile mirrors ensurepip: .pyc records written here would spell the stage paths — dead links in the mounted image. POSIX legs keep the old recipe exactly (ensurepip still rides the install). Recorded follow-up: whether ensurepip's --root placement ever landed pip in the POSIX staged trees either (the build-tree prefix rebase exists there too) — checked separately, not in this commit. * ci: spell the provenance exe with .exe on the windows leg (round-7) msys stat() transparency lets foo.exe satisfy `[ -s foo ]`, so the step's extensionless-or-.exe dance always kept the extensionless spelling; the native nm/objdump receive the argument verbatim, get ENOENT, and the script's discarded stderr read back as a stripped exe (five failed provenance checks on run 34758424181 — the exe itself was fine: the same build's place_pip had just run it). The name carries .exe on this leg by construction (Platform#exe_suffix) — spell it out, the ruby factory's ruby.exe spelling. The script now also prints nm's own diagnostic when the symbol-table capture comes back empty, so a silent nm failure never reads as "stripped" again. * ci: capture-then-test the disassembly greps (the SIGPIPE false negative) grep -q exits on its first match; under pipefail the still-writing printf then dies of SIGPIPE and the pipeline reads 141 — a false "not found". POSIX legs never noticed: their objdump honors --disassemble=main tightly (a screenful, under the pipe buffer). A binutils PE objdump's span runs from the symbol to the next data island — 4,700 lines on the windows leg — so grep matched early, printf was killed, and run 34759975017 failed the forwarding check while its own evidence printout showed main calling tebako_driver_boot at 140da1789. Same discipline as the nm side already documented in this script: capture first, then test. * windows-msys: bare boot pins driver + getpath warnings, not exit status (round-9) Run 34760958198's windows leg failed the bare scenario on its POSIX assertions: the child printed 'unreached' and exited 0, with only the driver warning and getpath's "Could not find platform independent libraries <prefix>" on stderr. Mechanism, established against the 3.14.7 sources: the encodings import at startup is unconditional (pylifecycle.c _PyUnicode_InitEncodings -> unicodeobject.c _PyCodec_InitRegistry, fatal on failure), so a child that ran user code necessarily FOUND a stdlib. The channel is upstream: Modules/getpath.py's winreg section (os_name == 'nt' and use_environment and winreg) appends the HKCU/HKLM SOFTWARE\Python\PythonCore\<ver>\PythonPath registry values to sys.path — and the GHA runner has a registered host python. Bare boot on windows is therefore environment-dependent by upstream design: exit status and the encodings fatal are not assertable there. The msys bare scenario now pins the deterministic surface instead: the driver's bare-boot warning (ours, every platform) plus getpath's platstdlib warning (the prefix fell back to the baked-in mount root, which never exists on the runner). POSIX keeps the strict trio (non-zero exit + driver warning + encodings fatal) — no registry fallback exists there. --------- Co-authored-by: tebako-ci <tebako@ribose.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Ports the MSYS2 sysconfig posix-mode pair into the 3.14 windows-msys series:
sysmodule_mingw_flags_msys.patch— MSYS20003verbatim: thesys._is_mingw/_is_mingw_ucrt/_use_alt_sepflags. Previously a deliberate skip (no consumer); 0020 consumessys._is_mingw. The flags are defined on every platform (0off-mingw), so consumers never seeAttributeError.sysconfig_posix_builds_msys.patch— MSYS20020verbatim: everyos.name == 'nt'special case inLib/sysconfig/__init__.pygainsand not sys._is_mingw; the two posix checks gainor sys._is_mingw.sysmodule_coredll_msys.patch— hunk 3 returns to its MSYS2-verbatim (0003-carrying) context: the diff body is now byte-identical with upstream's0034(verified bydiffagainst the fetched upstream patch). Header note updated.patch-3.14.yaml: both entries inserted at their MSYS2-number positions (0003 after 0002's pyport patch; 0020 after 0018 winreg); header comment carries the wall.Why — the wall (root-caused against the MSYS2 reference)
tebako-runtime-python run 34748169561 (the first
--enable-sharedleg, windows 3.14.7): the build linked the exe +libpython3.14.dll+ every stdlib extension cleanly, then died at thepybuilddir.txtrule:Upstream
get_config_vars()onos.name == 'nt'calls_init_non_posix, which never merges the Makefile-parsedbuild_time_vars— the Makefile variable set (LIBPL,BINDIR, …) exists only on the_init_posixpath, and_generate_posix_vars' post-write step indexesinstall_vars['LIBPL'].0020is MSYS2's answer to exactly this class. Two bonus corrections ride along:get_config_h_filenamestops readingPC/pyconfig.h(the MSVC config the nt branch was silently parsing) in favor of the build root's autotoolspyconfig.h, and the mingw interpreter gets the posix install schemes — the same layout the factory bakes into the env image.Verification
tools/apply 3.14.7 … --platform windows-msys+tools/lint— all 4 catalog versions apply-clean.Python/sysmodule.c+36/−0 (= 0003 alone; the 0034 transformation is unchanged) andLib/sysconfig/__init__.py14 lines (= 0020's 7 hunks).sysconfig.__main__._generate_posix_varsdriven underos.name='nt'+sys._is_mingw=1against the real cross-configuredMakefile/pyconfig.hcompletes —LIBPL/BINDIR/srcdir/EXT_SUFFIX(.cp314-mingw_x86_64_ucrt_gnu.pyd) all resolve,_sysconfigdata+pybuilddir.txtwritten. (Host-side simulation; the real PE exe can't run on macOS.)bundle exec rspec: 103/0.tools/validate_manifests: OK.Chain after merge
Tag
v0.3.4→release-src→ tebako-runtime-pythoncontract.ymlbump (source_release: v0.3.4) onfeat/windows-ucrt64-return→ the windows leg re-runs. Draft until the owner's go-ahead.