Support numba 0.68

[Goykhman]

No description provided.

[nelson2005]

From the fake Slim Shady:

pyproject.toml:12

On Windows this lets in the cell that fails in CI. numba 0.68 builds the archive member name with str(Path(...)), so there it carries backslashes and no member matches, source archives included. That one is numba's, reported as numba/numba#10889 with a fix in numba/numba#10896, but the fix only changes the separator, and a .pyc-only archive still has no .py member to hash.

[nelson2005]

From the fake Slim Shady:

test/core/test_no_cache_location.py:63-64

This hashing is what makes a .pyc-only archive crash at import on 0.68.0, and writing sources here is the only place the change answers it. numba 0.68's get_source_stamp opens the archived .py, a .pyc-only archive has none, and check_cache_location calls it outside its try, so the KeyError goes past the except at line 319 that turns a cache error into the fallback. On 0.67.0 the same archive imports and caches. That layout is the one compileall -d produces and the one the moved-archive remedy tells people to build, so following that remedy now fails the same way. Catching KeyError beside the OSError and treating it as a cache error, with the remedy pointing at source files on disk, would cover it on 0.68.0 as released.

import os, py_compile, subprocess, sys, tempfile, zipfile
from pathlib import Path
import numbox
pkg = Path(numbox.__file__).parent
tmp = Path(tempfile.mkdtemp())
archive = tmp / "numbox.zip"
with zipfile.ZipFile(archive, "w") as z:
    for src in sorted(pkg.rglob("*.py")):
        member = str(src.relative_to(pkg.parent))
        if src.parent == pkg / "core" / "bindings":
            pyc = py_compile.compile(str(src), cfile=str(tmp / "x.pyc"), dfile=os.path.join(str(archive), member), doraise=True)
            z.write(pyc, member + "c")
        else:
            z.write(src, member)
env = dict(os.environ, PYTHONPATH=str(archive), HOME=str(tmp), XDG_CACHE_HOME=str(tmp / "cache"))
env.pop("NUMBA_CACHE_DIR", None)
code = "import numbox.core.configurations, numbox.core.bindings.libm as m; print(m.__file__)"
sys.exit(subprocess.run([sys.executable, "-W", "always", "-c", code], env=env, cwd=str(tmp)).returncode)

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/x/numbox.zip/numbox/core/configurations.py", line 335, in <module>
  File "/tmp/x/numbox.zip/numbox/core/configurations.py", line 318, in uncached_where_no_cache_can_be_written
  File "/tmp/x/numbox.zip/numbox/core/configurations.py", line 76, in check_cache_location
  File "numba/core/caching.py", line 388, in get_source_stamp
    return _hash_zipped_source_file(self._zip_path, self._internal_path,
  File "numba/core/caching.py", line 175, in _hash_zipped_source_file
    with zf.open(internal_path) as f:
  File "/usr/lib/python3.12/zipfile/__init__.py", line 1617, in open
    zinfo = self.getinfo(name)
  File "/usr/lib/python3.12/zipfile/__init__.py", line 1545, in getinfo
    raise KeyError(
KeyError: "There is no item named 'numbox/core/bindings/__init__.py' in the archive"

[Goykhman]

Re-raised a more detailed error here.
EDIT: fixed here.

[nelson2005]

From the fake Slim Shady:

Verified at 843a85f on numba 0.68.0: the .pyc-only archive imports now and the warning names the missing sources. Two more places hit the same KeyError, and both scripts below run clean on 0.67.0.

An archive directory mixing .py members with one .pyc-only module still fails when that module is imported. _archived_module_files checks one member per directory. One member was enough while numba stat'ed the archive, but 0.68 hashes each module's own .py. So a .pyc-only member beside .py ones is never checked. Then numba's FunctionCache raises at the decorator, outside any try. The script from the report above, with only libm.py compiled:

import os, py_compile, subprocess, sys, tempfile, zipfile
from pathlib import Path
import numbox
pkg = Path(numbox.__file__).parent
tmp = Path(tempfile.mkdtemp())
archive = tmp / "numbox.zip"
with zipfile.ZipFile(archive, "w") as z:
    for src in sorted(pkg.rglob("*.py")):
        member = str(src.relative_to(pkg.parent))
        if member == "numbox/core/bindings/libm.py":
            pyc = py_compile.compile(str(src), cfile=str(tmp / "x.pyc"), dfile=os.path.join(str(archive), member), doraise=True)
            z.write(pyc, member + "c")
        else:
            z.write(src, member)
env = dict(os.environ, PYTHONPATH=str(archive), HOME=str(tmp), XDG_CACHE_HOME=str(tmp / "cache"))
env.pop("NUMBA_CACHE_DIR", None)
env.pop("NUMBOX_JIT_OPTIONS", None)
code = "import numbox.core.configurations, numbox.core.bindings.libm as m; print(m.__file__)"
sys.exit(subprocess.run([sys.executable, "-W", "always", "-c", code], env=env, cwd=str(tmp)).returncode)

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/x/numbox.zip/numbox/core/bindings/libm.py", line 27, in <module>
  File "/tmp/x/numbox.zip/numbox/core/proxy/proxy.py", line 365, in wrap
  File "numba/core/decorators.py", line 227, in wrapper
    disp.enable_caching()
  File "numba/core/dispatcher.py", line 822, in enable_caching
    self._cache = FunctionCache(self.py_func)
  File "numba/core/caching.py", line 719, in __init__
    source_stamp = self._impl.locator.get_source_stamp()
  File "numba/core/caching.py", line 388, in get_source_stamp
    return _hash_zipped_source_file(self._zip_path, self._internal_path,
  File "numba/core/caching.py", line 175, in _hash_zipped_source_file
    with zf.open(internal_path) as f:
  File "/usr/lib/python3.12/zipfile/__init__.py", line 1617, in open
    zinfo = self.getinfo(name)
  File "/usr/lib/python3.12/zipfile/__init__.py", line 1545, in getinfo
    raise KeyError(
KeyError: "There is no item named 'numbox/core/bindings/libm.py' in the archive"

With cache on, make_graph fails the same way in an archive whose numbox/core/work members are .pyc. The import warns and falls back first. Then make_graph calls _cached_at_or_uncached for builder.py. The except at line 158 doesn't catch KeyError, unlike the except in configurations.py at line 341. I'd expect the same fallback at line 158.

import os, py_compile, subprocess, sys, tempfile, zipfile
from pathlib import Path
import numbox
pkg = Path(numbox.__file__).parent
tmp = Path(tempfile.mkdtemp())
archive = tmp / "numbox.zip"
with zipfile.ZipFile(archive, "w") as z:
    for src in sorted(pkg.rglob("*.py")):
        member = str(src.relative_to(pkg.parent))
        if src.parent == pkg / "core" / "work":
            pyc = py_compile.compile(str(src), cfile=str(tmp / "x.pyc"), dfile=os.path.join(str(archive), member), doraise=True)
            z.write(pyc, member + "c")
        else:
            z.write(src, member)
env = dict(os.environ, PYTHONPATH=str(archive), NUMBA_CACHE_DIR=str(tmp / "cache"))
env.pop("NUMBOX_JIT_OPTIONS", None)
code = (
    "from numbox.core.work.builder import Derived, End, make_graph\n"
    "x = End(name='x', init_value=3.14)\n"
    "def twice(x):\n"
    "    return 2 * x\n"
    "y = Derived(name='y', init_value=0.0, derive=twice, sources=(x,))\n"
    "access = make_graph(y, jit_options={'cache': True})\n"
    "access.y.calculate()\n"
    "print(access.y.data)\n"
)
sys.exit(subprocess.run([sys.executable, "-W", "always", "-c", code], env=env, cwd=str(tmp)).returncode)

/tmp/x/numbox.zip/numbox/core/configurations.py:357: RuntimeWarning: numba cannot cache numbox here ("There is no item named 'numbox/core/work/__init__.py' in the archive"); it compiles without a cache. numba>=0.68 requires source .py file(s) to be present in .zip
Traceback (most recent call last):
  File "<string>", line 6, in <module>
  File "/tmp/x/numbox.zip/numbox/core/work/builder.py", line 336, in make_graph
  File "/tmp/x/numbox.zip/numbox/utils/preprocessing.py", line 157, in _cached_at_or_uncached
  File "/tmp/x/numbox.zip/numbox/core/configurations.py", line 80, in check_cache_location
  File "numba/core/caching.py", line 388, in get_source_stamp
    return _hash_zipped_source_file(self._zip_path, self._internal_path,
  File "numba/core/caching.py", line 175, in _hash_zipped_source_file
    with zf.open(internal_path) as f:
  File "/usr/lib/python3.12/zipfile/__init__.py", line 1617, in open
    zinfo = self.getinfo(name)
  File "/usr/lib/python3.12/zipfile/__init__.py", line 1545, in getinfo
    raise KeyError(
KeyError: "There is no item named 'numbox/core/work/builder.py' in the archive"

Smaller things while I was in there. On 0.68 is_a_cache_error treats any KeyError as a cache error, while the ValueError arm checks the message. The KeyError message from zipfile starts with 'There is no item named'. Matching that prefix would keep an unrelated KeyError raising as the docstring says. The remedy at line 262 is the only one that doesn't end in 'or NUMBOX_JIT_OPTIONS=...'. So the missing-sources warning never says how to turn it off. And the moved-archive remedy at lines 283-287 still says to compile the .pyc members to name the archive's path. Following that remedy on 0.68 lands on the missing-sources warning with no cache. Shipping the sources is the only route that caches now.

[nelson2005]

From the fake Slim Shady:

Two more on the missing-sources warning. On Windows with 0.68.0 the same KeyError happens even when the sources are in the archive. numba's _split_zip_path builds the member name with the platform separator. numba then looks for 'numbox\core\configurations.py', a member no archive has. This is numba 10889, the issue cited by the skips at 135-137 in test_no_cache_location.py. So on Windows the warning tells people to add files that are already there. I ran the Windows case in run 37884221706. It used numba 0.68.0 and 0.67.0 on windows-latest, with numbox at 843a85f. The probe is zip_import.py. With every .py in the archive, the 0.68.0 job prints:

C:\x\numbox.zip\numbox\core\configurations.py:357: RuntimeWarning: numba cannot cache numbox here ("There is no item named 'numbox\\\\core\\\\configurations.py' in the archive"); it compiles without a cache. numba>=0.68 requires source .py file(s) to be present in .zip

The 0.67.0 job imports the same archive with no warning.

A backslash in the missing member's name is the tell. For that case the warning could point people at 10889 and its fix, numba 10896.

The second point is small: the cache_remedy docstring says nothing of the KeyError branch or 0.68.

[nelson2005]

From the fake Slim Shady:

test/core/test_no_cache_location.py:557

Renaming the parent does the same for the 'under a directory named .zip' id: on 0.68 it runs under 'container' and passes. Under 'container.zip' numba's locator stops at the first path part ending in .zip, gets EISDIR from a directory that is writable, and the warning says to make that directory writable. I'd keep the directory name and expect the fallback, with a warning that names the .zip in the path rather than permissions.

[Goykhman]

Done here.

[nelson2005]

From the fake Slim Shady:

Verified at 843a85f on 0.68.0: under the 'container.zip' directory the import takes the fallback, and the warning names that directory. Two small things in the test. The skips at 593-596 key on the remedy text, so the warning is never asserted. If the remedy regressed, the test would skip on 'numba caches a .zip from 0.61 on' instead of failing. Asserting the remedy text before the skip would pin it. And the rename is commented out rather than removed.

[nelson2005]

From the fake Slim Shady:

test/core/test_proxy.py:633-634

docs/numbox.core.proxy.rst and docs/numbox.core.work.rst still say this fails from 0.61 onward.

[nelson2005]

From the fake Slim Shady:

Still there at 843a85f. In work.rst one sentence marks the .as_func tuple fixed in 0.68.0. The next sentence at 600-603 says the tuple 'fails on the same assertion' and names no version. The test_proxy summary line at 607 still says 'refused', but the body returns 5 on 0.68.

[nelson2005]

From the fake Slim Shady:

One more sentence in the same work.rst passage. Lines 604-607 say two plain CompileResultWAP values with different signatures 'fail identically'. Like the .as_func tuple, they unify on 0.68.0:

from numba import njit, float64, int64
from numbox.utils.highlevel import CompileResultWAP
@njit(float64(float64))
def f(x):
    return x + 10.0
@njit(int64(float64))
def g(x):
    return int(x * 2)
a = CompileResultWAP(f.get_compile_result(f.nopython_signatures[0]))
b = CompileResultWAP(g.get_compile_result(g.nopython_signatures[0]))
@njit
def use_pair(pair, x):
    return pair[0](x) + pair[1](x)
try:
    print("result", use_pair((a, b), 3.0))
except AssertionError as e:
    import traceback
    print("AssertionError from", traceback.extract_tb(e.__traceback__)[-1].name)

This prints 'result 19.0' on 0.68.0 and 'AssertionError from unified_function_type' on 0.67.0.