Something#2352
Draft
tbennun wants to merge 149 commits into
Draft
Conversation
The legacy schedule-tree frontend package is slated for removal in favor of nextgen, so nextgen must not depend on it. The utilities nextgen used move to their canonical new homes - structure member-access helpers to nextgen/semantics/structures.py and CallbackOutliner to nextgen/lowering/outliner.py - and the dependency inverts: the legacy structure_support/callback_support modules now re-export them (their legacy-only helpers stay put), so erasing the old package later only deletes shims. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
visit_ConsumeScope was an unimplemented stub (for SDFG-derived trees too). It now mirrors visit_MapScope: a fresh ConsumeEntry/ConsumeExit pair around the body, the consumed stream (the scope's Stream-typed tasklet read) sourced into the entry's fixed IN_stream connector through the shared input machinery, and popped-element reads routed through OUT_stream. All other reads and writes use the same IN_/OUT_ passthrough scheme as maps - the map exit-connection loop is extracted into a shared _connect_scope_exit helper, whose single-use access-node collapse reproduces the hand-built reference shape of tests/consume_test.py (tasklet-to-exit edges carrying the dynamic/WCR memlets). Tasklet scope wiring generalizes from MapEntry to any EntryNode. Leading state boundaries inside consume scopes are skipped (they encode hazards against pre-scope producers such as the stream push); interior boundaries still raise. The nextgen fibonacci consume program compiles and computes fib(10)=55 across 4 processing elements, and the consumescope statement-body form executes as well. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every interpreter-fallback site now carries a stable kebab-case category, rendered as a '[category]' prefix on the PythonCallbackNode reason (no new node field; batching's '; ' joins preserve one prefix per merged statement run). UnsupportedFeatureError gained an optional 'category' attribute so raise sites own their classification; fallback_to_callback resolves exception category > call-site default > 'uncategorized'. The registry totality net tags uncategorized escapes as 'safety-net' (highest bug suspicion), MarkOpaque tags non-CPA statements as 'opaque-syntax:<type>', and call dispatch distinguishes 'detected-callback' (preprocessing-wrapped, intended) from 'unknown-call:<qualname>' (missing-lowering worklist) and 'pyobject-propagation'. Taxonomy documented in lowering/dispatch.py. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…path _check_schedule_tree_parallel_lowering now compares the classic frontend's callback count (sdfg.callback_mapping, cross-checked against callback-typed symbols) with the nextgen tree's PythonCallbackNode count. The classic frontend is the ground truth for intent — callback auto-detection happens in shared preprocessing — and nextgen batching only merges adjacent callbacks, so more nextgen nodes than classic callback sites proves excess interpreter fallbacks: a nextgen lowering gap by definition. The error lists each node's category-prefixed reasons. New config frontend.stree_callback_check = error|warn|off (default error; warn is the triage escape hatch, off the transition emergency valve). Known discrepancies surfaced by the check (the initial gap list, real nextgen gaps, recorded in plan/plan.md): - tests/schedule_tree/schedule_test.py::test_for_in_map_in_for (unknown-call:dace.define_local + broadcast) - ::test_nesting_view/::test_nesting_nview (unknown-call:subset.reshape) - ::test_dyn_map_range (data-dependent-subscript in implicit loop body) - parallel_schedule_tree_test.py::test_parallel_path_pythonclass_argument (structure-member: PythonClass dynamic member assignment) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New config frontend.stree_report (string path, default off; env DACE_frontend_stree_report): when set, the parallel lowering check appends one JSON line per parsed program — program, owning test (PYTEST_CURRENT_TEST), classic/nextgen callback counts, discrepancy flag, category-prefixed reasons, per-category counts, and the legacy diagnostic counts (statement nodes, refsets, PythonClass containers). Single-line O_APPEND writes, pytest-xdist-safe; records are written before any raise so error-mode runs still contribute. New CLI: python -m dace.frontend.python.nextgen.coverage report <jsonl> aggregates by program (max over specializations), prints discrepant programs first, then top gap categories and unknown-call qualified names — the prioritized gap worklist. Triage recipe: DACE_frontend_stree_report=$TMP/stree.jsonl \ DACE_frontend_stree_callback_check=warn pytest ... ; then the report CLI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Broader-suite triage (tests/python_frontend/ + tests/numpy/, check=warn + report): 1052 programs, 313 discrepant — 303 of 1209 to_sdfg-exercising tests across 70 files would fail under the committed error default. Top categories: unknown-call 239 (dominated by the define_local/ndarray local- array declaration family), pyobject-propagation 68, memlet-parse 40, opaque-syntax:Assign 35 (tuple unpacking), opaque-syntax:For 24, type-inference 23, data-dependent-subscript 19. Full record and burn-down priority in plan/plan.md; default stays error per spec. Also hardens the check/report tests against ambient DACE_frontend_* env vars, which override set_temporary (config.py:255) and broke 4 tests during the env-driven triage run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every General Tests matrix job now runs with DACE_frontend_stree_callback_check=warn (the escape hatch from the committed error default, so the known discrepancy wave does not fail the full suite) and DACE_frontend_stree_report pointed at a jsonl the in-path parallel lowering check appends to across all test steps. At the end of each job the nextgen coverage CLI aggregates the collected lines into the GitHub job summary and uploads the raw jsonl plus the rendered report as a per-matrix-combination artifact. Also triggers General Tests on pushes to pyfrontend2 so the report is produced without an open PR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Opus 4.8
Co-authored-by: Opus 4.8
Step 1 keyed write-conflict resolution off the AugAssign token alone, matching classic. Every other spelling of the same reduction inside a map -- b = b + x, b = x + b, b = x + y + b -- still lowered as a silent data race. New CPA pass DetectAccumulations, ordered before DesugarStatements (whose AugAssign desugaring produces the very shape it matches) and before ANFTransform (which hoists compound operands into temporaries and can move the self-reference out of the statement entirely). That ordering constraint is why the detector belongs in canonicalization rather than at lowering. The pass only annotates, never rewrites: rewriting b = x + b into b += x swaps the operands of +, which concatenates Python sequences, where it does not commute. The swap happens at lowering instead, after resolve_access succeeds on the target -- something a compile-time list or string never does, so the swap only ever reaches numeric data. Forms with no WCR equivalent (b = b + x + y, needing inexact re-association; b = x - b, accumulator in a non-fold position; b = max(b, x), a call) are marked conflict_hazard and reported as warnings. They still lower as races, matching classic, but no longer silently: a racing read-modify-write produces a callback-free tree, which is exactly why this defect class was invisible to the callback discrepancy check. Policy moves out of rules/assign.py into lowering/mechanisms/conflict.py, so every write path consults one function instead of re-deriving the rule. For the normalized forms nextgen now emits WCR where classic emits none; test_matches_classic_wcr_decision is restricted to the += spelling accordingly. nextgen suite 286 passed. Targeted regression set (11 blast-radius files found by AST-walking tests/ for a self-referential write inside a map, plus tests/python_frontend and tests/schedule_tree): 67 failures before and after, identical sets, zero regressions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KJHNEtgKvbFSKmBHXKZcZ5
Reported as five advanced_indexing_test failures caused by the accumulation
work. Bisect says otherwise: they fail identically at HEAD~2, ~4, ~10 and
~20. The investigation found a worse defect underneath.
_indirect_subscript built the rewritten access by string round-trip,
ast.parse(f'{connector}[{index_code}]'). An unparsed index tuple carries
parentheses and a slice is only legal in a bare subscript, so
A[ind, 2:7:2, [15,10,1]] produced __in0[(__in1, 2:7:2, __in2)] and a raw
SyntaxError that escaped the categorized-fallback machinery. Now built as
an ast.Subscript directly.
That fix alone made five tests pass -- with trees that cannot become valid
SDFGs. indirect_index_reads accepted any index resolving to a container, so
a whole-array index (NumPy advanced indexing) was treated as scalar
indirection and given the base array's subset. A[ind] + B with A: int32[M],
ind: int32[N] emits ind[0:M]: it compiles and segfaults. A[indices, 4] and
A[rows, columns] fail SDFG validation as out-of-bounds memlets. Two of
those scored as successes under the callback discrepancy check -- the same
blind spot as the WCR bug, which measures whether we fell back, never
whether the dataflow we emitted instead was correct.
indirect_index_reads now collects only scalar index reads. Advanced
indexing falls through to resolve_access, which already rejects it as
arrdims with a categorized data-dependent-subscript error. Genuine
indirection is untouched -- x[A_col[j]] and A[ptr[i]:ptr[i+1]] index with
single elements.
Failure counts rise on purpose: advanced_indexing_test 21 -> 23, and
test_augmented_assignment_to_indirect_access joins them. All three were
passing with invalid trees, verified individually. Seven honest
data-dependent-subscript gap entries in place of two silent
miscompilations and five hard crashes.
nextgen advanced_indexing_test.py pins the boundary in both directions and
fails 4/5 without this change. nextgen suite 291 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KJHNEtgKvbFSKmBHXKZcZ5
advanced_indexing_test.py: 21 -> 6 failures, 19 tests fixed across the
numpy and python_frontend suites with zero regressions.
New mechanism lowering/mechanisms/advanced_indexing.py, ported from the
classic frontend, which is the only working reference (see below).
Reads (emit_gather): one map over the result index space, a pointer
connector on the base array with basic dimensions pinned and advanced
dimensions left whole, one element-read connector per index array, and a
tasklet __out = __arr[__inp0, ...]. Implements the full NumPy shape
algebra -- index-array broadcasting, the advanced chunk landing in place
when contiguous and at the front when not, scalar-indexed dimensions
counting as advanced, and newaxis insertion on either side of the chunk.
Writes (emit_scatter): the mirror, with the base array as an output
pointer connector and the element access in the tasklet code. An
accumulating scatter takes WCR unconditionally -- a stronger rule than
conflict.accumulation_wcr applies elsewhere, because the index array may
name the same element twice (A[[0, 0]] += 1) and no subset inspection can
rule that out.
Boolean masks (emit_masked_write): a mask makes the count of written
elements data-dependent but not their positions, so a masked write lowers
as a guarded update over the full array, needing neither a gather nor
dynamic allocation nor conflict resolution. Masked reads are rejected with
a categorized error, matching classic.
Nested occurrences (dispatch._materialize_advanced_indices): A[ind] + B
gathers into a temporary first. ANF cannot do this -- a data subscript is
a legal operand and canonicalization has no type information to tell
A[scalar_i] from A[index_array].
inference.parse_access restores ANF-hoisted literal index sequences before
parsing: A[:, (1, 2, 3)] becomes A[:, __anf0], and the memlet parser
recognizes an array index only from a literal or a registered container,
so it would otherwise emit a subset naming a value with no runtime
existence.
The deprecated schedule-tree frontend's advanced indexing turned out to be
modelling only: its boolean gather emits FrontendLibrary('boolean_mask_gather')
and tree_to_sdfg.visit_LibraryCall raises NotImplementedError. Verified end
to end. Its shape algebra was useful as a cross-check, nothing more.
The new gate executes and compares against NumPy rather than only
inspecting the tree, because the previous generation of this bug produced
trees that were callback-free and segfaulted. That caught a shared subset
object between the read and write memlets of the masked update that the
tree-only checks passed over.
Renamed from advanced_indexing_test.py: pytest cannot collect two
same-named test modules in one run, and the collision broke full-suite
collection.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KJHNEtgKvbFSKmBHXKZcZ5
cpa.is_dace_map_iterator tested rname(node.value) in ('dace.map', 'map')
textually, so `import dace as dc; for i in dc.map[0:H]:` was never
recognized: MarkOpaque wrapped the whole loop body as one interpreter
callback instead of a real MapScope. Widen the check (nextgen-side only,
per the plan's fallback option since preprocessing.py is shared with the
classic frontend and out of scope here) to also resolve the subscript's
root through the canonicalization context's global_vars, mirroring the
alias resolution passes._refers_to already performs for dace.tasklet/
dace.map decorators. Thread global_vars from PipelineContext down through
verify_canonical/_violations_in_statement (post-hoc contract check) and
from MarkOpaque.transform_statement (the actual decision point during
canonicalization), and from ProgramContext.globals into the lowering
rule (control_flow.lower_for).
Also widen the grammar to accept `dace.map[...] @ ScheduleType`, a BinOp
with ast.MatMult wrapping the map subscript (nested_name_accesses_test.py
::test_issue_2100's pattern) — is_dace_map_iterator previously required
an outer ast.Subscript. control_flow._lower_map_loop now splits the
iterator into its range subscript and an optional resolved schedule type
(mirroring newast.py's classic handling) and applies it to the emitted
Map; ANFTransform's For handling flattens the inner subscript's slice for
this shape too.
Before/after on the alias-bearing files (indirection_with_scalars_test.py,
indirections_test.py, nested_name_accesses_test.py) plus the full nextgen
suite: 31 failed/311 passed -> 18 failed/324 passed, 13 newly passing
(all opaque-syntax:For discrepancies), 0 regressions (identical failure
set across two runs). test_issue_2100's For statement is no longer
opaque, but the test still fails on unrelated pre-existing gaps
(dace.frontend.python.wrappers.ndarray/.fill() call dispatch, a c[:, :]
memlet-dimensionality bug) tracked separately in the plan. Coverage
report over the three files confirms opaque-syntax:For no longer appears
in the gap categories.
Note: this worktree was provisioned from main rather than frontend2-wip
(dace/frontend/python/nextgen/ didn't exist at its base commit); a prior
commit here merges frontend2-wip in to pick up the nextgen frontend
before this fix.
Root cause A (plan/eventual-sleeping-spring.md): `_lower_replacement_call` only ever consulted `Replacements.get(name)` (free functions), so every `A.copy()`/`A.fill(0)`/`A.astype(t)`/`A.transpose()`/`A.mean()`/`A.all()`/ `A.any()`/`A.conj()`/`p.copy()`-style bound-method call was typed (inference already consulted `get_method_descriptor_inference`) but never lowered, falling back to a Python callback every time. - `tn.ReplacementCallNode` gains an optional `receiver` field (the bound container's name); `arguments` carries the receiver as its first entry, mirroring classic's `newast.py` Call-visitor convention of prepending `self`. - `dispatch._lower_replacement_call` detects a bound-method call (an `ast.Attribute` callee whose base resolves, via `resolve_access`, to a whole `Name`/structure-member container -- never a `Subscript`, since a `_method_rep` replacement receives the full container by name and a subset receiver would silently operate on the whole array) and looks it up via `Replacements.get_method`/`get_method_descriptor_inference` instead. The existing `_expansion_viable` trial-run gate is reused unchanged, so view-producing methods (reshape/transpose/ravel/flatten's axis-less form) still correctly degrade to a callback rather than being forced through `ReplacementCallNode`. - `tree_to_sdfg.visit_ReplacementCallNode` dispatches through `Replacements.get_method` when `receiver` is set. - `verify.py` checks the receiver is a registered data argument. - `inference._bound_descriptor_of` admits one more level (`ast.Subscript`/ `ast.Attribute`, e.g. `A[0].sum()`) so such calls are at least typed; the early `return None` when no method-family inference matches now falls through to ufunc/free-function inference instead of hard-aborting the call. Verified via before/after runs of tests/numpy/ndarray_attributes_methods_test.py (methods only; the attribute tests -- .T/.real/.imag/.flat -- are a separate, untouched gap) and tests/python_frontend/nextgen/ under the default (strict) `stree_callback_check=error` config: nextgen suite stays green at 299 passed; the numpy gate file goes from 22 to 12 failing (10 newly passing: copy, astype, transpose x3, conj, mean, all, any, argmax_dim), with the after set a strict subset of the before set (zero regressions). Confirmed via the coverage report: unknown-call:<method> program count 18 -> 8, discrepant programs 22 -> 12. Remaining unfixed, all pre-existing/out-of-scope, not method-plumbing gaps: - .T/.real/.imag/.flat: the attribute family (root cause A's separate, unwired `_attr_rep` keyspace) -- explicitly out of scope per task. - A.fill(...): a bare (unassigned) call statement lowers with `target=None`, which `_lower_registry_call` rejects for ALL replacement families (not method-specific) as "unused result, preserve semantics via callback". - A.reshape()/.flatten()/.ravel()/.argmax()/.argmin() (axis-less): reshape is intentionally left to `_lower_reshape_call`'s view path; flatten/ravel and the axis-less argmax/argmin internally call the array-manipulation `flat()` helper, which records a view binding for contiguous arrays -- the SAME `_expansion_viable` gate this change reuses (not modified) correctly rejects these and falls back to a callback, as intended.
…/.imag/.flat) Inference (semantics/inference.py::_infer_attribute) and lowering (lowering/dispatch.py) previously never consulted the replacement registry's ATTRIBUTE family (Replacements.get_attribute / get_attribute_descriptor_inference), so A.T/A.real/A.imag/A.flat always fell back to a Python callback, triggering the schedule-tree parallel lowering discrepancy check on ordinary classic-frontend tests (classic supports these attributes natively). - inference.py: typed a small allowlisted set (SUPPORTED_DATA_ATTRIBUTES in nextgen/common.py) through get_attribute_descriptor_inference, scoped tightly to what lowering actually covers -- an attribute typed as 'data' with no lowering path would otherwise reach the generic elementwise mechanism and embed the unresolved attribute literally in generated tasklet code instead of degrading to a callback. - dispatch.py: resolve_attribute_data materializes the attribute into a fresh container, mirroring the _lower_reshape_call precedent. A.flat on a contiguous array binds a ViewNode directly (NumPy's flatiter aliases the source, and _expansion_viable correctly rejects view-producing deferred calls). Everything else (.T/.real/.imag, and non-contiguous .flat) defers to a ReplacementCallNode running the classic ATTRIBUTE implementation, after a scratch-SDFG viability trial (_run_attribute_trial, mirroring _expansion_viable). - op_repository.py/tree_to_sdfg.py: ReplacementCallNode.qualname only resolved through the free-function keyspace; added a small classname.@attr qualname convention (attribute_qualname/ ATTRIBUTE_QUALNAME_MARKER) so visit_ReplacementCallNode can also resolve ATTRIBUTE-family entries. - assign.py/returns.py: wired the new dedicated entry points at the points that matter for the gate tests -- bare `b = A.<attr>`, `return A.<attr>`, and `A.flat[idx] = ...` (rewrites the subscript base to the materialized view via rewrite_flat_subscript_base). Fixed (previously failing only via the discrepancy check): tests/numpy/ndarray_attributes_methods_test.py::test_T/test_real/ test_imag/test_lhs_flat, tests/numpy/advanced_indexing_test.py:: test_flat/test_flat_noncontiguous. Full nextgen suite stays green (299 passed) plus tests/schedule_tree/ (84 passed). Known remaining gap (deliberately out of scope): a nested subscript assignment target (A.flat[10:15][0:2] += 5, advanced_indexing_test.py:: test_aug_implicit_attribute) and an attribute base that isn't a plain dataref ((a @ b).T, attribute_test.py::test_attribute_of_expr) both fail at canonicalization, before semantic binding/lowering ever runs (_flatten_target/_as_dataref in canonical/passes.py only hoist Name-rooted attribute chains) -- a canonicalization-level fix, not a registry-wiring one. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…mlet_parser
A[:, (1, 2, 3)] and A[(1, 2, 3, 4, 5, 7)] as one dimension among several were
misparsed: _fill_missing_slices had no case for a raw ast.Tuple literal
surviving as a single dimension slot, so it fell into the scalar catch-all,
got eval()'d into a plain Python tuple, and was then reinterpreted by
_ndslice_to_subset as a (start, stop, step) range -- silently collapsing
"columns 1,2,3" into "column 1" for 3-tuples, or raising "Expected 3-tuple or
4-tuple" for longer ones.
Python's grammar makes "A[(1,2,3)]" and "A[1,2,3]" identical ASTs, so a bare
tuple that is the *entire* subscript is already unpacked into separate
per-dimension entries before _fill_missing_slices ever runs (see
astutils.subscript_to_ast_slice). The only way an ast.Tuple node can survive
as a single `dim` is if the source had an explicit nested tuple among
multiple indices/slices ("A[:, (1,2,3)]") or a trailing comma
("A[(1,2,3),]") -- both of which are unambiguously advanced (array-valued)
indexing in NumPy, never slice-range sugar. Classic's newast.py already
reached the right answer via a private preprocessing trick
(_parse_subscript_slice converts a nested literal tuple to a Python list
before calling ParseMemlet); nextgen's inference.py calls ParseMemlet
directly and had no equivalent, so it hit the shared bug. Fixing it in
_fill_missing_slices itself closes the gap for both frontends without
relying on caller-side preprocessing.
Fixes tests/numpy/advanced_indexing_test.py::test_multidim_tuple_index[False]
and ::test_multidim_tuple_index_longer. Full advanced_indexing_test.py goes
from 6 failed/29 passed to 4 failed/31 passed; the remaining 4 (test_flat,
test_flat_noncontiguous, test_aug_implicit, test_aug_implicit_attribute) are
pre-existing, unrelated gaps (A.flat attribute support and chained-subscript
augmented assignment). No regressions in tests/python_frontend/nextgen/ (299
passed before and after) or in the shared-parser consumer set (memlet_parser
direct consumers' test files: memlet_parser_test.py, slice_test.py,
negative_indices_test.py, slice_subset_aliasing_test.py, python_slice_test.py,
augmented_assignment_to_slice_test.py, subscript_regression_test.py --
identical failing-test-name sets before/after).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
_flatten had no ast.Dict branch, so any call with a dict-literal argument
(e.g. a plain-Python callback call callback({'a': x})) raised
_ShortCircuitHazard and the entire enclosing statement was left
non-canonical, later swallowed whole into one opaque-syntax callback.
Add a Dict branch mirroring the existing List/Tuple literal handling:
flatten keys/values to atom level and hoist the literal to a temporary
(dict-unpacking **rest entries, which have no atom form, still raise the
hazard and stay opaque). Deliberately does NOT touch cpa.is_flat, so a
bare dict assignment (d = {...}) is still marked opaque as before -- only
a dict flowing through a call argument position now separates into a
small opaque dict-construction statement plus a properly canonicalized
call, which resolves through the normal detected-callback path instead of
one coarse opaque-syntax:Assign blob.
Tests: tests/python_frontend/nextgen/callback_category_test.py gains
test_dict_literal_call_argument_category, asserting the call now carries
detected-callback provenance. tests/python_frontend/callback_autodetect_test.py
(the existing dict-literal-argument blast radius, incl. array-valued dict
entries and dict-typed kwargs) is unaffected -- identical failure set
before/after (2 pre-existing unrelated failures: test_gpu_callback, no GPU
in this env; test_two_callbacks, a pre-existing batching discrepancy).
Full nextgen suite: 299 -> 300 passed (the one new test), zero regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
inference.call_arguments and dispatch._replacement_arguments both refused
a static Python sequence (list/tuple) whose elements resolved to data
containers -- exactly the (A, B) in numpy.concatenate((A, B)), and
likewise for stack/vstack/hstack/dstack/select. The rejection comment
("Data elements cannot be passed by value") was written for scalars, not
for name lists: the classic frontend and the registry replacements
(_concat's `arrays: Tuple[Any]`, checked via `arrays[i] not in
sdfg.arrays`) already expect exactly this shape -- a tuple/list of
container name strings.
Both sites now resolve each element the same way a top-level argument is
resolved (reusing the existing per-argument `convert` closure), so a
data-container element is recorded by name into input_descs/data_arguments
instead of failing the whole conversion.
Tests: tests/python_frontend/nextgen/replacement_call_test.py gains
concatenate/hstack structure+execution tests. tests/numpy/concat_test.py
(the existing blast radius): 12 failed/1 passed -> 4 failed/9 passed.
The 4 remaining failures are unrelated, already-documented gaps verified
by direct probing of the registry replacements: numpy.stack and
numpy.vstack(1-D) record internal ArrayView bindings that the deferred
replacement-expansion trial deliberately rejects (view state cannot be
deferred); numpy.concatenate(..., out=c) is a bare statement call, which
hits the separate `target is None` short-circuit. None of these three are
in this item's scope. Full nextgen suite: 300 -> 303 passed, zero
regressions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…registry infer_call only recognized a bare `numpy.add(...)` call (isinstance(callee, numpy.ufunc)) -- an ast.Attribute call like `numpy.add.reduce(...)` resolves its callee to a bound method object, not the ufunc itself, so it was never typed at all. get_ufunc_descriptor_inference()/get_ufunc() were also always called with no method argument, so only the base 'ufunc' registry key ever resolved -- the registered reduce/accumulate/outer inference entries were dead code for nextgen. - InferenceService.resolve_ufunc_call (inference.py), shared by inference and dispatch: recognizes a direct ufunc call OR an ast.Attribute call whose attr is reduce/accumulate/outer and whose *base* resolves to a numpy.ufunc, returning (ufunc, method). infer_call now routes through get_ufunc_descriptor_inference(method) instead of the no-argument form. - dispatch.py: reduce/accumulate/outer calls need real reduction/scan/ broadcast dataflow that the lightweight single-tasklet elementwise mechanism cannot express, so they defer to the actual registry implementation (dace.frontend.python.replacements.ufunc) through the same deferred ReplacementCallNode expansion mechanism already used for other registry replacements (reused, not duplicated: _replacement_arguments for argument resolution, a shared _replacement_trial_scratch helper factored out of _expansion_viable for the build-time viability trial). - treenodes.ReplacementCallNode gains ufunc_name/ufunc_method fields; tree_to_sdfg.visit_ReplacementCallNode dispatches through Replacements.get_ufunc(method) with the ufunc calling convention (visitor, ast_node, sdfg, state, ufunc_name, args, kwargs) when set, and normalizes its List[UfuncOutput] (single-element) return to the same bare-string form the generic replacement convention already handles. - Direct (non-method) ufunc calls with keywords are UNCHANGED here (still rejected) -- that relaxation is the next commit (item 2); this commit only wires up the reduce/accumulate/outer *methods*, which need some keyword support (axis=/keepdims=/initial=) to do anything at all. BUG FOUND AND FIXED while wiring this up: _infer_ufunc_reduce_descriptor/ _infer_ufunc_accumulate_descriptor (dace/frontend/python/replacements/ufunc.py, pre-existing, previously dead code for nextgen and untested by any existing test) computed NumPy's actual integer-promotion dtype (e.g. int32 -> int64 for numpy.add.reduce), but the paired *implementation* (implement_ufunc_reduce/implement_ufunc_accumulate) always uses the input dtype verbatim, with no such promotion. Once reduce/accumulate became reachable, this mismatch pre-allocated the target container too wide; the raw (uncast) memlet copy between the narrower actual output and the wider declared target then read garbage from the extra bytes -- a silent miscompilation, not merely a precision difference, caught only by executing the nextgen-compiled SDFG directly and comparing against NumPy (the existing discrepancy-check gate cannot see it, being blind to numeric correctness). Fixed by dropping the promotion in the two inference functions to match what execution actually produces (removed the now-dead _infer_reduce_dtype helper). Confirmed no other call sites of get_ufunc_descriptor_inference exercise these two functions with any existing test coverage (dace/frontend/python/schedule_tree_frontend.py and .../schedule_tree/type_inference.py also consult them, but no test in tests/ reaches reduce/accumulate/outer through that alternate frontend). Tests: tests/python_frontend/nextgen/replacement_call_test.py gains reduce/accumulate/outer structure+execution tests (accumulate uses a 2-D array; 1-D degenerates implement_ufunc_accumulate's map to zero parameters, a separate pre-existing gap in the shared registry implementation, not a nextgen routing issue -- matches ufunc_support_test.py's own accumulate coverage, which is also never 1-D). Regression gates: tests/numpy/ufunc_support_test.py 30 failed/7 passed -> 11 failed/26 passed (the 11 are all keyword-argument forms, item 2's scope, unchanged here by design). tests/numpy/ufunc_nested_call_test.py (reduce/ accumulate via nested calls) 2 failed -> 0 failed. tests/numpy/ufunc_replicated_input.py: 1 pre-existing failure unchanged (unrelated target-is-None bare-statement gap, verified via backup/restore diff of failing-test-name sets). Full nextgen suite: 303 -> 307 passed (4 new tests), zero regressions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dispatch.py rejected ANY ufunc call with a keyword argument outright (out=/where=/axis=/keepdims=/initial=), even though the registry ufunc implementation (dace.frontend.python.replacements.ufunc.implement_ufunc) already supports all of them -- the rejection existed only because the lightweight local elementwise mechanism (a single tasklet expression) has no way to honor a keyword at all, not because the registry can't. The previous commit already built the machinery a keyworded ufunc call needs (deferred ReplacementCallNode expansion through the ufunc registry keyspace, for reduce/accumulate/outer). This commit just widens the routing condition so a direct call also takes that path whenever it carries a keyword, instead of returning False outright: `if ufunc_method is None and not call.keywords` picks the fast no-keyword elementwise path; everything else (any keyword, or a reduce/accumulate/outer method) defers to _lower_ufunc_replacement_call, which already rejects genuinely unsupported keywords via _SUPPORTED_UFUNC_KEYWORDS and falls back to a callback on any other viability failure. Tests: tests/python_frontend/nextgen/replacement_call_test.py gains test_ufunc_keyword_argument_execution (where=) and test_ufunc_unsupported_keyword_falls_back (an unrecognized keyword still degrades to a callback rather than being silently dropped). Regression gate: tests/numpy/ufunc_support_test.py 11 failed/26 passed (previous commit's state) -> 5 failed/32 passed. The 5 remaining failures are two unrelated, already-documented gaps: test_ufunc_add_out/out2/out3 (np.add(A, B, out=C) as a bare statement, no assignment -- the separate `target is None` short-circuit) and test_ufunc_add_reduce_simple2/simple3 (np.add.reduce on a scalar constant, not an array -- a pre-existing limitation of _infer_ufunc_reduce_descriptor's operand resolution, not a keyword-argument issue). Full nextgen suite: 307 -> 309 passed (2 new tests), zero regressions. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The result's element count is unknown until the mask is read at runtime, unlike every other advanced-indexing read this frontend supports. Resolves this with symbolic.UndefinedSymbol as the deferred-symbol placeholder: inference answers with an honestly deferred shape (replacing a previously silently-wrong one, which treated the mask as an ordinary same-shape index array), and a new fast-tracked lowering path (rules/assign.py::_lower_boolean_gather_assign, tried before generic inference runs) mints a real SDFG symbol from a runtime-computed element count and performs typing and lowering together for the bare top-level assignment form. Nested or combined boolean-mask uses are untouched and still fall back to a callback. Two configurable strategies (frontend.boolean_index_strategy): 'view' (default) does one pass into an upper-bound backing buffer, exposed as a view over the first M elements; 'exact' does two passes (a count-only reduction, then a compaction sized exactly to the count), never over-allocating. Both proven correct standalone first in tests/sdfg/deferred_symbol_boolean_filter_test.py before any frontend code was written. Building this surfaced two narrow, pre-existing DaCe bugs, fixed here since the feature depends on them: - dtypes.pointer never set self.typename (its __init__ doesn't call typeclass.__init__), so to_string()'s documented fallback raised a raw AttributeError instead of degrading gracefully -- crashed DeadDataflowElimination the first time it needed to stringify a Stream-typed connector while removing genuinely dead code. - tree_to_sdfg.py::visit_CopyNode only ever cached read access nodes, never writes, so a write immediately followed by a same-state read of the same name created a disconnected access node instead of reusing the one just written. Invisible to execution (per-state sequencing still orders things correctly) but made a live computation look like an unread dead end to DeadDataflowElimination's node-local liveness analysis, which then removed it. Fixed by mirroring the write-node caching idiom _connect_scope_exit already uses for the identical problem. New tests: tests/python_frontend/nextgen/boolean_gather_test.py (18), tests/sdfg/deferred_symbol_boolean_filter_test.py (2). Regression: nextgen suite 327 passed (+18), tests/sdfg/ 214 passed (4 pre-existing failures, confirmed identical on a temporarily-restored clean-HEAD version of the two changed core files), tests/passes/ 314 passed (8 pre-existing, same verification), tests/schedule_tree/ + tests/python_frontend/schedule_tree/ 420 passed / 0 failed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ndices, bounded subscript nesting Step 10: new lower_indirect_write mirrors the existing read-side indirection mechanism for write targets (hist[bin] += 1, scatter), wired into both plain and accumulating (WCR) subscript assignment. Step 11: guard resolve_access in the subscript-assign copy probe so a scalar-container index inside a dataflow scope falls through to the elementwise path instead of escaping to the top-level callback fallback. Step 12: bound the CPA grammar's subscript-nesting depth (previously the only unbounded recursion in an otherwise depth-1 grammar), restoring canonicalization totality; ANF now hoists deeper nesting into the scalar-container-index shape step 11 handles. Also fixes three bugs surfaced while extending coverage to a newaxis/Ellipsis/ advanced-indexing stress test: nested-SDFG connector cloning in visit_MapScope, basic-indexing newaxis/Ellipsis shape and stride squeeze (kept_dimensions), and advanced-indexing's AST-based (not range-based) kept-dimension computation. A fourth, deeper, pre-existing bug in _place_advanced_chunk (newaxis + multiple kept non-advanced dims) was found, confirmed independent of this change, and documented rather than fixed (see plan/plan.md). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
_reshape_operands checked the <x>.reshape(...) attribute-method branch
before the numpy.reshape(...) free-function branch. Since
numpy.reshape(A, s) is itself an Attribute call ending in .reshape
(Attribute(value=Name('numpy'), attr='reshape')), the method branch
fired first and returned (numpy-module, (A, s)) as (base, shape) --
treating the module as the reshaped array and misrouting every
numpy.reshape(A, s) call to a callback fallback. Check the
numpy.reshape qualname first so the free-function branch is reachable.
Also updates test_nonviable_replacement_falls_back, whose np.reshape(A,
(2, 3)) case now resolves through the fixed dedicated view path instead
of falling back; it's changed to a genuinely non-viable (element-count
mismatch) reshape that still exercises the deferred-replacement
viability rejection it was written to test. Adds
test_reshape_view_execution, executing both call forms and comparing
against NumPy.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
_lower_registry_call unconditionally required target is not None,
so a bare-statement call like A.fill(0) could never lower through the
registry path, even though A.fill's registered descriptor inference
already signals a zero-output/pure-side-effect result (`()`).
Two gaps stacked here, both fixed:
- _registry_inference conflated "registry function returned () (a
confirmed zero-output signature)" with "no registry entry matched"
-- both produced Optional[Inferred] = None. Inferred now has a
'none' kind for the former, via a new is_none_output property, so
callers can tell "typed call with nothing to assign" apart from
"untyped call, fall back to a callback".
- _lower_replacement_call only ever resolved the free-function
keyspace (Replacements.get(qualname)), so a method call like
A.fill(0) (registered under (class, method), not a qualname) was
unreachable regardless of target. It now falls back to the method
keyspace (Replacements.get_method) specifically for the bare
statement/zero-output case, passing the receiver as the first
argument (matching classic's convention). tree_to_sdfg's
visit_ReplacementCallNode gains the matching (class, method) lookup
fallback so the qualname it stores ('Array.fill') resolves the same
way at expansion time, and skips the result-copy when a mutating
replacement returns its own target container unchanged (the common
case for a receiver used as a bare statement's stand-in target).
A.fill(0) and dace.comm.Bcast(A) (a zero-output free function) now
lower without a callback; a discarded, ordinarily data-valued call
result (e.g. bare np.sum(A, axis=...)) still falls back, unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Single shared normalize_qualname() helper collapses a callee's real defining module path to the replacement-registry's shorter key (e.g. dace.frontend.python.wrappers.ndarray -> dace.ndarray), applied at the one source of truth (resolve_callee) instead of ad-hoc two-name retries duplicated across dispatch.py's creation and replacement-lookup call sites. Fixes previously-broken reachable-via-non-literal-spelling forms: aliased module imports (wrappers.ndarray via `import ... as wrappers`), `from ... import X as Y`, and stored callee references (`_nd = dace.ndarray; _nd(...)`) — the direct dace.X spelling already worked. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.