Support Qwen3.8-27B Q8 and BF16 processor loading and model discovery

[marksmeltzer]

Support Qwen3.8-27B Q8 and BF16 processor loading and model discovery

Important

Required dependency for sustained generation: MLX-LM PR #1872

This EXO contribution cannot provide the qualified sustained-decoding path without
the companion ArraysCache correction. This branch temporarily carries that same
correction through its Nix dependency patch; PR #1872 is the upstream MLX-LM change.

Motivation

EXO's VisionEncoder can fail to initialize the image processor for Qwen-family
model repositories even when the installed Transformers version has a compatible
processor registered for the model configuration.

On the tested Qwen3.8-27B assets, processor metadata names
Qwen3VLImageProcessor, while Transformers 5.6.2 maps Qwen3_5Config to
Qwen2VLImageProcessor. The fallback call to
AutoImageProcessor.from_pretrained without an explicit model configuration
raises an unrecognized-image-processor error. Supplying the model configuration
allows the existing Transformers mapping to resolve the processor.

This proposal combines the processor correction with model discovery for the
Q8 and BF16 variants. Sustained generation also requires the companion MLX-LM
cache correction described below. Short text generation already worked on the
baseline despite the caught processor error; processor initialization is not
the cause of the separate sustained-decoding resource failure.

Changes

• Load the processor repository's configuration with AutoConfig.from_pretrained
  and pass it to the existing AutoImageProcessor.from_pretrained fallback.
• Preserve the earlier MLX-VLM processor-loading paths and their precedence.
• Preserve the existing trust_remote_code=True setting on the fallback, and
  use it for the added configuration load. This adds a configuration-loading
  operation under that trust setting; it is not a security-hardening change.
• Include a focused regression exercising EXO's real VisionEncoder loader.
• Add bundled model cards for mlx-community/Qwen3.8-27B-8bit and
  mlx-community/Qwen3.8-27B-bf16, with registration coverage.
• Include the cache-dependency integration and reproduction artifacts needed
  to use these changes from the public PR checkout.

The cards, registration test, regression scripts and Nix dependency carrier are
included in this branch. No downloaded model metadata, weights, sharding,
dependency pins or distributed transport changes are proposed. The cache
correction belongs to MLX-LM; this temporary carrier includes only the adapted
ArraysCache.advance() correction, not unrelated fork changes.

Why It Works

The explicit configuration gives Transformers the model type needed for its
registered image-processor fallback. The change uses the library's existing
mapping instead of hard-coding a Qwen processor class or editing downloaded
model metadata.

With the same dependencies and assets, the baseline real EXO loader loaded
460.7 million vision parameters and then failed during processor resolution.
The corrected loader completed successfully. A separate factory-level probe
also processed a 64-by-64 RGB input with the explicit configuration.

These observations establish processor resolution and initialization, not
end-to-end image understanding or pixel-level equivalence between processors.

Compatibility

The added configuration load requires a loadable model configuration in the
selected processor repository. A processor-only repository without one could
previously resolve from processor metadata alone. Source review identified no
affected current EXO card, but universal processor-repository compatibility has
not been established.

Model Cards

The bundled cards make the two exact Hugging Face IDs discoverable through EXO's
normal catalog and placement workflow. They describe the models; they do not
contain weights, download weights by themselves, or fix runtime compatibility.
Users still need access to the model artifacts and enough memory for placement.

Both tested artifacts select the existing qwen3_5 implementation: 48 recurrent
layers and 16 full-attention layers. No new model architecture is introduced.
Card fields follow EXO's schema and the verified artifact configuration. The
cards advertise text/thinking and MlxMetal, not unqualified vision, CPU or CUDA
support. Their storage sizes come from the pinned safetensors indexes.
The model revisions in the test table identify the tested weights; a catalog
entry alone does not guarantee that a later download uses that same revision.
The 4-bit variant is outside this contribution's qualification.

Companion MLX-LM PR

Required MLX-LM dependency: PR #1872

The companion correction bounds lazy ArraysCache length and padding metadata
by attaching it to a populated cache-state tensor with mx.depends. EXO's tested
batch path evaluates token and logprob outputs without necessarily evaluating
every cache's complete state, which can leave metadata graphs growing during
decoding. This is separate from image-processor selection.

The complete Qwen result reported here used both corrections. Applying only the
processor change and model cards does not reproduce that qualification. The
MLX-LM cache correction is independently useful and does not require the EXO
processor change. The two PRs should link to each other while keeping source
ownership and test claims separate.

Reproducing the Environment

Reproduction starts from this public EXO PR checkout and public model artifacts.
No private repository, internal scripts, local journal or unpublished commit is
a prerequisite. The runtime, dashboard and test-environment Nix builds and
component checks passed on macOS 26.6.1 (25G76). Four-node qualification has not
been repeated on this branch. Full instructions and deterministic request
generation are included in nix/tests/README.qwen38.md and
nix/tests/qwen-requests.py.

1. Check out this PR from the public exo-explore/exo repository. Use the same
   PR revision on every participating node, rather than mixing released and
   patched EXO processes. Follow EXO's public macOS prerequisites and RDMA setup.
2. Build with the corrected MLX-LM dependency. The tested approach applies
   nix/arrays-cache-metadata.patch through the mlx-lm override in
   python/parts.nix. Both files are included in this PR. Installing MLX-LM into a
   separate virtual environment does not modify EXO's Nix-managed dependency.
3. For the Nix route, build the checked-out source with nix build path:.#exo
   and launch that resulting executable (./result/bin/exo) on each node.
   All dependency sources and patches must resolve publicly. A uv run exo
   launch does not apply a Nix-only patch override and is not an equivalent
   reproduction unless its own MLX-LM dependency is corrected and verified.
4. Obtain the exact Q8 or BF16 revision listed below from Hugging Face and make
   its local files discoverable to EXO. Use EXO's supported model storage and
   download workflow; machine-specific filesystem paths are not required.
   Confirm both catalog IDs are registered. A visible card is not proof that
   weights are available or that a model is serving.
5. To match the reported distributed environment, use four Mac Studio M3 Ultra
   nodes with Tensor/JACCL over Thunderbolt/RDMA. Place one variant at a time
   and confirm four assigned Ready runners with world size four before sending
   requests. Other placement modes and node counts were not qualified here.
6. Run the included processor and cache regressions using the Python environment
   consumed by that build, then run sustained text generation for each variant.
   The included request generator emits the retained sustained, recall and
   cached-extension payloads; all six model/scenario combinations were compared
   with the historical JSON and matched exactly. A short response alone does not
   exercise the sustained-decode gap.

Public upstream base: 21a54c5ea0230a3bec1e1a786d200126c7e34ec6. The existing
uv.lock is unchanged; MLX-LM remains pinned to
6a3df6cd6b00a347ee40f12d97a182aaf86ea599 with the Nix patch applied. The
historical distributed results below are supporting evidence, not fresh
qualification of this upstream port.

Test Plan

Manual Testing

Testing was performed on September 9, 2026 using four Mac Studio M3 Ultra nodes,
with four-way Tensor/JACCL over Thunderbolt/RDMA for distributed controls.
The processor reproduction itself is a component-level check, not a
four-node multimodal inference test.

Historical test environment (not yet reproduced from the public PR branch):

• EXO: reported package version 0.3.70, with both corrections applied.
• MLX: 0.32.0.dev20260522.
• MLX-LM: reported version 0.31.3, fork pin
  rltakashige/mlx-lm@6a3df6cd6b00a347ee40f12d97a182aaf86ea599.
• MLX-VLM: 0.4.4; Transformers: 5.6.2.

Exact model revisions retained for reproduction:

Model	Revision
mlx-community/Qwen3.8-27B-8bit	815b83c0df8ffd1d1b5244cf75fd6ef14fca9ef9
mlx-community/Qwen3.8-27B-bf16	6f265714824f3c38d4452baa1628aef3d9b9aae9

The real loader regression was demonstrated red then green against the Q8
assets, with dependencies and assets held constant. Its source is included as
nix/tests/qwen-vision-regression.py; it takes the
model ID as its argument and calls detect_vision_from_config and
VisionEncoder.ensure_loaded without mocking the processor. The cache regression
is included as nix/tests/arrays-cache-metadata-regression.py.
Its 256-step check evaluates a populated cache tensor rather than the complete
cache state, and verifies bounded metadata graphs and numerical values.

Additional controls on the combined candidate:

• Gemma 4 31B BF16 arithmetic and approximately 32k-token recall matched the
  production baseline. Arithmetic latency was 10.23 seconds versus 10.33 seconds
  on the baseline; both context controls completed in approximately 43 seconds.
• Qwen Q8 and BF16 completed sustained text generation and 210k-token cold
  recall, followed by a cached extension to 243k tokens.

The Qwen sustained results also included a separate MLX-LM cache correction.
They are supporting compatibility observations, not isolated evidence that
this processor change fixes long-generation stability. No full-window,
concurrency, overnight, generated-code correctness, or end-to-end multimodal
claim is made.

Automated Testing

The PR includes both standalone regression scripts, not just their results.
They run locally on an Apple Silicon Mac with the corrected MLX runtime;
neither requires a running EXO service or four-node placement.

Script	Requirements	What it checks
nix/tests/arrays-cache-metadata-regression.py	Patched MLX-LM and MLX; no model weights	After 256 advances evaluating cache[0], each metadata graph has at most eight edges and the final state and metadata values are correct.
nix/tests/qwen-vision-regression.py MODEL_ID	Local model config and weights; Transformers and MLX-VLM from the EXO build	Real qwen3_5 vision detection and VisionEncoder.ensure_loaded(), without a mocked processor. Loads vision weights and may take time and memory.

Running the Component Tests

Run from the public PR checkout on macOS, after installing EXO's documented Nix
prerequisites. Use the included scripts and dependency patch/override
described above. Use a normal Python invocation, without -O or PYTHONOPTIMIZE,
because the regressions include assertions.

The runtime virtual environment is exposed by the current Nix package as
exo.venv. Resolve it from this checkout instead of using system Python or
uv run, which could select a different MLX-LM build:

set -e
EXO_TEST_PYTHON_ENV="$(nix build --no-link --print-out-paths path:.#exo.venv)"
"$EXO_TEST_PYTHON_ENV/bin/python" nix/tests/arrays-cache-metadata-regression.py

Expected output:

arrays-cache metadata regression: PASS

For the processor test, first obtain the exact model revisions in the table
above. Set MODEL_ROOT below to an absolute local directory with EXO-discoverable
model directories, such as mlx-community--Qwen3.8-27B-8bit and
mlx-community--Qwen3.8-27B-bf16. The test itself is not a model-download script.
Both Q8 and BF16 loader invocations passed on the public-source contribution.

MODEL_ROOT="/absolute/path/to/local-models"
EXO_TEST_DASHBOARD="$(nix build --no-link --print-out-paths path:.#dashboard)"
(
  set -e
  export HOME="$(mktemp -d)"
  unset EXO_HOME EXO_MODELS_READ_ONLY_DIRS PYTHONPATH PYTHONOPTIMIZE
  export EXO_DEFAULT_MODELS_DIR="$HOME/models"
  export EXO_MODELS_DIRS="$MODEL_ROOT"
  export EXO_RESOURCES_DIR="$PWD/resources"
  export EXO_DASHBOARD_DIR="$EXO_TEST_DASHBOARD"
  printf 'Isolated test home: %s\n' "$HOME"
  "$EXO_TEST_PYTHON_ENV/bin/python" nix/tests/qwen-vision-regression.py \
    mlx-community/Qwen3.8-27B-8bit
  "$EXO_TEST_PYTHON_ENV/bin/python" nix/tests/qwen-vision-regression.py \
    mlx-community/Qwen3.8-27B-bf16
)

The subshell isolates EXO's user state and restores the caller's environment on
exit. Its temporary home is retained for inspection. EXO_MODELS_DIRS supplies
the local discovery path used by this loader check; clearing the read-only list
is confined to the test process. These are not recommended service settings,
and no model deletion or placement operation is performed by either script.

Expected processor output, one line per completed invocation:

Qwen vision loader regression: PASS (mlx-community/Qwen3.8-27B-8bit)
Qwen vision loader regression: PASS (mlx-community/Qwen3.8-27B-bf16)

An exception or nonzero exit is a failed check. A growing-metadata assertion
suggests checking whether the selected build actually includes the cache patch;
missing vision metadata suggests checking the model path and complete download.
A processor-resolution exception requires investigation with the exact dependency
versions. Do not treat any of these as a passing skip.

These component checks do not prove sustained generation, model-card registration,
or multimodal inference. Separate registration coverage is in
src/exo/shared/tests/test_qwen38_model_cards.py. Its native-suite invocation and
the opt-in four-node request/replay instructions are in
nix/tests/README.qwen38.md. The request generator does not place models or
verify topology; those are explicit manual prerequisites.

Recorded Results

• Real Q8 vision-loader regression: failed on baseline, passed with the
  processor correction using the same dependencies and model files.
• Historical combined development build: 472 tests passed, 3 skipped, 190 deselected;
  strict typecheck reported zero errors and warnings; repository lint passed.
  Formatting was applied in the native validation copy.
• Fresh public-source checks: Nix runtime, dashboard and test environment built;
  Q8 and BF16 real loaders passed; cache regression passed; both registration
  cases passed. Native suite: 471 passed, 3 skipped, 193 deselected. Strict
  typecheck: zero errors/warnings; Ruff passed; nix fmt applied.
• Native tests used the Nix test environment and explicit src bench tools
  targets with a private HOME and EXO_HOME unset. A bare pytest invocation hit
  upstream's separate cluster-harness import (exo_tools); that harness was
  not run. The source-suite approach follows upstream's macOS CI route.
• These are local checks, not GitHub CI results. Four-node sustained generation
  on this upstream port and human code/description review remain pending.

Attribution And AI Disclosure

• EXO contributors authored the existing vision integration, loader paths and
  model-card infrastructure reused by this proposal.
• The Transformers and MLX-VLM contributors provide the model/configuration
  mappings and image-processing implementations reused here. No new image
  processor implementation is claimed.
• The separate cache investigation benefited from Luka Afonin's
  MLX-LM #1632, the maintainer
  revision by zcbenz, thinkroth's analysis in
  #1641 and
  #1642, Pierre Lamy's
  bb615eb patch,
  and aidiffuser's subsequent batch-path report
  #1845. The adapted cache
  mechanism is their prior work, not an invention of this submission. Its source
  correction belongs in MLX-LM; an EXO dependency carrier must preserve that
  attribution. These references do not imply
  their authors reviewed or endorsed this PR.
• This work was conducted with GitHub Copilot under Mark Smeltzer's direction.
  AI assistance included source and prior-art analysis, implementation and
  regression drafting, command execution, interpretation of captured test
  results, and preparation of this description. The reported tests were
  executed by the agent against real software and hardware using deterministic
  scripts.

Related issue: #2302