diff --git a/CLAUDE.md b/CLAUDE.md index 3ec3b1e..40ed382 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -42,8 +42,12 @@ runs fully without ever signing in — community features are the only auth-gate surface. Two ways to classify: -- **AI Config mode** (no local model active): send the cropped image to an - OpenAI-compatible HTTP server (`/v1/chat/completions`). +- **Over HTTP** (`/v1/chat/completions` on an OpenAI-compatible server) — two + spellings of the same backend: **AI Config mode** (no active model, one + app-level config) and an active **openai-mode model** (`model_mode = + "openai"`), which carries its *own* `AIModelConfig` and headstamp list so + several such models can coexist, exactly as the Windows app's "OpenAI API" + Training Mode does. - **Local model mode**: run a PyTorch **ConvNeXt** model locally. The model can be one the user trained on the Train page, a pretrained model downloaded from the community, or one imported from a ZIP — running locally does **not** require @@ -234,7 +238,13 @@ sanctioned way for worker threads to update the UI. `ensure_initialized()` creates the DB, runs a one-shot import from legacy `data/config.json` (renaming it `.bak`), or seeds a default cartridge+model. Tables: `cartridges`, `models`, `headstamp_parents`, `headstamps`, - `slot_templates`, `settings`. + `slot_templates`, `settings`. One structural fix lives *outside* the + ladder: `_widen_model_mode_check` rebuilds `models` when its mode CHECK + predates `'openai'` — a CHECK can't be ALTERed, and the rebuild needs + `PRAGMA foreign_keys` toggled, which is a silent no-op inside the + transaction every ladder step runs in (with FKs on, `DROP TABLE models` + would cascade-delete every headstamp). Guarded structurally off + `sqlite_master`, like the DDL pass. - **`repository.py`** — `CartridgeRepo`, `ModelRepo`, `HeadstampRepo`, `HeadstampParentRepo`, `SlotTemplateRepo`, `SettingsRepo`. All SQL is **parameterized**. `SettingsRepo` is a typed key/value store (JSON-encoded @@ -248,12 +258,99 @@ sanctioned way for worker threads to update the UI. and the sorting-template API (see below). - **`models.py`** — dataclasses: `Model`, `Headstamp`, `Cartridge`, `SlotTemplate`, `TrainingConfig`, `AIModelConfig`, `ImageProcessingConfig`, plus normalizers - (`normalize_upload_mode`, `SUPPORTED_MODEL_MODES`, `SLOT_TEMPLATE_MODES`). + (`normalize_upload_mode`, `SLOT_TEMPLATE_MODES`) and the mode/ownership + vocabulary: `SUPPORTED_MODEL_MODES` (the trainable ConvNeXt backbones — + `train_page` assigns a mode straight into `training_config.model_name`, so + `"openai"` must never join this tuple), `OPENAI_MODEL_MODE` and + `MODEL_MODES` (what `ModelRepo` accepts), `is_openai_model`, and + `model_mode_label` — the user-facing spellings ("ConvNeXt-Tiny", + "OpenAI", the Windows app's Training Mode names) that every UI surface + prints while storage keeps the snake_case identifiers; the editor combo + carries the identifier as item *data*, and `_normalize_model_mode` + accepts the labels back so one leaking into a manifest still round-trips. - **`model_io.py`** (`sorter/data/model_io.py` — grouped with the rest of persistence, not a separate layer: it's a model persisted to a ZIP instead of SQLite) — model **ZIP** import/export; see the *Training & evaluation* entry below for what it does, kept there to stay next to the training workflow it feeds. +- **`winforms_import.py`** — one-shot import of an existing **WinForms ("AI + Brass Sorter") installation**, so a user moving off the Windows app doesn't + rebuild their setup by hand (#98). The legacy app keeps everything in its + *install directory* — `Data/ConfigDB.sjdb.json` (the whole database as one + JSON document, BOM-prefixed), `Data/Settings.json`, `training/images//` + and `training/models/.zip` — and **nothing in the registry**: + `HKCU\Software\AICaseSorter` exists but is empty, the only value under + `HKCU\Software\SJSeth\...` is an MSI-authored `DesktopFolder`, and the + uninstall entry's `InstallLocation` is blank, so a custom install is found + by asking the user, not by reading a key. + `survey()` reports what a root offers without importing anything (it is what + populates the dialog's counts and what keeps the first-run offer silent); + `import_installation()` does the work, per ticked item. + - **The selection is per model, not per category.** `ImportOptions.per_model` + maps a legacy model id to a `ModelSelection` (images / headstamps / + checkpoint), and `selection_for()` is the single place that resolves "what + do I bring for this model" — a real install holds years of models the user + doesn't want (sjseth on #125). Three states, and the difference is + load-bearing: **`None`** means no per-model choice was made, so every + surveyed model comes with the app-level flags (what every pre-tree caller + and every default `ImportOptions()` gets); **`{}`** means no models at all; + a populated map is the answer. The model's **row** is deliberately not a + selectable part — the images land in its folder, the headstamps hang off it, + the checkpoint is recorded on it — which is the inheritance the dialog's + tree makes structural instead of a rule (§5). + - **Model ids never collide, names can.** `ModelRepo.create` allocates the + rowid, so the legacy id survives only as a lookup key (`training/images/` + and the `winforms_imported_models` map) and an id conflict with a local + model is impossible by construction. The *name* is the one real conflict, so + a newly-created row goes through `model_io.unique_model_name` — the same + resolution the ZIP path has always applied, made public for this caller. + An update (UID or remembered pairing) keeps the local name, so re-running + the import stays idempotent rather than growing `(2)`s. + - **`survey(root, db=...)`** additionally resolves, per model, whether + importing it would create a row or refresh one (`LegacyModel.updates`), + via the same `_find_existing` the import uses. Asked early purely so the + dialog can say so before the user commits; without a `db` it stays None. + Two things it leans on and one it must not: + - Legacy `Models` rows are **the same PascalCase shape** as an export ZIP's + `ModelInfo`, so they go straight to `model_io.model_from_export_dict` — + `ModelType`/`ModelMode` int mapping and all — rather than being re-parsed. + `ModelType` 1/2 therefore lands as `ReadOnly`/`CommunityManaged`, i.e. a + community model stays non-trainable exactly as a download here would. + **This module is the only caller of `model_from_export_dict` with no clamp + of its own**, so every value in `_WINFORMS_MODELMODE_INT_TO_STR` has to be + a mode `ModelRepo` accepts — `test_model_io.py` pins that. `ModelMode` 2 + (OpenAI) maps to `"openai"`, a first-class mode here too, so the row + imports faithfully — its own `AIModelConfig` and headstamps included — + and needs no warning. The AI Config item still seeds the app-level + config, preferring the OpenAI-mode model's blob: the legacy app writes + one on every model and most are blank, so "first non-empty" picked the + wrong one. + - **`training/models/.zip` is a `torch.save` archive, not a ZIP of + anything** — it copies to `.pth` verbatim. The legacy **ML.NET** + pipeline writes its models beside it under the same extension, so + `_checkpoint_kind` looks inside (`*/data.pkl` = torch, + `TransformerChain/` = ML.NET) before copying. An ML.NET-only model is + imported as a **shell** — metadata, headstamps and images, no checkpoint — + because the images are the expensive part and the `NoLocalCheckpointError` + path already explains a model that can't classify yet. That warning lives + on `LegacyModel.warning`, not in an install-wide list + (`LegacySurvey.warnings` derives from the models), so the *survey* reports + on the whole install — which is what informs the pick — while the *import* + reports only on the models the user actually took. + - **Never destructive to the source.** Files are copied; nothing in the + install directory is written, moved or removed. Re-running is idempotent: + a community UID match or the per-root `winforms_imported_models` settings + map updates the row in place, so slot assignments and templates survive + and images already copied are skipped. + - **One bad row costs that row.** Each model imports inside its own nested + `db.transaction()` (a SAVEPOINT), counted into a scratch `ImportResult` + merged only on success, so a legacy row this app refuses is skipped with a + warning instead of rolling back an install's worth of images. + Slot assignments are **inverted on the way in** — the legacy DB stores a slot + listing its headstamps (`SlotConfigs[].Config`), ours stores a slot on the + headstamp row. `Defaults.IP_*` maps to `image_proc.linescan` **only**: the + legacy pipeline has no Hough stage, so writing its numbers into ours would + silently detune a working crop. ### Filesystem (`sorter/paths.py` — top level, not under `data/`) - **`paths.py`** — single source of truth for the on-disk layout (see §6) and @@ -279,10 +376,12 @@ sanctioned way for worker threads to update the UI. ### Active-model concept "Active model" = `settings.default_model_id`. When **absent**, the app is in -**AI Config mode** (cloud HTTP classification, headstamps in a settings key). -When **set**, that local model is active (Train live, local inference used, -headstamps in the `headstamps` table). Activating a model posts -`mode/changed`, which is what re-evaluates the mode pair (§5). +**AI Config mode** (HTTP classification via the app-level `config.api`, +headstamps in a settings key). When **set**, that model is active with its +headstamps in the `headstamps` table — a ConvNeXt model classifies locally +(Train live); an **openai-mode** model classifies over HTTP using its own +`ai_model_config` (AI Config live, editing that model's settings). Activating +a model posts `mode/changed`, which is what re-evaluates the mode pair (§5). ### Sorting templates A **sorting template** is a named snapshot of the Sort page's slot assignments, so @@ -382,8 +481,11 @@ between them from the Sort page's template dropdown. ### Classification (`sorter/ml/`) - **`classifier.py`** — `classify_active`: **the active model alone picks the - backend.** A model is active → local inference; AI Config mode (no active - model) → HTTP. Passes the trained `image_size` through. A local model whose + backend.** A ConvNeXt model is active → local inference; an openai-mode + model is active → HTTP with **that model's own** `ai_model_config` (the + passed app-level `api_cfg` is deliberately ignored there); AI Config mode + (no active model) → HTTP with the app-level config. Passes the trained + `image_size` through. A local model whose checkpoint is missing raises `NoLocalCheckpointError` — it does **not** degrade to HTTP. That fallback existed and was a trap: a renamed data folder or an images-only community share left `model_path` unusable and the app @@ -391,7 +493,9 @@ between them from the Sort page's template dropdown. surfacing only as a connection error naming a host the user wasn't knowingly using. Switching backends is the user's call, on the Models page. `active_model` / `uses_local_inference` / `has_local_checkpoint` / `checkpoint_problem` - expose the decision alone, so the UI can ask "does this need PyTorch?" and + expose the decision alone (`uses_local_inference` and `checkpoint_problem` + are both False/None for an openai model — no PyTorch, no checkpoint to + miss), so the UI can ask "does this need PyTorch?" and "can this model actually classify?" before starting a run — keep them in lock-step with `classify_active` or the install gate (§5) drifts from reality. `checkpoint_problem` also asks `torch_floor_problem`: a model records the @@ -589,9 +693,12 @@ runs the bus drain loop. `run_worker(fn, on_done, on_error)` is the standard helper for offloading blocking work to a thread and marshaling the result back through the bus. -**Neither of the mode pair is ever hidden**, and exactly one is *live*: -Train ⟺ `models.is_trainable(active model)`, AI Config ⟺ no active model — -so a community model leaves neither live. The other gets +**Neither of the mode pair is ever hidden**, and at most one is *live*: +Train ⟺ `models.is_trainable(active model)` (False for community *and* for +openai-mode models), AI Config ⟺ no active model **or an active openai-mode +model** (both classify over HTTP; the page's server fields bind to whichever +config is in effect via `AiSection.retarget()`, so an openai model's settings +are edited on its own row) — a community model leaves neither live. The other gets `_set_activity_unavailable`, which sets the dynamic property `unavailable` on the button — restyled `text_subtle` by `ui/theme.py` and re-inked by `_paint_sidebar_icon`, since a stylesheet can't reach a QIcon — and leaves @@ -669,7 +776,7 @@ modal), and never gate on `is_available()`. | **Train** | `train_page.py` | Feed→capture→classify→label→save loop; "Sort While Training"; launches training. | | **AI Config** | `ai_page.py` | HTTP server config (endpoint/key/model/prompt/encoding), headstamp manager, single-shot test. | | **Community** | `community_page.py` | Browse/search/download community models; share entry point. Auth-gated. | -| **Settings** | `settings_{camera,serial,imageproc}.py` + `app.py`'s Theme section | Camera, Serial, Image Processing, Theme — listed in `SETTINGS_SECTIONS`, reached by name. | +| **Settings** | `settings_{camera,serial,imageproc}.py` + `app.py`'s Theme section + `dialog_winforms_import.py` | Camera, Serial, Image Processing, Theme, Import from Windows — listed in `SETTINGS_SECTIONS`, reached by name. | Docks: `serial_monitor.py`, `history_view.py`, `help_viewer.py`, and the Themes panel in `app.py`. Dialogs are `dialog_*.py`. @@ -724,6 +831,19 @@ Themes panel in `app.py`. Dialogs are `dialog_*.py`. shipped and then reverted: JL lived with them and chose the bar. Don't reintroduce item widgets in these tables — `_pin_ai_row` and every sort destroy them, which is machinery the bar simply doesn't need. +- **One checkable tree, and it is a picker, not a table.** + `dialog_winforms_import.py`'s `QTreeWidget#importTree` is the only tree in the + app whose items carry check state, which is why `theme.py` needs an + `::indicator` block keyed on it (the `QCheckBox::indicator` rules can't reach + an item view's own indicator). It is not an exception to the bar convention + above: check state is item *data*, not an embedded widget, so nothing is + destroyed by a rebuild. The hierarchy carries meaning — a model's images, + headstamps and checkpoint are its children because they cannot exist without + it — and check propagation is manual (`_set_branch` down, `_refresh_ancestors` + up, both under `blockSignals`) rather than `ItemIsAutoTristate`, so exactly + one place decides what a half-ticked parent means. Rows with nothing behind + them are **omitted**, not disabled, so propagation never has to reason about + a child the user can't reach. - **The notify/confirm seam.** Anything that would open a native modal — `win.notify`, a page's `confirm` / `ask_text` / `ask_open_path` / `ask_save_path` / `ask_import_choice` — is an **instance attribute**, not a diff --git a/docs/guide/GUIDE.md b/docs/guide/GUIDE.md index 3d3257e..aa0a9fa 100644 --- a/docs/guide/GUIDE.md +++ b/docs/guide/GUIDE.md @@ -19,7 +19,8 @@ day, in the order you meet them. - [Models](#models) — the model library: activate, edit, import, export. - [Community](#community) — browse and install published models. - [Settings](#settings) — [Camera](#camera), [Serial](#serial), [Image - Processing](#image-processing) and [Theme](#theme). + Processing](#image-processing), [Theme](#theme) and [Import from + Windows](#import-from-windows). - [Getting help](#getting-help) — this guide, the [support package](#support-package) and [updates](#updates). @@ -428,14 +429,27 @@ here. The alternative to a local model: classification is sent to an OpenAI-compatible HTTP server, and this screen is where that server — and the headstamps it may answer with — is set up. It is Train's mirror in the -sidebar: live whenever no local model is active, dimmed when one is. +sidebar: live whenever classification runs over HTTP, dimmed when a local +ConvNeXt model does the work. + +Two things can put classification on HTTP, and this page serves both: + +- **AI Config mode** — no active model at all. The screen edits the + app-level server settings. +- **An active OpenAI model** — a model whose training mode is **OpenAI** on the + [Models](#models) page. Each OpenAI model carries its **own** server + settings and its own headstamp list, so several can coexist — different + cartridges, prompts, even different providers. While one is active this + screen edits *that model's* settings, and a caption above the form says + so by name. This mirrors the Windows app, where "OpenAI API" is a + Training Mode and its configuration lives on the model. ### When AI Config isn't the one classifying -Activate a local model and this screen swaps its form for a panel naming the -model that is classifying instead, with a button straight to the -[Models](#models) page. Select **Use AI Config** there to come back — the -server settings are still exactly as you left them. +Activate a local ConvNeXt model and this screen swaps its form for a panel +naming the model that is classifying instead, with a button straight to the +[Models](#models) page. Select **Use AI Config** — or an OpenAI model — +there to come back; the server settings are still exactly as you left them. ### Setting up the server @@ -459,9 +473,12 @@ this table, plus one synthetic row for AI Config mode. ### The model list The table lists each model's name, whether it is active, its cartridge, type -(yours or a community model), the ConvNeXt size it was built as, how many -training images it has, whether it has been trained, and when. Click a -column heading to sort by it. +(yours or a community model), the mode it was built as — a ConvNeXt size, or +**OpenAI** for a model that classifies over an HTTP server — how many training +images it has, whether it has been trained, and when. Click a column heading +to sort by it. An OpenAI model has nothing to train: activate it and the +[AI Config](#ai-config) page becomes the place its server settings and +headstamps live. The **Active** column marks the model the app currently classifies with: exactly one row reads **● ACTIVE** in the theme's action colour, and every @@ -639,6 +656,84 @@ the editor open so you can keep adjusting — **Close** ends the session; overwritten either way. A theme can also be exported to a file and imported on another machine. +### Import from Windows + +Copies your setup out of an installation of the Windows app (**AI Brass +Sorter**) so you do not have to build it again here. **Nothing in the Windows +app is changed, moved or deleted** — everything is copied, and that +installation keeps working exactly as it did. + +If the Windows app is installed in the usual place, this page finds it on its +own; otherwise choose the folder yourself — the one containing `Data` and +`training`. The same offer appears once, automatically, the first time you +start this app on a computer that has the Windows app on it. + +You tick what comes across, in a tree: + +- **Models** — with one branch per model in the Windows app. Under each model: + - **Training images** — usually the bulk of the data, and the slowest part + of the copy. + - **Headstamps and slot assignments** — that model's headstamp list, parent + classifications, and which bin each one drops into. + - **Trained model file** — its size is shown, because this is the big one. + Take it and the model classifies immediately; leave it and the model comes + across as a shell you retrain on the [Train](#train) page. +- **Image-processing settings** — the crop tuning from the Windows app. +- **Serial / board settings** — port, baud rate and the board's init values. +- **AI Config** — endpoint, model and prompt, if you classify over HTTP. + +**Pick the models you actually want.** A Windows install that has been in use +for a while usually holds models you have no interest in carrying forward, so +every model is its own tick. **Select all** / **Select none** are there so +choosing two out of fifteen is two clicks rather than thirteen, and the line +under the tree totals what you have chosen — models, images and trained model +files — before you start. + +Images, headstamps and the trained model file all belong to a model, so they sit +*under* it: untick a model and its whole branch goes with it. There is no way to +bring a model's images across without the model itself, because they land in its +folder. A model showing a half-filled tick is one where you have kept some parts +and not others. + +Anything this installation does not have is left out or greyed out rather than +offered as a tick that would import nothing — a model with no images has no +**Training images** row at all. + +Each model's row also says what importing it would **do to your library here**: +either *new model here*, or *updates '…'* naming the model it would refresh +instead of duplicating. + +A few things are worth knowing before you run it: + +- **Models trained with the Windows app's older ML.NET pipeline cannot + classify here.** They are still imported — with their headstamps and images + — but without a trained model file, so the [Train](#train) page is where + you pick them up. Their row says *ML.NET model, retrain needed* and has no + **Trained model file** to tick, so you can decide before you import; the + summary at the end names the ones you did import. +- **A model that came from the [Community](#community) stays read-only**, the + same as one downloaded here: the trained model file belongs to whoever + published it, so it is not trainable. Your own models stay trainable. +- **A Windows model set to classify over an OpenAI server imports as an + OpenAI model here too**, keeping its own endpoint, prompt and headstamps. + Activate it and the [AI Config](#ai-config) page edits its settings. +- **One model that cannot be imported does not stop the rest.** It is + skipped, the reason is listed at the end, and everything else still lands. + +Running the import a second time is safe. A model already brought across is +updated in place rather than duplicated, your slot assignments and sorting +templates survive, and images already copied are skipped. The import never +takes over an active model you have already chosen here. + +That also means you can import in passes — bring two models over, sort with +them, then come back for more. And a model whose trained model file you declined +the first time gets it on a later run if you tick it then. + +If a Windows model happens to share its name with a model you already have here, +the imported one is named `… (2)` rather than leaving you with two rows you +cannot tell apart. Your existing model is not touched. + + ## Getting help ### The guide diff --git a/src/sorter/data/db.py b/src/sorter/data/db.py index b8e9abe..d89b3c4 100644 --- a/src/sorter/data/db.py +++ b/src/sorter/data/db.py @@ -64,22 +64,18 @@ CREATE INDEX IF NOT EXISTS idx_slot_templates_scope ON slot_templates(model_id, mode); """ -SCHEMA_DDL = ( - """ -PRAGMA foreign_keys = ON; -PRAGMA journal_mode = WAL; - -CREATE TABLE IF NOT EXISTS cartridges ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - name TEXT NOT NULL UNIQUE -); - +# Split out of SCHEMA_DDL for the same reason as SLOT_TEMPLATES_DDL: the +# `_widen_model_mode_check` rebuild replays it (under a scratch table name), so +# one copy means the rebuilt table cannot drift from the schema. The mode CHECK +# stays in lock-step with models.MODEL_MODES — 'openai' classifies over an +# OpenAI-compatible HTTP server, the rest are trainable ConvNeXt backbones. +MODELS_DDL = """ CREATE TABLE IF NOT EXISTS models ( id INTEGER PRIMARY KEY AUTOINCREMENT, name TEXT NOT NULL, cartridge_id INTEGER NOT NULL REFERENCES cartridges(id) ON DELETE RESTRICT, model_mode TEXT NOT NULL - CHECK(model_mode IN ('convnext_tiny','convnext_small','convnext_base','convnext_large')), + CHECK(model_mode IN ('convnext_tiny','convnext_small','convnext_base','convnext_large','openai')), model_type TEXT NOT NULL DEFAULT 'Standard' CHECK(model_type IN ('Standard','ReadOnly','CommunityManaged')), community_model_uid TEXT, @@ -104,7 +100,20 @@ updated_at TEXT NOT NULL DEFAULT (datetime('now')) ); CREATE INDEX IF NOT EXISTS idx_models_cartridge ON models(cartridge_id); +""" + +SCHEMA_DDL = ( + """ +PRAGMA foreign_keys = ON; +PRAGMA journal_mode = WAL; +CREATE TABLE IF NOT EXISTS cartridges ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + name TEXT NOT NULL UNIQUE +); +""" + + MODELS_DDL + + """ -- Parent classifications: named groups that child headstamps roll up into. -- Scoped per-model. CREATE TABLE IF NOT EXISTS headstamp_parents ( @@ -378,6 +387,10 @@ def ensure_initialized(self, legacy_config_json: Path | None = None) -> None: # DDL is idempotent (IF NOT EXISTS) so re-running on an existing DB is safe. _execute_script(conn, SCHEMA_DDL) self._migrate_schema() + # After the ladder, not in it: the rebuild needs `PRAGMA foreign_keys` + # toggled, which is a silent no-op inside any transaction — and every + # ladder step runs inside one (see _migrate_schema). + self._widen_model_mode_check() if was_fresh: if legacy_config_json and Path(legacy_config_json).exists(): @@ -385,6 +398,54 @@ def ensure_initialized(self, legacy_config_json: Path | None = None) -> None: else: self._seed_defaults() + def _widen_model_mode_check(self) -> None: + """Rebuild `models` when its mode CHECK predates the 'openai' mode. + + SQLite cannot ALTER a CHECK constraint, so widening it is the + documented table-rebuild dance: create the current-DDL table under a + scratch name, copy every row, drop the old table, rename. Foreign + keys are switched off around it — with them on, `DROP TABLE models` + performs an implicit `DELETE FROM models`, and the children's + `ON DELETE CASCADE` would take every headstamp with it. + + The guard is structural, like the DDL pass: the table's own + `sqlite_master` text says whether 'openai' is already admitted, so + this is idempotent with no ladder bookkeeping — a fresh database (or + one already rebuilt) is a no-op, and there is deliberately nothing + here that can run twice destructively. `foreign_key_check` before + commit turns a botched copy into a rollback instead of a corrupt DB. + """ + conn = self.conn + row = conn.execute("SELECT sql FROM sqlite_master WHERE type='table' AND name='models'").fetchone() + sql = (row[0] or "") if row is not None else "" + if not sql or "'openai'" in sql or "model_mode" not in sql or "CHECK" not in sql.upper(): + return + + conn.execute("PRAGMA foreign_keys = OFF") + try: + with self.transaction(): + conn.execute("DROP TABLE IF EXISTS models_rebuild") + _execute_script( + conn, + MODELS_DDL.replace("CREATE TABLE IF NOT EXISTS models (", "CREATE TABLE models_rebuild (").replace( + "CREATE INDEX IF NOT EXISTS idx_models_cartridge ON models(cartridge_id);", "" + ), + ) + # Copy by the shared column set: `_migrate_schema` has already + # run, so a healthy table matches the DDL exactly — the + # intersection is belt-and-braces against a shape it missed. + shared = sorted(_columns(conn, "models") & _columns(conn, "models_rebuild")) + cols = ", ".join(shared) + conn.execute(f"INSERT INTO models_rebuild ({cols}) SELECT {cols} FROM models") + conn.execute("DROP TABLE models") + conn.execute("ALTER TABLE models_rebuild RENAME TO models") + conn.execute("CREATE INDEX IF NOT EXISTS idx_models_cartridge ON models(cartridge_id)") + problems = conn.execute("PRAGMA foreign_key_check").fetchall() + if problems: + raise RuntimeError(f"models rebuild broke {len(problems)} foreign key reference(s)") + finally: + conn.execute("PRAGMA foreign_keys = ON") + # ----- migration ---------------------------------------------------------- def _migrate_schema(self) -> None: diff --git a/src/sorter/data/model_io.py b/src/sorter/data/model_io.py index f86bf1b..f1c009c 100644 --- a/src/sorter/data/model_io.py +++ b/src/sorter/data/model_io.py @@ -38,7 +38,8 @@ from .. import paths from .models import ( - SUPPORTED_MODEL_MODES, + MODEL_MODES, + OPENAI_MODEL_MODE, AIModelConfig, CheckpointEnv, Headstamp, @@ -105,10 +106,14 @@ def model_to_export_dict(m: Model) -> dict[str, Any]: # The legacy app serialises its ModelMode enum as its integer value. Map back # to our snake_case backbone identifiers; non-ConvNeXt modes fall through to a # ConvNeXt this app can actually run. +WINFORMS_MODELMODE_OPENAI = 2 + _WINFORMS_MODELMODE_INT_TO_STR = { 0: "convnext_tiny", # DeepLearning (ResNet50) — can't run, fall back 1: "convnext_tiny", # Inception — fall back - 2: "openai", # OpenAI + # "OpenAI API" is a first-class mode here too (PR #125 review): the row + # keeps its own AIModelConfig and classifies over HTTP when active. + WINFORMS_MODELMODE_OPENAI: OPENAI_MODEL_MODE, 3: "convnext_large", 4: "convnext_tiny", # DeeperLearning (ResNet101) — fall back 5: "convnext_tiny", # Custom — fall back @@ -119,19 +124,23 @@ def model_to_export_dict(m: Model) -> dict[str, Any]: def _normalize_model_mode(raw: Any) -> str: - """Accept the snake_case string or the legacy enum int.""" + """Accept the mode string or the legacy enum int. + + Every branch lands in `MODEL_MODES`: `ModelRepo` rejects anything else, + and `winforms_import` hands the result straight to it. + """ if isinstance(raw, str): rl = raw.strip().lower() # Tolerate hyphenated/CamelCase variants # (`ConvNeXt-Tiny`, `convnext_tiny`, `ConvNeXtTiny`). rl = rl.replace("-", "_").replace(" ", "_") - if rl in SUPPORTED_MODEL_MODES: + if rl in MODEL_MODES: return rl # ConvNeXtTiny → convnext_tiny if rl.startswith("convnext") and not rl.startswith("convnext_"): tail = rl[len("convnext") :] candidate = f"convnext_{tail}" - if candidate in SUPPORTED_MODEL_MODES: + if candidate in MODEL_MODES: return candidate return "convnext_tiny" if isinstance(raw, int): @@ -361,8 +370,13 @@ def export_for_share( return zip_path, manifest_path -def _unique_model_name(base: str, repo: ModelRepo) -> str: - """Append (n) until the name is unique across all models.""" +def unique_model_name(base: str, repo: ModelRepo) -> str: + """Append (n) until the name is unique across all models. + + Public because `winforms_import` needs the same answer: importing a Windows + install onto a library that already holds a "9mm" must not leave the user + with two rows they cannot tell apart. + """ existing = {m.name.lower() for m in repo.list()} if base.lower() not in existing: return base @@ -504,7 +518,7 @@ def import_model( manifest = json.loads(zf.read(manifest_entry).decode("utf-8")) model = model_from_export_dict(manifest.get("ModelInfo") or {}) - if model.model_mode not in SUPPORTED_MODEL_MODES: + if model.model_mode not in MODEL_MODES: model.model_mode = "convnext_tiny" # Ownership is decided by how the archive reached this machine, not by @@ -548,7 +562,7 @@ def import_model( model_repo.update(saved) else: desired_name = model_name_override or model.name or "Imported" - model.name = _unique_model_name(desired_name, model_repo) + model.name = unique_model_name(desired_name, model_repo) saved = model_repo.create(model) # Decide target directories (per-model `/images` and diff --git a/src/sorter/data/models.py b/src/sorter/data/models.py index 119b772..b183b89 100644 --- a/src/sorter/data/models.py +++ b/src/sorter/data/models.py @@ -20,6 +20,39 @@ "convnext_large", ) +# A model that classifies over an OpenAI-compatible HTTP server instead of a +# local checkpoint. The legacy app treats "OpenAI API" as a Training Mode peer +# of the ConvNeXt sizes — several such models can coexist, each with its own +# cartridge, headstamp list and `AIModelConfig` (endpoint/key/model/prompt) — +# and this app mirrors that (PR #125 review). Deliberately NOT added to +# `SUPPORTED_MODEL_MODES`: that tuple doubles as the list of trainable +# backbones (`train_page` assigns `model_mode` straight into +# `training_config.model_name`), and an openai model has nothing to train. +OPENAI_MODEL_MODE = "openai" + +# Every mode a model row may persist: the trainable backbones plus openai. +# `ModelRepo` validates against this, not `SUPPORTED_MODEL_MODES`. +MODEL_MODES = (*SUPPORTED_MODEL_MODES, OPENAI_MODEL_MODE) + +# What the UI prints for a mode. Storage keeps the snake_case identifiers +# (they are also the torchvision backbone names the trainer passes through); +# these are the user-facing spellings, matching the Windows app's Training +# Mode dropdown. `model_io._normalize_model_mode` accepts the labels back, +# so one leaking into a manifest still round-trips. +MODEL_MODE_LABELS = { + "convnext_tiny": "ConvNeXt-Tiny", + "convnext_small": "ConvNeXt-Small", + "convnext_base": "ConvNeXt-Base", + "convnext_large": "ConvNeXt-Large", + OPENAI_MODEL_MODE: "OpenAI", +} + + +def model_mode_label(mode: str) -> str: + """The user-facing spelling of a mode; unknown values print as stored.""" + return MODEL_MODE_LABELS.get(mode, mode) + + MODEL_TYPES = ("Standard", "ReadOnly", "CommunityManaged") FEEDBACK_UPLOAD_MODES = ("Instant", "OnRunComplete", "Manual") @@ -469,6 +502,16 @@ def is_foreign_model(model: Model | None) -> bool: return bool(model is not None and model.model_type in FOREIGN_MODEL_TYPES) +def is_openai_model(model: Model | None) -> bool: + """True when `model` classifies over an OpenAI-compatible HTTP server. + + Such a model has no checkpoint, needs no PyTorch, and carries its server + settings in its own `ai_model_config` — the AI Config page edits them + while the model is active. + """ + return bool(model is not None and model.model_mode == OPENAI_MODEL_MODE) + + def is_trainable(model: Model | None) -> bool: """Can this model be trained (and have training images added) locally? @@ -478,5 +521,8 @@ def is_trainable(model: Model | None) -> bool: model was built from, and the next published update would overwrite the result anyway. Users who want to build on someone else's model export it and import it back as their own. + + Also False for an openai-mode model, whatever its ownership: there is no + local checkpoint to train — the "model" is an HTTP server configuration. """ - return model is not None and not is_foreign_model(model) + return model is not None and not is_foreign_model(model) and not is_openai_model(model) diff --git a/src/sorter/data/repository.py b/src/sorter/data/repository.py index e03db78..600146d 100644 --- a/src/sorter/data/repository.py +++ b/src/sorter/data/repository.py @@ -13,8 +13,8 @@ from .db import Database from .models import ( + MODEL_MODES, SLOT_TEMPLATE_MODES, - SUPPORTED_MODEL_MODES, Cartridge, Headstamp, HeadstampParent, @@ -135,7 +135,7 @@ def count_in_cartridge(self, cartridge_id: int) -> int: # ---- write --------------------------------------------------------------- def create(self, model: Model) -> Model: - if model.model_mode not in SUPPORTED_MODEL_MODES: + if model.model_mode not in MODEL_MODES: raise ValueError(f"Unsupported model_mode: {model.model_mode!r}") cur = self.db.conn.execute( """ @@ -181,7 +181,7 @@ def create(self, model: Model) -> Model: def update(self, model: Model) -> None: if not model.id: raise ValueError("Cannot update a model with no id") - if model.model_mode not in SUPPORTED_MODEL_MODES: + if model.model_mode not in MODEL_MODES: raise ValueError(f"Unsupported model_mode: {model.model_mode!r}") self.db.conn.execute( """ diff --git a/src/sorter/data/winforms_import.py b/src/sorter/data/winforms_import.py new file mode 100644 index 0000000..d3be932 --- /dev/null +++ b/src/sorter/data/winforms_import.py @@ -0,0 +1,1077 @@ +"""One-shot import of an existing WinForms ("AI Brass Sorter") installation. + +The legacy Windows app keeps **everything in its own install directory** — there +is no per-user data folder and, despite appearances, nothing usable in the +registry (see `find_installation`). The layout, as shipped by the 1.3.8 MSI: + + /Data/ConfigDB.sjdb.json the whole database, one JSON document + /Data/Settings.json three app-level toggles incl. baud rate + /training/images// training images, `{label}__{ticks}.jpg` + /training/models/.zip the trained checkpoint for model + +`ConfigDB.sjdb.json` is a single object whose top-level keys are the tables: +`Models`, `Headstamps`, `Cartridges`, `HeadStampParents`, `HeadStampParentLinks`, +`SlotConfigs`, `SortingTemplates`, `CommunityNotes` and a `Defaults` blob holding +the app-level settings. Model rows use the **same PascalCase shape** as the +`ModelInfo` block of an exported model ZIP, which is why this module can hand +them straight to `model_io.model_from_export_dict` instead of re-parsing them. + +**`training/models/.zip` is not a ZIP of anything.** It is a raw +`torch.save` archive — which is itself a zip container — so it drops in as our +`.pth` with a plain copy. The legacy app also ships an **ML.NET** pipeline +whose models live beside it under the same `.zip` extension (a +`TransformerChain/` tree wrapping a frozen TensorFlow graph). Those cannot +classify here, so `_checkpoint_kind` looks inside before copying and a model +that only has an ML.NET checkpoint is imported as a **shell**: metadata, +headstamps and images come across, and the user retrains into it. Importing it +anyway is the point — the images are the expensive part, and our +`NoLocalCheckpointError` path already explains a model that cannot classify yet. + +Everything here is **read-only with respect to the source install**: files are +copied out, nothing is moved, renamed or deleted. The legacy app keeps working. +""" + +from __future__ import annotations + +import json +import logging +import os +import shutil +import sqlite3 +import zipfile +from collections.abc import Callable +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any + +from .. import paths + +# The two per-model settings keys are private to `config`, and imported rather +# than re-spelled here: a key written under a name `Config` does not read back +# is a silently-dropped import, which no test of this module would catch. +from .config import ( + _PACKAGE_SLOTS_KEY, + _USE_PARENT_RUNTIME_KEY, + DEFAULT_INIT_SETTINGS, + Config, +) +from .db import Database +from .model_io import WINFORMS_MODELMODE_OPENAI, model_from_export_dict, unique_model_name +from .models import AIModelConfig, Model +from .repository import ( + CartridgeRepo, + HeadstampParentRepo, + HeadstampRepo, + ModelRepo, + SettingsRepo, +) + +log = logging.getLogger(__name__) + +# Relative layout inside a legacy install root. +CONFIG_DB_NAME = "Data/ConfigDB.sjdb.json" +SETTINGS_NAME = "Data/Settings.json" +IMAGES_SUBDIR = "training/images" +MODELS_SUBDIR = "training/models" + +# Where the MSI puts the app. The registry is deliberately not consulted: the +# uninstall entry's `InstallLocation` is empty, `HKCU\Software\AICaseSorter` +# exists but holds no values at all, and the only thing under +# `HKCU\Software\SJSeth\...` is an MSI-authored `DesktopFolder` path. Nothing +# there records where the app was installed or anything a user would want back, +# so a custom install is handled by letting the user point at the folder. +_INSTALL_SUBPATH = ("SJSeth", "AI Brass Sorter") + +# Only these land in `data/models//images/`. The legacy app writes JPEGs; +# the rest are here because its own importer accepted them. +_VALID_IMAGE_EXTS = {".jpg", ".jpeg", ".png", ".bmp", ".webp"} + +# Which legacy model ids an install has already brought across, so a second run +# updates rather than duplicates. Keyed by install root so two machines' folders +# imported in turn cannot collide on legacy id. +_IMPORTED_MODELS_KEY = "winforms_imported_models" +# Set once the first-run offer has been made, whether accepted or declined. +FIRST_RUN_SEEN_KEY = "winforms_import_offered" + +# `Defaults.IP_*` → our app-level `image_proc.linescan`. There is deliberately +# no Hough mapping: the legacy pipeline has no Hough stage, so its numbers mean +# nothing to ours and inventing a conversion would silently detune a working +# crop. `strategy` is left alone for the same reason. +_LINESCAN_FROM_LEGACY = { + "scan_precision": ("IP_Resolution", int), + "scan_sensitivity": ("IP_Sensitivity", float), + "padding_pct": ("IP_Padding", int), + "bg_cliff": ("IP_BgCliff", int), +} + +# What the user is told about a model whose only checkpoint is the legacy +# ML.NET pipeline's. A named constant rather than an inline f-string so a test +# can assert the exact message instead of sniffing for a substring of it — +# CodeQL reads `"ML.NET" in some_string` as a hostname allowlist check +# (`.NET` parses as a TLD) and flags it as incomplete URL sanitization. +MLNET_WARNING = ( + "'{name}' has only an ML.NET checkpoint, which cannot classify here — " + "it is imported without one so you can retrain it." +) + +# One unreadable row costs that row, not the whole import. +MODEL_FAILED_WARNING = "Could not import '{name}': {error}" + +# Every item the user can tick, in the order the dialog shows them — roughly by +# how much work each one saves. +ITEM_MODELS = "models" +ITEM_IMAGES = "training_images" +ITEM_HEADSTAMPS = "headstamps" +ITEM_IMAGE_PROC = "image_processing" +ITEM_SERIAL = "serial" +ITEM_AI_CONFIG = "ai_config" + +# The three things one model can bring across, each tickable on its own. The +# model's **row** is deliberately not among them: the images land in its folder, +# the headstamps hang off it and the checkpoint is recorded on it, so there is +# no meaningful "import the images but not the model". That is the inheritance +# the dialog's tree makes visible — untick a model and its whole branch goes. +PART_IMAGES = "images" +PART_HEADSTAMPS = "headstamps" +PART_CHECKPOINT = "checkpoint" + + +@dataclass(frozen=True) +class ModelSelection: + """What to bring across for one legacy model.""" + + images: bool = True + headstamps: bool = True + checkpoint: bool = True + + +@dataclass +class ImportOptions: + """Which checkboxes were ticked. + + `training_images` and `headstamps` are meaningless without `models` — the + images land in a model's folder and the headstamps hang off a model row — + so `normalized()` folds them off rather than making the dialog police it. + + `per_model` is how the dialog says "these two models, and for that one skip + the 200 MB checkpoint". Leaving it **None** means every model the survey + found, each taking the three app-level flags above — which is what a caller + that doesn't care about per-model choice (and every pre-tree caller) gets. + An **empty** dict is a real answer, not a missing one: no models at all. + """ + + models: bool = True + training_images: bool = True + headstamps: bool = True + image_processing: bool = False + serial: bool = False + ai_config: bool = False + per_model: dict[int, ModelSelection] | None = None + + def normalized(self) -> ImportOptions: + if self.models: + return self + return ImportOptions( + models=False, + training_images=False, + headstamps=False, + image_processing=self.image_processing, + serial=self.serial, + ai_config=self.ai_config, + ) + + def selection_for(self, legacy_id: int) -> ModelSelection | None: + """What to bring across for one legacy model, or None to skip it.""" + if not self.models: + return None + if self.per_model is None: + # No per-model choice was made, so the app-level flags stand for + # every model. The checkpoint has no app-level flag of its own — + # before per-model selection existed it always came across. + return ModelSelection( + images=self.training_images, + headstamps=self.headstamps, + checkpoint=True, + ) + return self.per_model.get(legacy_id) + + def any_selected(self) -> bool: + n = self.normalized() + models_wanted = n.models and (n.per_model is None or bool(n.per_model)) + return any((models_wanted, n.image_processing, n.serial, n.ai_config)) + + +@dataclass +class LegacyModel: + """One `Models` row plus what the filesystem says about it.""" + + legacy_id: int + name: str + raw: dict[str, Any] + cartridge_name: str + image_count: int = 0 + checkpoint: Path | None = None + checkpoint_kind: str = "none" # "torch" | "mlnet" | "none" + headstamp_count: int = 0 + # The local model this one would refresh instead of creating, when `survey` + # was given a `db` to check against. None means "a new model" — and so does + # a survey taken without a db, which is why the dialog is the only caller + # that passes one. + updates: str | None = None + # What needs saying about this model, if anything. Held per model rather + # than in one install-wide list so a warning about a model the user chose + # not to import doesn't follow them into the completion summary. + warning: str = "" + + @property + def has_usable_checkpoint(self) -> bool: + return self.checkpoint_kind == "torch" and self.checkpoint is not None + + @property + def community_uid(self) -> str: + """The community UID on the legacy row, or "" for a purely local model. + + Legacy rows are PascalCase; the snake_case spelling is accepted for the + same reason `model_from_export_dict` accepts it — a manifest that has + been round-tripped through this app once. + """ + value = self.raw.get("CommunityModelUID") or self.raw.get("community_model_uid") + return str(value).strip() if value else "" + + @property + def checkpoint_bytes(self) -> int: + """Size of a usable checkpoint, for a dialog that has to justify a wait.""" + if not self.has_usable_checkpoint or self.checkpoint is None: + return 0 + try: + return self.checkpoint.stat().st_size + except OSError: + return 0 + + @property + def was_openai_mode(self) -> bool: + """True when the legacy row classified over HTTP rather than locally.""" + mode = self.raw.get("ModelMode") + if isinstance(mode, bool): # bool is an int; a True here is not mode 1 + return False + if isinstance(mode, int): + return mode == WINFORMS_MODELMODE_OPENAI + return isinstance(mode, str) and mode.strip().lower() == "openai" + + +@dataclass +class LegacySurvey: + """What an install root offers, without importing any of it. + + Built by `survey()` and handed to the dialog so each checkbox can say how + much it would bring across. `is_empty` is what keeps the first-run offer + silent for an install holding nothing. + """ + + root: Path + models: list[LegacyModel] = field(default_factory=list) + has_serial: bool = False + has_image_processing: bool = False + has_ai_config: bool = False + + @property + def warnings(self) -> list[str]: + """Everything worth saying about this install, one line per model. + + Derived from the models rather than accumulated alongside them, so a + caller importing a subset can filter to the ones it actually took. + """ + return [m.warning for m in self.models if m.warning] + + @property + def total_images(self) -> int: + return sum(m.image_count for m in self.models) + + @property + def total_headstamps(self) -> int: + return sum(m.headstamp_count for m in self.models) + + @property + def is_empty(self) -> bool: + return not (self.models or self.has_serial or self.has_image_processing or self.has_ai_config) + + +@dataclass +class ImportResult: + models_imported: int = 0 + models_updated: int = 0 + images_copied: int = 0 + headstamps_imported: int = 0 + parents_imported: int = 0 + slots_assigned: int = 0 + checkpoints_copied: int = 0 + serial_imported: bool = False + image_processing_imported: bool = False + ai_config_imported: bool = False + activated_model_id: int | None = None + warnings: list[str] = field(default_factory=list) + + +# ----- discovery -------------------------------------------------------------- + + +def candidate_install_dirs() -> list[Path]: + """Well-known places the MSI lays the app down, most likely first. + + `%ProgramFiles%` is read from the environment rather than hard-coded so a + machine with a relocated Program Files still resolves, and the 32-bit view + is included because the same MSI has shipped both ways. + """ + roots: list[Path] = [] + for var in ("ProgramFiles", "ProgramW6432", "ProgramFiles(x86)"): + value = os.environ.get(var) + if value: + roots.append(Path(value)) + if not roots: # non-Windows, or a stripped environment + roots = [Path("C:/Program Files"), Path("C:/Program Files (x86)")] + seen: set[Path] = set() + out: list[Path] = [] + for base in roots: + candidate = base.joinpath(*_INSTALL_SUBPATH) + if candidate not in seen: + seen.add(candidate) + out.append(candidate) + return out + + +def looks_like_installation(root: Path | str) -> bool: + """True when `root` holds the one file an import cannot proceed without.""" + try: + return (Path(root) / CONFIG_DB_NAME).is_file() + except OSError: + return False + + +def find_installation() -> Path | None: + """The first candidate that actually holds a config DB, or None. + + Returning None is the normal case — a user who never ran the Windows app + must never see any of this (issue #98's last acceptance criterion). + """ + for candidate in candidate_install_dirs(): + if looks_like_installation(candidate): + return candidate + return None + + +# ----- reading the legacy database ------------------------------------------- + + +def _read_json(path: Path) -> dict[str, Any]: + """Parse one of the legacy JSON files. + + `utf-8-sig`: the app writes these with a BOM, which plain `utf-8` would + leave stuck to the front of the first key. + """ + with open(path, encoding="utf-8-sig") as fh: + data = json.load(fh) + return data if isinstance(data, dict) else {} + + +def _rows(db: dict[str, Any], table: str) -> list[dict[str, Any]]: + value = db.get(table) + if not isinstance(value, list): + return [] + return [r for r in value if isinstance(r, dict)] + + +def _checkpoint_kind(path: Path) -> str: + """Classify a `training/models/.zip`. + + A `torch.save` archive holds `/data.pkl`; the legacy ML.NET + pipeline writes a `TransformerChain/` tree instead. Anything unreadable is + "none" — a corrupt file must not stop the rest of the import. + """ + try: + with zipfile.ZipFile(path) as zf: + names = zf.namelist() + except (OSError, zipfile.BadZipFile): + return "none" + for name in names: + if name.replace("\\", "/").rsplit("/", 1)[-1] == "data.pkl": + return "torch" + for name in names: + if name.replace("\\", "/").startswith("TransformerChain/"): + return "mlnet" + return "none" + + +def _count_images(directory: Path) -> int: + try: + return sum(1 for p in directory.iterdir() if p.is_file() and p.suffix.lower() in _VALID_IMAGE_EXTS) + except OSError: + return 0 + + +def _legacy_ai_config(row: dict[str, Any]) -> dict[str, Any] | None: + """One model's `AIModelConfig`, or None when there is nothing worth taking. + + "Present" is not enough: the legacy app writes the blob on every model, so + an install's first row is usually an all-blank one. Treating that as the + config would let it win over the model that actually holds the endpoint and + report a successful import that changed nothing. + """ + value = row.get("AIModelConfig") + if not isinstance(value, dict) or not value: + return None + parsed = AIModelConfig.from_dict(value) + if not (parsed.endpoint_url or parsed.model or parsed.prompt): + return None + return value + + +def survey(root: Path | str, *, db: Database | None = None) -> LegacySurvey: + """Read an install root and report what could be imported. + + Never raises for a merely-incomplete install: a missing `Settings.json`, an + absent images folder or a model with no checkpoint are all normal and become + counts of zero. A missing or unparseable `ConfigDB.sjdb.json` is the one + hard error, because without it there is nothing to import at all. + + Pass `db` to also resolve, per model, whether importing it would create a + new row or refresh one already here (`LegacyModel.updates`). That is the + same question `_find_existing` answers during the import, asked early so the + dialog can show the answer before the user commits to anything. + """ + root = Path(root) + config_path = root / CONFIG_DB_NAME + if not config_path.is_file(): + raise FileNotFoundError(f"{root}: no {CONFIG_DB_NAME} — not an AI Brass Sorter installation") + raw = _read_json(config_path) + + cartridges = {int(c.get("Id") or 0): str(c.get("Name") or "") for c in _rows(raw, "Cartridges")} + headstamp_counts: dict[int, int] = {} + for hs in _rows(raw, "Headstamps"): + mid = int(hs.get("Model_Id") or 0) + headstamp_counts[mid] = headstamp_counts.get(mid, 0) + 1 + + out = LegacySurvey(root=root) + for row in _rows(raw, "Models"): + legacy_id = int(row.get("Id") or 0) + if not legacy_id: + continue + checkpoint = root / MODELS_SUBDIR / f"{legacy_id}.zip" + kind = _checkpoint_kind(checkpoint) if checkpoint.is_file() else "none" + entry = LegacyModel( + legacy_id=legacy_id, + name=str(row.get("Name") or f"Model {legacy_id}"), + raw=row, + cartridge_name=cartridges.get(int(row.get("CartridgeId") or 0)) or "Imported", + image_count=_count_images(root / IMAGES_SUBDIR / str(legacy_id)), + checkpoint=checkpoint if kind == "torch" else None, + checkpoint_kind=kind, + headstamp_count=headstamp_counts.get(legacy_id, 0), + ) + # An openai-mode row needs no warning: "openai" is a first-class mode + # here too, so it imports faithfully — config, headstamps and all. + if kind == "mlnet" and not entry.was_openai_mode: + entry.warning = MLNET_WARNING.format(name=entry.name) + out.models.append(entry) + + if db is not None: + model_repo = ModelRepo(db) + remembered = _imported_map(SettingsRepo(db), root) + for entry in out.models: + found = _find_existing(entry, model_repo=model_repo, remembered=remembered) + entry.updates = found.name if found is not None else None + + defaults = raw.get("Defaults") + defaults = defaults if isinstance(defaults, dict) else {} + out.has_serial = bool(defaults.get("DefaultSerialPort") or defaults.get("InitSettings")) + out.has_image_processing = any(key in defaults for key, _ in _LINESCAN_FROM_LEGACY.values()) + out.has_ai_config = any(_legacy_ai_config(m.raw) for m in out.models) + return out + + +# ----- the import ------------------------------------------------------------- + + +def _imported_map(settings: SettingsRepo, root: Path) -> dict[str, int]: + """legacy model id (as a string) → our model id, for this install root.""" + raw = settings.get(_IMPORTED_MODELS_KEY) or {} + if not isinstance(raw, dict): + return {} + scoped = raw.get(str(root)) + if not isinstance(scoped, dict): + return {} + return {str(k): int(v) for k, v in scoped.items() if isinstance(v, int)} + + +def _remember_import(settings: SettingsRepo, root: Path, legacy_id: int, model_id: int) -> None: + raw = settings.get(_IMPORTED_MODELS_KEY) or {} + if not isinstance(raw, dict): + raw = {} + scoped = raw.get(str(root)) + if not isinstance(scoped, dict): + scoped = {} + scoped[str(legacy_id)] = int(model_id) + raw[str(root)] = scoped + settings.set(_IMPORTED_MODELS_KEY, raw) + + +def _find_existing( + entry: LegacyModel, + *, + model_repo: ModelRepo, + remembered: dict[str, int], +) -> Model | None: + """The installed row this legacy model updates, or None to create one. + + Two ways to recognise it, in order. A **community UID** is authoritative and + cross-machine — the same shared model installed here from the Community page + is the same model, so re-importing it must not fork the user's slot layout. + Failing that, an earlier run of *this* import against *this* root recorded + the pairing, which is what makes running the import twice idempotent. + + Takes the `LegacyModel` rather than a parsed `Model` so `survey` can ask the + question too, without building one just to read a UID off it. + """ + if entry.community_uid: + found = model_repo.find_by_community_uid(entry.community_uid) + if found is not None: + return found + remembered_id = remembered.get(str(entry.legacy_id)) + if remembered_id: + return model_repo.get(remembered_id) + return None + + +def _copy_images(source: Path, dest: Path, *, progress: Callable[[str], None] | None) -> int: + """Copy the legacy `{label}__{ticks}.jpg` files across. + + Basename only and extension-checked: the source is a directory the user + pointed at, so it is treated with the same suspicion as a ZIP entry. Files + already present are skipped, which is what makes a re-run cheap instead of + re-copying a 12,000-image set. + """ + if not source.is_dir(): + return 0 + dest.mkdir(parents=True, exist_ok=True) + copied = 0 + for path in sorted(source.iterdir()): + if not path.is_file() or path.suffix.lower() not in _VALID_IMAGE_EXTS: + continue + target = dest / path.name + if target.exists() and target.stat().st_size == path.stat().st_size: + continue + try: + shutil.copy2(path, target) + except OSError as exc: # a locked or unreadable file loses that image, not the import + log.warning("winforms import: could not copy %s: %s", path, exc) + continue + copied += 1 + if progress and copied % 200 == 0: + progress(f"Copied {copied} images from '{source.name}'…") + return copied + + +def _import_headstamps( + entry: LegacyModel, + model_id: int, + raw_db: dict[str, Any], + *, + headstamp_repo: HeadstampRepo, + parent_repo: HeadstampParentRepo, + result: ImportResult, +) -> None: + """Headstamps, parent classifications and the slot layout for one model. + + Slot assignments are inverted on the way in: the legacy app stores them as + `SlotConfigs[].Config` (a slot listing its headstamps), while ours live on + the headstamp row as a single `slot` int. Package-mode rows are a separate + list in the legacy DB too, and are handled by the caller because they land + in a settings key rather than on a row. + """ + existing = {h.name: h for h in headstamp_repo.list_for_model(model_id)} + by_legacy_id: dict[int, str] = {} + for row in _rows(raw_db, "Headstamps"): + if int(row.get("Model_Id") or 0) != entry.legacy_id: + continue + name = str(row.get("Name") or "").strip() + if not name: + continue + by_legacy_id[int(row.get("Id") or 0)] = name + if name not in existing: + existing[name] = headstamp_repo.add(model_id, name) + result.headstamps_imported += 1 + + parents_by_legacy_id: dict[int, int] = {} + for row in _rows(raw_db, "HeadStampParents"): + if int(row.get("Model_Id") or 0) != entry.legacy_id: + continue + name = str(row.get("Name") or "").strip() + if not name: + continue + found = parent_repo.find_by_name(model_id, name) + if found is None: + found = parent_repo.add(model_id, name) + result.parents_imported += 1 + parents_by_legacy_id[int(row.get("Id") or 0)] = found.id + + for row in _rows(raw_db, "HeadStampParentLinks"): + if int(row.get("ModelId") or 0) != entry.legacy_id: + continue + parent_id = parents_by_legacy_id.get(int(row.get("ParentId") or 0)) + child_name = by_legacy_id.get(int(row.get("HeadStampId") or 0)) + if parent_id is None or child_name is None: + continue + child = existing.get(child_name) + if child is not None: + headstamp_repo.set_parent(child.id, parent_id) + + for slot_row in _rows(raw_db, "SlotConfigs"): + if int(slot_row.get("ModelId") or 0) != entry.legacy_id or slot_row.get("PackageMode"): + continue + slot = int(slot_row.get("SlotNumber") or 0) + for assignment in slot_row.get("Config") or []: + name = _assignment_name(assignment) + target = existing.get(name) if name else None + if target is not None: + headstamp_repo.update_slot(target.id, slot) + result.slots_assigned += 1 + for assignment in slot_row.get("ParentConfig") or []: + name = _assignment_name(assignment) + parent = parent_repo.find_by_name(model_id, name) if name else None + if parent is not None: + parent_repo.update_slot(parent.id, slot) + result.slots_assigned += 1 + + +def _assignment_name(assignment: Any) -> str: + """Pull the headstamp/parent name out of one `SlotConfigs` entry. + + The observed shape is `{"Headstamp": {"Name": ...}, "Counter": n}`, but the + parent list is empty in every install seen so far, so its exact spelling is + unconfirmed. Accepting the bare name, a `Parent` wrapper and a plain string + costs nothing and keeps an unexpected shape from silently dropping a slot. + """ + if isinstance(assignment, str): + return assignment.strip() + if not isinstance(assignment, dict): + return "" + for key in ("Headstamp", "Parent", "HeadStampParent"): + nested = assignment.get(key) + if isinstance(nested, dict): + return str(nested.get("Name") or "").strip() + return str(assignment.get("Name") or "").strip() + + +def _package_slots(raw_db: dict[str, Any], legacy_id: int) -> dict[str, list[str]]: + """The package-mode slot map for one model, in our settings-key shape.""" + out: dict[str, list[str]] = {} + for slot_row in _rows(raw_db, "SlotConfigs"): + if int(slot_row.get("ModelId") or 0) != legacy_id or not slot_row.get("PackageMode"): + continue + slot = int(slot_row.get("SlotNumber") or 0) + if slot <= 0: + continue + names = [n for n in (_assignment_name(a) for a in slot_row.get("Config") or []) if n] + if names: + out[str(slot)] = names + return out + + +_COUNT_FIELDS = ( + "models_imported", + "models_updated", + "headstamps_imported", + "parents_imported", + "slots_assigned", +) + + +def _merge_counts(into: ImportResult, delta: ImportResult) -> None: + for name in _COUNT_FIELDS: + setattr(into, name, getattr(into, name) + getattr(delta, name)) + + +def _import_one_model( + entry: LegacyModel, + raw_db: dict[str, Any], + *, + selection: ModelSelection, + model_repo: ModelRepo, + cart_repo: CartridgeRepo, + headstamp_repo: HeadstampRepo, + parent_repo: HeadstampParentRepo, + settings_repo: SettingsRepo, + root: Path, + remembered: dict[str, int], + result: ImportResult, +) -> int: + """Bring one legacy `Models` row across and return our model id. + + Raises rather than warns: the caller runs this inside its own SAVEPOINT and + turns a failure into a skipped model. + """ + model = model_from_export_dict(entry.raw) + model.name = entry.name + existing = _find_existing(entry, model_repo=model_repo, remembered=remembered) + model.cartridge_id = ( + existing.cartridge_id if existing is not None else cart_repo.get_or_create(entry.cartridge_name).id + ) + if existing is not None: + model.id = existing.id + model.name = existing.name + # The user's local API credentials outlive a re-import — unless they + # never entered any. A row first imported before "openai" was a mode + # here has a blank local config, and keeping that blank would discard + # the endpoint the legacy install actually holds. + if existing.ai_model_config.to_dict() != AIModelConfig().to_dict(): + model.ai_model_config = existing.ai_model_config + model.model_path = existing.model_path + model_repo.update(model) + saved_id = existing.id + result.models_updated += 1 + else: + # Not an update, so this is a genuinely new row — and the library may + # already hold a model of the same name that has nothing to do with it + # (the user built one here before importing, or imported a second + # machine's install). Two indistinguishable "9mm"s is the one conflict + # an import can actually create; the ZIP path already resolves it the + # same way. + model.name = unique_model_name(entry.name, model_repo) + saved_id = model_repo.create(model).id + result.models_imported += 1 + _remember_import(settings_repo, root, entry.legacy_id, saved_id) + + if selection.headstamps: + _import_headstamps( + entry, + saved_id, + raw_db, + headstamp_repo=headstamp_repo, + parent_repo=parent_repo, + result=result, + ) + package = _package_slots(raw_db, entry.legacy_id) + if package: + settings_repo.set(f"{_PACKAGE_SLOTS_KEY}:{saved_id}", package) + if bool(entry.raw.get("UseParentClassificationsAtRuntime")): + settings_repo.set(f"{_USE_PARENT_RUNTIME_KEY}:{saved_id}", True) + return saved_id + + +def _import_serial(defaults: dict[str, Any], settings_json: dict[str, Any], config: Config) -> None: + """Port, baud, slot count and the board's init values. + + The init values are the payload here: they are per-machine calibration the + user tuned against their own hardware. Only keys this app already knows are + taken — an unrecognised one would be written straight back to the board. + """ + serial = config.serial + port = str(defaults.get("DefaultSerialPort") or "").strip() + if port: + serial["port"] = port + baud = settings_json.get("serialBaudRate") + if isinstance(baud, int) and baud > 0: + serial["baud"] = baud + slots = defaults.get("SlotQuantity") + if isinstance(slots, int) and slots > 0: + serial["slot_quantity"] = slots + + init = dict(serial.get("init_settings") or {}) + for row in defaults.get("InitSettings") or []: + if not isinstance(row, dict): + continue + key = str(row.get("Key") or "").strip() + if key not in DEFAULT_INIT_SETTINGS: + continue + raw_value = row.get("Value") + try: + init[key] = int(str(raw_value).strip()) + except (TypeError, ValueError): + init[key] = str(raw_value) + serial["init_settings"] = init + + +def _import_image_processing(defaults: dict[str, Any], config: Config) -> bool: + """The line-scan tuning from `Defaults.IP_*`. + + Returns False when the install carries none of it. The per-model primer + settings are *not* handled here — `UsePrimerMask` / `HidePrimer` / + `PrimerMaskSize` ride along on the model row itself via + `model_from_export_dict`, which is where this app also keeps them. + """ + linescan = dict(config.image_proc.get("linescan") or {}) + touched = False + for field_name, (legacy_key, caster) in _LINESCAN_FROM_LEGACY.items(): + if legacy_key not in defaults: + continue + try: + linescan[field_name] = caster(defaults[legacy_key]) + except (TypeError, ValueError): + continue + touched = True + if touched: + config.image_proc["linescan"] = linescan + return touched + + +def _ai_config_candidates(survey_in: LegacySurvey, default_legacy_id: int = 0) -> list[LegacyModel]: + """Legacy models in the order their AI settings should be preferred. + + Per-model in the legacy app, app-level here (§4, "Active-model concept"), so + exactly one of them wins. A model set to OpenAI mode is the one that was + genuinely classifying over HTTP, so it outranks the legacy active model, + which outranks whatever happens to be declared first. + """ + + def rank(entry: LegacyModel) -> int: + if entry.was_openai_mode: + return 0 + return 1 if entry.legacy_id == default_legacy_id else 2 + + return sorted(survey_in.models, key=rank) + + +def _import_ai_config(survey_in: LegacySurvey, config: Config, default_legacy_id: int = 0) -> bool: + """The best `AIModelConfig` on any model, into the app-level `api`. + + `AIModelConfig.from_dict` already knows the legacy `OpenAI_*` key spellings. + """ + for entry in _ai_config_candidates(survey_in, default_legacy_id): + raw = _legacy_ai_config(entry.raw) + if raw is None: + continue + parsed = AIModelConfig.from_dict(raw) + api = config.api + if parsed.endpoint_url: + api["endpoint_url"] = parsed.endpoint_url + if parsed.api_key: + api["api_key"] = parsed.api_key + if parsed.model: + api["model"] = parsed.model + if parsed.prompt: + api["prompt"] = parsed.prompt + api["image_quality"] = int(parsed.image_quality or api.get("image_quality", 100)) + api["image_scale"] = int(parsed.image_scale or api.get("image_scale", 100)) + return True + return False + + +def import_installation( + root: Path | str, + *, + db: Database, + config: Config, + options: ImportOptions | None = None, + progress: Callable[[str], None] | None = None, +) -> ImportResult: + """Import the ticked items from a legacy install at `root`. + + Non-destructive to the source: every file is copied, nothing there is + written, moved or removed. + + `options.per_model` picks individual models, and per model which of its + images, headstamps and checkpoint come with it; a legacy install with years + of abandoned models in it is the normal case, not the exception. Models + absent from that map are not read at all. + + Re-running is safe. A model already brought across from this same root — or + one already installed under the same community UID — is refreshed in place, + so slot assignments and sorting templates survive and the library does not + grow duplicates. Images already copied are skipped by name and size. + + DB writes run inside one transaction; the bulk file copies deliberately do + not, since holding the SQLite write lock for the minutes a 12,000-image copy + takes would block the whole app. + + A model this app cannot accept is skipped with a warning rather than + failing the run — an install's worth of images is not worth losing to one + bad row. + """ + root = Path(root) + options = (options or ImportOptions()).normalized() + survey_in = survey(root) + # Resolved once, so the DB pass and the copy pass below cannot disagree + # about what the user asked for. A model absent from this map is one they + # unticked, and is skipped entirely — including its warning, which would + # otherwise report on a model they deliberately left behind. + selections = { + entry.legacy_id: selection + for entry in survey_in.models + if (selection := options.selection_for(entry.legacy_id)) is not None + } + result = ImportResult(warnings=[e.warning for e in survey_in.models if e.warning and e.legacy_id in selections]) + if not options.any_selected(): + return result + + settings_json: dict[str, Any] = {} + settings_path = root / SETTINGS_NAME + if settings_path.is_file(): + try: + settings_json = _read_json(settings_path) + except (OSError, json.JSONDecodeError) as exc: + result.warnings.append(f"Could not read {SETTINGS_NAME}: {exc}") + + raw_db = _read_json(root / CONFIG_DB_NAME) + defaults = raw_db.get("Defaults") + defaults = defaults if isinstance(defaults, dict) else {} + + settings_repo = SettingsRepo(db) + model_repo = ModelRepo(db) + cart_repo = CartridgeRepo(db) + headstamp_repo = HeadstampRepo(db) + parent_repo = HeadstampParentRepo(db) + + # legacy model id -> our model id, for the file copies after the transaction. + imported: dict[int, int] = {} + activate_legacy_id = int(defaults.get("DefaultModelId") or 0) + + with db.transaction(): + remembered = _imported_map(settings_repo, root) + for entry in survey_in.models: + selection = selections.get(entry.legacy_id) + if selection is None: + continue + if progress: + progress(f"Importing '{entry.name}'…") + # A nested transaction is a SAVEPOINT, so a row this app will + # not accept rolls back to just before itself. Counts go to a + # scratch result for the same reason — merged only once the + # row has actually committed. + delta = ImportResult() + try: + with db.transaction(): + saved_id = _import_one_model( + entry, + raw_db, + selection=selection, + model_repo=model_repo, + cart_repo=cart_repo, + headstamp_repo=headstamp_repo, + parent_repo=parent_repo, + settings_repo=settings_repo, + root=root, + remembered=remembered, + result=delta, + ) + except (ValueError, TypeError, sqlite3.Error) as exc: + log.warning("winforms import: skipping model %r: %s", entry.name, exc) + result.warnings.append(MODEL_FAILED_WARNING.format(name=entry.name, error=exc)) + continue + _merge_counts(result, delta) + imported[entry.legacy_id] = saved_id + + if options.serial: + _import_serial(defaults, settings_json, config) + result.serial_imported = True + if options.image_processing: + result.image_processing_imported = _import_image_processing(defaults, config) + if options.ai_config: + result.ai_config_imported = _import_ai_config(survey_in, config, activate_legacy_id) + if result.serial_imported or result.image_processing_imported or result.ai_config_imported: + config.save() + + # Only ever *adopt* the legacy app's active model, never override a + # choice already made here — the import is an offer, not a takeover. + if imported and settings_repo.get_active_model_id() is None: + target = imported.get(activate_legacy_id) + if target is not None: + settings_repo.set_active_model_id(target) + result.activated_model_id = target + + # ----- file copies, outside the write lock -------------------------------- + for entry in survey_in.models: + model_id = imported.get(entry.legacy_id) + selection = selections.get(entry.legacy_id) + if model_id is None or selection is None: + continue + paths.ensure_model_subtree(model_id) + if selection.images: + result.images_copied += _copy_images( + root / IMAGES_SUBDIR / str(entry.legacy_id), + paths.model_images_dir(model_id), + progress=progress, + ) + if selection.checkpoint and entry.has_usable_checkpoint and entry.checkpoint is not None: + dest = paths.model_trained_path(model_id) + # Same skip rule as the images: a re-run must not pay a multi- + # hundred-MB copy for a checkpoint that hasn't changed. Size is + # the same cheap proxy — a retrained model virtually never lands + # on the identical byte count. + try: + unchanged = dest.exists() and dest.stat().st_size == entry.checkpoint.stat().st_size + except OSError: + unchanged = False + if unchanged: + # The row still has to point at it — belt-and-braces for a + # half-finished earlier run that copied but never recorded. + with db.transaction(): + saved = model_repo.get(model_id) + if saved is not None and saved.model_path != str(dest): + saved.model_path = str(dest) + model_repo.update(saved) + continue + if progress: + progress(f"Copying the trained model for '{entry.name}'…") + try: + shutil.copy2(entry.checkpoint, dest) + except OSError as exc: + result.warnings.append(f"Could not copy the checkpoint for '{entry.name}': {exc}") + continue + result.checkpoints_copied += 1 + with db.transaction(): + saved = model_repo.get(model_id) + if saved is not None: + saved.model_path = str(dest) + model_repo.update(saved) + + log.info( + "winforms import from %s: %d new / %d updated models, %d images, %d checkpoints", + root, + result.models_imported, + result.models_updated, + result.images_copied, + result.checkpoints_copied, + ) + return result + + +# ----- first-run offer -------------------------------------------------------- + + +def _looks_unused(db: Database) -> bool: + """True while this app still looks freshly installed. + + Not "has no models": `Database.ensure_initialized` seeds a starter + cartridge and model on every fresh database, so the library is never empty. + What distinguishes a used install is a model the user has actually put to + work — one with a trained checkpoint on disk — or a deliberate choice of + active model. Either means the import offer would be noise. + """ + if SettingsRepo(db).get_active_model_id() is not None: + return False + return not any(m.model_path and Path(m.model_path).exists() for m in ModelRepo(db).list()) + + +def should_offer_first_run(db: Database) -> Path | None: + """The install to offer importing on this launch, or None to stay silent. + + None whenever the offer has already been made (accepted or declined — it + stays reachable from Settings either way), whenever this app is already in + use, or whenever there is simply no Windows install to find. + """ + settings = SettingsRepo(db) + if settings.get(FIRST_RUN_SEEN_KEY): + return None + if not _looks_unused(db): + return None + root = find_installation() + if root is None: + return None + try: + found = survey(root) + except (OSError, ValueError, json.JSONDecodeError): + return None + return None if found.is_empty else root + + +def mark_first_run_offered(db: Database) -> None: + SettingsRepo(db).set(FIRST_RUN_SEEN_KEY, True) diff --git a/src/sorter/ml/classifier.py b/src/sorter/ml/classifier.py index 144a704..0b5c6bb 100644 --- a/src/sorter/ml/classifier.py +++ b/src/sorter/ml/classifier.py @@ -2,8 +2,10 @@ Called from `RunController` so the run loop doesn't need to know which backend is active. **The active model alone decides the backend:** - - A model is active → local inference, always - - AI Config mode (no active model) → HTTP via `api_client.classify` + - A ConvNeXt model is active → local inference, always + - An openai-mode model is active → HTTP via `api_client.classify`, using + **that model's own** `ai_model_config` — never the app-level AI Config + - AI Config mode (no active model) → HTTP with the app-level config A local model whose checkpoint is missing raises `NoLocalCheckpointError`. It does **not** quietly become an HTTP classification. That fallback used to @@ -32,7 +34,7 @@ from .. import paths from ..data.db import Database -from ..data.models import Model +from ..data.models import Model, is_openai_model from ..data.repository import ModelRepo, SettingsRepo from . import api_client, local_inference @@ -71,20 +73,27 @@ def uses_local_inference(db: Database | None) -> bool: classify": a model with a missing checkpoint still routes locally, it just fails loudly. Callers that need to know whether it will actually work should ask `checkpoint_problem` first. + + False for an openai-mode model: it classifies over HTTP, so it needs no + PyTorch (the torch gate keys off this) and no inference device. """ - return active_model(db) is not None + model = active_model(db) + return model is not None and not is_openai_model(model) def checkpoint_problem(db: Database | None) -> str | None: """A user-facing explanation of why the active model can't classify. - None when there's nothing wrong — AI Config mode, or a local model whose - checkpoint is present. The UI uses this to refuse *before* the machine - feeds a case; `classify_active` raises the same text as a backstop for a - checkpoint that disappears mid-run. + None when there's nothing wrong — AI Config mode, an openai-mode model + (no checkpoint to miss), or a local model whose checkpoint is both present + and loadable by the installed PyTorch. The UI uses this to refuse *before* + the machine feeds a case; `classify_active` raises the same text as a + backstop for a checkpoint that disappears mid-run. """ model = active_model(db) - if model is None: + # An openai-mode model classifies over HTTP: no checkpoint to find, and + # no local torch to meet a floor with, so neither check below applies. + if model is None or is_openai_model(model): return None if not has_local_checkpoint(model): return _checkpoint_detail(model) @@ -172,12 +181,17 @@ def classify_active( """Classify `image_bgr` using whichever backend the active model selects. Raises `NoLocalCheckpointError` when a local model is active but its - checkpoint is missing. Uses HTTP only in AI Config mode (which includes - `db is None`, for tests that don't need a database). + checkpoint is missing. Uses HTTP in AI Config mode (which includes + `db is None`, for tests that don't need a database) with the app-level + `api_cfg`, and for an active openai-mode model with **its own** config — + the passed `api_cfg` is deliberately ignored there, so the app-level AI + Config can never leak into a model that carries its own server settings. """ model = active_model(db) if model is None: return api_client.classify(image_bgr, headstamps, api_cfg) + if is_openai_model(model): + return api_client.classify(image_bgr, headstamps, model.ai_model_config.to_dict()) # `not model.model_path` is folded into the guard (redundant with # `has_local_checkpoint`'s own check) so the type checker can narrow diff --git a/src/sorter/ui/ai_page.py b/src/sorter/ui/ai_page.py index 0dcb294..33b62b2 100644 --- a/src/sorter/ui/ai_page.py +++ b/src/sorter/ui/ai_page.py @@ -12,11 +12,18 @@ Unlike the settings pages, this one saves on the **Save** button, not on edit: a half-typed endpoint must not become the live one. -The page is a two-card stack, exactly as Train's is: the form when this is the -backend that classifies (no active model), or — when a local model is doing it -instead — a panel naming that model and offering the jump to the Models page. -The sidebar entry is never hidden (app.py's ``_apply_mode_visibility``), so -this page is what has to answer a click on the muted half of the pair. +The page is a two-card stack, exactly as Train's is: the form when HTTP is +what classifies — AI Config mode (no active model, app-level settings) or an +active **openai-mode model** (that model's own ``ai_model_config``) — or, when +a local ConvNeXt model is doing it instead, a panel naming that model and +offering the jump to the Models page. The sidebar entry is never hidden +(app.py's ``_apply_mode_visibility``), so this page is what has to answer a +click on the muted half of the pair. + +The server fields therefore have a **target**: the active openai model's row, +or the app-level ``config.api``. ``retarget()`` rebinds them only when the +target actually changes, so a mode/changed event can't discard a half-typed +endpoint (the old guarantee, kept per target). """ from __future__ import annotations @@ -44,6 +51,7 @@ QWidget, ) +from ..data.models import AIModelConfig, Model, is_openai_model from ..data.repository import ModelRepo from ..hardware.image_proc import apply_primer_mask, crop_headstamp from ..ml import api_client @@ -64,13 +72,30 @@ ) MODELS_JUMP_TEXT = "Go to Models" NAME_HINT = "Add creates a headstamp; Rename applies the typed name to the selected one." +# Caption over the server form saying whose settings these are — the whole +# point of per-model configs is that the answer isn't always the same. +TARGET_GLOBAL_TEXT = "App-level settings — used in AI Config mode (no active model)." +TARGET_MODEL_TEXT = "Settings of the active OpenAI model “{name}” — saved on its model row." def ai_config_mode(win: Any) -> bool: - """AI Config mode = no active local model. Only then do these settings apply.""" + """AI Config mode = no active local model. The app-level settings apply.""" return win.config.settings.get_active_model_id() is None +def openai_target(win: Any) -> Model | None: + """The active model, when it is an openai-mode one — the other HTTP target. + + None in AI Config mode and for any ConvNeXt/community model. Read fresh on + every call, like ``active_model_name``. + """ + model_id = win.config.settings.get_active_model_id() + if model_id is None: + return None + model = ModelRepo(win.config.db).get(model_id) + return model if is_openai_model(model) else None + + def active_model_name(win: Any) -> str | None: """Read fresh — the explainer names whatever is classifying right now.""" model_id = win.config.settings.get_active_model_id() @@ -88,6 +113,9 @@ def __init__(self, win: Any) -> None: self._win = win # Guards the slot spin box while it is being set to follow the selection. self._syncing = False + # Which config the server fields are bound to: an openai model's row + # id, or None for the app-level ``config.api``. Set by ``retarget()``. + self._target_model_id: int | None = None # Attribute, not method, so tests can replace it (ty rejects # assigning over a method) — same pattern as the dialogs' notify. self.confirm: Callable[[str, str], bool] = self._ask @@ -116,6 +144,12 @@ def _build_server_group(self) -> QGroupBox: column = QVBoxLayout(box) api = self._win.config.api + # Whose settings these are; ``retarget()`` keeps it truthful. + self.target_label = QLabel(TARGET_GLOBAL_TEXT, box) + self.target_label.setObjectName("mutedLabel") + self.target_label.setWordWrap(True) + column.addWidget(self.target_label) + form = QFormLayout() self.endpoint_edit = QLineEdit(str(api.get("endpoint_url", "")), box) self.model_edit = QLineEdit(str(api.get("model", "")), box) @@ -178,10 +212,49 @@ def _field_values(self) -> dict[str, Any]: "image_scale": int(self.scale_spin.value()), } + def _populate(self, values: dict[str, Any]) -> None: + self.endpoint_edit.setText(str(values.get("endpoint_url", ""))) + self.key_edit.setText(str(values.get("api_key", ""))) + self.model_edit.setText(str(values.get("model", ""))) + self.prompt_edit.setPlainText(str(values.get("prompt", ""))) + self.quality_spin.setValue(int(values.get("image_quality", 100))) + self.scale_spin.setValue(int(values.get("image_scale", 100))) + + def retarget(self) -> None: + """Bind the server fields to the active openai model, or the app config. + + A no-op while the target is unchanged, so refreshes triggered by + unrelated mode/changed events keep a half-typed edit. Switching + targets repopulates wholesale — the fields then show the other + config, not a mix. + """ + model = openai_target(self._win) + new_id = model.id if model is not None else None + if new_id == self._target_model_id: + return + self._target_model_id = new_id + if model is not None: + self._populate(model.ai_model_config.to_dict()) + self.target_label.setText(TARGET_MODEL_TEXT.format(name=model.name)) + else: + self._populate(dict(self._win.config.api)) + self.target_label.setText(TARGET_GLOBAL_TEXT) + def save(self) -> None: - self._win.config.api.update(self._field_values()) - self._win.config.save() - self._win.set_status("AI settings saved.") + values = self._field_values() + if self._target_model_id is None: + self._win.config.api.update(values) + self._win.config.save() + self._win.set_status("AI settings saved.") + return + repo = ModelRepo(self._win.config.db) + model = repo.get(self._target_model_id) + if model is None: # deleted underneath the open page + self._win.set_status("That model no longer exists — nothing saved.") + return + model.ai_model_config = AIModelConfig.from_dict(values) + repo.update(model) + self._win.set_status(f"AI settings saved to “{model.name}”.") # ----- headstamps --------------------------------------------------------- @@ -498,13 +571,15 @@ def is_available(self) -> bool: return self.stack.currentIndex() == 0 def refresh_mode(self) -> None: - """Re-read the active model and the (model-scoped) headstamp list. + """Re-read the active model, the headstamp list, and the config target. - The server fields aren't model-scoped, so they are left alone — a mode - change must not discard an edit that hasn't been saved yet. + ``retarget()`` rebinds the server fields only when the target really + changed — a mode/changed event for the *same* target must not discard + an edit that hasn't been saved yet. """ self.section.refresh_list() - if ai_config_mode(self._win): + self.section.retarget() + if ai_config_mode(self._win) or openai_target(self._win) is not None: self.stack.setCurrentIndex(0) return name = active_model_name(self._win) diff --git a/src/sorter/ui/app.py b/src/sorter/ui/app.py index c12c078..391c26d 100644 --- a/src/sorter/ui/app.py +++ b/src/sorter/ui/app.py @@ -24,6 +24,7 @@ import base64 import html import itertools +import logging import os import sys import threading @@ -84,6 +85,13 @@ from .community_page import build_community_page from .dialog_slot_assign import CATCH_ALL_HINT, SlotAssignDialog from .dialog_template import EditTemplateDialog, NewTemplateDialog +from .dialog_winforms_import import ( + SECTION_NAME as WINFORMS_IMPORT_SECTION, +) +from .dialog_winforms_import import ( + build_winforms_import_section, + maybe_offer_first_run, +) from .help_viewer import build_help_window, topic_for from .history_view import build_history_view from .icons import AI_CONFIG, COMMUNITY, MODELS, SETTINGS, SORT, TRAIN, app_icon @@ -106,6 +114,8 @@ from .torch_gate import TorchGate from .train_page import build_train_page +log = logging.getLogger(__name__) + PREVIEW_FPS = 20 SIDEBAR_WIDTH = 84 # Defensive: every SETTINGS_SECTIONS entry has a builder, so nothing renders @@ -137,9 +147,12 @@ # One line each, always set, saying only whether this entry is the live one. ACTIVITY_TOOLTIP_LIVE = "Classification uses this now" TRAIN_TOOLTIP_MUTED = "Activates when a local model is active — see Models" -AI_CONFIG_TOOLTIP_MUTED = "Activates when 'Use AI Config' is selected on Models" +AI_CONFIG_TOOLTIP_MUTED = "Activates when 'Use AI Config' or an OpenAI model is selected on Models" SIDEBAR_ICON_SIZE = 26 -SETTINGS_SECTIONS = ("Camera", "Serial", "Image Processing", "Theme") +# "Import from Windows" is last: it is a one-off errand, not a knob, and its +# name is also a GUIDE.md heading (help_viewer slugifies section names +# straight to an anchor, which tests/unit/ui/test_help.py pins). +SETTINGS_SECTIONS = ("Camera", "Serial", "Image Processing", "Theme", WINFORMS_IMPORT_SECTION) BAUD_CHOICES = (9600, 19200, 38400, 57600, 115200) # On every dock's tab: QtAds's drop overlays show where a panel *can* go once # a drag starts, but nothing hints that it can be dragged at all (JL). @@ -415,6 +428,9 @@ def __init__(self, config: Any, *, auto_connect: bool = True) -> None: self._auto_connect_serial() self._warm_device_indicator() QTimer.singleShot(2500, self, self._startup_update_check) + # After the shell is up, so the dialog has a parent to centre on. + # Silent unless a Windows install is actually there (issue #98). + QTimer.singleShot(0, self, self._offer_winforms_import) self._apply_auth_visibility() # ----- construction ------------------------------------------------------- @@ -932,6 +948,7 @@ def _build_settings_page(self) -> QWidget: "Serial": self._build_serial_page, "Camera": lambda: build_camera_section(self), "Image Processing": lambda: build_imageproc_section(self), + WINFORMS_IMPORT_SECTION: lambda: build_winforms_import_section(self), } for name in SETTINGS_SECTIONS: self.settings_list.addItem(name) @@ -1139,6 +1156,29 @@ def _on_signin_clicked(self) -> None: else: self.community_page.open_login() + # ----- import from the Windows app ---------------------------------------- + + def _offer_winforms_import(self) -> None: + """First-run offer. Silent, and cheap, when there is nothing to offer.""" + try: + maybe_offer_first_run(self) + except Exception: + # An import that cannot even be offered must not take the launch + # with it — Settings keeps the same dialog reachable. + log.exception("first-run Windows-app import offer failed") + + def after_winforms_import(self, result: Any) -> None: + """Re-read everything the import may have rewritten. + + It can touch the model library, the active model, every headstamp and + slot, and three settings sections at once, so this re-runs the same + refresh a mode switch does rather than trying to be surgical. + """ + self.config.load() + self._on_mode_changed() + self.models_page.refresh() + self.set_status("Imported from the Windows app.") + # ----- updates ------------------------------------------------------------ def open_update_dialog(self, *, check: bool = False) -> None: @@ -1664,16 +1704,19 @@ def _apply_mode_visibility(self) -> None: """The mode inks the Train / AI Config pair. **Neither is ever hidden** (JL: a hidden activity is one nobody finds). - Exactly one of the two is live — a trainable local model, or no active - model at all — and a community model makes it neither. The other goes - muted: still clickable, with the explainer behind it (train_page's - and ai_page's unavailable panels) saying why and what to do. + At most one of the two is live — a trainable local model makes it + Train, no active model or an active openai-mode model makes it AI + Config (both classify over HTTP, and the page edits whichever config + is in effect) — and a community model makes it neither. The other + goes muted: still clickable, with the explainer behind it + (train_page's and ai_page's unavailable panels) saying why and what + to do. """ - from ..data.models import is_trainable + from ..data.models import is_openai_model, is_trainable model = self._active_model() train_live = is_trainable(model) - ai_live = model is None + ai_live = model is None or is_openai_model(model) self._set_activity_unavailable("Train", not train_live) self._set_activity_unavailable(AI_CONFIG_ACTIVITY, not ai_live) self.sidebar_buttons["Train"].setToolTip(ACTIVITY_TOOLTIP_LIVE if train_live else TRAIN_TOOLTIP_MUTED) @@ -2170,11 +2213,19 @@ def notify(self, title: str, text: str) -> None: QMessageBox.warning(self, title, text) def _ai_credentials_missing(self) -> bool: - """AI Config mode can't classify without an API key and a model name. + """HTTP classification can't run without an API key and a model name. - Scoped to that mode: a local model never touches the HTTP client, so - an unset key there is no reason to refuse a run. + Scoped to the HTTP paths: a local model never touches the HTTP + client, so an unset key there is no reason to refuse a run. An + active openai-mode model is checked against **its own** config — the + same one `classify_active` will use — never the app-level one. """ + from ..data.models import is_openai_model + + model = classifier.active_model(self.db) + if is_openai_model(model): + cfg = model.ai_model_config if model is not None else None + return cfg is None or not (cfg.api_key and cfg.model) if classifier.uses_local_inference(self.db): return False api = self.config.api diff --git a/src/sorter/ui/dialog_model_editor.py b/src/sorter/ui/dialog_model_editor.py index 49c8c86..88ee22c 100644 --- a/src/sorter/ui/dialog_model_editor.py +++ b/src/sorter/ui/dialog_model_editor.py @@ -36,11 +36,12 @@ from ..data.db import Database from ..data.models import ( FEEDBACK_UPLOAD_MODES, - SUPPORTED_MODEL_MODES, + MODEL_MODES, AIModelConfig, ImageProcessingConfig, Model, TrainingConfig, + model_mode_label, ) from ..data.repository import CartridgeRepo, ModelRepo @@ -103,10 +104,18 @@ def __init__( form.addRow("Cartridge", self.cartridge_combo) self.mode_combo = QComboBox(self) - self.mode_combo.addItems(list(SUPPORTED_MODEL_MODES)) - if existing is not None and existing.model_mode in SUPPORTED_MODEL_MODES: - self.mode_combo.setCurrentText(existing.model_mode) - form.addRow("Model type", self.mode_combo) + # MODEL_MODES, not SUPPORTED_MODEL_MODES: "openai" is a legal mode — + # a model that classifies over an HTTP server (its settings live on + # the AI Config page while it is active) — it just isn't trainable. + # Display label vs stored identifier: the user sees "ConvNeXt-Tiny" / + # "OpenAI" (the Windows app's spellings), the row stores snake_case. + for mode in MODEL_MODES: + self.mode_combo.addItem(model_mode_label(mode), mode) + if existing is not None and existing.model_mode in MODEL_MODES: + self.mode_combo.setCurrentIndex(self.mode_combo.findData(existing.model_mode)) + # "Training mode", as the Windows app names it — "Model type" collided + # with the library table's Type column, which means ownership. + form.addRow("Training mode", self.mode_combo) self.primer_spin = QSpinBox(self) self.primer_spin.setRange(0, 512) @@ -177,8 +186,8 @@ def save(self) -> None: if cartridge_id is None: self.notify("Missing cartridge", "Pick a cartridge first.") return - mode = self.mode_combo.currentText() - if mode not in SUPPORTED_MODEL_MODES: + mode = self.mode_combo.currentData() + if mode not in MODEL_MODES: self.notify("Invalid model type", f"Unknown model: {mode}") return diff --git a/src/sorter/ui/dialog_winforms_import.py b/src/sorter/ui/dialog_winforms_import.py new file mode 100644 index 0000000..edf2d89 --- /dev/null +++ b/src/sorter/ui/dialog_winforms_import.py @@ -0,0 +1,691 @@ +"""Import an existing WinForms ("AI Brass Sorter") installation. + +Two ways in, one dialog. On first run `maybe_offer_first_run` opens it against +whatever `winforms_import.find_installation` turned up; Settings → Import from +Windows opens it against a folder the user picks. The picker is the same either +way — issue #98 asks for a per-item choice rather than an all-or-nothing +migration, so the dialog's job is to say what each item would cost and let the +user decline any of it. + +**Why a tree and not a list of checkboxes.** A real install accumulates years of +models, most of which the user has no interest in carrying forward (sjseth, +reviewing #125: "I actually only wanted to import a couple models… they may have +a lot of junk in the old system"). A flat "Models / Training images / Headstamps" +triple can only answer all-or-nothing, so the choice is per model, and per model +which of its images, headstamps and trained checkpoint come with it. + +The tree is also what makes the **inheritance** honest: those three hang off a +model row, so they are its children rather than siblings — untick the model and +its whole branch goes with it, which is a structure the user can see instead of +a rule the dialog has to explain. There is deliberately no leaf for the model's +own row: it is the branch. + +Threading follows CLAUDE.md §8: the import runs on a worker thread and only ever +puts a message on a ``queue.Queue``; a main-thread ``QTimer`` drains it into the +widgets. Nothing here touches a widget off the main thread. + +Seam discipline (§5): ``notify`` and ``ask_directory`` are instance attributes, +not methods, so an offscreen test can replace them and nothing blocks on a +native modal. +""" + +from __future__ import annotations + +import queue +import threading +from collections.abc import Callable +from pathlib import Path +from typing import Any + +from PySide6.QtCore import Qt, QTimer +from PySide6.QtWidgets import ( + QDialog, + QDialogButtonBox, + QFileDialog, + QFrame, + QHBoxLayout, + QLabel, + QMessageBox, + QProgressBar, + QPushButton, + QTreeWidget, + QTreeWidgetItem, + QVBoxLayout, + QWidget, +) + +from ..data import winforms_import +from ..data.winforms_import import ( + ImportOptions, + ImportResult, + LegacyModel, + LegacySurvey, + ModelSelection, +) +from .community_page import format_size + +TITLE = "Import from the Windows app" +# Settings section name — also a GUIDE.md heading, since help_viewer slugifies +# section names straight to an anchor. +SECTION_NAME = "Import from Windows" +# Dynamic property the Settings button carries so a test can find it. +IMPORT_BUTTON_ROLE = "winformsImport" + +FIRST_RUN_INTRO = ( + "An installation of the Windows app (AI Brass Sorter) was found on this " + "computer. Its models, training images and settings can be copied across " + "so you don't have to set everything up again." +) +SETTINGS_INTRO = "Copy models, training images and settings out of an installation of the Windows app." +# Said plainly and up front, because "import" reads as "move" to plenty of people. +NON_DESTRUCTIVE = "Nothing in the Windows app is changed, moved or deleted — everything is copied." + +_NOT_FOUND = ( + "No installation of the Windows app was found in the usual place. " + "If you have one somewhere else, choose its folder — the one containing " + "'Data' and 'training'." +) +# Said on the model row itself, because it is the answer to "will this tread on +# the models I already have here?" — and the answer is per model. +NEW_MODEL = "new model here" +UPDATES_MODEL = "updates '{name}'" +# Short form of `winforms_import.MLNET_WARNING`, for the row itself; the full +# sentence stays in the warning line under the tree. +MLNET_MARK = "ML.NET model, retrain needed" + +# Stands in for an unticked model while totting up what the selection costs. +_NOTHING = ModelSelection(images=False, headstamps=False, checkpoint=False) + + +def _muted(text: str, parent: QWidget) -> QLabel: + label = QLabel(text, parent) + label.setObjectName("mutedLabel") + label.setWordWrap(True) + return label + + +def describe_item(key: str, found: LegacySurvey) -> tuple[str, str]: + """(label, detail) for one top-level row, with what this install would bring. + + The counts are the whole point of surveying before importing: "Training + images" alone says nothing, "12,255 images" tells the user why the copy is + about to take a few minutes. + """ + if key == winforms_import.ITEM_MODELS: + trainable = sum(1 for m in found.models if m.has_usable_checkpoint) + return "Models", f"{len(found.models)} in the Windows app, {trainable} with a trained model file" + if key == winforms_import.ITEM_IMAGE_PROC: + return "Image-processing settings", "Crop tuning from the Windows app" + if key == winforms_import.ITEM_SERIAL: + return "Serial / board settings", "Port, baud rate and the board's init values" + if key == winforms_import.ITEM_AI_CONFIG: + return "AI Config", "Endpoint, model and prompt for classifying over HTTP" + return key, "" + + +def describe_model(entry: LegacyModel) -> tuple[str, str]: + """(label, detail) for one model's row. + + Everything needed to decide whether this model is worth carrying forward, + on the row itself — the counts included, because "38 images" is exactly how + a user recognises an abandoned experiment without expanding anything. The + fate comes last and is the answer to "will this tread on what I have here". + """ + fate = UPDATES_MODEL.format(name=entry.updates) if entry.updates else NEW_MODEL + detail = [entry.cartridge_name, f"{entry.image_count} image(s)", f"{entry.headstamp_count} headstamp(s)"] + if entry.has_usable_checkpoint: + detail.append(format_size(entry.checkpoint_bytes)) + elif entry.checkpoint_kind == "mlnet": + # There *is* a checkpoint, it just can't classify here — without this + # the missing "Trained model file" row reads as "never trained". + detail.append(MLNET_MARK) + detail.append(fate) + return entry.name, " · ".join(detail) + + +def describe_part(part: str, entry: LegacyModel) -> tuple[str, str]: + """(label, detail) for one model's images / headstamps / checkpoint row.""" + if part == winforms_import.PART_IMAGES: + return "Training images", f"{entry.image_count}" + if part == winforms_import.PART_HEADSTAMPS: + return "Headstamps and slot assignments", f"{entry.headstamp_count}" + if part == winforms_import.PART_CHECKPOINT: + return "Trained model file", format_size(entry.checkpoint_bytes) + return part, "" + + +def _state(checked: bool) -> Qt.CheckState: + return Qt.CheckState.Checked if checked else Qt.CheckState.Unchecked + + +def _children(item: QTreeWidgetItem) -> list[QTreeWidgetItem]: + return [item.child(i) for i in range(item.childCount())] + + +def _set_branch(item: QTreeWidgetItem, state: Qt.CheckState) -> None: + """Set `item` and everything under it, skipping what the user can't reach. + + A disabled row is one this install has nothing behind — carrying a tick into + it would make `selected_options` claim something that isn't there. + """ + if not item.isDisabled(): + item.setCheckState(0, state) + for child in _children(item): + _set_branch(child, state) + + +def _refresh_ancestors(item: QTreeWidgetItem) -> None: + """Recompute every parent above `item` from the children it actually has.""" + parent = item.parent() + while parent is not None: + states = {child.checkState(0) for child in _children(parent) if not child.isDisabled()} + if states == {Qt.CheckState.Checked}: + parent.setCheckState(0, Qt.CheckState.Checked) + elif states == {Qt.CheckState.Unchecked}: + parent.setCheckState(0, Qt.CheckState.Unchecked) + elif states: + parent.setCheckState(0, Qt.CheckState.PartiallyChecked) + parent = parent.parent() + + +def available_parts(entry: LegacyModel) -> tuple[str, ...]: + """The parts this legacy model actually has something to offer for. + + A part with nothing behind it gets no row at all rather than a disabled one: + every tick in the tree then means something, and the check propagation never + has to reason about a child the user cannot reach. + """ + parts: list[str] = [] + if entry.image_count: + parts.append(winforms_import.PART_IMAGES) + if entry.headstamp_count: + parts.append(winforms_import.PART_HEADSTAMPS) + if entry.has_usable_checkpoint: + parts.append(winforms_import.PART_CHECKPOINT) + return tuple(parts) + + +class WinFormsImportDialog(QDialog): + """Pick a folder, tick what to bring across, watch it happen.""" + + def __init__( + self, + win: Any, + root: Path | None, + parent: QWidget | None = None, + *, + first_run: bool = False, + ) -> None: + super().__init__(parent) + self._win = win + self._root = Path(root) if root else None + self._survey: LegacySurvey | None = None + self._first_run = first_run + self._running = False + self._events: queue.Queue[tuple[str, Any]] = queue.Queue() + self.result_summary: ImportResult | None = None + + self.notify: Callable[[str, str], None] = self._notify + self.ask_directory: Callable[[], str | None] = self._ask_directory + + self.setWindowTitle(TITLE) + # Wide enough for a model row's second column — name, cartridge, counts, + # checkpoint size and what the import would do to the library. Narrower + # and the last of those elides, which is the half a user is deciding on. + self.setMinimumWidth(760) + self._build_ui() + self._reload_survey() + + self._timer = QTimer(self) + self._timer.timeout.connect(self._drain) + self._timer.start(100) + + # ----- construction ------------------------------------------------------- + + def _build_ui(self) -> None: + column = QVBoxLayout(self) + column.setSpacing(10) + + column.addWidget(_muted(FIRST_RUN_INTRO if self._first_run else SETTINGS_INTRO, self)) + + folder_row = QHBoxLayout() + self.folder_label = QLabel("", self) + self.folder_label.setWordWrap(True) + folder_row.addWidget(self.folder_label, 1) + self.browse_button = QPushButton("Choose folder…", self) + self.browse_button.clicked.connect(self._choose_folder) + folder_row.addWidget(self.browse_button) + column.addLayout(folder_row) + + line = QFrame(self) + line.setFrameShape(QFrame.Shape.HLine) + line.setObjectName("sidebarSeparator") + column.addWidget(line) + + self.tree = QTreeWidget(self) + self.tree.setObjectName("importTree") + self.tree.setHeaderHidden(True) + # Two columns: what it is, then what it would bring. One column would + # elide a model's counts away exactly when they matter — the name is + # what has to survive, so it gets a column of its own. + self.tree.setColumnCount(2) + self.tree.setMinimumHeight(240) + self.tree.itemChanged.connect(self._on_item_changed) + # Column 0 is sized to what is *visible*, and a model's parts are a + # level deeper than anything measured while its branch was closed — + # without this they open elided to "Training ima…". + self.tree.itemExpanded.connect(lambda _item: self.tree.resizeColumnToContents(0)) + column.addWidget(self.tree, 1) + + # Populated by `_reload_survey`; every row is addressed through these + # rather than by walking the tree, so a test can tick one model. + self.items: dict[str, QTreeWidgetItem] = {} + self.model_items: dict[int, QTreeWidgetItem] = {} + self.part_items: dict[tuple[int, str], QTreeWidgetItem] = {} + + # Picking two models out of fifteen should not cost thirteen clicks. + select_row = QHBoxLayout() + select_row.addWidget(QLabel("Models:", self)) + self.all_models_button = QPushButton("Select all", self) + self.all_models_button.clicked.connect(lambda: self._set_all_models(True)) + self.no_models_button = QPushButton("Select none", self) + self.no_models_button.clicked.connect(lambda: self._set_all_models(False)) + select_row.addWidget(self.all_models_button) + select_row.addWidget(self.no_models_button) + select_row.addStretch(1) + column.addLayout(select_row) + + # What the current ticks add up to. A partial selection otherwise gives + # the user no idea what they have just signed up to wait for. + self.selection_label = _muted("", self) + column.addWidget(self.selection_label) + + self.warning_label = _muted("", self) + self.warning_label.setObjectName("warningLabel") + column.addWidget(self.warning_label) + + self.status_label = _muted("", self) + column.addWidget(self.status_label) + self.progress = QProgressBar(self) + self.progress.setRange(0, 0) # the copy has no useful total up front + self.progress.setVisible(False) + column.addWidget(self.progress) + + self.buttons = QDialogButtonBox(self) + self.import_button = self.buttons.addButton("Import", QDialogButtonBox.ButtonRole.AcceptRole) + self.import_button.setObjectName("action") + self.close_button = self.buttons.addButton( + "Not now" if self._first_run else "Close", + QDialogButtonBox.ButtonRole.RejectRole, + ) + self.import_button.clicked.connect(self.start_import) + self.close_button.clicked.connect(self.reject) + column.addWidget(self.buttons) + + # ----- the source folder -------------------------------------------------- + + def _reload_survey(self) -> None: + """Re-read the chosen folder and rebuild the tree from it.""" + self._survey = None + problem = "" + if self._root is not None: + try: + # `db` is what lets each model row say whether it would create a + # model here or refresh one that already exists. + self._survey = winforms_import.survey(self._root, db=self._win.db) + except (OSError, ValueError) as exc: + problem = str(exc) + + self._rebuild_tree() + if self._survey is None: + self.folder_label.setText(str(self._root) if self._root is not None else _NOT_FOUND) + self.warning_label.setText(problem) + self.import_button.setEnabled(False) + self._set_items_enabled(False) + return + + self.folder_label.setText(str(self._root)) + self.warning_label.setText("\n".join(self._survey.warnings)) + self._set_items_enabled(True) + self.import_button.setEnabled(not self._survey.is_empty) + self._refresh_selection_label() + + # ----- the tree ----------------------------------------------------------- + + def _rebuild_tree(self) -> None: + """Throw the rows away and build them from the current survey. + + Signals stay blocked throughout: every `setCheckState` here would + otherwise re-enter `_on_item_changed` and propagate against a tree that + is only half built. + """ + self.tree.blockSignals(True) + try: + self.tree.clear() + self.items = {} + self.model_items = {} + self.part_items = {} + found = self._survey + if found is None: + return + + models_row = self._add_item(winforms_import.ITEM_MODELS, checked=bool(found.models)) + models_row.setDisabled(not found.models) + for entry in found.models: + self._add_model(models_row, entry) + # Models open, each model closed: picking *which* models is the + # first job, and a 15-model install with every branch open would + # push the settings rows below the fold. Each model's own row + # already carries its counts, so nothing is hidden by this. + models_row.setExpanded(True) + + # Opt-in, not ticked by default: these overwrite values the user may + # already have tuned here, and unlike a model there is no second copy + # to fall back on. + for key, available in ( + (winforms_import.ITEM_IMAGE_PROC, found.has_image_processing), + (winforms_import.ITEM_SERIAL, found.has_serial), + (winforms_import.ITEM_AI_CONFIG, found.has_ai_config), + ): + row = self._add_item(key, checked=False) + # Nothing behind it, so nothing to offer — greyed out rather + # than a tick that would import nothing. + row.setDisabled(not available) + self.tree.resizeColumnToContents(0) + finally: + self.tree.blockSignals(False) + + def _add_item(self, key: str, *, checked: bool) -> QTreeWidgetItem: + found = self._survey + assert found is not None # only called from _rebuild_tree, past its guard + row = QTreeWidgetItem(self.tree, list(describe_item(key, found))) + row.setFlags(row.flags() | Qt.ItemFlag.ItemIsUserCheckable) + row.setCheckState(0, _state(checked)) + self.items[key] = row + return row + + def _add_model(self, parent: QTreeWidgetItem, entry: LegacyModel) -> None: + row = QTreeWidgetItem(parent, list(describe_model(entry))) + row.setFlags(row.flags() | Qt.ItemFlag.ItemIsUserCheckable) + row.setCheckState(0, Qt.CheckState.Checked) + self.model_items[entry.legacy_id] = row + for part in available_parts(entry): + leaf = QTreeWidgetItem(row, list(describe_part(part, entry))) + leaf.setFlags(leaf.flags() | Qt.ItemFlag.ItemIsUserCheckable) + leaf.setCheckState(0, Qt.CheckState.Checked) + self.part_items[(entry.legacy_id, part)] = leaf + + def _on_item_changed(self, item: QTreeWidgetItem, column: int) -> None: + """Push a tick down the branch and recompute the ones above it. + + Ticking a model means its whole branch; unticking it means none of it. + A parent left showing part of its branch is `PartiallyChecked`, which is + set here rather than through `ItemIsAutoTristate` so there is exactly one + place that decides what a parent's state means. + """ + if column != 0: + return + self.tree.blockSignals(True) + try: + state = item.checkState(0) + if state != Qt.CheckState.PartiallyChecked: + _set_branch(item, state) + _refresh_ancestors(item) + finally: + self.tree.blockSignals(False) + self._refresh_selection_label() + + def _set_all_models(self, checked: bool) -> None: + models_row = self.items.get(winforms_import.ITEM_MODELS) + if models_row is None or models_row.isDisabled(): + return + self.tree.blockSignals(True) + try: + _set_branch(models_row, _state(checked)) + finally: + self.tree.blockSignals(False) + self._refresh_selection_label() + + def _refresh_selection_label(self) -> None: + """Say what the current ticks add up to, in the units of the wait.""" + found = self._survey + if found is None: + self.selection_label.setText("") + return + chosen = self.selected_options() + models = chosen.per_model or {} + images = sum(e.image_count for e in found.models if (models.get(e.legacy_id) or _NOTHING).images) + checkpoints = sum( + 1 for e in found.models if e.has_usable_checkpoint and (models.get(e.legacy_id) or _NOTHING).checkpoint + ) + if not models: + self.selection_label.setText("No models selected.") + return + parts = [f"{len(models)} of {len(found.models)} model(s)"] + if images: + parts.append(f"{images} image(s)") + if checkpoints: + parts.append(f"{checkpoints} trained model file(s)") + self.selection_label.setText("Will import: " + ", ".join(parts) + ".") + + def _set_items_enabled(self, enabled: bool) -> None: + # The tree is one control: a running import must not let the user + # re-tick the selection it is halfway through acting on. + self.tree.setEnabled(enabled) + self.all_models_button.setEnabled(enabled) + self.no_models_button.setEnabled(enabled) + + def _choose_folder(self) -> None: + chosen = self.ask_directory() + if not chosen: + return + self._root = Path(chosen) + self._reload_survey() + + def selected_options(self) -> ImportOptions: + """Read the tree back as an `ImportOptions`. + + The per-model map is the authority on models; the three app-level flags + are set from it only so a summary or a log line reads sensibly. + """ + per_model: dict[int, ModelSelection] = {} + for legacy_id, row in self.model_items.items(): + if row.checkState(0) == Qt.CheckState.Unchecked: + continue + per_model[legacy_id] = ModelSelection( + images=self._part_checked(legacy_id, winforms_import.PART_IMAGES), + headstamps=self._part_checked(legacy_id, winforms_import.PART_HEADSTAMPS), + checkpoint=self._part_checked(legacy_id, winforms_import.PART_CHECKPOINT), + ) + return ImportOptions( + models=bool(per_model), + training_images=any(s.images for s in per_model.values()), + headstamps=any(s.headstamps for s in per_model.values()), + image_processing=self._item_checked(winforms_import.ITEM_IMAGE_PROC), + serial=self._item_checked(winforms_import.ITEM_SERIAL), + ai_config=self._item_checked(winforms_import.ITEM_AI_CONFIG), + per_model=per_model, + ) + + def _part_checked(self, legacy_id: int, part: str) -> bool: + """Is this part ticked? False when the model has no such row to tick.""" + row = self.part_items.get((legacy_id, part)) + return row is not None and row.checkState(0) == Qt.CheckState.Checked + + def _item_checked(self, key: str) -> bool: + row = self.items.get(key) + return row is not None and row.checkState(0) == Qt.CheckState.Checked + + # ----- running it --------------------------------------------------------- + + def start_import(self) -> None: + if self._running or self._root is None: + return + options = self.selected_options() + if not options.any_selected(): + self.notify(TITLE, "Tick at least one thing to import.") + return + + self._running = True + self._set_items_enabled(False) + self.browse_button.setEnabled(False) + self.import_button.setEnabled(False) + self.progress.setVisible(True) + self.status_label.setText("Reading the Windows app's data…") + + root = self._root + db = self._win.db + config = self._win.config + events = self._events + + def _work() -> None: + try: + result = winforms_import.import_installation( + root, + db=db, + config=config, + options=options, + progress=lambda message: events.put(("progress", message)), + ) + except Exception as exc: # surfaced in the dialog, never swallowed + events.put(("error", str(exc))) + return + events.put(("done", result)) + + threading.Thread(target=_work, daemon=True).start() + + def _drain(self) -> None: + while True: + try: + kind, payload = self._events.get_nowait() + except queue.Empty: + return + if kind == "progress": + self.status_label.setText(str(payload)) + elif kind == "error": + self._failed(str(payload)) + elif kind == "done": + self._succeeded(payload) + + def _failed(self, message: str) -> None: + self._running = False + self.progress.setVisible(False) + self.status_label.setText("") + self.browse_button.setEnabled(True) + self._reload_survey() + self.notify("Import failed", message) + + def _succeeded(self, result: ImportResult) -> None: + self._running = False + self.result_summary = result + self.progress.setVisible(False) + self.status_label.setText("") + self._win.after_winforms_import(result) + self.notify("Import complete", summarize(result)) + self.accept() + + # ----- seams -------------------------------------------------------------- + + def _notify(self, title: str, text: str) -> None: + QMessageBox.information(self, title, text) + + def _ask_directory(self) -> str | None: + path = QFileDialog.getExistingDirectory(self, "Choose the Windows app's folder") + return path or None + + def closeEvent(self, event: Any) -> None: + self._timer.stop() + super().closeEvent(event) + + def reject(self) -> None: + # A running import owns the DB and the file copies; closing the dialog + # out from under it would leave a half-copied model with no UI to say so. + if self._running: + return + self._timer.stop() + super().reject() + + +def summarize(result: ImportResult) -> str: + """Plain-language "here's what landed", for the completion box.""" + lines: list[str] = [] + if result.models_imported: + lines.append(f"{result.models_imported} model(s) imported") + if result.models_updated: + lines.append(f"{result.models_updated} model(s) updated") + if result.checkpoints_copied: + lines.append(f"{result.checkpoints_copied} trained model file(s) copied") + if result.images_copied: + lines.append(f"{result.images_copied} training image(s) copied") + if result.headstamps_imported: + lines.append(f"{result.headstamps_imported} headstamp(s) imported") + if result.parents_imported: + lines.append(f"{result.parents_imported} parent classification(s) imported") + if result.slots_assigned: + lines.append(f"{result.slots_assigned} slot assignment(s) restored") + if result.serial_imported: + lines.append("Serial / board settings imported") + if result.image_processing_imported: + lines.append("Image-processing settings imported") + if result.ai_config_imported: + lines.append("AI Config imported") + if not lines: + lines.append("Nothing needed importing — everything was already here.") + if result.warnings: + lines.append("") + lines.extend(result.warnings) + return "\n".join(lines) + + +# ----- entry points ----------------------------------------------------------- + + +def open_import_dialog(win: Any, root: Path | None, *, first_run: bool = False) -> WinFormsImportDialog: + dialog = WinFormsImportDialog(win, root, win, first_run=first_run) + dialog.exec() + return dialog + + +def maybe_offer_first_run(win: Any) -> WinFormsImportDialog | None: + """Offer the import once, on a launch where there is something to offer. + + Returns None — silently — for the overwhelmingly common case of a user who + never ran the Windows app. The offer is marked as made either way, since + Settings → Import from Windows keeps it reachable forever. + """ + root = winforms_import.should_offer_first_run(win.db) + if root is None: + return None + winforms_import.mark_first_run_offered(win.db) + return open_import_dialog(win, root, first_run=True) + + +def build_winforms_import_section(win: Any) -> QWidget: + """Settings → Import from Windows: the same dialog, on a folder you pick.""" + page = QWidget() + column = QVBoxLayout(page) + column.setSpacing(10) + column.setAlignment(Qt.AlignmentFlag.AlignTop) + + title = QLabel(TITLE, page) + title.setObjectName("sectionTitle") + column.addWidget(title) + column.addWidget(_muted(SETTINGS_INTRO, page)) + column.addWidget(_muted(NON_DESTRUCTIVE, page)) + + detected = winforms_import.find_installation() + column.addWidget(_muted(f"Found: {detected}" if detected else _NOT_FOUND, page)) + + button = QPushButton("Import from the Windows app…", page) + # `action` paints it as the primary; the second name is how a test finds + # it (`findChild`) without an attribute stuffed onto the page. + button.setObjectName("action") + button.setProperty("role", IMPORT_BUTTON_ROLE) + button.clicked.connect(lambda: open_import_dialog(win, winforms_import.find_installation())) + # In a row with a trailing stretch, not straight into the column: a primary + # button stretched to the full width of the settings pane reads as a banner. + button_row = QHBoxLayout() + button_row.addWidget(button) + button_row.addStretch(1) + column.addLayout(button_row) + return page diff --git a/src/sorter/ui/models_page.py b/src/sorter/ui/models_page.py index ea7967f..458e626 100644 --- a/src/sorter/ui/models_page.py +++ b/src/sorter/ui/models_page.py @@ -56,7 +56,7 @@ from .. import paths from ..data.model_io import ExportMode, export_model, find_update_target, import_model -from ..data.models import Model, is_foreign_model, is_trainable +from ..data.models import Model, is_foreign_model, is_trainable, model_mode_label from ..data.repository import ( CartridgeRepo, HeadstampRepo, @@ -545,7 +545,7 @@ def _add_model_row( ACTIVE_MARK if active else "", cartridge_name, describe_type(model), - model.model_mode, + model_mode_label(model.model_mode), str(image_count), trained, formatting.format_datetime(model.last_training_date), @@ -555,7 +555,7 @@ def _add_model_row( ACTIVE_MARK if active else "", cartridge_name.casefold(), describe_type(model).casefold(), - model.model_mode.casefold(), + model_mode_label(model.model_mode).casefold(), image_count, trained, # Sort on the raw stored date, not the locale-formatted @@ -663,7 +663,7 @@ def _announce_active(self, active_id: int | None) -> None: return model = self.models.get(active_id) if model is not None: - self._win.set_status(f"Active model: {model.name} ({model.model_mode}).") + self._win.set_status(f"Active model: {model.name} ({model_mode_label(model.model_mode)}).") # ----- create / edit / delete --------------------------------------------- diff --git a/src/sorter/ui/theme.py b/src/sorter/ui/theme.py index 328878e..09b94cb 100644 --- a/src/sorter/ui/theme.py +++ b/src/sorter/ui/theme.py @@ -268,6 +268,43 @@ def build_stylesheet(palette: dict[str, str]) -> str: QWidget#rowActions {{ background: transparent; }} QTreeWidget#modelTable QPushButton {{ padding: 2px 8px; }} +/* The Windows-import picker. The only tree in the app whose items are + checkable, so it needs the `::indicator` block the plain QCheckBox rules + below cannot reach — left to the platform style these all but vanish on a + dark surface and every row reads as a plain label. Same three states, same + `action` fill for "on", plus `indeterminate` for a model showing only part + of its branch. */ +QTreeWidget#importTree {{ + background-color: {c["bg_input"]}; + color: {c["text"]}; + border: 1px solid {c["border"]}; +}} +QTreeWidget#importTree::item {{ padding: 3px 4px; }} +/* A row this install has nothing behind. Without this it reads as available: + the widget's own `color` wins over the palette's disabled group, so the + `QCheckBox:disabled` rule below has no equivalent effect here. */ +QTreeWidget#importTree::item:disabled {{ color: {c["text_subtle"]}; }} +QTreeWidget#importTree::item:selected {{ + background-color: {c["bg_card_sel"]}; + color: {c["text_highlight"]}; +}} +QTreeWidget#importTree::indicator {{ + width: {INDICATOR_SIZE}px; + height: {INDICATOR_SIZE}px; + border: 1px solid {c["border_focus"]}; + border-radius: 3px; + background-color: {c["bg_surface"]}; +}} +QTreeWidget#importTree::indicator:disabled {{ border-color: {c["border"]}; }} +QTreeWidget#importTree::indicator:indeterminate {{ + border-color: {c["action"]}; + background-color: {c["bg_card_sel"]}; +}} +QTreeWidget#importTree::indicator:checked {{ + background-color: {c["action"]}; + border-color: {c["action"]}; +}} + QFrame#slotCard {{ background-color: {c["bg_card"]}; border: 1px solid {c["border"]}; diff --git a/src/sorter/ui/train_page.py b/src/sorter/ui/train_page.py index 56e4b8a..5a076e0 100644 --- a/src/sorter/ui/train_page.py +++ b/src/sorter/ui/train_page.py @@ -70,7 +70,7 @@ ) from .. import paths -from ..data.models import CheckpointEnv, Model, is_trainable +from ..data.models import CheckpointEnv, Model, is_openai_model, is_trainable, model_mode_label from ..data.repository import HeadstampRepo, ModelRepo, SettingsRepo from ..hardware import image_proc from ..ml import classifier, local_inference @@ -105,6 +105,13 @@ "on the Models page to train one here." ) UNAVAILABLE_TITLE_FOREIGN = "This model is managed by its publisher" +UNAVAILABLE_TITLE_OPENAI = "This model classifies over HTTP" +UNAVAILABLE_TEXT_OPENAI = ( + "“{name}” is an OpenAI model: an HTTP server does the recognising, so " + "there is nothing on this machine to train.\n\nIts server settings and " + "headstamps live on the AI Config page. To train locally, activate or " + "create a ConvNeXt model on the Models page." +) MODELS_JUMP_TEXT = "Open the Models page" @@ -366,7 +373,7 @@ def refresh(self) -> None: self._update_buttons(model) return - self.active_label.setText(f"{model.name} ({model.model_mode})") + self.active_label.setText(f"{model.name} ({model_mode_label(model.model_mode)})") # The DB owns the headstamp list, but a filename on disk is a label the # user has actually used. Back-fill anything the DB is missing (a # pre-existing folder, an images-only import) so the field, the Sort @@ -394,6 +401,9 @@ def _apply_availability(self, model: Model | None) -> None: if model is None: self.unavailable_title.setText(UNAVAILABLE_TITLE_AI) self.unavailable_text.setText(UNAVAILABLE_TEXT_AI) + elif is_openai_model(model): + self.unavailable_title.setText(UNAVAILABLE_TITLE_OPENAI) + self.unavailable_text.setText(UNAVAILABLE_TEXT_OPENAI.format(name=model.name)) else: self.unavailable_title.setText(UNAVAILABLE_TITLE_FOREIGN) self.unavailable_text.setText(FOREIGN_MODEL_TEXT.format(name=model.name)) diff --git a/tests/unit/data/test_db.py b/tests/unit/data/test_db.py index db015f5..f984523 100644 --- a/tests/unit/data/test_db.py +++ b/tests/unit/data/test_db.py @@ -453,3 +453,111 @@ def test_model_mode_check_constraint(tmp_path: Path) -> None: "INSERT INTO models(name, cartridge_id, model_mode) VALUES (?, ?, ?)", ("bad-mode", cart_id, "resnet50"), ) + + +# ----- widening the model_mode CHECK for 'openai' ----------------------------- + + +def _write_pre_openai_db(path: Path) -> None: + """A database as a pre-openai build's DDL laid it down. + + The load-bearing details: the four-mode CHECK on `models.model_mode` + (what `_widen_model_mode_check` exists to widen) and a child table whose + `ON DELETE CASCADE` references `models` — the rebuild must not fire it. + """ + conn = sqlite3.connect(path, isolation_level=None) + conn.executescript( + """ + PRAGMA foreign_keys = ON; + CREATE TABLE cartridges ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + name TEXT NOT NULL UNIQUE + ); + CREATE TABLE models ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + name TEXT NOT NULL, + cartridge_id INTEGER NOT NULL REFERENCES cartridges(id) ON DELETE RESTRICT, + model_mode TEXT NOT NULL + CHECK(model_mode IN ('convnext_tiny','convnext_small','convnext_base','convnext_large')), + model_type TEXT NOT NULL DEFAULT 'Standard' + CHECK(model_type IN ('Standard','ReadOnly','CommunityManaged')), + community_model_uid TEXT, + model_version INTEGER NOT NULL DEFAULT 1, + enable_image_processing INTEGER NOT NULL DEFAULT 1, + image_processing_json TEXT, + training_config_json TEXT, + ai_model_config_json TEXT, + use_primer_mask INTEGER NOT NULL DEFAULT 0, + hide_primer INTEGER NOT NULL DEFAULT 1, + primer_mask_size INTEGER NOT NULL DEFAULT 135, + last_training_date TEXT, + last_training_duration INTEGER NOT NULL DEFAULT 0, + trained_image_count INTEGER NOT NULL DEFAULT 0, + training_confusion_table TEXT, + feedback_loop_enabled INTEGER NOT NULL DEFAULT 0, + feedback_loop_confidence_floor INTEGER NOT NULL DEFAULT 95, + feedback_loop_upload_mode TEXT NOT NULL DEFAULT 'Manual', + model_path TEXT, + created_at TEXT NOT NULL DEFAULT (datetime('now')), + updated_at TEXT NOT NULL DEFAULT (datetime('now')) + ); + CREATE INDEX idx_models_cartridge ON models(cartridge_id); + CREATE TABLE headstamps ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + name TEXT NOT NULL, + model_id INTEGER NOT NULL REFERENCES models(id) ON DELETE CASCADE, + slot INTEGER NOT NULL DEFAULT 0, + parent_id INTEGER, + UNIQUE(model_id, name) + ); + CREATE TABLE settings ( + key TEXT PRIMARY KEY, + value TEXT NOT NULL + ); + """ + ) + conn.execute("INSERT INTO cartridges(id, name) VALUES (1, '9mm')") + conn.execute("INSERT INTO models(id, name, cartridge_id, model_mode) VALUES (1, 'Kept', 1, 'convnext_small')") + conn.execute("INSERT INTO headstamps(name, model_id, slot) VALUES ('WIN', 1, 3)") + conn.execute("PRAGMA user_version = 5") + conn.close() + + +def test_model_mode_check_is_widened_for_openai(tmp_path: Path) -> None: + """A pre-openai database is rebuilt so 'openai' rows can exist at all. + + The dangerous part is what must NOT happen: the rebuild drops the old + `models` table with `headstamps` still cascading off it, so a slip in the + foreign-key handling silently deletes every headstamp. + """ + path = tmp_path / "old.db" + _write_pre_openai_db(path) + + db = Database(path) + db.ensure_initialized() + + # The CHECK now admits openai — this used to raise IntegrityError. + from sorter.data.models import Model + + created = ModelRepo(db).create(Model(name="HTTP model", cartridge_id=1, model_mode="openai")) + assert created.id > 0 + + kept = ModelRepo(db).get(1) + assert kept is not None and kept.name == "Kept" and kept.model_mode == "convnext_small" + # The cascade did not fire during the rebuild. + assert [h.name for h in HeadstampRepo(db).list_for_model(1)] == ["WIN"] + # Foreign keys are back on for the rest of the app's lifetime. + assert db.conn.execute("PRAGMA foreign_keys").fetchone()[0] == 1 + db.close() + + +def test_model_mode_widening_is_idempotent(tmp_path: Path) -> None: + path = tmp_path / "old.db" + _write_pre_openai_db(path) + db = Database(path) + db.ensure_initialized() + first_sql = db.conn.execute("SELECT sql FROM sqlite_master WHERE name='models'").fetchone()[0] + db.ensure_initialized() # the guard sees 'openai' already admitted + assert db.conn.execute("SELECT sql FROM sqlite_master WHERE name='models'").fetchone()[0] == first_sql + assert ModelRepo(db).get(1) is not None + db.close() diff --git a/tests/unit/data/test_model_io.py b/tests/unit/data/test_model_io.py index 71af302..699db78 100644 --- a/tests/unit/data/test_model_io.py +++ b/tests/unit/data/test_model_io.py @@ -10,14 +10,17 @@ from sorter.data.db import Database from sorter.data.model_io import ( + _WINFORMS_MODELMODE_INT_TO_STR, + WINFORMS_MODELMODE_OPENAI, ExportMode, + _normalize_model_mode, export_model, find_update_target, import_model, model_from_export_dict, read_manifest, ) -from sorter.data.models import CheckpointEnv, Model +from sorter.data.models import MODEL_MODES, OPENAI_MODEL_MODE, CheckpointEnv, Model from sorter.data.repository import CartridgeRepo, HeadstampRepo, ModelRepo, SettingsRepo @@ -220,6 +223,22 @@ def test_unknown_model_mode_in_manifest_falls_back_to_tiny(tmp_path: Path) -> No assert _get_model(db, mid).model_mode == "convnext_tiny" +def test_every_legacy_model_mode_maps_to_one_this_app_accepts() -> None: + """`ModelRepo` rejects anything outside `MODEL_MODES`. + + `winforms_import` feeds `_normalize_model_mode` straight to it with no + clamp of its own, so a mapping to a mode this app does not accept is not a + cosmetic slip — it aborts that model's import. The legacy OpenAI mode (2) + once mapped to "openai" while `ModelRepo` still rejected it, and did + exactly that; "openai" is a first-class mode now, so the same mapping is + the correct one. + """ + for legacy_int, mode in _WINFORMS_MODELMODE_INT_TO_STR.items(): + assert mode in MODEL_MODES, f"ModelMode {legacy_int} maps to {mode!r}" + assert _normalize_model_mode(WINFORMS_MODELMODE_OPENAI) == OPENAI_MODEL_MODE + assert _normalize_model_mode("openai") == OPENAI_MODEL_MODE + + def test_winforms_pascal_manifest_picks_up_training_config(tmp_path: Path) -> None: """A legacy export uses PascalCase keys and ints for the ModelMode enum. This app must pull the training image size, model architecture, primer diff --git a/tests/unit/data/test_winforms_import.py b/tests/unit/data/test_winforms_import.py new file mode 100644 index 0000000..e6a5755 --- /dev/null +++ b/tests/unit/data/test_winforms_import.py @@ -0,0 +1,931 @@ +"""Tests for importing a legacy WinForms ("AI Brass Sorter") installation. + +Every fixture here is synthetic. The shapes are copied from a real 1.3.8 +install, but nothing in the suite touches `C:\\Program Files` — see +`_write_install` for the layout being mimicked. +""" + +from __future__ import annotations + +import json +import zipfile +from pathlib import Path +from typing import Any + +import pytest + +from sorter import paths +from sorter.data.config import Config +from sorter.data.db import Database +from sorter.data.models import is_trainable +from sorter.data.repository import ( + HeadstampParentRepo, + HeadstampRepo, + ModelRepo, + SettingsRepo, +) +from sorter.data.winforms_import import ( + CONFIG_DB_NAME, + FIRST_RUN_SEEN_KEY, + MLNET_WARNING, + MODEL_FAILED_WARNING, + ImportOptions, + ModelSelection, + candidate_install_dirs, + find_installation, + import_installation, + looks_like_installation, + mark_first_run_offered, + should_offer_first_run, + survey, +) + +# ----- synthetic legacy install ---------------------------------------------- + +# A model row as the legacy app writes it: PascalCase, enum ints for +# ModelType/ModelMode, and the two separate training-config blobs. +_MODEL_OWN: dict[str, Any] = { + "Id": 3, + "Name": "9mm Base Model", + "CartridgeId": 2, + "UsePrimerMask": False, + "PrimerMaskSize": 0, + "HidePrimer": True, + "EnableImageProcessing": True, + "ImageProcessingConfig": {"imageProcessingMode": 0, "edgeFilter1": 100}, + "AIModelConfig": None, + "PythonTrainingConfig": None, + "ModelTrainingConfig": None, + "UseParentClassificationsAtRuntime": False, + "FeedbackLoopEnabled": False, + "FeedbackLoopConfidenceFloor": 95, + "FeedbackLoopUploadMode": 0, + "CommunityModelUID": None, + "ModelVersion": 1, + "ModelType": 0, # Standard — the user's own + "ModelMode": 0, +} + +_MODEL_COMMUNITY: dict[str, Any] = { + "Id": 4, + "Name": "9mm Default", + "CartridgeId": 2, + "UsePrimerMask": False, + "PrimerMaskSize": 140, + "HidePrimer": True, + "EnableImageProcessing": True, + "ModelTrainingConfig": {"ModelName": "convnext_small", "ImageSize": 480}, + "UseParentClassificationsAtRuntime": True, + "FeedbackLoopEnabled": True, + "FeedbackLoopConfidenceFloor": 97, + "FeedbackLoopUploadMode": 0, + "CommunityModelUID": "12e3af83-9d6c-4f6c-b045-7a984420543d", + "ModelVersion": 16, + "ModelType": 1, # ReadOnly — a community download + "ModelMode": 7, # convnext_small +} + +# ModelMode 2 is the legacy OpenAI enum: this model classified over HTTP rather +# than from a local checkpoint. Reported on PR #125 against a real install. +_MODEL_OPENAI: dict[str, Any] = { + "Id": 5, + "Name": "9mm over HTTP", + "CartridgeId": 2, + "ModelType": 0, + "ModelMode": 2, + "AIModelConfig": { + "OpenAI_EndpointUrl": "http://192.168.1.9:1234/v1/chat/completions", + "OpenAI_Model": "qwen2-vl", + }, +} + +_DEFAULTS: dict[str, Any] = { + "SlotQuantity": 8, + "DefaultSerialPort": "COM3", + "DefaultModelId": 4, + "IP_Resolution": 5, + "IP_Sensitivity": 1.0, + "IP_Padding": 1, + "IP_BgCliff": 25, + "InitSettings": [ + {"Key": "sortspeed", "Value": "90"}, + {"Key": "feedsteps", "Value": "60"}, + {"Key": "cameraledlevel", "Value": "220"}, + {"Key": "notarealboardsetting", "Value": "1"}, + ], +} + + +def _torch_zip(path: Path) -> None: + """A stand-in for `training/models/.zip`. + + `torch.save` writes a zip container holding `/data.pkl`; that entry + is the whole signal `_checkpoint_kind` keys off, so the fixture needs + nothing else to be classified correctly. + """ + path.parent.mkdir(parents=True, exist_ok=True) + with zipfile.ZipFile(path, "w") as zf: + zf.writestr("trainedmodel/data.pkl", b"\x80\x02}q\x00.") + zf.writestr("trainedmodel/data/0", b"\x00" * 16) + + +def _mlnet_zip(path: Path) -> None: + """The legacy ML.NET pipeline's model, which shares the `.zip` extension.""" + path.parent.mkdir(parents=True, exist_ok=True) + with zipfile.ZipFile(path, "w") as zf: + zf.writestr("TransformerChain/Model.key", b"key") + zf.writestr("Schema", b"schema") + + +def _write_install( + root: Path, + *, + models: list[dict[str, Any]] | None = None, + headstamps: list[dict[str, Any]] | None = None, + parents: list[dict[str, Any]] | None = None, + parent_links: list[dict[str, Any]] | None = None, + slots: list[dict[str, Any]] | None = None, + defaults: dict[str, Any] | None = None, + settings_json: dict[str, Any] | None = None, +) -> Path: + """Lay down a legacy install tree under `root` and return it.""" + data = root / "Data" + data.mkdir(parents=True, exist_ok=True) + document = { + "Models": models if models is not None else [_MODEL_OWN, _MODEL_COMMUNITY], + "Cartridges": [{"Id": 2, "Name": "9mm"}], + "Headstamps": headstamps if headstamps is not None else [], + "HeadStampParents": parents or [], + "HeadStampParentLinks": parent_links or [], + "SlotConfigs": slots or [], + "SortingTemplates": [], + "CommunityNotes": [], + "Defaults": defaults if defaults is not None else dict(_DEFAULTS), + } + # The app writes these with a BOM; `utf-8-sig` on the read side is what + # keeps the first key from arriving as "\ufeffModels". + (root / CONFIG_DB_NAME).write_text(json.dumps(document), encoding="utf-8-sig") + if settings_json is not None: + (data / "Settings.json").write_text(json.dumps(settings_json), encoding="utf-8-sig") + return root + + +def _add_images(root: Path, legacy_id: int, names: list[str]) -> None: + folder = root / "training" / "images" / str(legacy_id) + folder.mkdir(parents=True, exist_ok=True) + for name in names: + (folder / name).write_bytes(b"\xff\xd8\xff\xe0jpeg-ish") + + +def _model(db: Database, name: str) -> Any: + """The imported row, by name. + + Not ``list()[-1]``: ``ModelRepo.list`` sorts by name, and a fresh DB is + seeded with a "Default" model that shares the table. + """ + found = next((m for m in ModelRepo(db).list() if m.name == name), None) + assert found is not None, f"{name!r} not imported" + return found + + +@pytest.fixture +def db(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Database: + """A real SQLite DB with the data root redirected under tmp_path.""" + monkeypatch.setenv("CASESORTER_DATA_DIR", str(tmp_path / "appdata")) + database = Database(tmp_path / "appdata" / "config" / "casesorter.db") + database.ensure_initialized() + return database + + +@pytest.fixture +def config(db: Database) -> Config: + return Config(db).load() + + +# ----- discovery -------------------------------------------------------------- + + +def test_candidate_dirs_follow_program_files_env(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("ProgramFiles", r"D:\Apps") + monkeypatch.delenv("ProgramW6432", raising=False) + monkeypatch.delenv("ProgramFiles(x86)", raising=False) + assert candidate_install_dirs() == [Path(r"D:\Apps") / "SJSeth" / "AI Brass Sorter"] + + +def test_find_installation_returns_none_when_absent(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """The whole feature has to stay invisible to a user with no Windows app.""" + monkeypatch.setenv("ProgramFiles", str(tmp_path / "nothing-here")) + monkeypatch.delenv("ProgramW6432", raising=False) + monkeypatch.delenv("ProgramFiles(x86)", raising=False) + assert find_installation() is None + + +def test_find_installation_picks_up_a_real_layout(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + root = tmp_path / "pf" / "SJSeth" / "AI Brass Sorter" + _write_install(root) + monkeypatch.setenv("ProgramFiles", str(tmp_path / "pf")) + monkeypatch.delenv("ProgramW6432", raising=False) + monkeypatch.delenv("ProgramFiles(x86)", raising=False) + assert find_installation() == root + assert looks_like_installation(root) + assert not looks_like_installation(tmp_path / "elsewhere") + + +# ----- survey ----------------------------------------------------------------- + + +def test_survey_counts_what_is_there(tmp_path: Path) -> None: + root = _write_install( + tmp_path / "legacy", + headstamps=[ + {"Id": 1, "Name": "GECO", "Model_Id": 3}, + {"Id": 2, "Name": "S&B", "Model_Id": 3}, + {"Id": 3, "Name": "FC", "Model_Id": 4}, + ], + ) + _add_images(root, 3, ["GECO__1.jpg", "S&B__2.jpg", "notes.txt"]) + _torch_zip(root / "training" / "models" / "4.zip") + + found = survey(root) + assert not found.is_empty + assert [m.legacy_id for m in found.models] == [3, 4] + own, community = found.models + assert own.image_count == 2 # notes.txt is not an image + assert own.headstamp_count == 2 + assert own.checkpoint_kind == "none" + assert community.headstamp_count == 1 + assert community.has_usable_checkpoint + assert found.total_images == 2 + assert found.total_headstamps == 3 + assert found.has_serial and found.has_image_processing + assert not found.has_ai_config + + +def test_survey_flags_an_mlnet_only_model(tmp_path: Path) -> None: + """An ML.NET checkpoint cannot classify here, and must not be copied as one.""" + root = _write_install(tmp_path / "legacy", models=[_MODEL_OWN]) + _mlnet_zip(root / "training" / "models" / "3.zip") + found = survey(root) + assert found.models[0].checkpoint_kind == "mlnet" + assert not found.models[0].has_usable_checkpoint + # Equality against the exported constant, not a substring probe: the + # message is one string owned by the module, and CodeQL reads a + # `"ML.NET" in ...` test as a hostname allowlist check. + assert MLNET_WARNING.format(name="9mm Base Model") in found.warnings + + +def test_a_warning_about_a_skipped_model_is_not_reported(tmp_path: Path, db: Database, config: Config) -> None: + """The survey warns about the whole install; the import warns about the pick. + + Telling a user their ML.NET model came across without a checkpoint, when + they deliberately unticked it, is a report on work that never happened. + """ + root = _write_install(tmp_path / "legacy") # _MODEL_OWN (3) + _MODEL_COMMUNITY (4) + _mlnet_zip(root / "training" / "models" / "4.zip") + assert MLNET_WARNING.format(name="9mm Default") in survey(root).warnings + + result = import_installation( + root, + db=db, + config=config, + options=ImportOptions(per_model={3: ModelSelection()}), + ) + + assert result.warnings == [] + + +def test_survey_does_not_warn_about_an_openai_mode_model(tmp_path: Path) -> None: + """ "openai" is a first-class mode now, so the row imports faithfully.""" + root = _write_install(tmp_path / "legacy", models=[_MODEL_OPENAI]) + found = survey(root) + assert found.warnings == [] + assert found.has_ai_config + + +def test_survey_rejects_a_folder_that_is_not_an_installation(tmp_path: Path) -> None: + with pytest.raises(FileNotFoundError): + survey(tmp_path) + + +def test_survey_is_empty_for_a_bare_installation(tmp_path: Path) -> None: + """Nothing to offer → the first-run check stays silent.""" + root = _write_install(tmp_path / "legacy", models=[], defaults={}) + assert survey(root).is_empty + + +# ----- importing models ------------------------------------------------------- + + +def test_import_brings_models_across_with_ownership_intact(tmp_path: Path, db: Database, config: Config) -> None: + root = _write_install(tmp_path / "legacy") + _torch_zip(root / "training" / "models" / "4.zip") + + result = import_installation(root, db=db, config=config, options=ImportOptions()) + + assert result.models_imported == 2 + assert result.models_updated == 0 + assert result.checkpoints_copied == 1 + + by_name = {m.name: m for m in ModelRepo(db).list()} + own = by_name["9mm Base Model"] + community = by_name["9mm Default"] + # ModelType 0 is the user's own work and stays trainable; ModelType 1 is + # someone else's checkpoint and must not be. + assert own.model_type == "Standard" + assert is_trainable(own) + assert community.model_type == "ReadOnly" + assert not is_trainable(community) + # ModelMode 7 is the legacy enum for convnext_small. + assert community.model_mode == "convnext_small" + assert community.community_model_uid == "12e3af83-9d6c-4f6c-b045-7a984420543d" + assert Path(community.model_path or "").is_file() + + +def test_import_accepts_an_openai_mode_model(tmp_path: Path, db: Database, config: Config) -> None: + """The whole import used to die here. + + `ModelMode` 2 mapped to a literal "openai", which `ModelRepo` rejects, so + one such row aborted the transaction and nothing at all was imported — + the failure reported against a real install on PR #125. + """ + root = _write_install(tmp_path / "legacy", models=[_MODEL_OWN, _MODEL_OPENAI]) + _add_images(root, 5, ["GECO__1.jpg"]) + + result = import_installation(root, db=db, config=config, options=ImportOptions()) + + assert result.models_imported == 2 + imported = _model(db, "9mm over HTTP") + # First-class: the row keeps its mode and its own server settings, so it + # classifies over HTTP here exactly as it did there (PR #125 review). + assert imported.model_mode == "openai" + assert not is_trainable(imported) + assert imported.ai_model_config.endpoint_url == "http://192.168.1.9:1234/v1/chat/completions" + assert imported.ai_model_config.model == "qwen2-vl" + assert result.images_copied == 1 + + +def test_one_unimportable_model_does_not_cost_the_others( + tmp_path: Path, db: Database, config: Config, monkeypatch: pytest.MonkeyPatch +) -> None: + """A single bad row is worth a warning, not an install's worth of images.""" + root = _write_install(tmp_path / "legacy") + _add_images(root, 4, ["FC__1.jpg"]) + + real_create = ModelRepo.create + + def refuse_one(self: ModelRepo, model: Any) -> Any: + if model.name == "9mm Base Model": + raise ValueError("Unsupported model_mode: 'nonsense'") + return real_create(self, model) + + monkeypatch.setattr(ModelRepo, "create", refuse_one) + result = import_installation(root, db=db, config=config, options=ImportOptions()) + + assert result.models_imported == 1 + assert MODEL_FAILED_WARNING.format(name="9mm Base Model", error="Unsupported model_mode: 'nonsense'") in ( + result.warnings + ) + names = {m.name for m in ModelRepo(db).list()} + assert "9mm Default" in names + # Rolled back to just before the failing row, counts included. + assert "9mm Base Model" not in names + assert result.images_copied == 1 + + +def test_import_copies_training_images(tmp_path: Path, db: Database, config: Config) -> None: + root = _write_install(tmp_path / "legacy", models=[_MODEL_OWN]) + _add_images(root, 3, ["GECO__1.jpg", "S&B__2.jpg"]) + + result = import_installation(root, db=db, config=config, options=ImportOptions()) + + assert result.images_copied == 2 + model = _model(db, "9mm Base Model") + from sorter import paths + + landed = sorted(p.name for p in paths.model_images_dir(model.id).iterdir()) + # The `{label}__{ticks}.jpg` names carry the labels, so they must survive. + assert landed == ["GECO__1.jpg", "S&B__2.jpg"] + + +def test_import_skips_an_mlnet_checkpoint_but_keeps_the_model(tmp_path: Path, db: Database, config: Config) -> None: + """A shell to retrain into: metadata and images yes, unusable weights no.""" + root = _write_install(tmp_path / "legacy", models=[_MODEL_OWN]) + _mlnet_zip(root / "training" / "models" / "3.zip") + _add_images(root, 3, ["GECO__1.jpg"]) + + result = import_installation(root, db=db, config=config, options=ImportOptions()) + + assert result.models_imported == 1 + assert result.checkpoints_copied == 0 + assert result.images_copied == 1 + imported = _model(db, "9mm Base Model") + assert imported.model_path is None + assert is_trainable(imported) + + +def test_import_leaves_the_source_installation_untouched(tmp_path: Path, db: Database, config: Config) -> None: + root = _write_install(tmp_path / "legacy") + _add_images(root, 3, ["GECO__1.jpg"]) + _torch_zip(root / "training" / "models" / "4.zip") + before = {p.relative_to(root): p.stat().st_size for p in root.rglob("*") if p.is_file()} + + import_installation(root, db=db, config=config, options=ImportOptions()) + + after = {p.relative_to(root): p.stat().st_size for p in root.rglob("*") if p.is_file()} + assert before == after + + +# ----- headstamps, parents and slots ----------------------------------------- + + +def test_import_restores_headstamps_parents_and_slots(tmp_path: Path, db: Database, config: Config) -> None: + root = _write_install( + tmp_path / "legacy", + models=[_MODEL_OWN], + headstamps=[ + {"Id": 1, "Name": "G.F.L. (Fiocci)", "Model_Id": 3}, + {"Id": 2, "Name": "GECO", "Model_Id": 3}, + {"Id": 9, "Name": "OTHER MODEL", "Model_Id": 99}, + ], + parents=[{"Id": 1, "Name": "FIOCCHI", "Model_Id": 3}], + parent_links=[{"Id": 1, "ParentId": 1, "HeadStampId": 1, "ModelId": 3}], + slots=[ + { + "SlotNumber": 2, + "ModelId": 3, + "PackageMode": False, + "Config": [{"Headstamp": {"Id": 1, "Name": "G.F.L. (Fiocci)"}, "Counter": 0}], + "ParentConfig": [], + }, + { + "SlotNumber": 5, + "ModelId": 3, + "PackageMode": False, + "Config": [], + "ParentConfig": [{"Name": "FIOCCHI"}], + }, + ], + ) + + result = import_installation(root, db=db, config=config, options=ImportOptions()) + + model = _model(db, "9mm Base Model") + headstamps = {h.name: h for h in HeadstampRepo(db).list_for_model(model.id)} + # A headstamp belonging to another model must not leak into this one. + assert set(headstamps) == {"G.F.L. (Fiocci)", "GECO"} + assert headstamps["G.F.L. (Fiocci)"].slot == 2 + assert headstamps["GECO"].slot == 0 + parents = HeadstampParentRepo(db).list_for_model(model.id) + assert [(p.name, p.slot) for p in parents] == [("FIOCCHI", 5)] + assert headstamps["G.F.L. (Fiocci)"].parent_id == parents[0].id + assert result.headstamps_imported == 2 + assert result.parents_imported == 1 + assert result.slots_assigned == 2 + + +def test_import_restores_package_mode_slots(tmp_path: Path, db: Database, config: Config) -> None: + """Package assignments are many-to-many and live in a settings key, not a row.""" + root = _write_install( + tmp_path / "legacy", + models=[_MODEL_OWN], + headstamps=[{"Id": 1, "Name": "GECO", "Model_Id": 3}], + slots=[ + { + "SlotNumber": 1, + "ModelId": 3, + "PackageMode": True, + "Config": [{"Headstamp": {"Name": "GECO"}}], + "ParentConfig": [], + }, + { + "SlotNumber": 2, + "ModelId": 3, + "PackageMode": True, + "Config": [{"Headstamp": {"Name": "GECO"}}], + "ParentConfig": [], + }, + ], + ) + + import_installation(root, db=db, config=config, options=ImportOptions()) + + model = _model(db, "9mm Base Model") + SettingsRepo(db).set_active_model_id(model.id) + assert config.package_slot_map() == {1: ["GECO"], 2: ["GECO"]} + # The standard-mode row is untouched by a package-mode assignment. + assert HeadstampRepo(db).list_for_model(model.id)[0].slot == 0 + + +def test_parent_runtime_preference_follows_the_model(tmp_path: Path, db: Database, config: Config) -> None: + root = _write_install(tmp_path / "legacy", models=[_MODEL_COMMUNITY]) + import_installation(root, db=db, config=config, options=ImportOptions()) + SettingsRepo(db).set_active_model_id(_model(db, "9mm Default").id) + assert config.use_parent_classifications + + +# ----- re-import -------------------------------------------------------------- + + +def test_reimport_updates_in_place_and_keeps_slots(tmp_path: Path, db: Database, config: Config) -> None: + """Running the import twice must not fork the library or lose a layout.""" + root = _write_install( + tmp_path / "legacy", + models=[_MODEL_OWN], + headstamps=[{"Id": 1, "Name": "GECO", "Model_Id": 3}], + slots=[ + { + "SlotNumber": 4, + "ModelId": 3, + "PackageMode": False, + "Config": [{"Headstamp": {"Name": "GECO"}}], + "ParentConfig": [], + } + ], + ) + _add_images(root, 3, ["GECO__1.jpg"]) + + first = import_installation(root, db=db, config=config, options=ImportOptions()) + before = len(ModelRepo(db).list()) + second = import_installation(root, db=db, config=config, options=ImportOptions()) + + assert first.models_imported == 1 + assert second.models_imported == 0 + assert second.models_updated == 1 + assert len(ModelRepo(db).list()) == before + # Already-copied images are skipped rather than re-copied. + assert second.images_copied == 0 + model = _model(db, "9mm Base Model") + assert HeadstampRepo(db).list_for_model(model.id)[0].slot == 4 + + +def test_reimport_recognises_an_already_installed_community_model(tmp_path: Path, db: Database, config: Config) -> None: + """A UID match wins over the per-root memory: same model, wherever it came from.""" + root_a = _write_install(tmp_path / "a", models=[_MODEL_COMMUNITY]) + root_b = _write_install(tmp_path / "b", models=[_MODEL_COMMUNITY]) + + import_installation(root_a, db=db, config=config, options=ImportOptions()) + count = len(ModelRepo(db).list()) + result = import_installation(root_b, db=db, config=config, options=ImportOptions()) + + assert result.models_updated == 1 + assert len(ModelRepo(db).list()) == count + + +# ----- settings --------------------------------------------------------------- + + +def test_serial_settings_import(tmp_path: Path, db: Database, config: Config) -> None: + root = _write_install(tmp_path / "legacy", settings_json={"serialBaudRate": 19200}) + + result = import_installation( + root, + db=db, + config=config, + options=ImportOptions(models=False, serial=True), + ) + + assert result.serial_imported + reloaded = Config(db).load() + assert reloaded.serial["port"] == "COM3" + assert reloaded.serial["baud"] == 19200 + assert reloaded.serial["slot_quantity"] == 8 + init = reloaded.serial["init_settings"] + assert init["sortspeed"] == 90 + assert init["cameraledlevel"] == 220 + # An unrecognised key would be written straight to the board on connect. + assert "notarealboardsetting" not in init + # Keys the legacy install didn't carry keep this app's defaults. + assert init["fan"] == 100 + + +def test_image_processing_import_touches_linescan_only(tmp_path: Path, db: Database, config: Config) -> None: + """The legacy pipeline has no Hough stage — its numbers must not reach ours.""" + root = _write_install(tmp_path / "legacy") + hough_before = dict(config.image_proc["hough"]) + + result = import_installation( + root, + db=db, + config=config, + options=ImportOptions(models=False, image_processing=True), + ) + + assert result.image_processing_imported + reloaded = Config(db).load() + assert reloaded.image_proc["linescan"]["scan_precision"] == 5 + assert reloaded.image_proc["linescan"]["bg_cliff"] == 25 + assert reloaded.image_proc["hough"] == hough_before + assert reloaded.image_proc["strategy"] == "hough" + + +def test_ai_config_import(tmp_path: Path, db: Database, config: Config) -> None: + model = dict(_MODEL_OWN) + model["AIModelConfig"] = { + "OpenAI_EndpointUrl": "http://192.168.1.5:8000", + "OpenAI_Model": "qwen-vl", + "OpenAI_SystemPrompt": "Read the headstamp: {{headstamps}}", + "ImageQuality": 90, + } + root = _write_install(tmp_path / "legacy", models=[model]) + + result = import_installation( + root, + db=db, + config=config, + options=ImportOptions(models=False, ai_config=True), + ) + + assert result.ai_config_imported + api = Config(db).load().api + assert api["endpoint_url"] == "http://192.168.1.5:8000" + assert api["model"] == "qwen-vl" + assert api["image_quality"] == 90 + + +def test_ai_config_import_reports_when_there_is_nothing_to_take(tmp_path: Path, db: Database, config: Config) -> None: + root = _write_install(tmp_path / "legacy") + result = import_installation( + root, + db=db, + config=config, + options=ImportOptions(models=False, ai_config=True), + ) + assert not result.ai_config_imported + + +def test_ai_config_prefers_the_model_that_classified_over_http(tmp_path: Path, db: Database, config: Config) -> None: + """The legacy app writes the blob on every model, most of them blank. + + Taking whichever is declared first would import an empty endpoint over the + one the user actually configured, and still report success. + """ + blank = dict(_MODEL_OWN) + blank["AIModelConfig"] = {"OpenAI_EndpointUrl": "", "OpenAI_Model": "", "OpenAI_SystemPrompt": ""} + root = _write_install(tmp_path / "legacy", models=[blank, _MODEL_OPENAI]) + + result = import_installation( + root, + db=db, + config=config, + options=ImportOptions(models=False, ai_config=True), + ) + + assert result.ai_config_imported + api = Config(db).load().api + assert api["endpoint_url"] == "http://192.168.1.9:1234/v1/chat/completions" + assert api["model"] == "qwen2-vl" + + +def test_ai_config_ignores_a_present_but_blank_blob(tmp_path: Path, db: Database, config: Config) -> None: + blank = dict(_MODEL_OWN) + blank["AIModelConfig"] = {"OpenAI_EndpointUrl": "", "OpenAI_Model": "", "ImageQuality": 90} + root = _write_install(tmp_path / "legacy", models=[blank]) + + assert not survey(root).has_ai_config + result = import_installation( + root, + db=db, + config=config, + options=ImportOptions(models=False, ai_config=True), + ) + assert not result.ai_config_imported + + +# ----- options and activation ------------------------------------------------- + + +def test_unticking_models_folds_off_its_dependents(tmp_path: Path, db: Database, config: Config) -> None: + root = _write_install( + tmp_path / "legacy", + headstamps=[{"Id": 1, "Name": "GECO", "Model_Id": 3}], + ) + _add_images(root, 3, ["GECO__1.jpg"]) + + result = import_installation( + root, + db=db, + config=config, + options=ImportOptions(models=False, training_images=True, headstamps=True, serial=True), + ) + + assert result.models_imported == 0 + assert result.images_copied == 0 + assert result.headstamps_imported == 0 + assert result.serial_imported + + +def test_per_model_selection_imports_only_the_chosen_models(tmp_path: Path, db: Database, config: Config) -> None: + """A real install carries years of models the user has no interest in.""" + root = _write_install(tmp_path / "legacy") + _add_images(root, 3, ["GECO__1.jpg"]) + _add_images(root, 4, ["FC__1.jpg", "FC__2.jpg"]) + + result = import_installation( + root, + db=db, + config=config, + options=ImportOptions(per_model={3: ModelSelection()}), + ) + + assert result.models_imported == 1 + assert result.images_copied == 1 # model 4 was not asked for + names = [m.name for m in ModelRepo(db).list()] + assert "9mm Base Model" in names + assert "9mm Default" not in names + + +def test_an_empty_per_model_map_means_no_models(tmp_path: Path, db: Database, config: Config) -> None: + """`None` is "no per-model choice was made"; `{}` is a real answer.""" + root = _write_install(tmp_path / "legacy") + before = len(ModelRepo(db).list()) + + options = ImportOptions(per_model={}, serial=True) + assert options.any_selected() # serial still is + + result = import_installation(root, db=db, config=config, options=options) + + assert result.models_imported == 0 + assert len(ModelRepo(db).list()) == before + assert result.serial_imported + + +def test_a_model_can_be_imported_without_its_parts(tmp_path: Path, db: Database, config: Config) -> None: + """Model row yes, images/headstamps/checkpoint no — each declinable alone.""" + root = _write_install( + tmp_path / "legacy", + models=[_MODEL_OWN], + headstamps=[{"Id": 1, "Name": "GECO", "Model_Id": 3}], + ) + _add_images(root, 3, ["GECO__1.jpg"]) + _torch_zip(root / "training" / "models" / "3.zip") + + result = import_installation( + root, + db=db, + config=config, + options=ImportOptions(per_model={3: ModelSelection(images=False, headstamps=False, checkpoint=False)}), + ) + + assert result.models_imported == 1 + assert result.images_copied == 0 + assert result.headstamps_imported == 0 + assert result.checkpoints_copied == 0 + imported = _model(db, "9mm Base Model") + assert not imported.model_path + assert HeadstampRepo(db).list_for_model(imported.id) == [] + + +def test_declining_a_checkpoint_leaves_the_source_file_alone(tmp_path: Path, db: Database, config: Config) -> None: + root = _write_install(tmp_path / "legacy", models=[_MODEL_OWN]) + checkpoint = root / "training" / "models" / "3.zip" + _torch_zip(checkpoint) + before = checkpoint.read_bytes() + + import_installation( + root, + db=db, + config=config, + options=ImportOptions(per_model={3: ModelSelection(checkpoint=False)}), + ) + + assert checkpoint.read_bytes() == before + copied = [p for m in ModelRepo(db).list() for p in paths.model_trained_dir(m.id).glob("*.pth")] + assert copied == [] + + +def test_a_name_already_taken_here_is_not_duplicated(tmp_path: Path, db: Database, config: Config) -> None: + """The one conflict an import can create: two rows the user cannot tell apart. + + Model ids never collide — `ModelRepo.create` allocates its own — so the name + is the whole of it, and it is resolved the way a ZIP import already resolves + it. + """ + repo = ModelRepo(db) + mine = repo.list()[0] + mine.name = "9mm Base Model" + repo.update(mine) + root = _write_install(tmp_path / "legacy", models=[_MODEL_OWN]) + + result = import_installation(root, db=db, config=config, options=ImportOptions()) + + assert result.models_imported == 1 + names = sorted(m.name for m in repo.list()) + assert names == ["9mm Base Model", "9mm Base Model (2)"] + # And the pre-existing row is untouched — the import took the new name. + still_there = repo.get(mine.id) + assert still_there is not None + assert still_there.name == "9mm Base Model" + + +def test_a_reimport_updates_rather_than_renaming(tmp_path: Path, db: Database, config: Config) -> None: + """Uniquifying must not turn idempotent re-runs into a pile of copies.""" + root = _write_install(tmp_path / "legacy", models=[_MODEL_OWN]) + import_installation(root, db=db, config=config, options=ImportOptions()) + first = [m.name for m in ModelRepo(db).list()] + + again = import_installation(root, db=db, config=config, options=ImportOptions()) + + assert again.models_imported == 0 + assert again.models_updated == 1 + assert [m.name for m in ModelRepo(db).list()] == first + + +def test_survey_says_what_each_model_would_do_to_the_library(tmp_path: Path, db: Database, config: Config) -> None: + root = _write_install(tmp_path / "legacy", models=[_MODEL_OWN]) + + assert survey(root).models[0].updates is None # no db asked, no answer + assert survey(root, db=db).models[0].updates is None + + import_installation(root, db=db, config=config, options=ImportOptions()) + + assert survey(root, db=db).models[0].updates == "9mm Base Model" + + +def test_nothing_ticked_is_a_no_op(tmp_path: Path, db: Database, config: Config) -> None: + root = _write_install(tmp_path / "legacy") + before = len(ModelRepo(db).list()) + result = import_installation( + root, + db=db, + config=config, + options=ImportOptions(models=False), + ) + assert result.models_imported == 0 + assert len(ModelRepo(db).list()) == before + + +def test_import_adopts_the_legacy_active_model(tmp_path: Path, db: Database, config: Config) -> None: + root = _write_install(tmp_path / "legacy") # Defaults.DefaultModelId == 4 + result = import_installation(root, db=db, config=config, options=ImportOptions()) + active = SettingsRepo(db).get_active_model_id() + assert active is not None + assert result.activated_model_id == active + assert active == _model(db, "9mm Default").id + + +def test_import_never_overrides_a_choice_already_made_here(tmp_path: Path, db: Database, config: Config) -> None: + """The import is an offer, not a takeover.""" + existing = ModelRepo(db).list()[0] + SettingsRepo(db).set_active_model_id(existing.id) + root = _write_install(tmp_path / "legacy") + + result = import_installation(root, db=db, config=config, options=ImportOptions()) + + assert result.activated_model_id is None + assert SettingsRepo(db).get_active_model_id() == existing.id + + +# ----- the first-run offer ---------------------------------------------------- + + +def test_first_run_offer_appears_once(tmp_path: Path, db: Database, monkeypatch: pytest.MonkeyPatch) -> None: + root = tmp_path / "pf" / "SJSeth" / "AI Brass Sorter" + _write_install(root) + monkeypatch.setenv("ProgramFiles", str(tmp_path / "pf")) + monkeypatch.delenv("ProgramW6432", raising=False) + monkeypatch.delenv("ProgramFiles(x86)", raising=False) + + assert should_offer_first_run(db) == root + mark_first_run_offered(db) + assert should_offer_first_run(db) is None + assert SettingsRepo(db).get(FIRST_RUN_SEEN_KEY) is True + + +def test_first_run_offer_stays_silent_for_an_app_already_in_use( + tmp_path: Path, db: Database, monkeypatch: pytest.MonkeyPatch +) -> None: + """A seeded fresh DB always has one model, so "has models" is not the test — + an active model, or a checkpoint on disk, is.""" + root = tmp_path / "pf" / "SJSeth" / "AI Brass Sorter" + _write_install(root) + monkeypatch.setenv("ProgramFiles", str(tmp_path / "pf")) + monkeypatch.delenv("ProgramW6432", raising=False) + monkeypatch.delenv("ProgramFiles(x86)", raising=False) + assert should_offer_first_run(db) == root + + SettingsRepo(db).set_active_model_id(ModelRepo(db).list()[0].id) + assert should_offer_first_run(db) is None + + +def test_first_run_offer_stays_silent_with_no_windows_app( + tmp_path: Path, db: Database, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setenv("ProgramFiles", str(tmp_path / "empty")) + monkeypatch.delenv("ProgramW6432", raising=False) + monkeypatch.delenv("ProgramFiles(x86)", raising=False) + assert should_offer_first_run(db) is None + + +def test_reimport_skips_an_unchanged_checkpoint(tmp_path: Path, db: Database, config: Config) -> None: + """Same rule as the images: a re-run must not re-copy a multi-hundred-MB + checkpoint that hasn't changed (real-install validation, PR #125).""" + root = _write_install(tmp_path / "legacy", models=[_MODEL_COMMUNITY]) + _torch_zip(root / "training" / "models" / "4.zip") + + first = import_installation(root, db=db, config=config, options=ImportOptions()) + second = import_installation(root, db=db, config=config, options=ImportOptions()) + + assert first.checkpoints_copied == 1 + assert second.checkpoints_copied == 0 + model = _model(db, "9mm Default") + assert model.model_path is not None and Path(model.model_path).is_file() diff --git a/tests/unit/ml/test_classifier.py b/tests/unit/ml/test_classifier.py index 53df2d9..dbb1979 100644 --- a/tests/unit/ml/test_classifier.py +++ b/tests/unit/ml/test_classifier.py @@ -128,3 +128,51 @@ def test_checkpoint_problem_explains_an_untrained_model(tmp_path: Path) -> None: assert "no trained model file" in problem # Must say it is NOT quietly switching backends. assert "AI Config" in problem + + +# ----- openai-mode models ----------------------------------------------------- + + +def _activate_openai_model(db: Database, **config_kwargs) -> int: + from sorter.data.models import AIModelConfig, Model + + cart_id = ModelRepo(db).list()[0].cartridge_id + model = ModelRepo(db).create( + Model( + name="HTTP model", + cartridge_id=cart_id, + model_mode="openai", + ai_model_config=AIModelConfig(**config_kwargs), + ) + ) + SettingsRepo(db).set_active_model_id(model.id) + return model.id + + +def test_openai_model_routes_to_http_with_its_own_config(tmp_path: Path) -> None: + """The passed api_cfg is the app-level one — an openai model must ignore it. + + Leaking the app-level config in would be the mirror image of the removed + HTTP fallback: cases silently classified against a server the active + model never named. + """ + db = _seed_db(tmp_path) + _activate_openai_model(db, endpoint_url="http://model-server:9", api_key="k", model="qwen") + image = np.zeros((10, 10, 3), dtype=np.uint8) + with patch("sorter.ml.classifier.api_client.classify", return_value=("FC", 88.0)) as m: + with patch("sorter.ml.classifier.local_inference.classify") as m_local: + result = classifier.classify_active(image, ["FC"], {"endpoint_url": "http://app-level"}, db) + assert result == ("FC", 88.0) + m_local.assert_not_called() + cfg = m.call_args.args[2] + assert cfg["endpoint_url"] == "http://model-server:9" + assert cfg["model"] == "qwen" + + +def test_openai_model_needs_no_torch_and_has_no_checkpoint_problem(tmp_path: Path) -> None: + """`uses_local_inference` is what the torch gate keys off; `checkpoint_problem` + is what refuses Start. Both must wave an openai model through.""" + db = _seed_db(tmp_path) + _activate_openai_model(db) + assert not classifier.uses_local_inference(db) + assert classifier.checkpoint_problem(db) is None diff --git a/tests/unit/ui/test_ai_page.py b/tests/unit/ui/test_ai_page.py index 991d127..afc683a 100644 --- a/tests/unit/ui/test_ai_page.py +++ b/tests/unit/ui/test_ai_page.py @@ -398,3 +398,75 @@ def test_test_shot_needs_credentials(section, window) -> None: section.test_button.click() assert window.notify.calls and window.notify.calls[0][0] == "AI not configured" + + +# ----- the active openai model as the config target --------------------------- + + +def _seed_openai_model(config: Config, name: str = "HTTP model", **cfg: Any) -> int: + from sorter.data.models import AIModelConfig, Model + from sorter.data.repository import CartridgeRepo, ModelRepo + + cart = CartridgeRepo(config.db).list()[0] + model = ModelRepo(config.db).create( + Model(name=name, cartridge_id=cart.id, model_mode="openai", ai_model_config=AIModelConfig(**cfg)) + ) + SettingsRepo(config.db).set_active_model_id(model.id) + return model.id + + +def test_form_targets_the_active_openai_model(page, window, config) -> None: + """An active openai model keeps the page live — and rebinds the server + fields to *its* config, not the app-level one.""" + _seed_openai_model(config, endpoint_url="http://model-box:9", model="qwen-vl") + + page.refresh_mode() + + assert page.is_available() + assert page.section.endpoint_edit.text() == "http://model-box:9" + assert page.section.model_edit.text() == "qwen-vl" + assert "HTTP model" in page.section.target_label.text() + + +def test_save_writes_to_the_model_row_not_the_app_config(page, window, config) -> None: + from sorter.data.repository import ModelRepo + + model_id = _seed_openai_model(config, endpoint_url="http://old") + page.refresh_mode() + app_endpoint_before = config.api["endpoint_url"] + + page.section.endpoint_edit.setText("http://new-endpoint:8000") + page.section.model_edit.setText("gpt-5") + page.section.save() + + saved = ModelRepo(config.db).get(model_id) + assert saved is not None + assert saved.ai_model_config.endpoint_url == "http://new-endpoint:8000" + assert saved.ai_model_config.model == "gpt-5" + # The app-level config the AI Config mode uses is untouched. + assert Config(config.db).load().api["endpoint_url"] == app_endpoint_before + + +def test_refresh_keeps_unsaved_edits_while_the_target_is_unchanged(page, window, config) -> None: + """The old guarantee, kept per target: a mode/changed event for the same + target must not discard a half-typed endpoint.""" + _seed_openai_model(config) + page.refresh_mode() + + page.section.endpoint_edit.setText("http://half-typed") + page.refresh_mode() + + assert page.section.endpoint_edit.text() == "http://half-typed" + + +def test_deactivating_rebinds_the_app_level_settings(page, window, config) -> None: + _seed_openai_model(config, endpoint_url="http://model-box:9") + page.refresh_mode() + assert page.section.endpoint_edit.text() == "http://model-box:9" + + SettingsRepo(config.db).clear_active_model() + page.refresh_mode() + + assert page.is_available() + assert page.section.endpoint_edit.text() == str(config.api["endpoint_url"]) + assert page.section.target_label.text() == ai_page.TARGET_GLOBAL_TEXT diff --git a/tests/unit/ui/test_app.py b/tests/unit/ui/test_app.py index 49d0dc4..18f95ee 100644 --- a/tests/unit/ui/test_app.py +++ b/tests/unit/ui/test_app.py @@ -174,8 +174,9 @@ def test_menus(window) -> None: def test_theme_section_hosts_the_theme_combo(window) -> None: - window.sidebar_buttons["Settings"].click() - window.settings_list.setCurrentRow(window.settings_list.count() - 1) + # By name, not by row: Theme stopped being the last section when "Import + # from Windows" was added, and this test is about the section's content. + window._open_settings_section("Theme") assert window.settings_list.currentItem().text() == "Theme" page = window.settings_pages.currentWidget() diff --git a/tests/unit/ui/test_e2e.py b/tests/unit/ui/test_e2e.py index 9699073..5ef304f 100644 --- a/tests/unit/ui/test_e2e.py +++ b/tests/unit/ui/test_e2e.py @@ -344,7 +344,7 @@ def fill_editor(dialog: Any) -> None: dialog.notify = lambda title, text: pytest.fail(f"{title}: {text}") dialog.name_edit.setText("Demo model") dialog.cartridge_combo.setCurrentText("9x19") - dialog.mode_combo.setCurrentText("convnext_small") + dialog.mode_combo.setCurrentText("ConvNeXt-Small") dialog.save() script_dialog(monkeypatch, ModelEditorDialog, fill_editor) diff --git a/tests/unit/ui/test_models.py b/tests/unit/ui/test_models.py index 792a5bd..f8933dd 100644 --- a/tests/unit/ui/test_models.py +++ b/tests/unit/ui/test_models.py @@ -162,7 +162,9 @@ def test_rows_carry_the_facts_the_tk_cards_showed(page, config) -> None: row = names(page).index("Range brass") assert cell(page, row, "Cartridge") == cartridge.name assert cell(page, row, "Type") == "Standard" - assert cell(page, row, "Mode") == "convnext_small" + # The display label, not the stored identifier — the Windows app's + # Training Mode spelling. + assert cell(page, row, "Mode") == "ConvNeXt-Small" assert cell(page, row, "Images") == "0" assert cell(page, row, "Trained") == "no" @@ -676,7 +678,7 @@ def editor(page, existing: Model | None = None) -> tuple[ModelEditorDialog, _Rec def test_the_editor_creates_a_model(page, config) -> None: dialog, notified = editor(page) dialog.name_edit.setText(" Range brass ") - dialog.mode_combo.setCurrentText("convnext_small") + dialog.mode_combo.setCurrentText("ConvNeXt-Small") dialog.primer_spin.setValue(120) dialog.hide_primer_check.setChecked(False) @@ -1029,3 +1031,26 @@ def test_the_hint_line_is_left_to_what_a_row_cannot_carry(page, config) -> None: select_name(page, "Someone else's") assert page.hint_label.text() == FOREIGN_NOTICE + + +def test_editor_offers_openai_and_persists_it(page) -> None: + """ "openai" is a first-class mode in Create/Edit Model (PR #125 review) — + the Windows app's "OpenAI API" Training Mode, imported or created here.""" + from sorter.data.models import is_trainable + + dialog, _recorder = editor(page) + offered_data = [dialog.mode_combo.itemData(i) for i in range(dialog.mode_combo.count())] + offered_text = [dialog.mode_combo.itemText(i) for i in range(dialog.mode_combo.count())] + assert "openai" in offered_data + # Shown with the user-facing spelling; the identifier is the item data. + assert "OpenAI" in offered_text + + dialog.name_edit.setText("HTTP model") + dialog.mode_combo.setCurrentText("OpenAI") + dialog.save() + + assert dialog.saved_id is not None + saved = ModelRepo(page.db).get(dialog.saved_id) + assert saved is not None + assert saved.model_mode == "openai" + assert not is_trainable(saved) diff --git a/tests/unit/ui/test_sort.py b/tests/unit/ui/test_sort.py index 23646e3..398b233 100644 --- a/tests/unit/ui/test_sort.py +++ b/tests/unit/ui/test_sort.py @@ -1025,3 +1025,27 @@ def test_a_failed_camera_start_reaches_the_status_bar(window) -> None: assert "Settings" in window.statusBar().currentMessage() assert "Camera" in window.statusBar().currentMessage() assert window._camera_state[1] is False + + +def test_an_openai_model_keeps_ai_config_live_and_train_muted(window, config) -> None: + """The mode pair's third state (PR #125 review): an active openai model + classifies over HTTP, so AI Config stays the live surface — editing that + model's own settings — while Train gets the openai explainer.""" + from sorter.data.models import AIModelConfig, Model + from sorter.data.repository import CartridgeRepo, ModelRepo, SettingsRepo + + cart = CartridgeRepo(config.db).list()[0] + model = ModelRepo(config.db).create( + Model(name="HTTP model", cartridge_id=cart.id, model_mode="openai", ai_model_config=AIModelConfig()) + ) + SettingsRepo(config.db).set_active_model_id(model.id) + mode_changed(window) + + assert not window.sidebar_buttons["AI Config"].property("unavailable") + assert window.sidebar_buttons["Train"].property("unavailable") + assert window.ai_page.is_available() + assert "HTTP model" in window.ai_page.section.target_label.text() + + window.sidebar_buttons["Train"].click() + assert not window.train_page.is_available() + assert "HTTP" in window.train_page.unavailable_title.text() diff --git a/tests/unit/ui/test_winforms_import_dialog.py b/tests/unit/ui/test_winforms_import_dialog.py new file mode 100644 index 0000000..930e6a8 --- /dev/null +++ b/tests/unit/ui/test_winforms_import_dialog.py @@ -0,0 +1,449 @@ +"""The Windows-app import dialog: what it offers, and what it refuses to offer. + +Offscreen, no display. The dialog's `notify` / `ask_directory` seams are +replaced so nothing opens a native modal (§5), and the import itself runs +against a synthetic install tree — nothing here touches `C:\\Program Files`. +""" + +from __future__ import annotations + +import json +import time +import zipfile +from pathlib import Path +from typing import Any + +import pytest + +pytest.importorskip("PySide6") + +from sorter.data import winforms_import +from sorter.data.repository import ModelRepo, SettingsRepo +from sorter.ui.app import SETTINGS_SECTIONS +from sorter.ui.dialog_winforms_import import ( + IMPORT_BUTTON_ROLE, + SECTION_NAME, + WinFormsImportDialog, + build_winforms_import_section, + maybe_offer_first_run, + summarize, +) + +_MODEL = { + "Id": 3, + "Name": "9mm Base Model", + "CartridgeId": 2, + "ModelType": 0, + "ModelMode": 0, +} +# A second model, so "pick a couple out of the junk" has something to pick from. +_MODEL_OTHER = { + "Id": 4, + "Name": "223 Remington", + "CartridgeId": 5, + "ModelType": 0, + "ModelMode": 0, +} + + +def _write_install( + root: Path, + *, + images: list[str] | None = None, + with_defaults: bool = True, + models: list[dict[str, Any]] | None = None, + images_by_id: dict[int, list[str]] | None = None, + headstamps: list[dict[str, Any]] | None = None, +) -> Path: + """A synthetic legacy install. `images` is sugar for model 3's folder.""" + (root / "Data").mkdir(parents=True, exist_ok=True) + document: dict[str, Any] = { + "Models": models if models is not None else [_MODEL], + "Cartridges": [{"Id": 2, "Name": "9mm"}, {"Id": 5, "Name": "223"}], + "Headstamps": headstamps if headstamps is not None else [{"Id": 1, "Name": "GECO", "Model_Id": 3}], + "HeadStampParents": [], + "HeadStampParentLinks": [], + "SlotConfigs": [], + "Defaults": {"DefaultSerialPort": "COM3", "SlotQuantity": 8} if with_defaults else {}, + } + (root / winforms_import.CONFIG_DB_NAME).write_text(json.dumps(document), encoding="utf-8-sig") + per_model = dict(images_by_id or {}) + per_model.setdefault(3, list(images or [])) + for legacy_id, names in per_model.items(): + folder = root / "training" / "images" / str(legacy_id) + folder.mkdir(parents=True, exist_ok=True) + for name in names: + (folder / name).write_bytes(b"\xff\xd8\xff\xe0jpeg-ish") + return root + + +def _label(item: Any) -> str: + """Both columns of a row — what it is, then what it would bring.""" + return f"{item.text(0)} — {item.text(1)}" + + +def _checked(item: Any) -> bool: + from PySide6.QtCore import Qt + + return bool(item.checkState(0) == Qt.CheckState.Checked) + + +def _partial(item: Any) -> bool: + from PySide6.QtCore import Qt + + return bool(item.checkState(0) == Qt.CheckState.PartiallyChecked) + + +def _set_checked(item: Any, checked: bool) -> None: + """Tick a row the way a click does — through the signal, so it propagates.""" + from PySide6.QtCore import Qt + + item.setCheckState(0, Qt.CheckState.Checked if checked else Qt.CheckState.Unchecked) + + +def _torch_zip(path: Path) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + with zipfile.ZipFile(path, "w") as zf: + zf.writestr("trainedmodel/data.pkl", b"\x80\x02}q\x00.") + + +def _quiet(dialog: WinFormsImportDialog) -> list[tuple[str, str]]: + """Replace the modal seam and hand back what it was told.""" + seen: list[tuple[str, str]] = [] + dialog.notify = lambda title, text: seen.append((title, text)) + return seen + + +def _run_and_wait(qapp: Any, dialog: WinFormsImportDialog, seen: list[tuple[str, str]]) -> None: + """Start the import and pump until the worker's queue has been drained. + + The dialog drains on a 100 ms QTimer that an offscreen test never runs, so + `_drain` is called directly — the same main-thread entry point the timer + uses. A failure arrives through `notify` rather than as a result, so `seen` + is what turns a worker exception into a legible assertion instead of a + timeout. + """ + deadline = time.monotonic() + 10.0 + dialog.start_import() + while time.monotonic() < deadline: + qapp.processEvents() + dialog._drain() + if dialog.result_summary is not None: + return + if any(title == "Import failed" for title, _ in seen): + raise AssertionError(f"import failed: {seen}") + time.sleep(0.01) + raise AssertionError(f"import did not finish (notified: {seen})") + + +# ----- what the dialog offers ------------------------------------------------- + + +def test_counts_come_from_the_survey(qapp, window, tmp_path: Path) -> None: + """Each part row carries the count for the tick that decides it.""" + root = _write_install(tmp_path / "legacy", images=["GECO__1.jpg", "GECO__2.jpg"]) + dialog = WinFormsImportDialog(window, root, window) + _quiet(dialog) + + assert _label(dialog.part_items[(3, winforms_import.PART_IMAGES)]) == "Training images — 2" + assert _label(dialog.part_items[(3, winforms_import.PART_HEADSTAMPS)]).endswith("— 1") + # And on the model's own row, so an abandoned experiment is recognisable + # without expanding it. + assert "2 image(s)" in _label(dialog.model_items[3]) + assert dialog.import_button.isEnabled() + dialog.close() + + +def test_every_model_gets_its_own_branch(qapp, window, tmp_path: Path) -> None: + """The point of the tree: two models, picked apart from one another.""" + root = _write_install( + tmp_path / "legacy", + models=[_MODEL, _MODEL_OTHER], + images_by_id={3: ["GECO__1.jpg"], 4: ["FC__1.jpg", "FC__2.jpg"]}, + headstamps=[{"Id": 1, "Name": "GECO", "Model_Id": 3}], + ) + dialog = WinFormsImportDialog(window, root, window) + _quiet(dialog) + + assert set(dialog.model_items) == {3, 4} + assert "9mm Base Model" in _label(dialog.model_items[3]) + # Model 4 has images but no headstamps, so it gets no headstamp row at all. + assert (4, winforms_import.PART_IMAGES) in dialog.part_items + assert (4, winforms_import.PART_HEADSTAMPS) not in dialog.part_items + assert (3, winforms_import.PART_HEADSTAMPS) in dialog.part_items + dialog.close() + + +def test_a_part_the_model_has_nothing_for_gets_no_row(qapp, window, tmp_path: Path) -> None: + """A tick that would import nothing is worse than no tick at all.""" + root = _write_install(tmp_path / "legacy", with_defaults=False) + dialog = WinFormsImportDialog(window, root, window) + _quiet(dialog) + + # No images on disk and no torch checkpoint — only headstamps to offer. + assert (3, winforms_import.PART_IMAGES) not in dialog.part_items + assert (3, winforms_import.PART_CHECKPOINT) not in dialog.part_items + assert (3, winforms_import.PART_HEADSTAMPS) in dialog.part_items + # Same rule one level up, for the app-level settings rows. + assert dialog.items[winforms_import.ITEM_SERIAL].isDisabled() + assert dialog.items[winforms_import.ITEM_AI_CONFIG].isDisabled() + dialog.close() + + +def test_unticking_a_model_takes_its_branch_with_it(qapp, window, tmp_path: Path) -> None: + """The inheritance is the tree's shape, not a rule the dialog polices.""" + root = _write_install( + tmp_path / "legacy", + models=[_MODEL, _MODEL_OTHER], + images_by_id={3: ["GECO__1.jpg"], 4: ["FC__1.jpg"]}, + ) + dialog = WinFormsImportDialog(window, root, window) + _quiet(dialog) + + _set_checked(dialog.model_items[3], False) + + assert not _checked(dialog.part_items[(3, winforms_import.PART_IMAGES)]) + assert _checked(dialog.part_items[(4, winforms_import.PART_IMAGES)]) + # One of two models left ticked, so the parent says so. + assert _partial(dialog.items[winforms_import.ITEM_MODELS]) + assert set(dialog.selected_options().per_model or {}) == {4} + dialog.close() + + +def test_unticking_the_models_row_clears_every_model(qapp, window, tmp_path: Path) -> None: + root = _write_install(tmp_path / "legacy", models=[_MODEL, _MODEL_OTHER], images=["GECO__1.jpg"]) + dialog = WinFormsImportDialog(window, root, window) + _quiet(dialog) + + _set_checked(dialog.items[winforms_import.ITEM_MODELS], False) + + assert not any(_checked(row) for row in dialog.model_items.values()) + assert dialog.selected_options().per_model == {} + assert not dialog.selected_options().any_selected() + dialog.close() + + +def test_select_none_then_select_all_walks_the_whole_branch(qapp, window, tmp_path: Path) -> None: + """Two models out of fifteen must not cost thirteen clicks.""" + root = _write_install(tmp_path / "legacy", models=[_MODEL, _MODEL_OTHER], images=["GECO__1.jpg"]) + dialog = WinFormsImportDialog(window, root, window) + _quiet(dialog) + + dialog._set_all_models(False) + assert dialog.selected_options().per_model == {} + + dialog._set_all_models(True) + assert set(dialog.selected_options().per_model or {}) == {3, 4} + assert _checked(dialog.part_items[(3, winforms_import.PART_IMAGES)]) + dialog.close() + + +def test_a_model_can_come_without_its_checkpoint(qapp, window, tmp_path: Path) -> None: + """The 200 MB item is the one a user most wants to decline individually.""" + root = _write_install(tmp_path / "legacy", images=["GECO__1.jpg"]) + _torch_zip(root / "training" / "models" / "3.zip") + dialog = WinFormsImportDialog(window, root, window) + _quiet(dialog) + + _set_checked(dialog.part_items[(3, winforms_import.PART_CHECKPOINT)], False) + chosen = dialog.selected_options() + + assert chosen.per_model is not None + assert chosen.per_model[3].checkpoint is False + assert chosen.per_model[3].images is True + # The model is still coming, just not its checkpoint. + assert _partial(dialog.model_items[3]) + dialog.close() + + +def test_the_model_row_says_what_it_would_do_to_the_library(qapp, window, tmp_path: Path, monkeypatch) -> None: + """Will this tread on what I already have? — answered per model, up front.""" + monkeypatch.setenv("CASESORTER_DATA_DIR", str(tmp_path / "appdata")) + root = _write_install(tmp_path / "legacy", images=["GECO__1.jpg"]) + first = WinFormsImportDialog(window, root, window) + seen = _quiet(first) + assert "new model here" in _label(first.model_items[3]) + _run_and_wait(qapp, first, seen) + first.close() + + again = WinFormsImportDialog(window, root, window) + _quiet(again) + assert "updates '9mm Base Model'" in _label(again.model_items[3]) + again.close() + + +def test_a_folder_that_is_not_an_installation_says_so(qapp, window, tmp_path: Path) -> None: + dialog = WinFormsImportDialog(window, tmp_path / "not-it", window) + _quiet(dialog) + assert not dialog.import_button.isEnabled() + assert dialog.warning_label.text() + dialog.close() + + +def test_choose_folder_reloads_the_counts(qapp, window, tmp_path: Path) -> None: + root = _write_install(tmp_path / "legacy", images=["GECO__1.jpg"]) + dialog = WinFormsImportDialog(window, None, window) + _quiet(dialog) + assert not dialog.import_button.isEnabled() + + dialog.ask_directory = lambda: str(root) + dialog._choose_folder() + + assert dialog.import_button.isEnabled() + assert _label(dialog.part_items[(3, winforms_import.PART_IMAGES)]) == "Training images — 1" + dialog.close() + + +def test_the_selection_line_totals_what_is_ticked(qapp, window, tmp_path: Path) -> None: + root = _write_install( + tmp_path / "legacy", + models=[_MODEL, _MODEL_OTHER], + images_by_id={3: ["GECO__1.jpg"], 4: ["FC__1.jpg", "FC__2.jpg"]}, + ) + dialog = WinFormsImportDialog(window, root, window) + _quiet(dialog) + assert "2 of 2 model(s)" in dialog.selection_label.text() + assert "3 image(s)" in dialog.selection_label.text() + + _set_checked(dialog.model_items[4], False) + + assert "1 of 2 model(s)" in dialog.selection_label.text() + assert "1 image(s)" in dialog.selection_label.text() + dialog.close() + + +def test_nothing_ticked_is_refused_rather_than_run(qapp, window, tmp_path: Path) -> None: + root = _write_install(tmp_path / "legacy") + dialog = WinFormsImportDialog(window, root, window) + seen = _quiet(dialog) + dialog._set_all_models(False) + for key in (winforms_import.ITEM_IMAGE_PROC, winforms_import.ITEM_SERIAL, winforms_import.ITEM_AI_CONFIG): + _set_checked(dialog.items[key], False) + + dialog.start_import() + + assert dialog.result_summary is None + assert seen and "at least one" in seen[0][1] + dialog.close() + + +# ----- running it ------------------------------------------------------------- + + +def test_import_lands_and_the_shell_is_refreshed(qapp, window, tmp_path: Path, monkeypatch) -> None: + monkeypatch.setenv("CASESORTER_DATA_DIR", str(tmp_path / "appdata")) + root = _write_install(tmp_path / "legacy", images=["GECO__1.jpg"]) + _torch_zip(root / "training" / "models" / "3.zip") + dialog = WinFormsImportDialog(window, root, window) + seen = _quiet(dialog) + + _run_and_wait(qapp, dialog, seen) + + result = dialog.result_summary + assert result is not None + assert result.models_imported == 1 + assert result.images_copied == 1 + assert result.checkpoints_copied == 1 + assert any(m.name == "9mm Base Model" for m in ModelRepo(window.db).list()) + assert seen and seen[-1][0] == "Import complete" + dialog.close() + + +def test_only_the_ticked_models_are_imported(qapp, window, tmp_path: Path, monkeypatch) -> None: + """sjseth's ask on #125: bring across a couple, leave the junk behind.""" + monkeypatch.setenv("CASESORTER_DATA_DIR", str(tmp_path / "appdata")) + root = _write_install( + tmp_path / "legacy", + models=[_MODEL, _MODEL_OTHER], + images_by_id={3: ["GECO__1.jpg"], 4: ["FC__1.jpg", "FC__2.jpg"]}, + ) + dialog = WinFormsImportDialog(window, root, window) + seen = _quiet(dialog) + _set_checked(dialog.model_items[4], False) + + _run_and_wait(qapp, dialog, seen) + + result = dialog.result_summary + assert result is not None + assert result.models_imported == 1 + assert result.images_copied == 1 # model 4's two images stayed behind + names = [m.name for m in ModelRepo(window.db).list()] + assert "9mm Base Model" in names + assert "223 Remington" not in names + dialog.close() + + +def test_a_declined_checkpoint_is_not_copied(qapp, window, tmp_path: Path, monkeypatch) -> None: + monkeypatch.setenv("CASESORTER_DATA_DIR", str(tmp_path / "appdata")) + root = _write_install(tmp_path / "legacy", images=["GECO__1.jpg"]) + _torch_zip(root / "training" / "models" / "3.zip") + dialog = WinFormsImportDialog(window, root, window) + seen = _quiet(dialog) + _set_checked(dialog.part_items[(3, winforms_import.PART_CHECKPOINT)], False) + + _run_and_wait(qapp, dialog, seen) + + result = dialog.result_summary + assert result is not None + assert result.models_imported == 1 + assert result.images_copied == 1 + assert result.checkpoints_copied == 0 + imported = next(m for m in ModelRepo(window.db).list() if m.name == "9mm Base Model") + assert not imported.model_path + dialog.close() + + +def test_summary_says_what_landed() -> None: + result = winforms_import.ImportResult(models_imported=2, images_copied=40, warnings=["a warning from the survey"]) + text = summarize(result) + assert "2 model(s) imported" in text + assert "40 training image(s) copied" in text + assert "a warning from the survey" in text + + +def test_summary_of_an_import_that_had_nothing_to_do() -> None: + assert "already here" in summarize(winforms_import.ImportResult()) + + +# ----- the first-run offer ---------------------------------------------------- + + +def test_first_run_offer_stays_silent_with_no_installation(qapp, window, monkeypatch) -> None: + """The one case that matters most: a user who never ran the Windows app.""" + monkeypatch.setattr(winforms_import, "find_installation", lambda: None) + assert maybe_offer_first_run(window) is None + + +def test_first_run_offer_is_marked_as_made(qapp, window, tmp_path: Path, monkeypatch) -> None: + root = _write_install(tmp_path / "legacy") + monkeypatch.setattr(winforms_import, "find_installation", lambda: root) + opened: list[Path | None] = [] + # Not `exec()`: a modal event loop offscreen never returns. + monkeypatch.setattr( + "sorter.ui.dialog_winforms_import.open_import_dialog", + lambda win, root_, **kw: opened.append(root_), + ) + + maybe_offer_first_run(window) + + assert opened == [root] + assert SettingsRepo(window.db).get(winforms_import.FIRST_RUN_SEEN_KEY) is True + # Offered once, ever — Settings keeps it reachable after that. + assert maybe_offer_first_run(window) is None + + +# ----- the Settings section --------------------------------------------------- + + +def test_settings_section_is_listed_and_reachable(qapp, window) -> None: + assert SECTION_NAME in SETTINGS_SECTIONS + window._open_settings_section(SECTION_NAME) + assert window.settings_list.currentItem().text() == SECTION_NAME + + +def test_settings_section_offers_the_import(qapp, window) -> None: + from PySide6.QtWidgets import QPushButton + + page = build_winforms_import_section(window) + buttons = [b for b in page.findChildren(QPushButton) if b.property("role") == IMPORT_BUTTON_ROLE] + assert len(buttons) == 1