diff --git a/documents/DATA_MIGRATIONS.md b/documents/DATA_MIGRATIONS.md index bb42c73b..7b7dcff9 100644 --- a/documents/DATA_MIGRATIONS.md +++ b/documents/DATA_MIGRATIONS.md @@ -33,12 +33,116 @@ Operational CLI migrations (OB-001, OB-002) are documented here but are **not** | **MIG-C** | Resolve Qdrant to Mongo | CLI | `runQdrantResolveToMongo` | register point UUIDs onto `qdrantChunkIds` | Keep | | **MIG-D** | Validate Qdrant from Mongo | CLI | `runQdrantValidateFromMongo` | Mongo wins metadata; delete orphan points | Keep | | **ADM-001** | Platform admin `isAdmin` backfill | Startup | `migratePlatformAdmins` in `src/helpers/migrate-platform-admins.ts` | GlobalUsers matching `CHARISMA_RUSDIYANTO_PUID` / `RICHARD_TAPE_PUID` → `isAdmin: true` | Operational — keep unless product changes | +| **GPF-001** | Guided Pathway alert course isolation | Superseded | Historical implementation in `guided-pathway-flag-collection-mongo.ts` | shared `guided-pathway-flags` rows → hashed per-course collections | Superseded by GPF-002; hash recognition remains migration-only | +| **GPF-002** | Guided Pathway registered collection normalization | Startup + operation gate | `migrateGuidedPathwayFlagsToCourseCollections` in `src/db/mongo/guided-pathway-flag-collection-mongo.ts` | shared and hashed rows → readable `activeCourse.collections.guidedPathwayFlags` targets; lease/result → `application-migrations` | Operational — retain until every environment passes the GPF-002 postchecks and retained legacy data is resolved | | **SQ-001** | Scenario Questions collection backfill | Lazy (first API call) | `ensureScenarioQuestionsCollection` in `src/db/mongo/scenario-questions-mongo.ts` | missing `activeCourse.collections.scenarioQuestions` → creates `{courseName}_scenario_questions` + `$set` the field | Keep while any pre-feature course document may lack `collections.scenarioQuestions` | | **SQ-004** | Scenario Progress collection backfill | Lazy (first progress API call) | `ensureScenarioProgressCollection` in `src/db/mongo/scenario-progress-mongo.ts` | missing `activeCourse.collections.scenarioProgress` → creates `{courseName}_scenario_progress` + `$set` the field | Keep while any course may lack `collections.scenarioProgress` | | **GP-001** | Remove legacy off-topic pathway | Lazy (pathways ensure/list/seed) | `healRemoveOffTopicPathway` in `src/db/mongo/pathways-mongo.ts` | `{courseName}_pathways` docs with `id: 'off-topic'` → deleted | Keep while legacy courses may still hold the seed; audit then remove | --- +## GPF-001: Guided Pathway alert course isolation + +**Status:** Superseded by GPF-002 + +GPF-001 moved the original global `guided-pathway-flags` rows into +`guided-pathway-flags-course-` namespaces. It also made the hash authoritative +instead of reading `activeCourse.collections.guidedPathwayFlags`. Some environments may +already contain those namespaces, so GPF-002 still recognizes them as migration sources. +GPF-001 must not run independently again. + +--- + +## GPF-002: Guided Pathway registered collection normalization + +**Status:** Active (startup migration with operation-level gate) + +**Collections:** legacy `guided-pathway-flags`, legacy `guided-pathway-flags-course-`, `active-course-list`, durable migration state in `application-migrations`, and readable registered course targets such as `{courseName}_guided-pathway-flags` + +### Behavior + +1. Create/verify a partial unique index on the non-empty string `activeCourse.collections.guidedPathwayFlags` field. This is the cross-process catalog guard that prevents two active courses from owning the same automatic-alert namespace. +2. Treat a valid non-hash registered value as authoritative. When the field is missing or still points at a GPF-001 hash, choose `{courseName}_guided-pathway-flags` and persist it only after source rows are copied and verified. A later course rename does not recompute the stored name. +3. Preflight every target. Reject protected names, collisions with another registered course collection, duplicate Guided Pathway registrations, and a physical target containing any row whose `courseId` differs from the target course. The checks and logs use counts/identity metadata only and never emit alert content. +4. Copy matching rows from the course's GPF-001 hash namespace first, then the older shared collection, in 200-row `_id` batches. Each destination operation is an upsert with `$setOnInsert`, so the per-course source wins a duplicate legacy `_id` while an existing readable-target document remains authoritative. Verify every destination `_id` with its owning `courseId` before switching the catalog field. +5. Compare-and-set `collections.guidedPathwayFlags`, invalidate the course collection-name cache only after success, and then verify that the catalog still owns the target. Re-verify exact source `_id` batches before deleting them. Drop only namespaces that are empty; a missing namespace is an idempotent cleanup result. +6. Retain malformed global rows and non-empty hash namespaces without an active catalog owner for manual recovery. Never guess ownership or attach orphan data to a current course. +7. Do not register or create storage for an untouched existing course with no registration, no shared/hash source, and no pre-existing readable namespace. Alert creation uses the provisioning resolver when storage is first needed. Course/admin list, count, backup, and cross-course aggregation use read-only resolution and include only existing registered collections. + +Startup invokes GPF-002 after academic-period initialization. Guided Pathway operations also await the migration gate. The exported method retains its historical name for façade compatibility. + +### Cross-process coordination + +GPF-002 stores one record at `application-migrations._id = 'GPF-002'`: + +- The owner acquires `state: 'running'` with a random `ownerId` and a renewable five-minute `leaseUntil`. +- Other application instances poll the durable record. A failed record, an expired lease, or a running record with no lease can be claimed for retry. +- The owner renews the lease between bounded migration/copy/cleanup phases. Losing the lease fails the attempt before it can report completion. +- Success persists `state: 'complete'`, the count-only result, and `completedAt`, then removes owner/lease fields. Later processes and restarts return that persisted result without rerunning data movement. +- Failure marks the record `failed` and removes the lease so a later operation can retry. A process-local promise coalesces callers only within one application instance and is discarded on failure. + +### Idempotency and failure safety + +Destination writes are insert-only upserts by Mongo `_id`; retrying after a partial copy does not duplicate an alert and does not overwrite a newer target decision, admin review, or reveal-audit history with a stale legacy snapshot. A source row is never deleted before target verification, compare-and-set catalog registration, and a second ownership check. Per-course unique alert-id and deduplication indexes remain the runtime guards. + +GPF-001 application instances ignore the registered field and can continue writing hashes. Because a completed GPF-002 record makes later runs a no-op, any old process writing after completion would strand new rows in a migration-only source. Run GPF-002 only after old instances stop accepting chat traffic; a rolling mixed-version migration is unsupported. + +### Deployment preconditions + +1. Take and validate a recoverable full database backup. +2. Stop every pre-GPF-002 application instance from accepting chat/write traffic before a new instance starts the migration. +3. Ensure the deployment identity can read/write `application-migrations`, create the catalog and per-course indexes, create target collections, update `active-course-list`, and delete/drop only verified legacy sources. +4. Do not manually mark the migration complete. If startup fails, inspect the count-only migration error/result and retry the same build after correcting the cause. + +### Verification (Mongo shell) + +```js +db.getCollection('guided-pathway-flags').countDocuments({ + courseId: { $type: 'string' } +}) + +db.getCollection('active-course-list').countDocuments({ + 'collections.guidedPathwayFlags': /^guided-pathway-flags-course-[a-f0-9]{24}$/ +}) + +db.getCollection('active-course-list').aggregate([ + { $match: { 'collections.guidedPathwayFlags': { $type: 'string' } } }, + { $group: { _id: '$collections.guidedPathwayFlags', owners: { $addToSet: '$id' }, count: { $sum: 1 } } }, + { $match: { count: { $gt: 1 } } } +]) + +db.getCollection('application-migrations').findOne( + { _id: 'GPF-002' }, + { _id: 1, state: 1, completedAt: 1, result: 1 } +) + +db.getCollection('active-course-list').getIndexes().filter( + ({ name }) => name === 'guided_pathway_flag_collection_unique' +) +``` + +The first two counts and the duplicate-registration aggregation should be empty/zero for migratable active-course data. The migration record must be `state: 'complete'`, and the named partial unique catalog index must exist. Review the count-only `result.retainedLegacyRows`, `result.retainedHashedCollections`, and `result.orphanCourseCollections` values. Non-empty global or hash sources require manual ownership review; do not delete them merely to satisfy the count. + +For every active course with a non-empty registration, also verify operationally that the named collection exists and contains no row with a different `courseId`. Perform that check with counts/projections only; do not print messages, user identifiers, deduplication keys, or reveal events. + +### Rollback + +Rollback requires restoring the pre-deployment database backup together with the pre-GPF-002 application build. Redeploying the old build alone is unsafe because new writes use readable registered targets and verified legacy rows may already have been removed. If an attempt fails before completion, prefer correcting the cause and retrying the same GPF-002 build; do not edit the lease/result record or move rows manually without a separate recovery plan. + +### Sunset criteria + +Keep the operation gate, shared/hash source discovery, and durable migration record handling until every deployed environment has: + +- a `complete` GPF-002 record from the current migration implementation; +- zero migratable active-course rows in the shared source and zero hash registrations; +- no duplicate/unsafe registered namespaces and the partial unique registry index present; +- reviewed and resolved every retained malformed/orphan source; and +- passed an agreed rollback-support window with no pre-GPF-002 application version eligible for redeployment. + +After those conditions are documented, a separate change may remove global/hash discovery and the operation-level migration gate. Keep registry-authoritative resolution, the unique catalog index, and read-only versus provisioning resolution after migration code is retired. + +--- + ## SP-001: System prompt v1 → v2 **Status:** Active (lazy migrate + lazy unset) diff --git a/documents/ENDPOINT_ARCHITECTURE.md b/documents/ENDPOINT_ARCHITECTURE.md index b1521e22..5437ded3 100644 --- a/documents/ENDPOINT_ARCHITECTURE.md +++ b/documents/ENDPOINT_ARCHITECTURE.md @@ -26,6 +26,7 @@ All API routes are prefixed with `/api/`. Page routes are served from `/` and `/ | `/api/rag` | ragAppRoutes | Document upload, retrieval, search, wipe | | `/api/courses` | mongodbRoutes | Courses, flags, objectives, materials, monitor | | `/api/courses` | writingFeedbackRoutes | Optional staff writing-feedback workspace | +| `/api/admin` | adminCourseRoutes | Platform-admin course catalog and cross-course Guided Pathway review | | `/api/course` | courseEntryRoutes | Course entry, enter-by-code, current course | | `/api/user` | userManagementRoutes | User profile, onboarding, activity | | `/api/health` | healthRoutes | Health check | @@ -67,7 +68,7 @@ All course-scoped pages use the same HTML shell; the frontend parses the URL to | `GET /course/:courseId/instructor/assistant-prompts` | Assistant prompts | | `GET /course/:courseId/instructor/system-prompts` | System prompts | | `GET /course/:courseId/instructor/scenario-questions` | Scenario Questions (Practice Scenarios authoring). Requires `scenarioGeneration` Extra Feature. Query: `?browse=questions`, `?topicOrWeekId=`, `?generate=1`, `?questionId=` | -| `GET /course/:courseId/instructor/pathway-library` | Capability-gated Guided Pathway Library; redirects to Dashboard when disabled | +| `GET /course/:courseId/instructor/pathway-library` | Capability-gated Guided Pathway Library for faculty instructors/platform admins; teaching assistants redirect to Dashboard | | `GET /course/:courseId/instructor/course-information` | Legacy redirect → dashboard (metadata in Advanced Settings; course code in topbar) | | `GET /course/:courseId/instructor/about` | About page | | `GET /course/:courseId/instructor/onboarding/course-setup` | Onboarding | @@ -323,17 +324,114 @@ Live Canvas OAuth routes are intentionally absent from this table until the priv | PUT | `/api/courses/:courseId/topic-or-week-instances/:topicOrWeekId/items/:itemId/struggle-topics/:struggleTopicId` | Yes | Instructor | Update struggle topic (response includes `changed`) | | DELETE | `/api/courses/:courseId/topic-or-week-instances/:topicOrWeekId/items/:itemId/struggle-topics/:struggleTopicId` | Yes | Instructor | Delete struggle topic (response includes `changed`) | -#### Flags (student creates; instructor manages) +#### Manual flags (explicit report; instructor manages) | Method | Path | Auth | Role | Description | |--------|------|------|------|-------------| | POST | `/api/courses/:courseId/flags` | Yes | Student or Instructor | Create flag (shared) | | GET | `/api/courses/:courseId/flags` | Yes | Instructor | List flags | | GET | `/api/courses/:courseId/flags/with-names` | Yes | Instructor | List flags with names | +| GET | `/api/courses/:courseId/flags/validate` | Yes | Instructor | Validate flag collection integrity | +| GET | `/api/courses/:courseId/flags/statistics` | Yes | Instructor | Flag counts for the course | +| GET | `/api/courses/:courseId/flags/student/:userId` | Yes | **Record owner or course staff** | One student's flag history | | GET | `/api/courses/:courseId/flags/:flagId` | Yes | Instructor | Get flag report | -| PUT | `/api/courses/:courseId/flags/:flagId` | Yes | Instructor | Update flag | +| PUT | `/api/courses/:courseId/flags/:flagId` | Yes | Instructor (faculty, TA, admin) | Update flag (`unresolved` / `resolved` only; blocked when `escalated`) | +| PATCH | `/api/courses/:courseId/flags/:flagId/escalate` | Yes | Instructor (faculty, TA, admin) | Escalate unresolved manual flag to platform admins | | PATCH | `/api/courses/:courseId/flags/:flagId/response` | Yes | Instructor | Update response | +`GET /flags/student/:userId` is student-facing — a student reads their own history — so it uses +`requireSelfOrInstructorForCourseAPI` rather than an instructor-only guard: the record owner passes, +course staff pass, and every other authenticated caller receives `403`. The target user id arrives in +the path and is untrusted, so course scope alone is not sufficient authorization. + +The literal `/flags/validate`, `/flags/statistics`, `/flags/with-names`, and `/flags/student/:userId` +routes must stay declared **above** `/flags/:flagId`. Express matches in declaration order, so a +literal route registered after the capture is shadowed and never runs. + +#### Guided Pathway Library and automatic alerts + +Guided Pathway configuration is separate from manual student-created flags. Faculty instructors +and platform admins may configure pathways; teaching assistants cannot. `enabled` controls whether +a pathway can trigger. The independent `notifyInstructorOnTrigger` setting controls whether a +successful trigger creates an automatic alert, and defaults to `true` for new, seeded, and legacy +records where the field is missing. Manually created and seeded pathways use the same evaluator. +When course staff exercise a notification-enabled pathway in chat, +the server records a course-local `instructor-test` alert; the client cannot request or forge test mode. + +| Method | Path | Auth | Role | Description | +|--------|------|------|------|-------------| +| GET | `/api/courses/:courseId/pathways` | Yes | Faculty instructor or **Admin** | List course pathways | +| POST | `/api/courses/:courseId/pathways` | Yes | Faculty instructor or **Admin** | Create a pathway; notification defaults on | +| PUT | `/api/courses/:courseId/pathways/reorder` | Yes | Faculty instructor or **Admin** | Reorder pathways | +| PUT | `/api/courses/:courseId/pathways/:pathwayId` | Yes | Faculty instructor or **Admin** | Update configuration, including either independent switch | +| DELETE | `/api/courses/:courseId/pathways/:pathwayId` | Yes | Faculty instructor or **Admin** | Delete a pathway definition | +| POST | `/api/courses/:courseId/pathways/reset` | Yes | Faculty instructor or **Admin** | Restore platform defaults with notification on | +| GET | `/api/courses/:courseId/guided-pathway-flags` | Yes | Faculty instructor or **Admin** | Paginated anonymous owning-course alert list, including labelled instructor tests; optional `status` | +| PATCH | `/api/courses/:courseId/guided-pathway-flags/:flagId/decision` | Yes | Faculty instructor or **Admin** | Atomic pending decision; student body `{ decision: 'escalate' \| 'dismiss' }`; instructor tests permit `dismiss` only | +| GET | `/api/admin/guided-pathway-flags` | Yes | **Admin** | Cross-course anonymous student-alert queue with period/course/pathway/status/reviewer/date filters; instructor tests excluded | +| PATCH | `/api/admin/guided-pathway-flags/:courseId/:flagId/review` | Yes | **Admin** | Mark an escalated student alert reviewed in its owning course without deleting it; tests rejected | +| POST | `/api/admin/guided-pathway-flags/:courseId/:flagId/reveal-identity` | Yes | **Admin** | Audit an escalated student-alert reveal in its owning course, then return only the current roster display name; tests rejected | + +#### Manual flag escalations (platform admin) + +| Method | Path | Auth | Role | Description | +|--------|------|------|------|-------------| +| GET | `/api/admin/manual-flags` | Yes | **Admin** | Cross-course escalated manual flag queue; optional `reviewState`, period/course/date filters | +| PATCH | `/api/admin/manual-flags/:courseId/:flagId/review` | Yes | **Admin** | Mark an escalated manual flag reviewed | + +**Unified instructor Flag Management UI** merges manual flags and course-scoped Guided Pathway alerts client-side. RBAC split: TAs may list/resolve/escalate manual flags (`requireInstructorForCourseAPI`) but cannot access Guided Pathway alert APIs (`requireInstructorOrAdminForCourseAPI`). + +Automatic alerts are created when an enabled pathway with `notifyInstructorOnTrigger` wins for an eligible chat sender. Enrolled non-staff users produce production `student` alerts; course staff (listed faculty instructors, TAs, and platform admins) produce non-escalatable `instructor-test` alerts. Staff are classified before enrollment so dual-role users never get a production alert. TAs may trigger test alerts while chatting but still cannot list or act on GP flags in Flag Management (`requireInstructorOrAdminForCourseAPI`). The stored `studentUserId` field holds the triggering user's id for production student alerts only; admin identity reveal resolves the display name from the course roster, then `active-users` when the sender is staff not on the roster. + +**Guided Pathway category filters** (faculty/admin Flag Management UI only) are client-side: checkboxes mirror the current Pathway Library; each GP flag is classified by its persisted `pathwayId` against that library. Flags whose `pathwayId` is missing or no longer in the library appear under **Others** — never by title inference. + +List and action responses use an explicit anonymous projection: `origin`, pathway/course snapshots, +exact message, trigger/decision/review times, state, and staff reviewer display names. They never +include a student or tester user ID, PUID, chat/request identifiers, deduplication key, or reveal +audit events. The exact message is not automatically redacted and can still identify its author if +the author writes personal information in it. Existing rows with no `origin` are returned as +`origin: 'student'`. + +Production student alerts have `pending`, `escalated`, and `dismissed` states. Instructor decisions +are final in this version, and completed records remain viewable. Escalation is an internal decision: +EngE-AI surfaces it to platform admins but does not contact LTIC. Admin identity reveal is available +only on escalated student records, requires confirmation in the client, is re-masked after refresh, +and fails closed when the audit write fails. Student records retain a restricted internal +`studentUserId` only for this audited reveal path. + +At creation, instructor-test records store neither `studentUserId` nor a separate raw trigger-actor +identity. A listed instructor's ID may participate in the opaque deduplication digest but is not +returned as trigger identity. A later dismissal retains the ordinary authorized decision-actor audit +fields; those describe who made the decision, not who originally triggered the test. +Tests are visible only in the owning course, show a `Test` label and `Instructor test message`, and +offer only `Mark test complete` (the dismiss transition). Server guards reject test escalation, +admin review, and identity reveal with `409` before mutation, audit, or roster access. TA membership, +platform-admin +privilege without explicit instructor listing, outsiders, and missing course/user context do not +create tests. Students and teaching assistants cannot call these APIs; automatic alerts never enter +Student Flag History. + +Each course stores automatic alerts separately from manual flags in the physical collection named by +`activeCourse.collections.guidedPathwayFlags`. New registrations default to the readable +`${courseName}_guided-pathway-flags` name, but the stored registry value remains authoritative after +a rename. Course routes resolve only that registered collection, while the platform-admin queue +aggregates existing registered active-course collections server-side. Alert creation may provision a +missing legacy-course target; list, count, backup, and admin aggregation paths do not create empty +collections. Including `courseId` in admin action paths makes equal alert ids in different courses +unambiguous. GPF-001 hash namespaces are migration inputs only; GPF-002 moves shared/hash rows to +registered targets under a Mongo-backed lease. See +[DATA_MIGRATIONS.md](DATA_MIGRATIONS.md#gpf-002-guided-pathway-registered-collection-normalization). + +`GET /api/admin/course-selection` also returns +`data.guidedPathwayEscalationsAwaitingReview`, the combined count of unreviewed escalated **Guided +Pathway alerts and manual flags** awaiting platform-admin review (GP count plus manual count from +dedicated Mongo count helpers — no list rows fetched for the badge). Instructor tests are excluded +from the GP portion of this count. The admin course-selection page renders that count as a bell +badge between the welcome text and logout. Clicking the bell toggles a side-by-side escalations +panel (same anonymous admin queue, prefiltered to escalated items needing review); the badge +refreshes from the same course-selection count after review actions. There is no polling, email, or +external notification. + #### Monitor (instructor roster; post-period analytics) | Method | Path | Auth | Role | Description | @@ -794,7 +892,17 @@ console.log(res.status, await res.json()); ### Chat RAG flow (`POST /api/chat/:chatId`) -On each student message, `ChatApp` orchestrates retrieval through two RAG classes (shared `RAGModule` from `RAGApp`): +The browser includes a stable opaque `clientMessageId` for each deliberate send and reuses it when +retrying the same failed transport. The server binds it to the authenticated student, course, chat, +and exact message before hashing it; a unique Mongo key prevents duplicate automatic pathway alerts. + +Before RAG, an enabled Guided Pathway may intercept the message and return its predefined response. +When its independent notification setting is on, the chat route attempts to create one anonymous +alert in a separate failure boundary for enrolled users and course staff senders. An alert-write failure never blocks the predefined safety or +redirection response. Trigger metadata remains backend-only and is not stored on `ChatMessage` or +returned to the student. + +When no pathway intercepts, `ChatApp` orchestrates retrieval through two RAG classes (shared `RAGModule` from `RAGApp`): 1. **`RAGApp.retrieveForChat`** — vector search with published-item filter (skipped in developer mode) 2. **`ragPrompts.formatRetrievedContext`** — wraps chunks in `...` diff --git a/documents/FLAG_ARCHITECTURE.md b/documents/FLAG_ARCHITECTURE.md new file mode 100644 index 00000000..b85ddc64 --- /dev/null +++ b/documents/FLAG_ARCHITECTURE.md @@ -0,0 +1,59 @@ +# Flag architecture + +EngE-AI has two course-scoped flag workflows. They share a discoverable domain folder but intentionally retain separate storage schemas, privacy boundaries, and lifecycles. + +## Boundaries + +| Layer | Responsibility | +| --- | --- | +| `src/flags` | Persistence-neutral contracts, trigger and transition policy, and failure-isolated orchestration | +| `src/db/mongo` | Registered collection resolution, indexes, CRUD, safe projections, and migration behavior | +| `src/routes` | HTTP validation, course RBAC, session-owned actors, and response mapping | +| `src/guided-pathways` | Pathway configuration, prompt/classifier behavior, and winning-trigger selection | + +Routes call the `EngEAI_MongoDB` facade. Domain services depend on small writer contracts rather than importing Mongo implementation types. + +## Workflow comparison + +| Concern | Manual flag | Guided Pathway flag | +| --- | --- | --- | +| Creation | Explicit report action | Automatic persistence after a notification-enabled winning pathway | +| Course registry | `collections.flags` | `collections.guidedPathwayFlags` | +| State | `unresolved` / `resolved` | Student: `pending` / `escalated` / `dismissed`; instructor test: `pending` / `dismissed` | +| Identity | Reporter identity supports student history and instructor enrichment | Student identity is restricted to audited reveal; instructor tests store no raw trigger identity | +| Cross-course admin workflow | None | Student alerts only; instructor tests are excluded | + +## Manual flags + +Manual flags are submitted explicitly from chat and stored in the course collection registered as `activeCourse.collections.flags`. They carry the reporting user's course-local identifier, appear in student history, may be enriched with roster names for authorized instructors, and transition between `unresolved` and `resolved`. + +The accepted categories and lifecycle transition policy live in `src/flags/manual-flag-policy.ts`; persistence remains in `src/db/mongo/flag-mongo.ts`. The legacy HTTP handlers remain in `src/routes/route-mongo.ts`. Moving that large endpoint family into `src/routes/mongo/manual-flag-routes.ts` is deferred until it has dedicated route-contract coverage. + +## Guided Pathway flags + +Guided Pathway flags are automatic records created only after the real evaluator selects a winning pathway whose `notifyInstructorOnTrigger` value is true. The persistence attempt is failure-isolated: a Mongo failure must not replace or suppress the pathway's predefined response. Manually created and seeded pathways use the same evaluator and persistence path. + +### Server-derived origin + +The chat route resolves the actor from the current `activeCourse` and `GlobalUser`; request fields never select an origin. + +- An enrolled non-staff user becomes `origin: 'student'`. +- Any course staff member (`isCourseStaff`: listed faculty instructor, TA, or platform admin) becomes `origin: 'instructor-test'`. This check runs before enrollment, so dual-role staff remain test actors. +- An outsider or a request without valid course/user context does not create an automatic alert. +- A legacy document with no `origin` is normalized to `student` at the safe-view boundary. + +Student rows retain internal `studentUserId` for the existing audited reveal workflow. At creation, instructor-test rows omit `studentUserId` and do not persist a separate trigger-actor identity field. The trigger actor ID may contribute to the opaque deduplication digest, which also includes origin so equivalent student and test triggers cannot collide, but it is never returned as trigger identity. A later dismissal retains the ordinary authorized decision-actor audit fields; those describe the decision, not the original trigger. + +### Visibility and lifecycle + +Course queue reads use inclusion-only anonymous projections. They expose the origin, pathway/course snapshots, exact message, status, and relevant decision timestamps/names, but exclude user IDs, raw chat/request identifiers, deduplication material, and reveal audit events. Exact user-authored text can still be self-identifying. + +Student alerts retain the production `pending` → `escalated` or `dismissed` lifecycle. Instructor tests appear only in the owning course queue, are labelled as tests, and can be marked complete only through the dismiss transition. Server-side guards reject test escalation, administrator review, and identity reveal before an audit or roster read. Every global admin list, total, facet, reviewer filter, and bell-count query includes only `origin: 'student'` or a legacy missing origin. + +### Collection authority and isolation + +Automatic alerts do not share the manual-flag collection. Each course owns a collection registered in `activeCourse.collections.guidedPathwayFlags`. New registrations default to `${courseName}_guided-pathway-flags`; once stored, the registry value is authoritative and survives course renames. A partial unique catalog index prevents two active courses from registering the same non-empty automatic-alert namespace. Resolution also rejects protected names, collisions with other registered course collections, and physical targets containing another course's rows. + +Alert creation uses a provisioning resolver that can register, create, and index storage. Course/admin list, count, backup, and cross-course aggregation paths use existing registered namespaces only, so reading an untouched legacy course does not create an empty collection. Queries retain a `courseId` predicate inside the selected collection as defense in depth. Generic course updates strip browser-provided `collections` values; registry changes belong to server provisioning and migration code. + +GPF-002 moves data from the former shared collection and GPF-001 hash collections into the registered target. A Mongo-backed lease serializes application instances, insert-only `_id` upserts preserve any newer target lifecycle state, source rows are deleted only after target and catalog verification, and malformed/orphan data is retained for recovery. See [DATA_MIGRATIONS.md](DATA_MIGRATIONS.md#gpf-002-guided-pathway-registered-collection-normalization). diff --git a/documents/MONGO_DATA_LAYER.md b/documents/MONGO_DATA_LAYER.md index d10f4bcc..ce041044 100644 --- a/documents/MONGO_DATA_LAYER.md +++ b/documents/MONGO_DATA_LAYER.md @@ -44,6 +44,18 @@ - **GP-001** — lazy `deleteMany({ id: 'off-topic' })` on ensure/list/seed. Registry: [DATA_MIGRATIONS.md](DATA_MIGRATIONS.md#gp-001-remove-legacy-off-topic-pathway). - **Lazy provision** — `ensurePathwaysCollection` creates collection + indexes; new-course seed / Reset inserts platform cards + evaluation shell. - **Chat threads** (`chat-mongo.ts` on `{courseName}_users.chats[]`) — conversation-level starring has been retired. New records and API responses omit `isPinned`; legacy embedded values are ignored on reads and may remain inert in MongoDB without a destructive migration. Optional `pinnedMessageId` continues to represent the separate message-level pin feature. +- **Guided Pathway alerts** (`guided-pathway-flag-mongo.ts` + `guided-pathway-flag-collection-mongo.ts`): + - Each course owns a separate collection registered in `activeCourse.collections.guidedPathwayFlags`. New registrations default to the readable `{courseName}_guided-pathway-flags` name. The stored value, not a recomputed name, remains authoritative after a course rename. GPF-001 `guided-pathway-flags-course-` names are migration sources only. Automatic alerts stay separate from `activeCourse.collections.flags` manual-flag storage and are never queried by Student Flag History or `/flags/with-names`. + - Manual `{courseName}_flags` documents use `FlagReport.status` of `unresolved`, `resolved`, or `escalated`. Escalation stores optional `escalatedAt` / `escalatedBy`; platform-admin review stores `adminReviewedAt` / `adminReviewedBy`. No migration required for existing `unresolved` / `resolved` rows. + - A partial unique index on non-empty string `activeCourse.collections.guidedPathwayFlags` registrations enforces one catalog owner per physical namespace across processes. Provisioning also rejects protected names, collisions with any other registered course collection, and physical collections containing rows for another `courseId`. Generic course updates strip client-provided `collections`; only server-owned create/provision/migration paths can change registry entries. + - **Startup/operation migration (GPF-002)** copies rows from both the former global `guided-pathway-flags` collection and GPF-001 hashed collections into the registered readable target. A durable `application-migrations` record with `_id: 'GPF-002'` provides a renewable cross-process lease and persisted completion state; a process-local promise only coalesces callers within one application instance. Operations await this gate until migration completion. + - GPF-002 uses 200-row, `_id`-keyed, insert-only `$setOnInsert` upserts. Existing target documents are not replaced, so a newer decision, admin review, or reveal audit cannot be reverted by a stale legacy snapshot. The migration verifies target ownership and every copied `_id`, compare-and-set switches the catalog, rechecks catalog ownership, and only then deletes verified source rows. It drops only empty legacy namespaces and retains malformed/orphan data for manual recovery. See [DATA_MIGRATIONS.md](DATA_MIGRATIONS.md#gpf-002-guided-pathway-registered-collection-normalization). + - Every new row has explicit `origin: 'student' | 'instructor-test'`; safe reads normalize a missing legacy origin to `student`. Student rows store the exact message, restricted `studentUserId`, opaque deduplication hash, decision/review actors and times, and append-only identity-reveal audit events. At creation, instructor-test rows omit `studentUserId` and any separate trigger-actor identity field; the trigger actor ID participates only in the opaque deduplication digest. A later dismissal may add the ordinary authorized decision-actor audit fields. PUID and raw client/chat identifiers are never stored. + - Instructor/admin list delegates use inclusion projections and map to `GuidedPathwayFlagView`; student/tester identity, deduplication data, and reveal events cannot reach normal API responses. Admin reveal first atomically appends its audit event, then resolves and returns only the current course-roster display name, falling back to `active-users` when the sender is staff not on the roster. Audit failure returns no name. + - A unique deduplication index makes transport retries an idempotent no-op. Additional per-course indexes cover status/date, pathway/status/date, and escalated/unreviewed admin queries. There is no TTL because completed decisions remain viewable. + - Production student decisions are atomic `pending` to `escalated`/`dismissed` transitions. An instructor test is course-only and can transition from `pending` to `dismissed`; escalation, admin review, and identity reveal reject it before any mutation, audit write, or roster read. Global admin rows, totals, facets, reviewer facets, and awaiting-review counts apply a student-or-missing-origin filter, so tests never enter the global workflow. Admin review remains a soft completion marker; neither workflow hard-deletes alert rows. + - Alert creation uses a provisioning resolver that may register, create, and index missing legacy-course storage. Course/admin list, count, backup, and aggregation paths use read-only resolution and do not create or index empty collections. Platform-admin listing and pending-count operations build a server-owned `$unionWith` pipeline over existing registered active-course collections; request input never supplies a physical namespace. + - Course backup reads the anonymous projection, including safe `origin`, from that course's existing registered collection. Restarting onboarding or deleting a course drops only the registered owned alert collection after counting its rows; a missing namespace is an idempotent no-op and invalidates its process-local index memo. - **Topic/week embedded content** (`topic-week-mongo.ts` on `active-course-list`): - **`learningObjectives[]`** per `items[]` — instructor CRUD; flattened via `getAllLearningObjectives` for system-prompt injection. - **`instructorStruggleTopics[]`** per `items[]` — instructor CRUD (`/struggle-topics` API); gated by `features.memoryAgent`; flattened via `getAllInstructorStruggleTopics` for memory-agent catalog only (not main chat system prompt). diff --git a/documents/RESPONSIVE_DESIGN.md b/documents/RESPONSIVE_DESIGN.md index c1eab56d..3c537b26 100644 --- a/documents/RESPONSIVE_DESIGN.md +++ b/documents/RESPONSIVE_DESIGN.md @@ -188,28 +188,102 @@ The student mode uses a consistent mobile header pattern across welcome screen, --- +## Instructor Flag Management + +**Flag Management** (`flag-instructor.html`, `flag-instructor.css`) uses the shared **page shell** (`.page-frame` > `.page-shell` > `.page-header`). Workflow nav tiles sit in the page header (outline/green, filled when active). At **768px**, tiles wrap under the title with 44px touch targets. + +**Filters** sit in page content, first below the header (not in the header, not a modal). Source, category, and period controls apply with Clear/Apply. Custom date fields stack to one column at ≤768px. + +--- + ## Instructor Dashboard -The instructor dashboard (`dashboard-instructor.html`, `dashboard.css`) uses a two-column topbar on desktop and stacks on mobile. `.dashboard-page-header` is `position: sticky` inside `.dashboard-grid-view` (the scrollport), with opaque `var(--chat-bg)` chrome and negative margins matching grid padding so cards cannot peek beside it. +The instructor dashboard (`dashboard-instructor.html`, `dashboard.css`) uses the shared **page shell**. The topbar is `.page-header` plus `.dashboard-topbar` (title + course-code flip). Welcome/date live in the scroll body under the header. ### Desktop (≥768px) -- **Topbar**: sticky flex row — left column (title, welcome, date) + right column (course-code flip widget, 200px). +- **Topbar**: sticky flex row — left column (title) + right column (course-code flip widget, 200px). - **Advanced Settings**: static section title + divider; three inline accordion cards (Model Settings, Advanced Features, Course Information). -- **Enter animation**: topbar first, greeting/date at 0.08s, card grid at 0.18s, Advanced Settings section at 0.28s. Topbar keyframes end at `transform: none` so sticky keeps working after the enter anim. +- **Enter animation**: header via `.page-header`; greeting/date at 0.08s, card grid at 0.18s, Advanced Settings section at 0.28s. ### Mobile (≤768px) -- **Topbar**: sticky; `flex-direction: column`; course-code flip `align-self: flex-start` (left-aligned under greeting block). Sticky bleed margins match the tighter grid padding (`1.25rem` / `1rem`). +- **Topbar**: sticky; `flex-direction: column`; course-code flip `align-self: flex-start` (left-aligned under the title). - **Advanced Settings**: three inline accordions (Model Settings, Advanced Features, Course Information) expand inside each card with a smooth height transition; `prefers-reduced-motion` collapses instantly. - **Feature rows**: Model pickers and Advanced Feature toggles share a wrapping flex layout at every width. Controls stay beside their title where space permits, wrap internally when possible, then move to a right-aligned line below the title. -- **Hamburger**: shown in title row via `.dashboard-mobile-menu-btn`. +- **Hamburger**: shown in the title row via `.instructor-mobile-hamburger-btn`. - **Accordions**: full-width; toggle min-height 44px for touch. Course Information was removed from the instructor sidebar footer; course code lives in the dashboard topbar and metadata in the Course Information accordion. --- +## Admin course selection (`admin-course-selection.html`) + +Styles: [`public/styles/course-selection.css`](public/styles/course-selection.css) (split layout) and [`public/styles/admin-guided-pathway-flags.css`](public/styles/admin-guided-pathway-flags.css) (escalations queue). + +### Desktop + +- **Bell**: toggles a **1:1 flex split** of `.admin-course-selection-wrapper` — courses on the left, escalations panel on the right (`flex: 1 1 0` each). Active bell uses palette light brown (`#ECE5DD` / `--background-2`). +- **Independent scroll**: the page does not scroll as a whole; the course column and the escalations list each scroll in their own overflow region. +- **Panel**: 1rem gap from the course column; 10px radius; **green header** on a **white** body. Header is a single row — title, All courses pill, compact Refresh / Hide. Bell, **Hide**, or **Escape** closes; focus returns to the bell. Closed panel uses `hidden` + `inert`. +- **Splitter**: a drag handle between the columns resizes them. Both columns have **min-width 40%**. Arrow keys on the handle nudge by 2%. Hidden on ≤768px. +- **Period cards**: when split, course containers use **100% of the left column** (not the default 80% page width). + +### Mobile (≤768px) + +- Escalations open as a **ModalOverlay** (same 768px cutoff as the View Diagram artefact modal). Course list stays full width; no stacked split and no splitter. +- Overlay X, backdrop click, Escape, or the bell closes it. Nested identity-reveal confirms still stack on top. +- `prefers-reduced-motion`: desktop split transitions remain disabled. + +--- + +## Page shell (reusable layout) + +Generic scrollport + sticky header module: [`public/styles/page-shell.css`](public/styles/page-shell.css). Loaded from `instructor-mode.html` for instructor features today; student/admin can link the same file later without renaming classes. + +### Structure + +```text +.page-frame optional full-height outer wrapper + .page-shell scrollport (1200px cap, 3rem / 1rem gutters) + .page-header optional sticky title row + bleed + (feature content) +``` + +Modifiers: `.page-shell--wide` (1680px, Writing Feedback), `.page-shell--column` + `.page-shell-body` (System Prompts editor column). + +### Tokens (on `.page-shell`, overridable per instance) + +| Token | Desktop | Mobile | +|-------|---------|--------| +| `--page-shell-max-width` | `1200px` | — | +| `--page-shell-pad-x` | `3rem` | `1rem` | +| `--page-shell-pad-bottom` | `20px` | shell uses `2rem` bottom pad | +| `--page-header-pad-top` | `1.5rem` | `1.25rem` | +| `--page-header-pad-bottom` | `1rem` | `0.75rem` | +| `--page-header-margin-bottom` | `3rem` | — | +| `--page-header-title-size` | `2rem` | — | +| `--page-header-title-weight` | `700` | — | + +### Sticky header (`.page-header`) + +- Bleed: `margin` / `padding` use `--page-shell-pad-x` so the title aligns with body content. +- `background: var(--chat-bg)` so scrolled content does not peek beside the sticky title. +- Enter animation: `page-header-in` (`0.45s ease-out`, ends at `transform: none` for sticky). + +Instructor hamburger styling stays in [`instructor-mode.css`](public/styles/instructor-mode.css) (`#main-content-area .page-header.mobile-header-bar`). + +### Adopting on a new page + +1. Link `/styles/page-shell.css`. +2. Wrap content: `page-frame` > `page-shell` > `page-header` + body. +3. Override tokens on `.page-shell` if needed (e.g. `--page-shell-max-width: none`). + +### `prefers-reduced-motion` + +- Header enter animation collapses to `0.01ms` in `page-shell.css`. + --- ## Marketing homepage (`/`, `/team`, `/docs`, `/pages/ai-disclaimer.html`) diff --git a/jest.config.cjs b/jest.config.cjs index c0399c13..a2c283c6 100644 --- a/jest.config.cjs +++ b/jest.config.cjs @@ -2,7 +2,7 @@ module.exports = { preset: 'ts-jest', testEnvironment: 'node', - roots: ['/src'], + roots: ['/src', '/public/scripts'], testMatch: ['**/__tests__/**/*.test.ts'], moduleFileExtensions: ['ts', 'js', 'json'], clearMocks: true, diff --git a/package-lock.json b/package-lock.json index 466ac8ab..44f5dac9 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "tlef-EngE-AI", - "version": "1.8.4", + "version": "1.11.4", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "tlef-EngE-AI", - "version": "1.8.4", + "version": "1.11.4", "license": "ISC", "dependencies": { "@qdrant/js-client-rest": "^1.15.1", diff --git a/package.json b/package.json index 245037ad..8f6717f7 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "tlef-EngE-AI", - "version": "1.7.27", + "version": "1.11.4", "description": "", "main": "dist/server.js", "scripts": { diff --git a/public/components/assistant-prompts/assistant-prompts-instructor.html b/public/components/assistant-prompts/assistant-prompts-instructor.html index 0c7dde3a..058dbe0e 100644 --- a/public/components/assistant-prompts/assistant-prompts-instructor.html +++ b/public/components/assistant-prompts/assistant-prompts-instructor.html @@ -1,7 +1,7 @@ -
+
-
-
+
+
diff --git a/public/components/dashboard/dashboard-instructor.html b/public/components/dashboard/dashboard-instructor.html index a4654cf1..6550ac92 100644 --- a/public/components/dashboard/dashboard-instructor.html +++ b/public/components/dashboard/dashboard-instructor.html @@ -1,15 +1,15 @@ -
-
-
+
+
+
- -

Dashboard

+

Dashboard

diff --git a/public/components/documents/documents-instructor.html b/public/components/documents/documents-instructor.html index 3a5bf518..b6908269 100644 --- a/public/components/documents/documents-instructor.html +++ b/public/components/documents/documents-instructor.html @@ -1,8 +1,8 @@ -
+
-
-
+
+
diff --git a/public/components/monitor/monitor-instructor.html b/public/components/monitor/monitor-instructor.html index 51b93a60..3ceff5ac 100644 --- a/public/components/monitor/monitor-instructor.html +++ b/public/components/monitor/monitor-instructor.html @@ -1,9 +1,9 @@ -
+
-
+
-
+
diff --git a/public/components/pathways/pathway-library-instructor.html b/public/components/pathways/pathway-library-instructor.html index e19fced6..1a143972 100644 --- a/public/components/pathways/pathway-library-instructor.html +++ b/public/components/pathways/pathway-library-instructor.html @@ -1,8 +1,11 @@ -
+
-
-

Pathway Library

+
+ +

Pathway Library

+
+
+ Notify instructor when triggered + + Creates an anonymous Guided Pathway Alert when this active pathway is triggered. + +
+
+ On + +
+
+
Call to Action diff --git a/public/components/report/flag-instructor.html b/public/components/report/flag-instructor.html index 2ea589fd..3853d155 100644 --- a/public/components/report/flag-instructor.html +++ b/public/components/report/flag-instructor.html @@ -1,8 +1,7 @@ -
+
-
- -
+
+
-
-
+
+
-
+ 0 + +
-
-
- - + 0 + +
-
- -
-
- - - - - - -
-
+
+ +
+ +

- -
- +
+
diff --git a/public/components/scenarios/scenario-questions-grid.html b/public/components/scenarios/scenario-questions-grid.html index c9b05d5f..b7a32ec2 100644 --- a/public/components/scenarios/scenario-questions-grid.html +++ b/public/components/scenarios/scenario-questions-grid.html @@ -1,14 +1,11 @@
-
-

Scenario Questions

-
-

Scenario Questions

-
View: diff --git a/public/components/scenarios/scenario-questions-instructor.html b/public/components/scenarios/scenario-questions-instructor.html index 3544e5e3..f3cf50d6 100644 --- a/public/components/scenarios/scenario-questions-instructor.html +++ b/public/components/scenarios/scenario-questions-instructor.html @@ -1,4 +1,4 @@ -
+
diff --git a/public/components/system-prompts/system-prompts-instructor.html b/public/components/system-prompts/system-prompts-instructor.html index 32e06c58..12026184 100644 --- a/public/components/system-prompts/system-prompts-instructor.html +++ b/public/components/system-prompts/system-prompts-instructor.html @@ -1,7 +1,7 @@ -
+
-
-
+
+
@@ -53,6 +53,7 @@

System Prompts

+
@@ -88,5 +89,6 @@

System Prompts

+
diff --git a/public/components/writing-feedback/writing-feedback.html b/public/components/writing-feedback/writing-feedback.html index 2f904fb5..8096c085 100644 --- a/public/components/writing-feedback/writing-feedback.html +++ b/public/components/writing-feedback/writing-feedback.html @@ -11,10 +11,10 @@ --> -
-
+
+
-
+
Welcome, +
-
- -
+
+
+ +
-
+
+
+
-
- -

Loading academic periods and courses…

-
+
+ +

Loading academic periods and courses…

+
- + + + +