Fix/ollama health monitoring and timeouts

[Ayan-josh-05]

fix: keep field-mapping and translation usable when Ollama is down or cold-loading

The problem

Both field_mapping_poc and translation call the same local Ollama instance. Two things went wrong, and they made each other worse:

1. A stopped Ollama looked identical to a slow one.

The health probe had no connect timeout, so when Ollama wasn't running, the probe just hung.

/health stayed stuck on "initializing" forever, and /map and /translate kept accepting requests that queued up behind a call that was never going to return — each one holding the service's single request lock.

2. Timing out a request made the model load slower.

Ollama treats the client closing a connection as a cancellation.

If our timeout fired while the model was cold-loading (2–4 minutes on CPU), Ollama aborted the load. The retry then started a fresh load, which timed out again — creating a loop that never let the model finish loading.

The fix

Split the timeout into two phases instead of one flat number.

Connect timeout: 5s

Either Ollama is listening or it isn't. This can't abort a model load, because if the socket connected, the load hasn't started yet.

Read timeout: 600s

A cold load or a long document has to be waited out, never raced. This is a last-resort circuit breaker for a wedged server, not a per-request deadline.

Probe in two steps so each status means one thing

• /api/tags just lists model files on disk. It's cheap and can't trigger a load. This answers: "Is the server up?"
• A real one-token chat ping answers: "Is the model ready?"

That gives three honest statuses:

• unreachable — Server is down
• initializing — Server is up, model hasn't answered yet (usually cold-loading)
• ok — Model is loaded and responding

Previously, a single probe couldn't tell the first two states apart, so a perfectly healthy service dipped to "initializing" during every ping.

Reject early when Ollama is genuinely down

/map, /translate/text, and /translate/files now return 503 with Retry-After: 30 before taking the request lock, so callers fail fast instead of piling up.

This fires only on unreachable — not on "anything that isn't ok".

initializing requests do succeed if you let them wait, and rejecting them would turn "slow" into "broken".

Changes by area

field_mapping_poc

Brought up to the same standard translation already had:

• Added the background health monitor and a real /health. It used to return a hardcoded {"status": "healthy"} that told you nothing about Ollama.
• Default num_ctx raised from 8192 → 32768 to match translation. Ollama treats a different num_ctx as a different runtime, so the mismatch was forcing a full model reload (~2 min) every time the two services took turns calling it.
• The monitor now starts on a half-interval offset, so the two services' pings land staggered instead of hitting the shared model at the same moment.
• Read timeouts are no longer retried — we already waited the full window, and the timeout just aborted whatever Ollama was loading, so a retry is strictly worse.
• Connect timeouts are still retried: they fail fast and Ollama may just be mid-restart.

translation

Closed the gaps found while porting the pattern across:

• Model calls now go through a configured client with the connect/read split above, instead of the untimed module-level chat(). Without any read bound, a wedged Ollama held _translate_lock until the process was restarted.
• Added the two-phase probe and the fail-fast gate.
• Pulled the retry/backoff logic out into _backoff() (shared by both failure paths, was previously duplicated).

Small fixes

• surya-inference/entrypoint.sh: mkdir -p "$MODEL_DIR" before using it.
• frontend/.env.example: 8000 → 8080, the actual gateway port.
• Deleted field_mapping_poc/.env.example — stale. It listed DATABASE_URL, ENCRYPTION_KEY, and DEBUG, none of which that service reads. The root .env.example is the real one.
• cancel() + await in Surya OCR warm-up shutdown keeps the asyncio side tidy by ensuring there is no unfinished task left when the event loop closes. It also makes shutdown deterministic from the async side and keeps the implementation consistent with stop_monitoring().

Cold load

Restart Ollama, then immediately call /map.

/health should report initializing and the request should succeed after the load finishes (~3 min) rather than being rejected or timing out.

Shutdown

docker compose down

Notes for reviewers

• 600s is a deliberate range, not a guess. It has to sit above a real cold load (~215s measured) but not too far past the gateway's 300s proxy timeout. Past that point nobody is waiting for the answer, but the call is still holding the lock. It's env-overridable for slower hosts.
• One caveat on translation: a long document can legitimately generate for a long time (num_predict=16384 at ~3.5 tok/s on CPU). A large enough document could hit the 600s read ceiling and get cut off mid-answer. If long translations start failing right around 600s, OLLAMA_TIMEOUT_SECONDS is the knob — Ollama isn't broken.