feat(core): support Responses API with configuration and documentation

apps/chrome-extension/src/extension/recorder/generators/shared/testGenerationUtils.ts

@@ -1,3 +1,4 @@
+import type { MessageContent } from '@midscene/core/ai-model';
 import type { ChromeRecordedEvent } from '@midscene/recorder-ui';
 import { recordLogger } from '../../logger';
 import type {
@@ -169,8 +170,8 @@ export const createMessageContent = (
   promptText: string,
   screenshots: string[] = [],
   includeScreenshots = true,
-): Array<string | Record<string, any>> => {
-  const messageContent: Array<string | Record<string, any>> = [
+): MessageContent[] => {
+  const messageContent: MessageContent[] = [
     {
       type: 'text',
       text: promptText,
@@ -186,10 +187,8 @@ export const createMessageContent = (
 
     screenshots.forEach((screenshot) => {
       messageContent.push({
-        type: 'image_url',
-        image_url: {
-          url: screenshot,
-        },
+        type: 'image',
+        url: screenshot,
       });
     });
   }

apps/chrome-extension/src/extension/recorder/utils.ts

@@ -1,5 +1,6 @@
 import {
-  type AIArgs,
+  type MessageContent,
+  type ModelCallMessages,
   callAIWithObjectResponse,
   callAIWithStringResponse,
   getModelRuntime,
@@ -9,7 +10,6 @@ import type { IModelConfig } from '@midscene/shared/env';
 import { message } from 'antd';
 import { saveAs } from 'file-saver';
 import JSZip from 'jszip';
-import type { ChatCompletionContentPart } from 'openai/resources/index';
 import type { RecordingSession } from '../../store';
 import { recordLogger } from './logger';
 import { withRecorderMessageTimeout } from './messageTimeout';
@@ -338,7 +338,7 @@ export const generateRecordTitle = async (
       const screenshots = getScreenshotsForLLM(events);
 
       // Create the message content
-      const messageContent: ChatCompletionContentPart[] = [
+      const messageContent: MessageContent[] = [
         {
           type: 'text',
           text: `Generate a concise title (5-7 words) and brief description (1-2 sentences) for a browser recording session with the following events:\n\n${JSON.stringify(summary, null, 2)}\n\nRespond with a JSON object containing "title" and "description" fields. The title should be action-oriented and highlight the main task accomplished. The description should provide slightly more detail about what was done.`,
@@ -354,10 +354,8 @@ export const generateRecordTitle = async (
 
         screenshots.forEach((screenshot) => {
           messageContent.unshift({
-            type: 'image_url',
-            image_url: {
-              url: screenshot,
-            },
+            type: 'image',
+            url: screenshot,
           });
         });
       }
@@ -628,7 +626,7 @@ const generateAIMindmap = async (
       };
     });
 
-    const prompt: AIArgs = [
+    const prompt: ModelCallMessages = [
       {
         role: 'system',
         content: `You are an expert test automation analyst who creates detailed Mermaid mindmaps that preserve the complete sequence and details of user interactions.

apps/site/docs/en/faq.md

@@ -128,15 +128,15 @@ MIDSCENE_MODEL_API_KEY="<your-azure-api-key>"
 
 In other words, other settings such as `MIDSCENE_MODEL_NAME` and `MIDSCENE_MODEL_FAMILY` should still follow the corresponding model section in [Supported models and setup](./model-common-config). Azure is only a model provider with different authentication, not a special model.
 
-This uses the normal OpenAI-compatible path and sends `POST /openai/v1/chat/completions` with `Authorization: Bearer ...`. Do not append `/chat/completions` to `MIDSCENE_MODEL_BASE_URL`. For most `/openai/v1` endpoints you do not need `api-version`.
+This uses the normal OpenAI-compatible path with `Authorization: Bearer ...`. Depending on `MIDSCENE_MODEL_PROTOCOL`, requests go to `/openai/v1/chat/completions` (the default) or `/openai/v1/responses`. Confirm that your Azure model and endpoint support the selected protocol. Do not append `/chat/completions` or `/responses` to `MIDSCENE_MODEL_BASE_URL`. For most `/openai/v1` endpoints you do not need `api-version`.
 
 If your resource still rejects the request with `400 Missing required query parameter: api-version`, the `/openai/v1` surface on that specific resource has not GA'd yet. Inject the query parameter through `defaultQuery`:
 
 ```bash
 MIDSCENE_MODEL_INIT_CONFIG_JSON='{"defaultQuery":{"api-version":"preview"}}'
 ```
 
-Use the `api-version` value your resource expects (`preview`, or a dated version like `2025-01-01-preview` shown in the Azure portal). This turns every request into `.../openai/v1/chat/completions?api-version=preview`.
+Use the `api-version` value your resource expects (`preview`, or a dated version like `2025-01-01-preview` shown in the Azure portal). The query parameter is appended to the selected protocol's request path, for example `.../openai/v1/chat/completions?api-version=preview` or `.../openai/v1/responses?api-version=preview`.
 
 If an Azure-compatible gateway only accepts the `api-key` header, use this fallback:
 
@@ -158,11 +158,19 @@ Azure AD / keyless auth (`DefaultAzureCredential`) is not supported. Use an API
 
 ## Clicks are offset when using Azure OpenAI
 
-With a GPT-5 family model, you may find that the same script clicks the correct spot on the official OpenAI API but a consistently offset spot on Azure OpenAI. The offset scales with resolution: it appears at large screenshots (e.g. `1920x1080`) and disappears at small ones (e.g. `1280x600`).
+With a GPT-5 family model, you may find that the same script clicks the correct spot on the official OpenAI API but a consistently offset spot on Azure OpenAI's Chat Completions API. The offset scales with resolution: it appears at large screenshots (e.g. `1920x1080`) and disappears at small ones (e.g. `1280x600`).
 
-The cause is image handling on the Azure side. GPT-5 returns absolute coordinates based on the screenshot it actually sees, and Midscene sends the screenshot with `"detail": "original"` so the model sees the full-resolution image (see the [GPT-5 notes](./model-common-config#gpt)). Azure does not honor `"detail": "original"`, so it downscales large images server-side (the short side is capped at 768). The model then answers in the downscaled coordinate space while Midscene maps coordinates against the original resolution, producing a proportional offset. You can confirm `original` is not taking effect by checking token usage: when `original` works, image token consumption is noticeably higher.
+We observed that this issue is related to image handling in Azure's Chat Completions API. GPT-5 returns absolute coordinates based on the screenshot it actually sees, and Midscene sends the screenshot with `"detail": "original"` so the model sees the full-resolution image (see the [GPT-5 notes](./model-common-config#gpt)). In affected Chat Completions requests, Azure does not honor `"detail": "original"`, so it downscales large images server-side (the short side is capped at 768). The model then answers in the downscaled coordinate space while Midscene maps coordinates against the original resolution, producing a proportional offset. You can confirm `original` is not taking effect by checking token usage: when `original` works, image token consumption is noticeably higher.
 
-There are two ways to work around it:
+We have verified that `"detail": "original"` works correctly with Azure's Responses API, so we recommend switching to the Responses protocol first:
+
+```bash
+MIDSCENE_MODEL_PROTOCOL="openai-responses"
+```
+
+Confirm that your model supports the Responses API and use its corresponding Base URL. See [Protocol type](./model-config#model-api-type) for configuration details.
+
+If you cannot switch protocols yet, you can also:
 
 1. Use the official OpenAI GPT-5, or configure a separate model dedicated to grounding (localization) and keep the Azure GPT-5 only as the planning model.
 2. Pre-shrink the screenshot with the `screenshotShrinkFactor` agent option so the image stays under Azure's downscale threshold and no server-side resizing happens. See [`screenshotShrinkFactor`](./reference/#common).