Skip to content

fix(sync): content_preview / content_text から HTML タグを除去 #498

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
otomatty merged 1 commit into from
Apr 6, 2026
Merged

fix(sync): content_preview / content_text から HTML タグを除去 #498

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 83 additions & 0 deletions server/hocuspocus/src/extractPlainTextFromYXml.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,25 @@ import * as Y from "yjs";
import { buildContentPreview, extractTextFromYXml } from "./extractPlainTextFromYXml.js";

describe("extractTextFromYXml", () => {
it("returns empty string for an empty fragment / 空の fragment では空文字を返す", () => {
const doc = new Y.Doc();
const fragment = doc.getXmlFragment("default");
expect(extractTextFromYXml(fragment)).toBe("");
});

it("extracts plain text from a simple paragraph / 単純なパラグラフからテキストを抽出する", () => {
const doc = new Y.Doc();
doc.transact(() => {
const fragment = doc.getXmlFragment("default");
const p = new Y.XmlElement("paragraph");
fragment.push([p]);
const t = new Y.XmlText();
t.insert(0, "Hello world");
p.push([t]);
});
expect(extractTextFromYXml(doc.getXmlFragment("default")).trim()).toBe("Hello world");
});

it("does not insert a newline inside a paragraph between inline bold and following text", () => {
const doc = new Y.Doc();
doc.transact(() => {
Expand Down Expand Up @@ -49,6 +68,70 @@ describe("extractTextFromYXml", () => {
expect(plain.includes("First") && plain.includes("Second")).toBe(true);
expect(/\n/.test(plain)).toBe(true);
});

it("strips formatting attributes from XmlText delta (Tiptap marks) / XmlText の書式属性を除去する", () => {
const doc = new Y.Doc();
doc.transact(() => {
const fragment = doc.getXmlFragment("default");
const p = new Y.XmlElement("paragraph");
fragment.push([p]);
const t = new Y.XmlText();
t.insert(0, "Hello ");
t.insert(6, "world", { bold: true });
t.insert(11, "!");
p.push([t]);
});
const plain = extractTextFromYXml(doc.getXmlFragment("default")).trim();
expect(plain).toBe("Hello world!");
expect(plain).not.toContain("<bold>");
expect(plain).not.toContain("</bold>");
});

it("strips italic and other mark attributes / italic 等の書式属性も除去する", () => {
const doc = new Y.Doc();
doc.transact(() => {
const fragment = doc.getXmlFragment("default");
const p = new Y.XmlElement("paragraph");
fragment.push([p]);
const t = new Y.XmlText();
t.insert(0, "normal ");
t.insert(7, "italic", { italic: true });
t.insert(13, " ");
t.insert(14, "bold-italic", { bold: true, italic: true });
p.push([t]);
});
const plain = extractTextFromYXml(doc.getXmlFragment("default")).trim();
expect(plain).toBe("normal italic bold-italic");
expect(plain).not.toContain("<italic>");
expect(plain).not.toContain("<bold>");
});

it("handles nested elements (e.g. list items) / ネストされた要素を処理する", () => {
const doc = new Y.Doc();
doc.transact(() => {
const fragment = doc.getXmlFragment("default");
const list = new Y.XmlElement("bulletList");
const item1 = new Y.XmlElement("listItem");
const p1 = new Y.XmlElement("paragraph");
const t1 = new Y.XmlText();
t1.insert(0, "Item 1");
p1.push([t1]);
item1.push([p1]);

const item2 = new Y.XmlElement("listItem");
const p2 = new Y.XmlElement("paragraph");
const t2 = new Y.XmlText();
t2.insert(0, "Item 2");
p2.push([t2]);
item2.push([p2]);

list.push([item1, item2]);
fragment.push([list]);
});
const plain = extractTextFromYXml(doc.getXmlFragment("default"));
expect(plain).toContain("Item 1");
expect(plain).toContain("Item 2");
});
});

describe("buildContentPreview", () => {
Expand Down
13 changes: 12 additions & 1 deletion server/hocuspocus/src/extractPlainTextFromYXml.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,15 +30,26 @@ function isInlineXmlElement(node: Y.XmlElement): boolean {

/**
* Y.Doc の XmlFragment（または XmlElement 根）からプレーンテキストを再帰的に抽出する。
* `Y.XmlText.toString()` / `toJSON()` は `<bold>` 等の HTML タグを返すため、
* `toDelta()` を使い `insert` 文字列のみを連結する。
*
* Recursively extract plain text from a Y.XmlFragment or Y.XmlElement subtree.
* Uses `toDelta()` instead of `toString()` / `toJSON()` because the latter
* returns HTML-like tags (`<bold>`, `<italic>`, etc.) for formatted text.
*/
export function extractTextFromYXml(node: Y.XmlFragment | Y.XmlElement): string {
let text = "";

for (let i = 0; i < node.length; i++) {
const child = node.get(i);
if (child instanceof Y.XmlText) {
text += child.toString();
// toDelta() を使い書式属性なしの純粋なテキストのみを抽出する。
// Use toDelta() to extract pure text without formatting attributes.
for (const op of child.toDelta()) {
if (typeof op.insert === "string") {
text += op.insert;
}
}
} else if (child instanceof Y.XmlElement) {
const inner = extractTextFromYXml(child);
const suffix = isInlineXmlElement(child) ? " " : "\n";
Expand Down
4 changes: 3 additions & 1 deletion server/hocuspocus/src/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,8 @@ async function loadDocumentFromDb(pageId: string): Promise<Y.Doc> {

async function saveDocumentToDb(pageId: string, document: Y.Doc): Promise<void> {
const encodedState = Buffer.from(Y.encodeStateAsUpdate(document));
// Y.Doc からプレーンテキストを抽出（HTML タグなし・toDelta() ベース）
// Extract plain text from Y.Doc (without HTML tags, using toDelta())
const contentText = extractTextFromYXml(document.getXmlFragment("default"));
const contentPreview = buildContentPreview(contentText);
const client = await getPool().connect();
Expand All @@ -207,8 +209,8 @@ async function saveDocumentToDb(pageId: string, document: Y.Doc): Promise<void>
VALUES ($1, $2, 1, $3, NOW())
ON CONFLICT (page_id) DO UPDATE
SET ydoc_state = EXCLUDED.ydoc_state,
version = page_contents.version + 1,
content_text = EXCLUDED.content_text,
version = page_contents.version + 1,
updated_at = NOW()
`,
[pageId, encodedState, contentText],
Expand Down
10 changes: 9 additions & 1 deletion src/lib/collaboration/CollaborationManager.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -296,7 +296,15 @@ export class CollaborationManager {
const parts: string[] = [];
const walk = (node: Y.XmlFragment | Y.XmlElement | Y.XmlText) => {
if (node instanceof Y.XmlText) {
parts.push(node.toJSON());
// toDelta() を使い書式属性なしの純粋なテキストのみを抽出する。
// toString() / toJSON() は <bold> 等の HTML タグを返すため使用しない。
// Use toDelta() to extract pure text without formatting attributes.
// toString() / toJSON() return HTML-like tags (<bold>, etc.) so we avoid them.
for (const op of node.toDelta()) {
if (typeof op.insert === "string") {
parts.push(op.insert);
}
}
Comment on lines +303 to +307

@gemini-code-assist gemini-code-assist Bot Apr 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

node.toDelta() は、テキスト内の書式（太字、斜体など）が切り替わる箇所で複数の op を返します。現在の実装では、各 op.insert を個別に parts 配列に push していますが、最終的に parts.join("\n")（307行目）で連結されるため、書式の境界ごとに意図しない改行が挿入されてしまいます。

XmlText ノード内のテキストは、改行を挟まずに結合してから parts に追加することで、以前の toJSON() を使用していた時と同様の構造（1つの XmlText ノードにつき1つの parts 要素）を維持しつつ、HTML タグのみを除去できます。

        let text = "";
        for (const op of node.toDelta()) {
          if (typeof op.insert === "string") {
            text += op.insert;
          }
        }
        parts.push(text);

} else {
for (const child of node.toArray()) {
if (
Comment on lines +303 to 310

@devin-ai-integration devin-ai-integration Bot Apr 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Client-side extractText inserts spurious newlines between formatted text runs

The old code pushed one string per XmlText node via node.toJSON(), so parts.join("\n") only added newlines between block-level text nodes (paragraphs). The new code pushes one string per delta operation (one per formatting run), but parts.join("\n") is unchanged. This means for text like "Hello world!", toDelta() produces 3 entries ("Hello ", "world", "!"), and the join produces "Hello \nworld\n!" instead of "Hello world!".

This corrupts content_text sent to the server in local-mode saves (used for full-text search at server/api/src/routes/search.ts:53), breaks content duplication detection (detectContentDuplication), and also affects the fireAndForgetSave on page unload. The server-side implementation at server/hocuspocus/src/extractPlainTextFromYXml.ts:45 correctly uses parts.join("") with explicit newline insertion between block elements.

(Refers to lines 289-307)

Prompt for agents 
In CollaborationManager.ts, the extractText method at line 281 now pushes one string per delta op into `parts`, but still uses `parts.join("\n")` to concatenate. This inserts newlines between formatting runs within the same XmlText node (e.g. bold/italic text). The server-side implementation in extractPlainTextFromYXml.ts solves this correctly by using `parts.join("")` with explicit newline insertion between block elements.

Two approaches to fix:
1. Align the client implementation with the server-side one: use `parts.join("")` and insert explicit newlines between block-level elements (like XmlElement children that are not the last child).
2. Alternatively, import and reuse `extractTextFromYXml` from a shared location, though currently the server utility is in server/hocuspocus/src/ which isn't accessible from the frontend. Consider extracting the utility to a shared package or duplicating the correct algorithm.

The key change needed: within the XmlText branch, accumulate all delta inserts into a single string (or push them individually), but do NOT join all parts with "\n". Only add newlines between block-level elements.
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Expand Down
Loading