Skip to content

fix: sync service otel and CI #5390

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
404Wolf wants to merge 2 commits into
base: main
Choose a base branch
from
+41 −8
Open

fix: sync service otel and CI #5390

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9b6c873
fix prod sync service
404Wolf Aug 4, 2026
bb0e449
record full error objects
404Wolf Aug 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions apps/web/src/features/block-md/definition.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,9 @@ export const definition = defineBlock({
registerDocumentSpan(documentId, rootSpan);
}
return rootSpan.span('doc.load', async (loadSpan) => {
// The long-lived root may not be retained, so keep completed loads
// independently searchable by document.
loadSpan.setAttr('document.id', documentId);
const loadBundle = () =>
loadSpan.span('doc.load.bundle', async (bundleSpan) => {
const result = await fetchDocumentLoadBundle(documentId);
Expand Down
26 changes: 21 additions & 5 deletions packages/observability/src/span.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,10 +99,26 @@ export class SpanImpl implements Span {
}

private static normalizeError(error: unknown): Error | string {
return error instanceof Error ||
typeof error === "string" ||
(typeof error === "object" && error !== null && "message" in error)
? (error as Error | string)
: String(error);
if (error instanceof Error || typeof error === "string") return error;

if (
typeof error === "object" &&
error !== null &&
"message" in error &&
typeof error.message === "string"
) {
return new Error(error.message);
}

if (typeof error === "object" && error !== null) {
try {
const message = JSON.stringify(error);
if (message !== undefined) return new Error(message);
} catch {
return new Error("Unserializable thrown object");
}
}

return new Error(String(error));
Comment on lines +104 to +122

@coderabbitai coderabbitai Bot Aug 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Make normalizeError safe for every unknown value.

JSON.stringify(error) can return undefined without throwing. Line 122 then falls through to String(error) instead of returning the intended "Unserializable thrown object" fallback. A custom message getter or string conversion can also throw. This can make SpanImpl.error throw while it is handling the original error.

Keep message access and final coercion inside guarded fallbacks.

Proposed fix 
 private static normalizeError(error: unknown): Error | string {
-	if (error instanceof Error || typeof error === "string") return error;
-
-	if (
-		typeof error === "object" &&
-		error !== null &&
-		"message" in error &&
-		typeof error.message === "string"
-	) {
-		return new Error(error.message);
-	}
-
-	if (typeof error === "object" && error !== null) {
-		try {
-			const message = JSON.stringify(error);
-			if (message !== undefined) return new Error(message);
-		} catch {
-			return new Error("Unserializable thrown object");
+	try {
+		if (error instanceof Error || typeof error === "string") return error;
+		if (typeof error === "object" && error !== null) {
+			const message = "message" in error ? error.message : undefined;
+			if (typeof message === "string") return new Error(message);
+			try {
+				const serialized = JSON.stringify(error);
+				return new Error(serialized ?? "Unserializable thrown object");
+			} catch {
+				return new Error("Unserializable thrown object");
+			}
 		}
+		return new Error(String(error));
+	} catch {
+		return new Error("Unserializable thrown value");
 	}
-
-	return new Error(String(error));
 }
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/observability/src/span.ts` around lines 104 - 122, Update
normalizeError so every operation on an unknown thrown value is guarded: safely
read the object’s message property, use the "Unserializable thrown object"
fallback when JSON.stringify returns undefined or throws, and protect the final
String(error) coercion with a fallback that cannot throw. Ensure normalizeError
always returns an Error, including for throwing getters, unserializable objects,
and values with failing string conversion.

@whutchinson98 whutchinson98 Aug 6, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

testing reply

@coderabbitai coderabbitai Bot Aug 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@whutchinson98, reply received.

You are interacting with an AI system.

@whutchinson98 whutchinson98 Aug 6, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

again

@coderabbitai coderabbitai Bot Aug 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@whutchinson98, reply received.

You are interacting with an AI system.

}
}
14 changes: 14 additions & 0 deletions packages/observability/src/telemetry.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,20 @@ describe("Telemetry", () => {
expect(exported?.events[0]?.name).toBe("exception");
});

test("records structured non-Error values", () => {
const span = Telemetry.span("edit.apply");
span.error([{ code: "ImportFailed", detail: "missing dependency" }]);
span.end();

const [exported] = spanExporter.getFinishedSpans();
expect(exported?.status.message).toBe(
'[{"code":"ImportFailed","detail":"missing dependency"}]',
);
expect(exported?.events[0]?.attributes?.["exception.message"]).toBe(
'[{"code":"ImportFailed","detail":"missing dependency"}]',
);
});

test("emits structured logs", () => {
Telemetry.warn("WAL flush not acked", {
"document.id": "doc-1",
Expand Down
2 changes: 1 addition & 1 deletion services/sync-service/wrangler.docker.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ name = "sync-service-prod2"

[env.prod.vars]
INTERNAL_API_SECRET_KEY = "SYNC_SERVICE_KEY_PROD"
ENVIRONMENT = "production"
ENVIRONMENT = "prod"
SPS_URL = "https://search-processing.macro.com"

[env.prod.durable_objects]
Expand Down
4 changes: 2 additions & 2 deletions services/sync-service/wrangler.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ main = "build/worker/shim.mjs"
compatibility_date = "2025-03-05"

[build]
command = "cargo install -q worker-build@0.8.4 --locked && worker-build --profile sync-service-release"
command = "cargo install -q worker-build@0.8.4 --locked --force && worker-build --profile sync-service-release"

[observability.logs]
enabled = true
Expand Down Expand Up @@ -138,7 +138,7 @@ name = "sync-service-prod2"

[env.prod.vars]
INTERNAL_API_SECRET_KEY = "SYNC_SERVICE_KEY_PROD"
ENVIRONMENT = "production"
ENVIRONMENT = "prod"
# The proxy injects the Datadog key; the exporter appends the signal path.
OTEL_EXPORTER_OTLP_ENDPOINT = "https://macro-prox-prod.macroverse.workers.dev/i/otlp"
SPS_URL = "https://search-processing.macro.com"
Expand Down
Loading