Repository navigation
docs: publish AI-agent Markdown mirror - #179
Conversation
|
@baalimago not sure how useful this is, but i think it could be a good addition . |
There was a problem hiding this comment.
Very solid effort, well needed!
Tiny scope-creep, but I recommend adding support for https://acceptmarkdown.com/ in the server somehow. So basically Accept: text/markdown parsing -> parse markdown render of the same docs. Might require rewrite of the server to make it work automatically?
|
The problem is, this would need a real server instead of a static page. |
|
True, but as we're in no rush, I think we should scale up and go for a refactor which supports it. |
|
@baalimago they literally can't drift because the markdown pages get auto-generated based on the existing content.... |
|
Ohh so they go to |
|
Yes. |
Why
Wasmer documentation is currently published for people reading in a browser. AI agents can use those pages, but they have to scrape rendered HTML, filter out navigation and layout markup, and guess what interactive MDX components mean.
This PR gives agents a direct Markdown version at
/ai-docs. The goal is not to maintain a second copy of the docs. Authors keep working in the existingcontent/tree, and the Markdown mirror is derived during the build so it cannot quietly drift away from the website.What changed
The new generator reads every Markdown and MDX page, converts the document into a syntax tree, and writes a matching
.mdfile underpublic/ai-docs. The existing static export then publishes those files under the same subpath.Custom components have explicit Markdown counterparts beside their React implementations. Tabs become labeled sections, steps become numbered lists, callouts become blockquotes, cards become links, file trees become text blocks, and imported images are copied into the mirror. Reusable MDX fragments are expanded in place.
This conversion happens at the AST level instead of rendering React to HTML and converting the HTML back. That preserves Markdown structure and makes the supported component surface deliberate. If a new component, dynamic attribute, or executable expression has no safe Markdown representation, the generator stops with the source file and location rather than publishing partial content.
Internal documentation links are rewritten to point directly at
/ai-docs/*.mdfiles. The generator also validates generated page and asset targets, and two stale documentation routes now resolve to their current destinations.The browser footer now includes a visible “Markdown version (for AI)” link with explanatory hover text. The Markdown landing page also ends with a generated, product-grouped index of every document, so an agent can discover the entire mirror from one request instead of guessing routes.
Generation runs automatically before
pnpm devandpnpm build. The generated directory is ignored by Git; only the generator and serializers are committed.Result
The current documentation produces 88 Markdown pages and 26 imported assets. Visiting
/ai-docsredirects to/ai-docs/index.md, while direct page requests are served astext/markdown. The root Markdown file now provides a complete directory of all 88 pages.Verification
pnpm test:ai-docs(six tests)pnpm build/ai-docs/index.md, confirmed the Markdown content type, and checked the generated product sectionsLinear: https://linear.app/wasmer/issue/ECO-485/publish-an-ai-agent-markdown-mirror-of-docswasmerio