Skip to content

whitepaper.md

Latest commit

 

History

History
267 lines (211 loc) · 11.7 KB

whitepaper.md

File metadata and controls

267 lines (211 loc) · 11.7 KB
Raw

A Dual-Layer Framework for AI-Optimized Web Crawling and Structured Content Delivery

Whitepaper: Langshake & Extended Sitemap Protocol
Version: 1.0 (April 2025)
Authors: Langshake team

Abstract

The rise of Large Language Models (LLMs) and AI agents is reshaping how content is consumed online. Traditional HTML-based crawling methods are inefficient, expensive, and prone to hallucination or misinterpretation. This whitepaper proposes a hybrid standard: combining the Langshake micro-standard with an Extended Sitemap Protocol. Together, these frameworks provide both global and per-page LLM-ready structured content in JSON format. The proposal includes a verification system using hash validation or Merkle roots to ensure data integrity and defend against schema manipulation. This dual approach offers developers, AI systems, and search engines a path toward a scalable, efficient, and trustworthy machine-readable web.

1. Purpose & Motivation

Web crawling today involves parsing unstructured HTML, which is costly and imprecise. While Schema.org and JSON-LD have helped, they often bloat pages and are easily manipulated. AI and LLMs require structured, reliable content at scale — but the current ecosystem doesn't deliver it efficiently.

The combination of Langshake and the Extended Sitemap Protocol aims to:

  • Reduce server and compute overhead for AI agents and crawlers
  • Give developers full control over what content is exposed to machines
  • Ensure integrity via hashes and verifiable structures
  • Support both whole-site and per-page metadata

2. Core Components

2.1 Langshake .well-known/llm.json

Langshake is a global summary format that provides LLM-friendly site metadata, FAQs, links to modular JSON content files, and optional LLM-specific context. These files are typically generated by the Langshake CLI, which extracts Schema.org-compliant data from HTML and outputs it as standalone files. Example:

{
  "version": "1.0",
  "site": {
    "name": "Example Corp",
    "description": "Tools for developers",
    "language": "en"
  },
  "modules": [
    "https://example.com/langshake/article.json",
    "https://example.com/langshake/products.json",
    "https://example.com/langshake/docs.json"
  ],
  "llm_context": {
    "summary": "We are a privacy-first AI tools company.",
    "principles": ["Transparency", "Open Source", "Non-tracking"],
    "usage_notes": "We prioritize ethical AI use cases and do not support military applications."
  },
  "verification": {
    "strategy": "merkle",
    "merkleRoot": "abc123...",
    "lastVerified": "2025-04-10"
  }
}

Note: The modules array uses absolute URLs (e.g., https://example.com/langshake/article.json) to ensure maximum compatibility and ease of crawling for AI agents and search engines. Absolute URLs remove ambiguity, work across different contexts, and are considered best practice for machine-readable site manifests.

⚠️ While the llm_context field allows developers to share nuanced or mission-driven content with LLMs, it is not validated by external schema verification. It is recommended that AI platforms and agents treat this field with caution and avoid using it for factual grounding unless verified through external trust or citation mechanisms.

Discovery: LLM/AI agents should look for the .well-known/llm.json file at the root of the domain for global site metadata and module discovery. This approach follows web standards (RFC 8615) and has zero SEO impact.

2.2 LangShake Extended Sitemap Protocol

Extends sitemap.xml with the <langshake:schema-url> and <langshake:checksum> tags, using the langshake namespace to provide per-page Schema.org-compliant JSON references and integrity verification. Example:

<url>
  <loc>https://example.com/article</loc>
  <lastmod>2025-04-16T17:58:00Z</lastmod>
  <changefreq>weekly</changefreq>
  <priority>0.8</priority>
  <langshake:schema-url>https://example.com/langshake/article.json</langshake:schema-url>
  <langshake:checksum>8f7a9b3cf5a...a8b9c07e8f</langshake:checksum>
</url>

2.3 Per-Page JSON-LD Output Format & Checksum (Langshake Standard)

Each per-page JSON-LD file is output in a universal, verifiable format:

  • The output is always an array, even if there is only one JSON-LD object.
  • The last element of the array is always an object { "checksum": "..." }.
  • The checksum is calculated from the array of JSON-LD objects (excluding the checksum object itself).
  • The original JSON-LD objects are not mutated or wrapped.

Single JSON-LD Object Example

Extracted:

[
  {
    "@context": "http://schema.org",
    "@type": "Article",
    "headline": "LangShake: Revolutionizing LLM Training Data",
    "description": "A comprehensive guide to implementing the LangShake protocol for efficient AI data collection",
    "articleBody": "The rise of Large Language Models has created a need for more efficient data collection methods. LangShake addresses this by...",
    "datePublished": "2025-04-16T17:58:00Z",
    "dateModified": "2025-04-16T17:58:00Z",
    "image": "https://example.com/images/langshake-header.jpg",
    "author": {
      "@type": "Person",
      "name": "Jane Smith",
      "url": "https://example.com/team/jane-smith"
    },
    "publisher": {
      "@type": "Organization",
      "name": "Example Corp",
      "logo": {
        "@type": "ImageObject",
        "url": "https://example.com/logo.png"
      }
    },
    "keywords": ["LLM", "AI", "data collection", "web standards"],
    "mainEntityOfPage": {
      "@type": "WebPage",
      "@id": "https://example.com/article"
    }
  }
]

Output:

[
  {
    "@context": "http://schema.org",
    "@type": "Article",
    "headline": "LangShake: Revolutionizing LLM Training Data",
    "description": "A comprehensive guide to implementing the LangShake protocol for efficient AI data collection",
    "articleBody": "The rise of Large Language Models has created a need for more efficient data collection methods. LangShake addresses this by...",
    "datePublished": "2025-04-16T17:58:00Z",
    "dateModified": "2025-04-16T17:58:00Z",
    "image": "https://example.com/images/langshake-header.jpg",
    "author": {
      "@type": "Person",
      "name": "Jane Smith",
      "url": "https://example.com/team/jane-smith"
    },
    "publisher": {
      "@type": "Organization",
      "name": "Example Corp",
      "logo": {
        "@type": "ImageObject",
        "url": "https://example.com/logo.png"
      }
    },
    "keywords": ["LLM", "AI", "data collection", "web standards"],
    "mainEntityOfPage": {
      "@type": "WebPage",
      "@id": "https://example.com/article"
    }
  },
  {
    "checksum": "8f7a9b3cf5a...a8b9c07e8f"
  }
]

Multiple JSON-LD Objects Example

Extracted:

[
  { "@type": "WebPage", "name": "A" },
  { "@type": "WebPage", "name": "B" }
]

Output:

[
  { "@type": "WebPage", "name": "A" },
  { "@type": "WebPage", "name": "B" },
  { "checksum": "..." }
]

How to Verify the Checksum

  1. Read the file as an array.
  2. Remove the last element (the checksum object).
  3. Calculate the checksum on the remaining array.
  4. Compare to the value in the removed checksum object.

This format is universal, easy to verify, and works for both single and multiple JSON-LD objects.

2.4 Extended Sitemap Protocol Example (unchanged)

<url>
  <loc>https://example.com/article</loc>
  <lastmod>2025-04-16T17:58:00Z</lastmod>
  <changefreq>weekly</changefreq>
  <priority>0.8</priority>
  <langshake:schema-url>https://example.com/langshake/article.json</langshake:schema-url>
  <langshake:checksum>8f7a9b3cf5a...a8b9c07e8f</langshake:checksum>
</url>

3. Verification Strategy

To prevent abuse (fake prices, fake reviews), both systems support validation:

  • Per-JSON Hashing: Each JSON file includes a checksum object as the last element of the array. During validation, this object must be removed before recalculating the hash.

  • Merkle Root Aggregation: The CLI aggregates all per-page hashes into a Merkle tree, and includes the root in .llm.json. This enables fast multi-file verification.

  • Random Sampling: Crawlers validate 5–10% of pages by extracting the schema from the actual HTML and comparing the hash.

  • Validation Rules: If the hashes don't match, crawlers may de-rank the data, issue warnings, or block further trust in that site's structured content.

  • Caution with Context Fields: Fields like llm_context are unverified and should not be used for factual reasoning or truth-grounding by default.

4. Developer Workflow

  1. Author schema data in your source code or CMS.

  2. Run langshakeit:

    • Scans built HTML output (e.g., from Next.js, Astro)
    • Extracts and caches all JSON-LD schema data
    • Generates per-page JSON-LD files in /langshake/, each ending with a verifiable checksum
    • Computes a Merkle root from all per-page checksums
    • Outputs the global .well-known/llm.json index with module links, Merkle root, and optional llm_context
    • Auto-detects your site's public base URL for accurate module referencing
    • Uses smart caching to only update changed files

  3. Optionally, provide an llm_context.json file to describe your site's principles and intended LLM usage. This content is embedded into .llm.json, but is not subject to checksum validation.

  4. Run shakeproof:

    • Re-extracts Schema.org data from rendered HTML
    • Recalculates checksums and compares with your published JSON files
    • Verifies Merkle root integrity
    • Benchmarks extraction speed, schema match accuracy, and system resource usage
    • Outputs both JSON results and a human-readable Markdown summary

  5. LLM/AI discovery is handled solely via the .well-known/llm.json file located at the root of your domain, in compliance with RFC 8615.

5. Benefits

  • Efficiency: Minimizes HTML parsing with structured JSON metadata, reducing computational overhead and speeding up data collection processes.

  • Integrity: Checksum verification ensures trustworthy data by validating content hasn't been modified, creating a more reliable training dataset.

  • Compatibility: Builds on established Sitemap.xml and Schema.org standards, making adoption straightforward for websites already using these technologies.

  • Developer-Friendly: CLI tool abstracts complexity; caching speeds up local builds.

  • Context-Ready: Developers can add nuance for LLMs in a dedicated field.

  • Future-Proof: Modular, extendable spec allows for Merkle, LLM summaries, and more.

6. Next Steps

  • Generate the LangShake Extended Sitemap Protocol
  • Submit the Extended Sitemap Protocol proposal to the W3C
  • Improve the Open-source:
  • Attract more webmasters and contributors to the project
  • Partner with:
    • AI/LLM platforms for .llm.json adoption and verification support
    • SEO and structured data platforms to recognize Langshake as a trust signal

Conclusion

By merging Langshake's developer-first JSON interface with the Extended Sitemap Protocol's page-level precision and hash validation, this proposal delivers a unified standard for AI-native web content. It empowers developers, protects against abuse, and enables the next generation of trusted, structured, machine-readable websites.

Contact & Contribution

To get involved, visit: github.com/langshake or contact the authors directly. All specs and tools are MIT-licensed and open for contribution.