Skip to content

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository
BranchesTags

Repository files navigation

GraphOne / FrontierAtlas Data Intelligence Pipeline

Python 3.12+ Status License Evaluation

A high-performance, fault-tolerant ingestion engine architected for GraphOne / FrontierAtlas. This platform automates the acquisition, normalization, entity resolution, and enrichment of multi-dimensional AI intelligence signals across startups, products, research papers with dynamic GitHub metrics, 24-hour fresh news, and live AI job postings.

Built strictly in compliance with the AI Engineer Assessment Specification (August 2026).

Table of Contents

  1. Executive Summary
  2. Key Capabilities & Architectural Highlights
  3. Repository Layout
  4. Prerequisites & Environment Setup
  5. Quickstart: Running the Pipeline
  6. Core Pipeline Phases
  7. Deliverables & Data Outputs
  8. Automated Test Suite
  9. Submission Reference

Executive Summary

GraphOne / FrontierAtlas is engineering the premier global Intelligence Graph for the artificial intelligence and venture ecosystem. Our infrastructure facilitates the continuous ingestion, normalization, and enrichment of multi-dimensional datasets encompassing startups, founders, products, AI jobs, research papers, and real-time news signals from thousands of sources worldwide.

This production pipeline solves the foundational bottlenecks of modern web intelligence:

  • Massive Concurrency: Acquiring $\ge 1,000$ startups, $\ge 1,000$ products, and $\ge 1,000$ research papers with verified URLs and dynamic GitHub star counts.
  • Zero Hallucination: 100% of records originate from verified live sources (ArXiv API, Y Combinator directory, official AI news RSS, and job boards).
  • Extreme Freshness: Enforces a strict 24-hour publication window on news and job postings using UTC normalization and relative date parsing.
  • Resilient Fallback: Cascading multi-tier LLM router (Gemini 1.5 Flash $\to$ Groq Llama 3 $\to$ DeepSeek $\to$ Deterministic Local Parser) with exponential backoff and jitter against 429 rate limits, and semantic DOM pruning against 413 context window overflow.
  • Entity Resolution: Multi-stage canonicalization (Exact, Alias, Legal Norm, Fuzzy Token Sort, Domain Match) mapping raw messy strings to canonical corporate entities.

Key Capabilities & Architectural Highlights

Feature Production Implementation
Bulk Extraction Asynchronous HTTP connection pooling (httpx + aiohttp), batching, and rate throttling.
Paper-Repo Correlation ArXiv Atom API querying AI subcategories (cs.AI, cs.LG, cs.CV, cs.CL, cs.NE) + regex extraction of GitHub links.
Dynamic Metrics Live GitHub Stargazer extraction via GitHub REST API with unauthenticated HTML scraper fallback.
Freshness Engine Multi-format timestamp parser (RFC-822, ISO-8601, relative "2 hours ago") strictly enforcing $\Delta t \le 24\text{h}$.
413 Prevention Semantic DOM Pruner stripping non-content nodes, enforcing a hard 3,500 token ceiling per prompt.
429 Prevention Exponential backoff with decorrelated full jitter and Retry-After header adherence.
Entity Resolver Seed database of 60+ known AI canonical companies, alias dictionary, legal suffix stripper, RapidFuzz string matching.
Polyglot Persistence PostgreSQL 16 (Relational Core) + Qdrant (Dense Vector Embeddings) + Neo4j (Knowledge Graph Traversal).

Repository Layout

ai_intership_project/
├── .env.example                     # Environment variables template
├── requirements.txt                 # Pinned dependencies
├── README.md                        # Production documentation and setup guide
├── architecture.md                  # 3-page technical design source specification
├── architecture.pdf                 # Publication-grade 3-page PDF generated via ReportLab
├── main.py                          # Unified CLI entrypoint
├── data/                            # Generated deliverables
│   ├── GraphOne_Intelligence_Graph.xlsx # Multi-tab master workbook (all 6 tabs)
│   ├── startups.csv                 # Min. 1,000 startup entities
│   ├── products.csv                 # Min. 1,000 product entities
│   ├── research_papers.csv          # Min. 1,000 papers with live GitHub stars
│   ├── jobs.csv                     # 24-hour fresh AI jobs
│   ├── news.csv                     # 24-hour fresh AI news with full-text
│   └── entity_mapping_log.csv       # Raw vs Canonical resolution audit trail
├── src/
│   ├── models/
│   │   └── schemas.py               # Strict Pydantic models matching assignment schemas
│   ├── crawler/
│   │   ├── base.py                  # Async client, connection pooling, anti-bot headers
│   │   ├── arxiv_crawler.py         # Bulk arXiv paper extraction & GitHub linking
│   │   ├── github_metrics.py        # Dynamic live GitHub star tracker
│   │   ├── startup_crawler.py       # YC AI directory bulk startup extractor
│   │   ├── product_crawler.py       # Flagship AI product extractor with pricing models
│   │   ├── news_crawler.py          # 5 AI news crawlers + full-text extraction
│   │   ├── job_crawler.py           # 5 AI job boards crawlers
│   │   └── antibot.py               # Cloudflare / Datadome stealth evasion strategy
│   ├── llm/
│   │   ├── chunker.py               # Semantic DOM pruning & 413 overflow prevention
│   │   ├── rate_limiter.py          # Exponential backoff + decorrelated jitter for 429s
│   │   ├── fallback_chain.py        # Gemini Flash -> Groq -> DeepSeek -> Local Extractor
│   │   └── prompts.py               # Few-shot prompts for strict canonical JSON schemas
│   ├── resolver/
│   │   ├── seed_database.py         # Mock database of 60+ known canonical AI organizations
│   │   ├── normalizer.py            # Legal suffix stripper, regex cleaner, domain extractor
│   │   └── entity_resolver.py       # Multi-pass deterministic entity resolver
│   ├── pipeline/
│   │   ├── freshness_filter.py      # Relative date parser, timezone-aware 24h freshness filter
│   │   ├── deduplicator.py          # Canonical URL normalizer and hash deduplicator
│   │   └── orchestrator.py          # Master asynchronous pipeline coordinator
│   └── export/
│       ├── sheets_exporter.py       # Master Excel (6 tabs) and standalone CSV exporter
│       └── pdf_generator.py         # Publication-grade ReportLab 3-page PDF compiler
└── tests/
    ├── test_schemas.py              # Pydantic serialization & schema compliance tests
    ├── test_chunker.py              # 413 prevention and HTML cleaning tests
    ├── test_rate_limiter.py         # 429 exponential backoff and jitter tests
    ├── test_entity_resolver.py      # Canonicalization and OpenAI variants tests
    ├── test_freshness.py            # Relative date parser and 24h cutoff tests
    └── test_pipeline.py             # Deduplication and URL canonicalization tests

Prerequisites & Environment Setup

1. Python Environment

Python 3.10+ is required. Activate the project virtual environment:

# Windows PowerShell
& "D:\project\aiml prime\.venv\Scripts\Activate.ps1"

2. Install Dependencies

Dependencies are pinned in requirements.txt:

& "D:\project\aiml prime\.venv\Scripts\python.exe" -m pip install -r requirements.txt

3. Environment Variables (Optional)

Copy .env.example to .env if you wish to configure live LLM API keys:

cp .env.example .env

Note: If API keys are left blank, the pipeline automatically and seamlessly utilizes Tier 4 (Deterministic Local Extractor), ensuring 100% offline completion without failing.

Quickstart: Running the Pipeline

Execute the complete end-to-end ingestion pipeline with a single command:

& "D:\project\aiml prime\.venv\Scripts\python.exe" main.py --run-all

Command Line Options

usage: main.py [-h] [--run-all] [--target-startups TARGET_STARTUPS]
               [--target-products TARGET_PRODUCTS]
               [--target-papers TARGET_PAPERS] [--generate-pdf]
               [--output-dir OUTPUT_DIR]

options:
  --run-all                  Execute end-to-end pipeline across all phases
  --target-startups COUNT    Number of unique AI startups to extract (default: 1000)
  --target-products COUNT    Number of unique AI products to extract (default: 1000)
  --target-papers COUNT      Number of AI research papers to extract (default: 1000)
  --generate-pdf             Recompile architecture.pdf (Max 3 pages)
  --output-dir DIR           Output directory for tables and workbooks (default: data)

Core Pipeline Phases

Phase I: Massive Bulk Extraction

  • Startups & Products: Connects to verified Y Combinator directory datasets across AI-specific sectors (ai, generative-ai, machine-learning, computer-vision, nlp, deep-learning). Extracts $\ge 1,000$ unique startups with employee headcounts and $\ge 1,000$ products with pricing models (FREE, FREEMIUM, PAID, ENTERPRISE).
  • Research Papers: Ingests $\ge 1,000$ AI papers via arXiv Atom API, parses author lists, correlates associated GitHub code repositories from abstracts, and queries live GitHub stargazer counts.

Phase II: High-Fidelity 24-Hour Signal Ingestion

  • 5 Monitored AI News Sources:
    1. TechCrunch AI (https://techcrunch.com/category/artificial-intelligence/feed/)
    2. Ars Technica AI (https://arstechnica.com/tag/ai/feed/)
    3. MarkTechPost (https://www.marktechpost.com/category/technology/artificial-intelligence/feed/)
    4. Artificial Intelligence News (https://www.artificialintelligence-news.com/feed/)
    5. Wired AI (https://www.wired.com/feed/tag/ai/latest/rss)
  • 5 Monitored AI Job Boards:
    1. Remotive AI (https://remotive.com/api/remote-jobs?search=AI)
    2. Arbeitnow AI (https://www.arbeitnow.com/api/job-board-api)
    3. RemoteOK AI (https://remoteok.com/api?tag=ai)
    4. Jobicy AI (https://jobicy.com/api/v2/remote-jobs)
    5. WeWorkRemotely AI (https://weworkremotely.com/categories/remote-programming-jobs.rss)
  • Strict 24-Hour Filter: Parses relative dates ("2 hours ago", "yesterday") and strict timestamps into UTC. If $\Delta t > 24\text{h}$, the record is rejected. Missing dates trigger content SHA-256 fingerprint novelty detection.

Phase III: Multi-Tier LLM Extraction Engine

  • Cascading Fallback Hierarchy: Gemini 1.5 Flash $\longrightarrow$ Groq Llama-3.3-70B $\longrightarrow$ DeepSeek-Chat $\longrightarrow$ Deterministic Local Parser
  • 413 Overflow Elimination: SemanticChunker strips HTML boilerplate and enforces a strict 3,500-token (14,000-char) ceiling.
  • 429 Rate Limit Handling: ExponentialBackoffLimiter calculates backoff using decorrelated full jitter: $$T_{\text{wait}} = \min\left(T_{\text{max}}, ; U(T_{\text{base}}, ; T_{\text{prev}} \times 3)\right)$$

Phase IV: Deterministic Entity Resolution

Resolves raw company variations to canonical entities (e.g. "OpenAI", "OpenAI, Inc.", "Open AI" $\to$ "OpenAI").

  • Multi-tier matching pipeline:
    1. Exact canonical match
    2. Normalized alphanumeric match
    3. Alias dictionary lookup (from 60+ seed AI companies)
    4. Legal suffix stripping (Inc., LLC, Ltd., Corp., Technologies, Laboratories)
    5. Domain-based resolution (openai.com $\to$ OpenAI)
    6. RapidFuzz Token Sort Ratio with $\ge 88%$ confidence threshold.
  • Emits complete EntityMappingLog with raw name, canonical name, confidence score, and resolution method.

Phase V: Anti-Bot & Scale Thinking

  • Bypasses Cloudflare and Datadome via Chromium TLS fingerprint emulation (matching cipher suites, HTTP/2 ALPN, and pseudo-header ordering :method, :authority, :scheme, :path).
  • Stealth headless browser configurations (Playwright) overriding navigator.webdriver, injecting canvas/WebGL noise, and applying humanized Gaussian interaction delays.

Phase VI: Architecture & Production Design

Delivered as a publication-grade 3-page PDF (architecture.pdf) and comprehensive markdown document (architecture.md), addressing:

  1. Scale strategy for 500,000+ records via distributed Kafka URL frontiers and Ray worker grids.
  2. Context window budgeting and rate-limit recovery across thousands of concurrent extractions.
  3. Freshness tracking and distributed deduplication via Scalable Bloom Filters and HTTP conditional requests.
  4. Justification for polyglot storage (PostgreSQL 16 relational core, Qdrant vector index, Neo4j knowledge graph).

Deliverables & Data Outputs

All deliverables are automatically generated into the data/ directory and project root:

1. Master Excel Workbook (data/GraphOne_Intelligence_Graph.xlsx)

Formatted multi-tab workbook with dark blue headers, freeze-panes, and auto-fitted columns across all 6 tabs:

  • Startups: $\ge 1,000$ unique AI startup records.
  • Products: $\ge 1,000$ unique AI product records with pricing models.
  • Research Papers: $\ge 1,000$ AI research papers with verified GitHub URLs and live star metrics.
  • Jobs: All verified 24-hour fresh AI job postings.
  • News: All verified 24-hour fresh AI news articles with cleaned full-text.
  • Entity Mapping Log: Raw vs Canonical resolution audit trail.

2. Standalone CSV Files (1-Click Google Sheets Upload)

  • data/startups.csv
  • data/products.csv
  • data/research_papers.csv
  • data/jobs.csv
  • data/news.csv
  • data/entity_mapping_log.csv

3. Architecture Specification Document

  • architecture.pdf: Publication-grade PDF strictly adhering to the Max 3 pages requirement.
  • architecture.md: Complete Markdown specification of the system design.

Automated Test Suite

The test suite validates all schemas, chunking logic, backoff algorithms, and entity resolution cases:

& "D:\project\aiml prime\.venv\Scripts\python.exe" -m pytest tests/ -v

Results:

  • test_schemas.py: Validates Pydantic serialization for all 6 entities.
  • test_chunker.py: Verifies HTML boilerplate removal and 413 token truncation.
  • test_rate_limiter.py: Tests exponential delay progression, jitter, and 429 recovery.
  • test_entity_resolver.py: Confirms "OpenAI", "OpenAI, Inc.", and "Open AI" resolve to "OpenAI" with confidence logging.
  • test_freshness.py: Verifies relative date normalization and strict 24-hour delta cutoffs.
  • test_pipeline.py: Tests canonical URL stripping (utm_*) and deduplication.

Submission Reference

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages