SANKET (संकेत) is an open-source, low-latency, privacy-preserving AI accessibility system designed to recognize Indian Sign Language (ISL) from camera video input and translate recognized signs into accessible text and speech on commodity edge hardware.
- Active Phase: Phase 2 — Real-Time Sanket MVP
- Active Milestone: Milestone 2 — Real-Time Sanket MVP (Issues #51–#60)
-
Current Objective: Convert the validated Phase 1 offline recognition pipeline into a usable, low-latency real-time camera prototype operating at
$\ge 20$ FPS with end-to-end latency$< 100\text{ ms}$ . - Phase 0 Status: Completed and Verified (Foundational scaffolding, security policies, dataset licensing audits, initial 16-sign vocabulary, and signer-independent protocols).
-
Phase 1 Status: Completed and Certified (14 of 14 tasks merged):
- Dataset ingestion with SHA-256 integrity verification (
sanket.data.ingestion). - Dataset inspection and label balance profiling (
sanket.data.inspection). - Signer-independent train/validation/test splitting (
sanket.data.splits). - Uniform temporal subsampling and letterbox video preprocessing (
sanket.vision.preprocessing). - MediaPipe 3D hand landmark extraction (
sanket.vision.landmarks). - Upper-body pose reference anchor investigation (
sanket.vision.pose). - Wrist-centered and palm-scale coordinate normalization (
sanket.vision.normalization). - 645-dimensional geometric and temporal feature vector extraction (
sanket.features). - Random Forest baseline implementation (93.3% Top-1 Accuracy, 0.73 ms p95 latency).
- SVM baseline with RBF kernel and standard scaling (
sanket.models.svm). - MLP neural network baseline with Adam optimization (
sanket.models.mlp). - Unified evaluation pipeline and JSON manifest generation (
sanket.evaluation.evaluator). - Confusion matrix and phonological error diagnostics (
sanket.evaluation.diagnostics). - Cross-baseline comparative study and model selection (ADR-0003,
EXP-P1-001).
- Dataset ingestion with SHA-256 integrity verification (
SANKET follows a modular, feed-forward pipeline separating vision acquisition from feature extraction, inference, and accessible presentation:
flowchart LR
A["Camera Input\n(OpenCV VideoCapture)"] --> B["Preprocessing\n(Letterbox & Mirror)"]
B --> C["Landmark Extraction\n(MediaPipe Hands & Pose)"]
C --> D["Coordinate Normalization\n(Wrist Origin & Palm Scale)"]
D --> E["Feature Extraction\n(645-dim Geometry & Moments)"]
E --> F["ML Inference\n(Random Forest Classifier)"]
F --> G["Temporal Smoothing\n(EMA & Hysteresis Gating)"]
G --> H["Accessible UI\n(HUD & Text Subtitles)"]
- Project Overview: Executive summary, problem statement, ISL linguistic context, and phased roadmap.
- Project Context: Authoritative architectural boundaries, linguistic phonology, and research foundations.
- System Architecture: Layered design, package decomposition, class diagrams, and latency budgets.
- Technology Stack: Runtimes, core libraries, testing frameworks, CI configuration, and hardware targets.
- Developer Onboarding: Environment setup, daily workflows, test commands, and branch conventions.
- Coding Standards: Python 3.10+ typing, Google docstrings, vectorization, and exception design.
- Testing & Benchmarking Guide: Testing pyramid, synthetic data fixtures, coverage, and latency profiling.
- Contributing Guidelines: Contribution standards, issue linking, and human merge control.
- Security Policy: Vulnerability disclosure, least-privilege CI, and biometric privacy mandate.
- Documentation Format Guide: Formatting, typography, and attribution standards for repository documents.
- Contributor & Agent Guide (AGENTS.md): Operational constraints, modular rules, and reusable agent skills.
- Accessible Design System: High-contrast color palette, video HUD layout, and subtitles.
- API Documentation: Complete Python class/function reference and CLI guide.
- Data Persistence & Storage: File-based storage, split JSON schemas, and
.joblibserialization. - Module Breakdown: In-depth documentation of all internal
sanket.*packages. - Business Logic & Mathematics: Normalization formulas, joint angles, and smoothing math.
- Upstream Integrations: MediaPipe Hands/Pose, OpenCV, and Scikit-learn adapter patterns.
- Security & Privacy Governance: Zero-biometric-retention model and data protection.
- Performance & Latency: Sub-100ms latency budgets and empirical CPU benchmarks.
- Error Handling & Resilience: Exception hierarchy, missing landmark imputation, and OOV rejection.
- Deployment & Packaging: Local installation, Docker containerization, and edge distribution.
- Environment Specifications: Operating systems, hardware requirements, and environment variables.
- Release Engineering: Semantic versioning, milestone checklists, and tag protocols.
- Changelog: Chronological log of releases and phase deliverables.
- Troubleshooting Guide: Common failure modes and actionable remedies.
- Runtime Monitoring: Real-time telemetry, throughput tracking, and drift detection.
- Backup & Recovery: Model weight retention, split verification, and disaster recovery.
- Incident Response Plan: Playbooks for privacy breaches, credential leaks, and CI errors.
- Master Roadmap: Seven-phase engineering and research roadmap.
- Architecture Decision Records:
- Dataset Register: Authoritative dataset catalog and licensing records.
- Initial 16-Sign Vocabulary: Closed vocabulary definition and linguistic mapping.
- Evaluation Methodology: Formal evaluation metrics and latency protocols.
- Comparative Baseline Study: Empirical report across RF, SVM, and MLP.
- Project Development Log: Chronological development log.
- Project Recovery Audit: Repository audit and baseline verification report.
# 1. Clone repository
git clone https://github.com/GITLUVV/Sanket.git
cd Sanket
# 2. Set up virtual environment
python -m venv venv
# Windows: .\venv\Scripts\Activate.ps1 | Linux: source venv/bin/activate
# 3. Install in editable mode with development tooling
pip install --upgrade pip
pip install -e ".[dev]"
# 4. Verify environment setup and test suite
python scripts/verify_setup.py
pytest -q
ruff check .Important
SANKET strictly enforces a Zero-Biometric-Retention Mandate. Live camera streams reside exclusively in volatile memory for landmark localization and are evicted immediately without touching persistent storage. No raw facial or bodily recordings of identifiable individuals are ever committed to this repository.
This project is licensed under the Apache License 2.0. See the LICENSE file for details. Third-party datasets retain their respective original licenses as recorded in the Dataset Register.