Skip to content

README.md

Latest commit

 

History

History
336 lines (218 loc) · 8.24 KB

README.md

File metadata and controls

336 lines (218 loc) · 8.24 KB
Raw

OpenMed Jupyter Notebooks

This directory contains comprehensive Jupyter notebooks demonstrating OpenMed's capabilities.

Script Quick Starts

📚 Available Notebooks

🆕 PII_Detection_Complete_Guide.ipynb

Complete guide to PII detection and de-identification (v0.5.0+)

A comprehensive tutorial covering everything about PII functionality:

  • Basic PII Extraction - Detect PII entities in clinical text
  • Smart Entity Merging - Fix fragmentation issues (NEW in v0.5.0)
  • De-identification Methods - Mask, remove, replace, hash, shift dates
  • Re-identification - Reverse de-identification with mappings
  • Batch Processing - Process multiple texts efficiently
  • Confidence Thresholding - Control precision vs recall
  • Custom Patterns - Add domain-specific PII patterns
  • Clinical Use Cases - Real-world examples (discharge summaries, research datasets, HIPAA compliance)
  • Visualization - Display results with highlighting

Topics covered:

  • 48 executable cells with detailed explanations
  • Smart merging comparison (before/after)
  • All 5 de-identification methods demonstrated
  • HIPAA compliance checklist
  • Production-ready examples
  • Best practices and security considerations

Recommended for:

  • Healthcare data scientists
  • Clinical researchers
  • HIPAA compliance officers
  • Anyone working with protected health information (PHI)

Deidentification_Cookbook.ipynb

Task-oriented de-identification recipes (copy-paste)

Short, self-contained recipes for the most common real workflows:

  • De-identify a list or CSV of clinical strings
  • Batch-redact a directory of text files with BatchProcessor
  • Reversible replace + re-identify round-trip
  • Per-language model selection with DEFAULT_PII_MODELS

Uses synthetic PHI only and a lightweight 44M English model for CPU-friendly runs.

Recommended for: Engineers wiring de-identification into a pipeline who want a quick, runnable starting point.

🌍 Multilingual_PII_Detection_Guide.ipynb

Multilingual PII detection across 9 supported languages

Walk through language-aware extraction and de-identification for:

  • English, French, German, Italian, Spanish, and Portuguese multilingual support
  • Dutch, Hindi, and Telugu flagship PII models
  • Locale-specific dates, phones, addresses, postcodes, and validators
  • Cross-language comparisons and custom model selection

Recommended for:

  • Teams processing multilingual clinical text
  • Reviewers validating new PII releases
  • Anyone comparing locale-specific pattern behavior

getting_started.ipynb

Introduction to OpenMed basics

Learn the fundamentals of OpenMed:

  • Installation and setup
  • Basic text analysis with analyze_text()
  • Model discovery and selection
  • Output formatting (JSON, HTML, CSV)
  • Entity grouping and filtering

Recommended for: First-time users, quick start

Sentence_Detection_Batching.ipynb

Advanced sentence detection and batch processing

Master efficient text processing:

  • Sentence boundary detection with pySBD
  • Batch processing multiple documents
  • Performance optimization
  • Memory-efficient processing

Recommended for: Production deployments, large-scale processing

Medical_Tokenizer_Demo.ipynb

Medical-aware tokenization

Explore medical text tokenization:

  • Medical term boundary preservation
  • Tokenization comparison
  • Custom tokenizer configuration

Recommended for: NLP engineers, model developers

Medical_Tokenizer_Benchmark.ipynb

Performance benchmarking

Compare tokenization performance:

  • Speed benchmarks
  • Accuracy metrics
  • Resource usage

Recommended for: Performance optimization

ZeroShot_NER_Tour.ipynb

Zero-shot NER capabilities

Explore zero-shot learning:

  • No training required
  • Custom entity types
  • Domain adaptation

Recommended for: Research, custom entity detection

🚀 Getting Started

Prerequisites

# Install OpenMed
uv pip install "openmed[hf]"

# Install Jupyter
uv pip install jupyter

# For PII notebooks, you may need:
uv pip install matplotlib ipython

Running the Notebooks

  1. Clone the repository:

    git clone https://github.com/maziyarpanahi/openmed.git
cd openmed/examples/notebooks

  2. Start Jupyter:

    jupyter notebook

  3. Open a notebook and run cells sequentially

Environment Setup

Some notebooks may require a HuggingFace token for accessing gated models:

import os
os.environ['HF_TOKEN'] = 'your_token_here'

Get your token from: https://huggingface.co/settings/tokens

📋 Notebook Quick Reference

Notebook Difficulty Topics Duration
PII_Detection_Complete_Guide Intermediate PII, De-identification, HIPAA 30-45 min
getting_started Beginner Basics, Installation 10-15 min
Sentence_Detection_Batching Intermediate Batch processing 15-20 min
Medical_Tokenizer_Demo Intermediate Tokenization 10-15 min
Medical_Tokenizer_Benchmark Advanced Performance 15-20 min
ZeroShot_NER_Tour Advanced Zero-shot NER 20-30 min

🎯 Learning Paths

Path 1: Getting Started with OpenMed

  1. getting_started.ipynb - Learn basics
  2. Sentence_Detection_Batching.ipynb - Batch processing

Path 2: PII Detection & De-identification

  1. getting_started.ipynb - Basics first
  2. PII_Detection_Complete_Guide.ipynb - Complete PII guide
  3. Practice with your own clinical notes

Path 3: Advanced NER

  1. getting_started.ipynb - Foundation
  2. ZeroShot_NER_Tour.ipynb - Zero-shot capabilities
  3. Medical_Tokenizer_Demo.ipynb - Tokenization deep dive

Path 4: Production Deployment

  1. Sentence_Detection_Batching.ipynb - Batch processing
  2. Medical_Tokenizer_Benchmark.ipynb - Performance tuning
  3. PII_Detection_Complete_Guide.ipynb - End-to-end de-identification workflows

💡 Tips for Using Notebooks

Best Practices

  1. Run cells sequentially - Dependencies may exist between cells
  2. Check imports - Ensure all required packages are installed
  3. Monitor memory - Some models are large (500MB-1GB+)
  4. Use GPU if available - Speeds up inference significantly
  5. Save outputs - Notebooks auto-save, but export important results

Common Issues

Issue: ImportError

# Solution: Install missing package
!pip install package-name

Issue: Model download slow

# Solution: Use HuggingFace cache
export HF_HOME="/path/to/cache"

Issue: Out of memory

# Solution: Use smaller batch size or CPU
batch_size=1  # or device="cpu"

Customization

All notebooks are designed to be modified:

  • Change model names to try different models
  • Adjust confidence thresholds
  • Use your own clinical text
  • Modify visualization styles

🔗 Additional Resources

📝 Contributing

Have a notebook idea or improvement?

  1. Fork the repository
  2. Create a new notebook in this directory
  3. Follow the existing style and structure
  4. Submit a pull request

Notebook Template Structure

# Title

## Overview
- What this notebook covers
- Prerequisites
- Expected outcomes

## Setup
- Imports
- Configuration

## Main Content
- Section 1: Topic A
- Section 2: Topic B
- ...

## Summary
- Key takeaways
- Next steps

📄 License

All notebooks are released under the same license as OpenMed (Apache 2.0).

Last Updated: 2026-01-13

OpenMed Version: v0.5.0+