[kenrogers]

Summary

Implements a scheduled GitHub Action that monitors merged PRs across all Stacks Network and Hiro Systems repositories and automatically creates documentation update issues when changes require doc updates.

How It Works

1. Scheduled Monitoring: Runs every 6 hours via cron
2. PR Discovery: Polls GitHub API for recently merged PRs across monitored repositories
3. Intelligent Analysis: Uses Claude Code Action to analyze PR changes and determine documentation impact
4. Automated Issue Creation: Creates specific, actionable GitHub issues with recommended documentation updates

Features

✅ Automated monitoring every 6 hours
✅ Manual trigger support for testing (configurable lookback period)
✅ Smart PR analysis to determine documentation impact
✅ Specific, actionable issues with recommended changes
✅ Priority-based issue creation
✅ Skips PRs with no user-facing impact

Files Added

• repos-to-monitor.txt - List of repositories to monitor
• .github/workflows/smart-docs-monitor.yml - Main GitHub Action workflow
• CLAUDE.md - Comprehensive documentation sync guidelines for Claude

Testing

Test manually before the first scheduled run:

# Test with default 6-hour lookback
gh workflow run smart-docs-monitor.yml

# Test with 24-hour lookback for more results
gh workflow run smart-docs-monitor.yml -f hours_back=24

Example Output

The workflow will create issues like:

Title: [Docs Update] stacks-core PR #456: Add new Clarity built-in function

With detailed information about:

• What changed in the PR
• Specific documentation files that need updates
• Recommended text changes with examples
• Priority level and reasoning

Benefits

🎯 Proactive: Catches documentation needs automatically
⚡ Efficient: No manual PR monitoring required
📋 Actionable: Creates specific issues with recommended changes
🔍 Comprehensive: Monitors all major Stacks repositories
🎛️ Flexible: Schedule and lookback period are configurable

---

🤖 Generated with Claude Code

[claude]

Code Review: Automated Documentation Sync Monitor

Overall Assessment

This is a well-designed automation system with good code quality and a sound approach. However, there are several important issues to address before deployment.

---

Critical Issues

1. API Key Configuration Mismatch (.github/workflows/smart-docs-monitor.yml:89)

• Workflow references CLAUDE_CODE_OAUTH_TOKEN
• PR description says to set up ANTHROPIC_API_KEY
• These are different secrets
• Fix: Clarify which secret is needed (should be CLAUDE_CODE_OAUTH_TOKEN for claude-code-action)

2. Incomplete Repository List (repos-to-monitor.txt)

• File lists only 8 repositories
• CLAUDE.md claims to monitor 29 repositories (21 Stacks Network + 8 Hiro Systems)
• Current: 3 stacks-network + 1 stacks-sbtc + 3 hirosystems
• Fix: Add remaining ~21 repositories or update CLAUDE.md to reflect actual scope

---

Security Concerns

3. Overly Broad Permissions (.github/workflows/smart-docs-monitor.yml:13-15)

• Has contents: write but only needs issues: write
• Workflow doesn't commit or push changes
• Fix: Remove contents: write permission

4. No API Rate Limit Handling

• Could make hundreds of calls (30 PRs × 29 repos)
• No rate limit checks or backoff logic
• Recommendation: Add rate limit handling

---

Design & Best Practices

5. Lenient Error Handling (.github/workflows/smart-docs-monitor.yml:76-78)

• Errors logged but workflow continues silently
• Won't detect systematic failures
• Recommendation: Track and report failed repos

6. Hardcoded Model Version (.github/workflows/smart-docs-monitor.yml:159)

• Uses claude-sonnet-4-20250514 (future date model)
• May not exist or become outdated
• Recommendation: Use claude-sonnet-4 or make configurable

7. Truncated PR Descriptions (.github/workflows/smart-docs-monitor.yml:70)

• Limited to 200 characters
• May lose important context
• Recommendation: Increase to 500-1000 characters

8. No Duplicate Prevention

• Doesn't check for existing issues
• Rerunning creates duplicates
• Recommendation: Search before creating issues

---

Performance & Testing

9. Sequential Polling

• Polls 29 repos sequentially
• Could be slow
• Recommendation: Use GitHub search API for batch queries

10. No Testing Strategy

• No dry-run mode
• Must call API and create real issues to test
• Recommendation: Add dry_run input, unit tests, testing docs

---

Documentation

11. Inconsistent Docs (CLAUDE.md)

• Says "And 12 additional repositories" without listing them
• Fix: List all repos explicitly

12. Unpinned Action Version (.github/workflows/smart-docs-monitor.yml:88)

• Uses @v1 tag instead of commit hash
• Recommendation: Pin for security

---

Positive Highlights

• Excellent CLAUDE.md documentation with clear examples
• Good separation of concerns
• Manual trigger support for testing
• Conservative approach to issue quality
• Well-structured prompts

---

Recommended Changes Before Merge

Must fix:

• Resolve API key naming inconsistency
• Complete repository list or update docs
• Reduce permissions to issues: write only

Should fix:

• Add duplicate issue detection
• Add dry-run mode
• Add rate limit handling
• Fix model version identifier

Nice to have:

• Improve error reporting
• Increase PR description limit
• Optimize repo polling

---

Great work on this automation! Address the critical issues and this will be a valuable addition.

[claude]

Code Review - Smart Docs Monitor

Great work on implementing an automated documentation sync system! This is a valuable addition that will help keep docs in sync with code changes.

Strengths

1. Well-structured implementation - Clean separation between PR discovery, analysis, and issue creation
2. Comprehensive CLAUDE.md - Excellent detailed guidelines with clear examples of good vs bad issues
3. Flexible configuration - Configurable lookback period and manual trigger support
4. Error handling - Try-catch blocks prevent one repo failure from breaking the entire workflow

Issues and Recommendations

CRITICAL: Output Assignment Issue

Location: .github/workflows/smart-docs-monitor.yml:84

The return prCount statement won't properly set the step output. GitHub Actions requires explicit output setting.

Fix: Add core.setOutput('pr_count', prCount) before the return statement, then update line 87 condition to: if: steps.batch.outputs.pr_count > 0

Security: Token Permissions

Location: .github/workflows/smart-docs-monitor.yml:13-14

Currently only issues: write is specified. Add contents: read permission for repository contents.

Potential: Rate Limiting

Location: .github/workflows/smart-docs-monitor.yml:43-80

With 12 repos and 30 PRs per repo, you could hit rate limits. Consider adding delay between API calls or using GraphQL for more efficient batch queries. The try-catch is adequate for now.

Code Quality: Model Version

Location: .github/workflows/smart-docs-monitor.yml:160

The model claude-sonnet-4-20250514 is dated May 2025. Verify this is the correct model identifier or consider using a more stable reference.

Enhancement: PR Description Truncation

Location: .github/workflows/smart-docs-monitor.yml:70

The substring(0, 200) truncation might cut off important context mid-sentence. Consider truncating at word boundaries.

Testing Recommendations

1. Test with zero PRs - Verify graceful exit when no merged PRs found
2. Test with API failures - Simulate GitHub API errors
3. Test manual trigger - Run with hours_back=24
4. Monitor first runs - Watch for rate limiting or timeout issues

Documentation Quality

The CLAUDE.md file is excellent with clear examples, specific guidelines, proper priority levels, and detailed issue format templates.

Overall Assessment

This is a solid implementation that addresses a real need. The main issue is the output assignment bug which will prevent the conditional from working properly.

Priority:

• HIGH: Fix the output assignment bug
• MEDIUM: Add explicit contents: read permission
• LOW: Improve description truncation and verify model name

[claude]

Code Review: Automated Documentation Sync Monitor

Overview

This PR introduces an automated system to monitor merged PRs across Stacks Network and Hiro Systems repositories and create documentation update issues. Overall, this is a well-designed solution with clear documentation and good structure.

Strengths

1. Comprehensive Documentation: The CLAUDE.md file is exceptionally thorough
2. Flexible Design: Supports both scheduled runs and manual triggers
3. Smart Filtering: Built-in logic to skip PRs that do not require documentation updates
4. Clear Issue Templates: Provides specific, actionable issue formats

Code Quality Recommendations

Error Handling (.github/workflows/smart-docs-monitor.yml:32-35,77-79): Add try-catch for file reads and better error tracking
Rate Limiting: Add GitHub API rate limit checks before processing repositories
Timeout Protection: Add timeout-minutes: 30 to Claude analysis step
Input Validation: Validate hours_back parameter (should be 1-168)
Secret Validation: Check if CLAUDE_CODE_OAUTH_TOKEN exists before running

Potential Issues

1. No error handling if repos-to-monitor.txt does not exist or is malformed
2. If no PRs found, workflow provides no feedback
3. PR body content included without sanitization (potential injection risk)

Performance Considerations

• API calls are sequential - consider parallelizing where possible
• Large batches could create very large pr-batch.md files - consider pagination
• max-turns 40 may be excessive for typical batches

Test Coverage

Missing automated tests for workflow logic. Recommend:

• Test workflow to validate repos-to-monitor.txt format and YAML validity
• Dry-run mode that analyzes but does not create issues

Documentation Quality

Excellent overall. Minor suggestions:

• Add troubleshooting section to CLAUDE.md
• Document token permissions requirements
• Include example workflow run outputs

Priority Recommendations

High Priority:

1. Add input validation for hours_back
2. Add error handling for file reads
3. Validate CLAUDE_CODE_OAUTH_TOKEN exists
4. Add timeout to Claude analysis step

Medium Priority:
5. Add rate limit monitoring
6. Add dry-run testing mode
7. Document token setup requirements

Low Priority:
8. Consider pagination for large PR batches
9. Add workflow test suite

Overall Assessment

This is a well-designed and valuable addition to the documentation workflow. The code is clean, the documentation is thorough, and the approach is sound. With the suggested error handling and validation improvements, this will be a robust automation system.

Recommendation: Approve with minor improvements

[claude]

PR Review: Automated Documentation Sync Monitor

Overall Assessment

✅ Excellent implementation - This is a well-designed automation system that will significantly improve documentation maintenance across the Stacks ecosystem.

---

🎯 Strengths

1. Architecture & Design

• Clean separation of concerns: PR discovery → analysis → issue creation
• Smart use of Claude Code Action for intelligent PR analysis
• Configurable schedule (6-hour cron) with manual trigger support
• Proper permissions model (read contents, write issues)

2. Comprehensive Documentation (CLAUDE.md)

• Exceptionally detailed guidelines (322 lines)
• Clear examples of good vs. bad issue creation
• Specific criteria for what requires documentation
• Priority-based triage system

3. Smart PR Batching

• Efficient batch processing reduces API calls
• Collects PRs from multiple repos in one pass
• Passes structured data to Claude for analysis

4. Action Workflow (.github/workflows/smart-docs-monitor.yml)

• Well-structured with clear steps
• Good error handling in PR fetching script
• Reasonable default values (6-hour lookback)

---

🔍 Issues & Recommendations

CRITICAL: Incomplete Repository List

repos-to-monitor.txt lists only 7 repositories, but CLAUDE.md documents 29 repositories:

• Stacks Network: Claims 21 repositories but only lists 3
• Hiro Systems: Claims 8 repositories but only lists 3

Missing repos include:

• stacks-blockchain-docker, clarunit, bitcoin, c32check, stacks, send-many-stx-cli
• stacks.js-starters, vitest-environment-clarinet
• Plus the 12 additional supporting repositories and 3 additional supporting repositories mentioned

Impact: High - The system will not monitor most intended repositories.

Fix: Add all intended repositories to repos-to-monitor.txt or update CLAUDE.md to match reality.

[claude]

HIGH: Rate Limiting & API Quota

Issue: Polling 29 repos every 6 hours (4x/day) = 116 API calls/day minimum, plus PR diff fetches, documentation searches, and issue creation API calls

Recommendation:

1. Add rate limit handling in the GitHub Script step
2. Consider increasing the interval to 12 hours for initial deployment
3. Add retry logic with exponential backoff
4. Monitor GitHub API quota usage

---

MEDIUM: Error Handling & Observability

Current state: Silent failures - if a repo errors, the workflow continues without notification

Issues:

• No aggregated error reporting
• Difficult to debug issues post-run

Recommendations:

1. Collect errors and report them in workflow summary
2. Set job output for error count
3. Consider failing the workflow if critical repos fail (e.g., stacks-core, clarinet)
4. Add structured logging for troubleshooting

---

MEDIUM: Security & Token Scope

Observation: Uses CLAUDE_CODE_OAUTH_TOKEN secret

Recommendations:

1. Document required token scopes in PR description or README
2. Verify token has minimal required permissions
3. Consider using GitHub App installation token for better security and rate limits
4. Add note about token rotation/expiration

---

LOW: Duplicate Issue Prevention

Current: Relies on Claude to check similar issues do not already exist

Issue: Claude has no programmatic way to enforce this consistently

Recommendation: Add a pre-check step before Claude analysis to fetch recently created auto-generated issues and pass this to Claude or filter PRs already covered.

[claude]

LOW: Testing & Validation

Missing:

• No example test run output
• No integration tests for the workflow
• No dry-run mode

Recommendations:

1. Add a dry_run input that analyzes but does not create issues
2. Test with hours_back=168 (1 week) to validate on real data
3. Document first manual test run results in PR

---

LOW: Code Quality - Workflow Script

Suggestions:

1. Extract magic numbers to constants (DEFAULT_HOURS_BACK = 6, MAX_PRS_PER_REPO = 30, etc.)
2. DRY principle: PR count logging duplicated
3. Add JSDoc comments for type safety

---

📋 Additional Recommendations

1. Add Workflow Status Badge in README.md or CLAUDE.md
2. Add Metrics Collection: Track PRs analyzed, issues created, false positive rate, average time to documentation update
3. Consider PR Size Filtering: Very large PRs (100+ files) may be refactors - consider skipping or flagging these separately
4. Add Issue Template: Create .github/ISSUE_TEMPLATE/auto-generated-docs.md to standardize format

---

🧪 Testing Checklist

Before merging:

• Complete repos-to-monitor.txt with all 29 intended repos
• Verify CLAUDE_CODE_OAUTH_TOKEN secret is set
• Test manual trigger with hours_back=24
• Review first batch of created issues for quality
• Verify labels are applied correctly
• Check rate limit consumption during test run
• Validate error handling with an invalid repo in the list

---

🚀 Deployment Recommendation

Staged Rollout:

1. Week 1: Monitor only 3-5 critical repos (stacks-core, clarinet, stacks.js)
2. Week 2: Review issue quality, tune Claude prompt if needed
3. Week 3: Add remaining Stacks Network repos
4. Week 4: Add remaining Hiro Systems repos
5. Ongoing: Monitor and iterate based on feedback

---

✅ Conclusion

This is a high-quality PR that demonstrates excellent engineering and documentation practices. The primary blocker is completing the repository list. Once addressed, this will be a valuable addition to the Stacks documentation workflow.

Recommendation: Approve with changes requested

The framework is solid - just needs the repository list completed and improved error handling before going live.