fix(scripts): detect and auto-fix quiz answer-length bias using Claude API#307
fix(scripts): detect and auto-fix quiz answer-length bias using Claude API#307javy0107 wants to merge 6 commits into
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughA new Python CLI scans Quiz Bias Detection Script
Quiz Bias Fixer
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
scripts/check_quiz_bias.py (1)
15-61: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low valueDefensively validate that options are strings.
The function assumes all items in the
optionslist are strings (callslen()on line 42–43). If any option is not a string, aTypeErrorwill be raised and not caught by the exception handler on line 20, crashing the script. While the quiz.json schema should enforce strings, a defensive check would improve robustness.💡 Optional defensive improvement
for i, q in enumerate(questions): if not isinstance(q, dict): continue options = q.get("options", []) + if not all(isinstance(opt, str) for opt in options): + continue correct = q.get("correct", -1)However, if all quiz.json files follow the schema (as expected), this check is not necessary.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/check_quiz_bias.py` around lines 15 - 61, The check_quiz function assumes all items in the options list are strings when calling len() on them at lines 42-43, but does not validate this assumption. If any option is not a string, a TypeError will be raised and crash the script since it is not caught by the existing exception handler. Before calculating correct_len and the distractors list, add a defensive check to ensure that each item in the options list is a string, and skip the question if any option fails this validation. This should be done after retrieving the options from the question dictionary but before attempting to access their lengths.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/check_quiz_bias.py`:
- Around line 1-13: Remove the unused `os` import statement from the imports
section. The `os` module is imported on line 8 but is never referenced anywhere
in the file, so it should be deleted to keep the imports clean and avoid
unnecessary dependencies.
---
Nitpick comments:
In `@scripts/check_quiz_bias.py`:
- Around line 15-61: The check_quiz function assumes all items in the options
list are strings when calling len() on them at lines 42-43, but does not
validate this assumption. If any option is not a string, a TypeError will be
raised and crash the script since it is not caught by the existing exception
handler. Before calculating correct_len and the distractors list, add a
defensive check to ensure that each item in the options list is a string, and
skip the question if any option fails this validation. This should be done after
retrieving the options from the question dictionary but before attempting to
access their lengths.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 4f5a63f0-8ac0-48fc-a01f-80f6c928f0a8
📒 Files selected for processing (1)
scripts/check_quiz_bias.py
| #!/usr/bin/env python3 | ||
| """ | ||
| Check quiz.json files for answer-length bias. | ||
| Flags questions where the correct answer is significantly longer than distractors. | ||
| """ | ||
|
|
||
| import json | ||
| import os | ||
| import sys | ||
| from pathlib import Path | ||
|
|
||
| THRESHOLD = 1.5 # correct answer is 1.5x longer than average distractor length | ||
|
|
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Check if Python version requirement is declared
fd -e 'toml|cfg|txt|ini|yaml|yml' . | head -20 | xargs grep -l 'python' 2>/dev/null | head -3Repository: rohitg00/ai-engineering-from-scratch
Length of output: 174
🏁 Script executed:
#!/bin/bash
# Read the check_quiz_bias.py file to verify the claims
echo "=== check_quiz_bias.py content ==="
cat scripts/check_quiz_bias.py
echo -e "\n=== Searching for Python version requirements ==="
find . -maxdepth 2 \( -name "setup.py" -o -name "pyproject.toml" -o -name "setup.cfg" -o -name ".python-version" \) -exec cat {} +
echo -e "\n=== Checking if 'os' module is used in check_quiz_bias.py ==="
grep -n '\bos\.' scripts/check_quiz_bias.py || echo "No 'os.' usage found"Repository: rohitg00/ai-engineering-from-scratch
Length of output: 3042
Remove unused os import on line 8.
The os module is imported but never used in the file. Additionally, line 15 uses the list[dict] type hint syntax (PEP 585), which requires Python 3.9+.
Suggested fix
import json
-import os
import sys🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/check_quiz_bias.py` around lines 1 - 13, Remove the unused `os`
import statement from the imports section. The `os` module is imported on line 8
but is never referenced anywhere in the file, so it should be deleted to keep
the imports clean and avoid unnecessary dependencies.
|
Added scripts/fix_quiz_bias.py, an AI powered fixer that uses Claude API to rewrite biased questions with balanced answer lengths. |
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
scripts/fix_quiz_bias.py (1)
122-140: 🧹 Nitpick | 🔵 TrivialConsider safeguards for the destructive in-place rewrite.
This run makes one API call per biased question (the PR reports ~1,730) with no rate limiting, no retry/backoff, and overwrites
quiz.jsonfiles in place with no backup or dry-run. A transient failure or a bad model response mid-run leaves the working tree in a partially-modified, hard-to-audit state. Consider a--dry-runflag, writing to a temp file + atomic rename, and basic retry/backoff before adding this to any automated pipeline.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/fix_quiz_bias.py` around lines 122 - 140, The main flow in main() performs destructive in-place updates to quiz.json without safeguards, so add a --dry-run mode, write changes to a temporary file and atomically rename it in fix_quiz, and add basic retry/backoff around the per-question API calls. Use the existing main, fix_quiz, and is_biased flow to keep behavior unchanged by default while making the rewrite auditable and safer for failures.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/fix_quiz_bias.py`:
- Around line 78-90: The API request in the quiz-bias fixer can hang
indefinitely and still crashes on non-HTTP failures, so update the request path
in the function that wraps urllib.request.urlopen to pass an explicit timeout
and catch urllib.error.URLError as well as response-shape errors from
data["content"][0]["text"] (for example KeyError and IndexError). Keep the
existing “API error…keeping original” fallback in that same try/except block so
any timeout, network issue, or malformed response returns the original question
instead of aborting the run.
- Around line 131-135: The `main` logic is reading the same quiz file multiple
times with `json.load(open(path))`, which both leaks file descriptors and
duplicates the detection work already handled by `fix_quiz`. Update the
biased-question detection in `main` to read each file once using a closed file
handle, or remove the inline `issues` scan entirely and rely on `fix_quiz` for
the bias check, keeping the behavior centered around `main`, `fix_quiz`, and
`is_biased`.
- Around line 93-119: The `fix_quiz` flow writes `rewrite_question` output back
to `questions[i]` without checking that the returned object still matches the
quiz schema or actually removes the bias. Add validation in `fix_quiz` after
`rewrite_question(q, api_key)` to ensure the rewritten question is a dict with a
valid `options` list and `correct` index, and re-run `is_biased` before
accepting it. If the rewrite is invalid or unchanged, leave the original
question in place and avoid modifying the file; only persist the JSON when at
least one question was safely fixed.
---
Nitpick comments:
In `@scripts/fix_quiz_bias.py`:
- Around line 122-140: The main flow in main() performs destructive in-place
updates to quiz.json without safeguards, so add a --dry-run mode, write changes
to a temporary file and atomically rename it in fix_quiz, and add basic
retry/backoff around the per-question API calls. Use the existing main,
fix_quiz, and is_biased flow to keep behavior unchanged by default while making
the rewrite auditable and safer for failures.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 87fff995-0a0c-493d-92b2-39fda541f786
📒 Files selected for processing (1)
scripts/fix_quiz_bias.py
There was a problem hiding this comment.
♻️ Duplicate comments (1)
scripts/fix_quiz_bias.py (1)
123-135: 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy liftValidate the rewritten question before overwriting the file.
rewrite_questioncan return the original object (on any API/parse error) or a structurally divergent object from Claude, yet lines 125–126 assign it unconditionally and incrementfixed. Two consequences remain:
- If the model drops
correct, changes the option count, or shifts the correct index, the bias isn't actually removed and the file may break the canonical schema.fixedis overcounted because error fallbacks (which return the original question) still increment it, and the file at lines 128–131 is rewritten even when nothing meaningfully changed.Validate structure and re-run
is_biasedbefore acceptingquestions[i], and skip the write when nothing changed.🛡️ Sketch
if is_biased(options, correct): print(f" Fixing Q#{i}: {q.get('question', '')[:60]}...") - questions[i] = rewrite_question(q, api_key) - fixed += 1 + new_q = rewrite_question(q, api_key) + new_opts = new_q.get("options", []) if isinstance(new_q, dict) else [] + if (isinstance(new_q, dict) and isinstance(new_opts, list) + and len(new_opts) == len(options) + and new_q.get("correct") == correct + and not is_biased(new_opts, new_q["correct"])): + questions[i] = new_q + fixed += 1 + else: + print(f" Skipped Q#{i}: rewrite failed validation") + + if fixed == 0: + return 0🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/fix_quiz_bias.py` around lines 123 - 135, In fix_quiz_bias.py, the loop that calls rewrite_question and updates questions[i] should only accept a rewritten question if it preserves the expected schema and is no longer biased; otherwise keep the original item unchanged. Add a validation step in the bias-fix flow around rewrite_question and is_biased so that errors or structurally divergent Claude output do not increment fixed. Also track whether any question actually changed and skip the json.dump write block when nothing meaningful was modified.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Duplicate comments:
In `@scripts/fix_quiz_bias.py`:
- Around line 123-135: In fix_quiz_bias.py, the loop that calls rewrite_question
and updates questions[i] should only accept a rewritten question if it preserves
the expected schema and is no longer biased; otherwise keep the original item
unchanged. Add a validation step in the bias-fix flow around rewrite_question
and is_biased so that errors or structurally divergent Claude output do not
increment fixed. Also track whether any question actually changed and skip the
json.dump write block when nothing meaningful was modified.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: c982fe97-198a-4c2a-a74d-092dea40770b
📒 Files selected for processing (1)
scripts/fix_quiz_bias.py
javy0107
changed the title
…fix Python 3.9+ type hints
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
scripts/fix_quiz_bias.py (1)
16-16: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider builtin generics over
typing.List.The commit message notes type hints were adjusted for Python 3.9+, but
is_biasedstill usestyping.List. On 3.9+ the builtinlist[str]is preferred, letting you drop this import.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/fix_quiz_bias.py` at line 16, Update the type hints in is_biased to use the Python 3.9+ builtin generic list[str] instead of typing.List, and remove the now-unused List import from the top of the script. Keep the change localized to the is_biased signature and any related annotations so the module consistently uses builtin generics.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/fix_quiz_bias.py`:
- Line 150: The print statement in the quiz bias fixer is using an unnecessary
f-string prefix with no interpolations, which triggers Ruff F541. Remove the
extraneous f prefix from the affected print calls in the validation flow,
including the one in the same block as the “Validation failed — keeping
original” message and the other matching occurrence, while leaving the string
content unchanged.
- Around line 53-56: Update is_valid_question in fix_quiz_bias.py to also verify
that every entry in q["options"] is a string, not just that options is a list
with the expected length. Add an element-type guard alongside the existing
list/count checks so invalid rewrites are rejected before is_biased and its
len(options[correct]) access run. Use the is_valid_question and is_biased
symbols to place the fix and keep the validation consistent.
- Around line 25-28: The API key handling in fix_quiz_bias.py currently falls
back to sys.argv[1], which should be removed. Update the startup logic so the
script reads the key only from ANTHROPIC_API_KEY (using the existing os.environ
access) and does not accept a positional argument path; keep the rest of the
flow unchanged and ensure the key lookup remains in the main argument-loading
block.
---
Nitpick comments:
In `@scripts/fix_quiz_bias.py`:
- Line 16: Update the type hints in is_biased to use the Python 3.9+ builtin
generic list[str] instead of typing.List, and remove the now-unused List import
from the top of the script. Keep the change localized to the is_biased signature
and any related annotations so the module consistently uses builtin generics.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 6ce92540-4153-4490-b76a-c36143c42b6d
📒 Files selected for processing (1)
scripts/fix_quiz_bias.py
|
|
||
| # validate before overwriting | ||
| if not is_valid_question(rewritten, correct, len(options)): | ||
| print(f" Validation failed — keeping original") |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Remove the extraneous f prefix.
These strings contain no placeholders, so the f prefix is dead (Ruff F541).
🧹 Proposed fix
- print(f" Validation failed — keeping original")
+ print(" Validation failed — keeping original")- print(f" Bias not resolved — keeping original")
+ print(" Bias not resolved — keeping original")Also applies to: 155-155
🧰 Tools
🪛 Ruff (0.15.18)
[error] 150-150: f-string without any placeholders
Remove extraneous f prefix
(F541)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/fix_quiz_bias.py` at line 150, The print statement in the quiz bias
fixer is using an unnecessary f-string prefix with no interpolations, which
triggers Ruff F541. Remove the extraneous f prefix from the affected print calls
in the validation flow, including the one in the same block as the “Validation
failed — keeping original” message and the other matching occurrence, while
leaving the string content unchanged.
Source: Linters/SAST tools
1 participant
Fixes #281
What this PR does:
Adds two scripts to detect and fix answer-length bias in quiz.json files.
Scripts:
scripts/check_quiz_bias.py — scans all 338 quiz.json files, detects 1730 biased questions where the correct answer is 1.5x or more longer than the average distractor. Can be added to CI as a blocking check.
scripts/fix_quiz_bias.py — AI-powered fixer that rewrites biased questions using Claude API with balanced answer lengths while keeping meaning intact.
Usage:
python3 scripts/check_quiz_bias.py
export ANTHROPIC_API_KEY=your_key_here
python3 scripts/fix_quiz_bias.py