fix: improve font encoding handling for better text extraction

[ajroetker]

Summary

This PR improves font encoding handling to fix text extraction issues in PDFs with complex font configurations. It includes two key fixes:

1. Prioritize ToUnicode CMap - Check ToUnicode CMap first before falling back to Encoding dictionaries
2. Properly apply BaseEncoding + Differences - When using Encoding dictionaries, apply BaseEncoding first then overlay Differences array

Problem

When extracting text from certain PDFs (e.g., scanned legal documents), characters were being incorrectly decoded:

• '0' was appearing as 'M' because ToUnicode CMap was being ignored
• Case numbers like 1:15-cv-07433-LAP were corrupted because Differences were applied without the BaseEncoding foundation

Solution

Fix 1: ToUnicode Priority

Per the PDF specification (section 9.10.2), ToUnicode CMap should be the primary source for mapping character codes to Unicode. This change checks ToUnicode first:

func (f Font) getEncoder() TextEncoding {
    toUnicode := f.V.Key("ToUnicode")
    if toUnicode.Kind() == Stream {
        if m := readCmap(toUnicode); m != nil {
            return m
        }
    }
    // Fall back to Encoding-based decoding
    ...
}

Fix 2: BaseEncoding + Differences

Per PDF spec section 9.6.6, when an Encoding dictionary is present:

1. Start with BaseEncoding (e.g., WinAnsiEncoding, MacRomanEncoding)
2. Apply Differences array on top to override specific character codes

The new dictEncoder:

• Builds a complete 256-entry lookup table at initialization
• Copies BaseEncoding first (defaulting to PDFDocEncoding)
• Applies Differences entries on top
• Uses O(1) lookup instead of O(n) scanning during decoding

func newDictEncoder(enc Value) *dictEncoder {
    e := &dictEncoder{}
    
    // Start with base encoding
    baseEnc := enc.Key("BaseEncoding")
    switch baseEnc.Name() {
    case "WinAnsiEncoding":
        baseTable = &winAnsiEncoding
    // ... other encodings
    }
    copy(e.table[:], baseTable[:])
    
    // Apply Differences on top
    diff := enc.Key("Differences")
    // ... apply differences
    
    return e
}

Testing

Tested with legal document PDFs (court documents) that previously had incorrect text extraction:

• Before: 104 pages with font encoding corruption
• After: 2 pages with font encoding corruption (98% reduction)

All characters including numbers (0-9), punctuation, and case numbers are now correctly extracted.

[Update] Review feedback addressed (2026-09-03)

All 8 review points are addressed (rebased onto master + one new commit):

1. Identity-H — returns nopEncoder when ToUnicode is absent/unparseable, with a comment noting that proper 2-byte CID decoding is out of scope.
2. Unknown named encodings — the default case returns nopEncoder again (same for the unexpected-kind default at the bottom of getEncoder).
3. MacExpertEncoding — now logged via DebugOn; falls back to StandardEncoding with a comment explaining why no substitute table is accurate.
4. Default BaseEncoding — added a full standardEncoding table (Table D.2) in text.go and made it the dictEncoder default, so 0x27/0x60 and the 0x80–0x9F range are now correct.
5. /.notdef — explicitly mapped to noRune in the Differences loop.
6. readCmap panics — both beginbfchar/beginbfrange panics converted to the existing ok = false / return nil pattern, with DebugOn logging.
7. Tests — added encoding_test.go: unit tests for newDictEncoder (BaseEncoding + Differences overlay, gap via restarted code counter, out-of-range codes, /.notdef, StandardEncoding default) plus two synthetic-PDF tests: ToUnicode precedence (CMap remaps A→Z over WinAnsi) and malformed-ToUnicode fallback to /Encoding (exercises the de-panicked readCmap path). Also updated the existing TestDictEncoder to the new constructor.
8. Rebase — rebased onto current master; go build, go vet, and the full test suite (including the new master tests) pass.

Follow-up hardening (commit a47f781)

A further review pass of the expanded ToUnicode parsing surface found three more robustness gaps, now fixed with regression tests:

• Codespace ranges wider than 4 bytes (e.g. <0000000000> <FFFFFFFFFF>) previously panicked with index-out-of-range on the fixed [4] space table — reachable from any untrusted PDF now that ToUnicode is parsed for every font. Now rejected cleanly.
• Truncated CMaps — endbfchar/endbfrange never reset the section counter (only endcodespacerange did), so a stream cut off inside a section, or a stray end operator, was treated as successfully parsed and every code decoded to U+FFFD. A CMap ending inside an open section (or containing no bfchar/bfrange mappings at all) now returns nil so extraction falls back to /Encoding.
• Differences code counter now keeps advancing past out-of-range codes, so a bad start code no longer drops later entries that land back in range.

Follow-up hardening, round 2 (commit e21cd86)

A second review pass over the now-always-parsed ToUnicode path found two panic/DoS vectors and one fallback gap, all pre-existing but newly reachable from every font. Fixed with regression tests:

• Unbounded section counts — readCmap looped on the untrusted entry count (Pop on an exhausted stack returns zero values), so 9223372036854775807 beginbfchar endbfchar in a tiny stream allocated unbounded entries. Counts are now capped against the operand stack size, and the open section kind is tracked so beginbfchar ... endbfrange is rejected rather than matched.
• Decode-time panics — utf16Decode indexed past the end on odd-length destination strings (e.g. a bfchar destination of (Z)), and bfrange scaling indexed b[len(b)-1] on an empty destination. Both now decode to U+FFFD instead of panicking.
• CMaps without a codespace range can never match a code, so they now count as a parse failure and extraction falls back to /Encoding rather than emitting only replacement characters.

Follow-up hardening, round 3 (commit 01784f4)

• Interpreter panics contained at readCmap — Interpret panics on malformed PostScript (unmatched end, begin on a non-dict, currentdict with no dictionary); readCmap now recovers and reports parse failure so extraction falls back to /Encoding instead of crashing Content(). This completes item 6: no malformed ToUnicode input can now escape the CMap parsing boundary.
• Entry counts validated against the section's own operands (stack depth recorded at each begin*), so an entry missing an operand can't pass by borrowing unrelated stack values.
• Destinations that decode to no UTF-16 units (e.g. a malformed single-byte bfchar destination) now yield U+FFFD instead of silently deleting the character; genuinely empty destinations still map to nothing.
• Renamed a local that shadowed the package-level name type.

[ledongthuc]

Thanks for this PR — the direction is correct and both fixes are spec-compliant (ToUnicode precedence per §9.10.2, and BaseEncoding + Differences per §9.6.6). I verified locally: it builds, go vet is clean, the existing tests pass, and it merges cleanly.

I'd like a few changes before merging:

1. Identity-H fallback — Identity-H is a 2-byte CMap for Type0/CID fonts, so decoding byte-by-byte through pdfDocEncoding produces garbage when ToUnicode is absent. This predates the PR, but since this line is being touched, can we leave it as nopEncoder for this case (or explicitly note it's out of scope)?

2. Unknown named encodings (default case) — this changed from nopEncoder to pdfDocEncoding, which silently alters behavior for Symbol, ZapfDingbats, StandardEncoding, Identity-V, etc. None of those equal PDFDocEncoding; Symbol/ZapfDingbats especially will decode to wrong (but plausible-looking) Latin text. Recommend keeping nopEncoder for genuinely unknown names, or adding proper tables.

3. MacExpertEncoding fallback — silently mapping it to pdfDocEncoding is inaccurate; please at least log via DebugOn or add a real table.

4. Default BaseEncoding = PDFDocEncoding — per spec the default should be the font's built-in encoding (usually StandardEncoding). The comment acknowledges the simplification, but consider adding a StandardEncoding table since the 0x80–0x9F range differs from PDFDocEncoding.

5. /.notdef in Differences — nameToRune has no .notdef, so a /.notdef entry silently keeps the BaseEncoding value instead of mapping to noRune.

6. readCmap can panic — readCmap has panic("missing beginbfchar") / panic("missing beginbfrange"). Now that ToUnicode is parsed for every font, this expands the panic surface; Content() has no recover(), so a malformed ToUnicode CMap could crash. Recommend converting those to the existing ok = false / return nil pattern.

7. Tests — the PR reports a big improvement but has no regression tests. Could you add a unit test for newDictEncoder (BaseEncoding + Differences, incl. a gap and an out-of-range code) and a ToUnicode-precedence test (e.g., a synthetic PDF like ps_test.go)?

8. Minor: the branch is ~20 commits behind master; it merges cleanly, but a rebase would be nice.

None of these are hard blockers except arguably (2) and (6). Happy to iterate.