Space out JLCPCB website searches, and report and back off from its refusals (#110)

[danielmeza]

Fixes #110

EasyEDA / LCSC searches of JLCPCB's website endpoint now go through a slower rate-limit bucket of their own. A 403 or 429 from the endpoint is now reported as "JLCPCB is rate-limiting; try again shortly" in the Part Explorer and in the MCP output, instead of the provider silently dropping out. After a refusal, the app backs off for a short, bounded time. It is still a failure, never "no results", and it is never cached.

What I measured

I sent seven live requests in total: six raw probes, plus one search through the real app at the end.

The probe used the app's exact request (POST, JSON body, the honest kicad-ultra/1.0 User-Agent):

#	Sent at	Status
1	0.00 s	200
2	0.91 s (straight after #1)	200
3	4.54 s	200
4	8.29 s	200
5	8.73 s (straight after #4)	200
6	9.66 s (straight after #5)	200

• That is six requests in 10.1 s, three of them within 1.4 s, and none was refused.
• The responses come through Akamai (Akamai-GRN, Akamai-Cache-Status and Server-Timing: ak_p headers).
• They carry no rate-limit headers: no Retry-After and no X-RateLimit-*.

What #107's log shows. Its 403 came to a request sent 0.5 s after the previous one: NE555 at 16:24:36.307, then NE555DR at 16:24:36.812. Five earlier runs that day sent the same pattern (14:39 to 15:03), and all of them were answered.

So the threshold is not a simple per-second rate, and I could not reproduce the refusal. It may depend on longer-window volume from this IP, since several agents searched from this machine that day, or on how Akamai classifies the client. I did not try to provoke it: that takes the kind of volume this unofficial endpoint must not be sent, and a block would also hit the user's own browsing of jlcpcb.com from the same IP.

The numbers follow from that:

• Bucket: one request at once, then one every 3 s. That allows at most two within any second and five within any 10 s, which is below every pattern measured to work.
• Not guaranteed: the bucket cannot rule out a refusal, because the endpoint's real threshold is unknown. The reporting and the back-off below handle the refusals that still happen.

Changes

• Per-provider buckets.
  • ProviderSearchOptions.ProviderRateLimits gives EasyEDA / LCSC its own ProviderRateLimit(BurstLimit: 1, ReplenishmentPeriod: 3 s, QueueLimit: 4).
  • Every other provider keeps the defaults (3 at once, then one per 2 s).
  • ProviderRateLimiter builds each partition from GetRateLimit(providerId).
  • It is still one token per provider call. Nothing here assumes that one call makes one HTTP request (Octopart: keep the results when Nexar refuses a field to the token's plan (#109) #114's Nexar retry).
• The refusal.
  • JlcpcbWebsiteSearchClient throws the new ProviderRateLimitedException for 403 or 429. Its message is the user-facing sentence, and it carries the status and any Retry-After.
  • Retry-After is read as a delay, or as a date measured against the response's own Date header, so the local clock does not matter.
  • EasyEdaProvider logs the status and rethrows, following the narrow-catch rule.
  • The official API's 403 is unchanged: it is a documented permission error there, not rate limiting.
• The back-off, in ProviderRateLimiter: BackOff, GetBackoff and Answered.
  • After a refusal, the provider is not asked again for 30 s.
  • The wait doubles per further refusal before an answer, up to 2 min. A longer Retry-After is honoured, up to the same cap.
  • A refusal that arrives while a back-off is running counts once: it was a request already on its way.
  • The aggregator checks the back-off before waiting for a token, and again after getting one. A request that waited through another request's refusal therefore does not go out.
  • A cached answer is still served during the back-off.
• Reporting. IPartAggregatorService.StreamAllProvidersAsync now yields a ProviderSearchOutcome per provider: ProviderResults(Parts) or ProviderRateLimited(ProviderId, ProviderName, Reason, RetryAt). SearchAllProvidersAsync returns AggregatedSearchResult(Parts, RateLimited).
  • Part Explorer:
    • the refusal is a ProviderLeftOut update in the same ordered stream as the results, so Switch() still drops a superseded search's updates;
    • the existing JLCPCB notice switches to "JLCPCB is rate-limiting; try again shortly". It shows even if the notice had been hidden, names the search and says what time the app will ask again;
    • the status line adds "Left out: EasyEDA / LCSC (JLCPCB is rate-limiting; try again shortly)";
    • the next search that completes without a refusal restores the normal notice. Hide dismisses the refusal notice; the tooltip now says what Hide does in each state.
  • MCP:
    • the data-source lines gain a "Not answered (these providers' results are missing from this answer, not empty):" section, for example - EasyEDA / LCSC: JLCPCB is rate-limiting; try again shortly. This server will not ask it again for 30 s; search again after that.;
    • a search_components or get_component_details call that found nothing while a provider was left out returns isError: true.
• Other failures are unchanged. Other HTTP errors, parse errors and timeouts are still only logged and left out. Only rate limiting is reported, which is what JLCPCB website endpoint returns 403 after a few quick searches #110 asks for.
• Docs: CLAUDE.md's provider section and the README's endpoint paragraph.

Decisions to review

• The back-off is per provider, so it also holds the GUI's official-API lookups (LCSC numbers, with credentials) for its duration, and reports them as rate-limited too. Those lookups are rare, and both routes are JLCPCB's. Splitting the back-off by source would need the aggregator to know which route a query takes. The bucket covers both routes for the same reason.
• Burst 1, not 2. A burst of 2 plus a refill tick can put three requests within a fraction of a second, which was never measured. The cost of burst 1 is small: a second search within 3 s waits for up to 3 s, while the other providers' results still stream in straight away.
• isError: true only when nothing was found. With some results and a provider left out, the call succeeds, and the "Not answered" line says what is missing.
• The empty-results placeholder in the grid still reads "No component search results" when EasyEDA / LCSC was refused. The notice directly above it says the results are missing "because JLCPCB turned a request down as too frequent, not because nothing matched", and so does the status line. I left the generic placeholder alone.
• Per process: the GUI and a --mcp server each keep their own bucket and back-off, as they already did for the cache.

Verification

Build gates, on the branch rebased onto 14e51ff:

• dotnet build -c Release and -c Debug: 0 warnings, 0 errors;
• dotnet format --severity warn --verify-no-changes: clean;
• SampleConsole --test-parser: exit 0.

Offline harness. A scratch console references the UI project, and nothing of it is committed. It installed a stub as the handler of EasyEdaProvider's own static HttpClient, so the provider, the website client and the whole pipeline ran unchanged. Every other HttpClient went to the discard port. All 29 checks pass:

• Spacing, in Program.BuildMcpServiceProvider's container with the real options, for seven distinct MCP-style searches back to back:
  • the stub received them at 0.00, 3.08, 6.18, 9.28, 12.38, 15.48 and 18.48 s;
  • that is at most one within any second, and four within any 10 s;
  • a repeated query is served from the cache, with no request.
• The refusal path, with a manual clock for the back-off:
  • a 403 gives no parts, plus ProviderRateLimited with the exact reason and a 30 s RetryAt;
  • during the back-off, a new query sends no request and is still reported, while a cached query is served;
  • after 31 s, the refused query is asked again, so the failure was not cached. A second 403 in a row gives 60 s;
  • an answer resets the doubling;
  • a 429 with Retry-After: 90 gives 90 s, and Retry-After: 3600 is capped at 2 min;
  • a Retry-After date is measured against the response's Date header;
  • a 500 is still only logged, and is not cached.
• The MCP text, from McpServer's own tool handler: search_components and get_component_details both return isError=true with the "Not answered" line and "for 30 s". A cached search has no such line.
• Waiting through a refusal: two concurrent searches, the first answered 403 after 1 s. Only the first request went out, and both searches report the rate limit.

The real binaries, with the endpoint stubbed. A DOTNET_STARTUP_HOOKS assembly swaps in the stub before Main runs, and nothing reaches JLCPCB. Each run used a temp HOME and XDG dirs, and a dead D-Bus.

• --mcp over stdio: I piped initialize, notifications/initialized, then NE555 (200), LM358 (403) and get_component_details TL072.
  • The stub saw NE555 at 17:01:09.844 and LM358 at 17:01:12.908, 3.06 s apart.
  • TL072 sent no request, because of the back-off.
  • stdout had exactly four lines, all valid JSON-RPC 2.0. The second and third tool calls returned isError: true with the line above.
• GUI on a private Xvfb (:110), driven with XTest:
  • searching ne555 showed the stub row;
  • lm358 got a 403. The notice switched to "JLCPCB is rate-limiting; try again shortly", with "The search for 'lm358' has no EasyEDA / LCSC results because JLCPCB turned a request down as too frequent, not because nothing matched. The app leaves JLCPCB alone until 4:57:03 PM…". The status line read "Found 0 results across providers for 'lm358'. Left out: EasyEDA / LCSC (JLCPCB is rate-limiting; try again shortly).";
  • tl072, during the back-off, showed the same notice for 'tl072', with no request;
  • ne555 came from the cache and restored the normal notice;
  • after hiding the notice and waiting out the back-off, a new 403 brought the refusal notice back, with 4:58:20 PM, which is 60 s after the second refusal;
  • Hide then dismissed it;
  • the log has no crash markers. I stopped the app myself.

Live, once: the real --mcp binary, one search_components NE555 against the real endpoint.

• It answered 200, with 25 parts. The first two were C695838 NE555DR UMW 194291 0.0616 USD and C7593 NE555DR Texas Instruments 164286 0.1205 USD.
• stdout was clean JSON-RPC.

What was not verified

• A real refusal was not reproduced. I don't know JLCPCB's actual threshold or how long its block lasts, and I did not try to provoke either. The 403 handling was tested only against the stub. It is based on Show CAD availability as the provider reports it, Unknown distinct from none (#48) #107's log, which recorded only the status code (no body, no headers).
• Whether the 3 s spacing prevents the kind of 403 Show CAD availability as the provider reports it, Unknown distinct from none (#48) #107 saw is unknown. Its trigger looks like more than per-second rate.
• The official API path's interaction with the back-off was checked only by reading the code (no credentials).
• Not tested on Windows or macOS.

Overlap

• Rebased onto 14e51ff (docs: CLAUDE.md after the second agent round (#92–#111) #113 docs, Octopart: keep the results when Nexar refuses a field to the token's plan (#109) #114 Octopart) without conflicts. Octopart: keep the results when Nexar refuses a field to the token's plan (#109) #114 touches only OctopartProvider.
• MainWindow.axaml: one tooltip binding and a comment on the JLCPCB notice. MainWindow.axaml.cs is untouched (Browser downloads ignore the download folder set in Settings #108).
• CLAUDE.md "Tracked work" still lists "JLCPCB website endpoint returns 403 after a few quick searches #110 (JLCPCB rate limiting)" as in progress. I left that line alone, because Settings: a relative download folder is accepted, and saving resets the Web Browser tab #112's PR will likely edit it too.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DHQsqcKVEiAKvCS98Hn4Yh

[danielmeza]

Reviewed and merging.

• Rate limit: EasyEDA / LCSC gets its own, slower bucket: one request at once, then one per 3 s. That sits below every request pattern JLCPCB answered, and the endpoint could not be probed further without abusing it.
• Reporting: a 403 or 429 from the website endpoint becomes ProviderRateLimitedException and is reported, not dropped. The official API's 403 keeps its documented meaning.
• Back-off: it lives in the limiter, under a lock. A refusal arriving during an active back-off is not counted twice, waits double from 30 s up to a 2-minute cap, and Retry-After is honoured up to that cap.
• UI and MCP: the aggregator now yields one outcome per provider. The ReactiveUI search pipeline keeps its shape: outcomes turn into updates inside the same ordered stream, after the clear, and Switch() is untouched. The Part Explorer notice and status line and the MCP "Not answered" section all say which provider was left out and until when.
• Verification: the spacing, the refusal path and the back-off were checked offline against a stub, and in the real GUI and --mcp binaries through a startup-hook stub, with only one live search. The branch is on current master and CI is green.