Generated by scripts/make_results.py 008-sites from the committed run
checks/results/008-sites/. Do not edit by hand.
- scrapeplanner 0.1.0.dev0, commit
b3278bba2561 - GitHub Actions runner (Linux 7.0.0-1017-azure (Linux)), Python 3.12.3
- run started 2026-11-04T20:52:01+00:00, finished 2026-11-04T20:56:15+00:00 (UTC)
- options: delay=1.0, items=1000, max_requests=10
- one homepage or public listing page per site; the tool read robots.txt first and stopped at the first block or challenge
Requests counts every HTTP request the tool sent for that site, robots.txt and redirect hops included. Time is the tool's own wall time, including the polite delay between requests (and robots.txt crawl-delay where set).
- Sites analysed: 30
- Verdicts: good fit 21, with care 3, not a fit 6
- Platforms: unknown 9, not checked 6, Shopify 4, Next.js 2, WooCommerce 1, Greenhouse 1, Lever 1, Ashby 1, WordPress 1, Sphinx 1, Lit 1, Nuxt 1, Hugo 1
- Recommended method: HTML 13, Feed / API 5, Structured data 3, Embedded JSON 2, Browser 1
- Anti-bot vendor seen: Cloudflare 13, Akamai 1; challenge page served: Cloudflare 4
- Requests per site: 1 to 8, 122 in total; median time 4.5 s
- A platform was named on 15 of the 24 sites that served a page; 6 sites served no page to look at (blocked, challenged or disallowed).
- 15 sites carry an expected platform written in
checks/request.ymlbefore the run (from the URL, the software a portal publicly runs on, or public knowledge). Of the 14 of them that served a page, 13 matched. - Mismatches: data-gov (expected CKAN, found unknown)
| Site | Category | Platform | Expected | Method found | Anti-bot vendor | Verdict | Requests | Time (s) | Why |
|---|---|---|---|---|---|---|---|---|---|
| allbirds | e-commerce | Shopify | Shopify | Feed / API | Cloudflare (CDN only) | good fit | 5 | 5.8 | Shopify /products.json: products, variants, prices, images |
| gymshark | e-commerce | Shopify (Next.js) | Shopify | Embedded JSON | none | with care | 4 | 5.0 | script#NEXT_DATA (761,736 bytes); run stopped at HTTP 403 on https://www.gymshark.com/products.json?limit=250 |
| ugmonk | e-commerce | Shopify | Shopify | Feed / API | Cloudflare (CDN only) | good fit | 5 | 5.5 | Shopify /products.json: products, variants, prices, images |
| deathwish-coffee | e-commerce | Shopify | Shopify | Feed / API | Cloudflare (CDN only) | good fit | 5 | 5.2 | Shopify /products.json: products, variants, prices, images |
| porter-york | e-commerce | not checked | - | - | Cloudflare (challenge) | not a fit | 2 | 1.2 | cloudflare challenge page (HTTP 403: challenge script _cf_chl_opt) |
| woocommerce | e-commerce | WooCommerce | WooCommerce | HTML | none | good fit | 4 | 4.0 | CSS selectors on server-rendered HTML (15,268 chars of text) |
| gitlab-greenhouse | jobs | Greenhouse (Remix) | Greenhouse | Feed / API | none | good fit | 5 | 4.7 | Greenhouse job-board API: every open job with its fields |
| palantir-lever | jobs | Lever | Lever | Feed / API | Cloudflare (present) | with care | 5 | 8.0 | Lever postings API: open jobs with their fields |
| linear-ashby | jobs | Ashby | Ashby | Browser | Cloudflare (CDN only) | with care | 4 | 3.4 | render pages in a headless browser (highest cost); run stopped at HTTP 401; cloudflare in front of the site on https://api.ashbyhq.com/robots.txt |
| remoteok | jobs | unknown | - | Structured data | Cloudflare (CDN only) | good fit | 8 | 9.8 | Product in json-ld (1 on the page) |
| weworkremotely | jobs | not checked | - | - | Cloudflare (challenge) | not a fit | 2 | 1.2 | cloudflare challenge page (HTTP 403: challenge script _cf_chl_opt) |
| himalayas | jobs | not checked | - | - | Cloudflare (challenge) | not a fit | 2 | 1.4 | cloudflare challenge page (HTTP 403: challenge script _cf_chl_opt) |
| data-gov | open-data | unknown | CKAN | HTML | none | good fit | 5 | 41.2 | CSS selectors on server-rendered HTML (12,159 chars of text) |
| nyc-open-data | open-data | not checked | Socrata | - | Cloudflare (present), Akamai (CDN only) | not a fit | 4 | 3.9 | HTTP 403; akamai in front of the site |
| bbc-news | news | Next.js | - | Embedded JSON | none | good fit | 3 | 2.6 | script#NEXT_DATA (146,500 bytes) |
| guardian | news | unknown | - | HTML | none | good fit | 7 | 8.3 | CSS selectors on server-rendered HTML (21,231 chars of text) |
| reuters | news | not checked | - | - | none | not a fit | 1 | 0.4 | page disallowed by robots.txt |
| apnews | news | not checked | - | - | Cloudflare (challenge) | not a fit | 2 | 1.2 | cloudflare challenge page (HTTP 403: challenge script _cf_chl_opt) |
| techcrunch | news | WordPress | WordPress | HTML | Cloudflare (CDN only) | good fit | 5 | 4.5 | CSS selectors on server-rendered HTML (12,717 chars of text) |
| hacker-news | news | unknown | - | HTML | none | good fit | 4 | 90.4 | CSS selectors on server-rendered HTML (4,068 chars of text) |
| python-docs | docs | Sphinx | Sphinx | HTML | none | good fit | 3 | 2.2 | CSS selectors on server-rendered HTML (3,196 chars of text) |
| mdn | docs | Lit | - | HTML | none | good fit | 5 | 5.0 | CSS selectors on server-rendered HTML (6,085 chars of text) |
| react-dev | docs | Next.js | Next.js | HTML | none | good fit | 4 | 3.4 | CSS selectors on server-rendered HTML (8,090 chars of text) |
| nuxt-docs | docs | Nuxt | Nuxt | Structured data | none | good fit | 5 | 4.8 | Article in json-ld (1 on the page) |
| kubernetes-docs | docs | Hugo | Hugo | HTML | none | good fit | 4 | 4.6 | CSS selectors on server-rendered HTML (26,469 chars of text) |
| craigslist-sf | listings | unknown | - | HTML | none | good fit | 6 | 7.3 | CSS selectors on server-rendered HTML (4,617 chars of text) |
| books-toscrape | practice | unknown | - | HTML | none | good fit | 3 | 2.6 | CSS selectors on server-rendered HTML (1,810 chars of text) |
| quotes-toscrape | practice | unknown | - | HTML | none | good fit | 3 | 2.9 | CSS selectors on server-rendered HTML (1,664 chars of text) |
| webscraper-io | practice | unknown | - | Structured data | none | good fit | 4 | 3.4 | Product in microdata (3 on the page) |
| scrapethissite | practice | unknown | - | HTML | Cloudflare (CDN only) | good fit | 3 | 3.1 | CSS selectors on server-rendered HTML (18,132 chars of text) |
- data-gov: Labelled CKAN before the run from public knowledge. The page served in this run has no CKAN markers (no generator tag, no /api/3/action/ links, no CKAN assets; the front end is USWDS with htmx), so the label is out of date. See docs/incidents.md, I-008.
Per-site details: checks/results/008-sites/<site>/report.txt (terminal report)
and report.json (full JSON). Labels: CDN only means only CDN headers were
seen, not bot management; challenge means the vendor served a block or
interstitial page and the run stopped there.