Skip to content

Blockscout backend build: re-resolve hackney and certifi, clearing the last eMudhra trust store - #369

Merged
Greaker merged 1 commit into
developmentfrom
feature/blockscout-backend-certifi
Nov 11, 2026
Merged

Greaker merged 1 commit into
developmentfrom
feature/blockscout-backend-certifi

Conversation

@Greaker

@Greaker Greaker commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

What

build-images.sh gets a third guarded backend patch: the Blockscout backend's deps stage now runs mix deps.update hackney certifi and asserts the resulting mix.lock entries, so the backend image no longer carries certifi 2.17.0 (which still holds the distrusted "emSign ECC Root CA - G3"). The register row for the backend goes from open to fixed, and the source-build item lists the new step.

Why a build step

Upstream's v12.0.4 lockfile resolves hackney 4.8.5, which requires certifi ~> 2.17.0. certifi 2.18.0 needs hackney 4.8.6 or later, so no upstream tag clears it yet. The step follows the #364 style: an exact-match guard on the upstream Dockerfile line (RUN mix local.hex --force, exactly one), then, in the same RUN, the update followed by assertions on the six lock entries it changes. The build fails loudly if upstream or hex resolves anything differently.

Package Before After
hackney 4.8.5 4.10.2
certifi 2.17.0 2.18.0
h2 0.12.4 0.15.1
quic 2.1.1 2.6.1
ssl_verify_fun 1.1.7 1.1.9
webtransport 0.4.8 0.5.5

Evidence (local sandbox, in-place upgrade, data kept)

  • Image built through ./sandbox.sh build-images; the pod runs digest sha256:b500012… (checked via the pod's imageID, after replacing the same-tag registry copy and the Kind node's cached image).
  • CA stores read in the running pod, by openssl and by full certificate DER against the certifi 2.17.0 list: castore 1.0.22, certifi 2.18.0 and the Alpine bundle (20261020-r0) hold 118 certificates each, with no match for "emSign ECC Root CA - G3" (or C1/C3). "emSign Root CA - G1", which is not distrusted, remains in all three.
  • hackney paths exercised: indexing at head (JSON-RPC to Besu through the :ethereum_jsonrpc pool; new blocks and transactions appeared in API v2 within seconds), API v2, BENS lookups (domains:lookup, batch-resolve-names seen on the BENS pod), the verifier microservice (a fresh standard-input verification of a throwaway contract succeeded, and verify-latest returned 13/13), a small concurrent search load, and a 15-minute soak with periodic transactions (the container was OOM-killed once by a diagnostic Erlang VM started inside it, so the soak is measured from that restart: 16 minutes, no further restart, head indexed at the chain head).
  • Startup log set is identical to the baseline pod's (the same five error lines and one warning: rate-limit config URL unset, ETS table races at start, static manifest, genesis data). No new error, no HTTP/TLS errors.
  • hackney's TLS defaults, checked on the exact hackney/certifi/h2/quic beams from the image under the same OTP: verification on by default (a self-signed server is refused), insecure and {verify, verify_none} work.

Not exercised: outbound HTTPS to public hosts (the sandbox has no egress, same as before).

hackney 4.8.5 → 4.10.2, what changes

  • max_body_size (4.9.0): buffered bodies are capped at 512 MiB by default (was unlimited); a chunked body can no longer bypass the cap (4.10.0).
  • Redirects (4.9.0): basic_auth and cookies are no longer resent to another port, or over HTTP after HTTPS.
  • Pinned cacerts (4.9.0) now truly limits trust, with no fallback to public roots; HTTPS proxies get the same certificate checks.
  • Pooling (4.9.0 to 4.10.2): a slow host no longer stalls other hosts' checkouts; max_per_host slots are released on failed handshakes, timeouts and killed callers; socket options in connect_options are no longer dropped on pooled connections; TLS 1.3 session resumption no longer causes bad_record_mac on parallel connects.
  • 4.10.1: the insecure option and {verify, verify_none} actually disable verification (they were ignored before).
  • HTTP/2 and HTTP/3 (4.10.0): requests wait for a free stream instead of failing; refused streams are retried once; quic/h2/webtransport carry DoS fixes. The sandbox backend talks HTTP/1.1 in-cluster.
  • 4.8.6: certifi 2.18.0 (drops the compromised eMudhra roots).

Rollout impact

Area Impact
Backend image same v12.0.4-kind tag, new digest; the build needs network access (hex.pm)
Existing sandbox build-images skips the registry push when the tag exists, so replace the registry copy (for example registry-reset, then build-images) and restart the backend; the Kind node's cached image of the same tag also has to be evicted
Database / chain / config none; no migration, no env change
Other services none
Dependency note hackney 4.10.2 was published one day before this change

Docs: docs/KNOWN_ISSUES.md (register row, source-build item, same-tag note).

…e last eMudhra trust store

The upstream v12.0.4 lockfile resolves hackney 4.8.5, which pins certifi to
~> 2.17.0, and certifi 2.17.0 still holds the distrusted root. The deps stage now
runs mix deps.update hackney certifi (hackney 4.10.2, certifi 2.18.0) and asserts
the six resulting lock entries, so the build stops if resolution changes. The
register row for the backend moves from open to fixed after the three stores
(castore, certifi, Alpine bundle) were read clean in the running pod.
@Greaker
Greaker merged commit 2d5d74c into development Nov 11, 2026
6 checks passed
@Greaker
Greaker deleted the feature/blockscout-backend-certifi branch November 11, 2026 12:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant