Feat analytics by sirpy · Pull Request #19 · GoodDollar/antseed-integration

sirpy · 2026-07-24T14:02:38Z

No description provided.

Copilot

Pull request overview

Adds an analytics aggregation flow to the Cloudflare Worker to track daily G$ deposits/streaming and Base “AI credits used” from Antseed Channels, persisted in KV and exposed via versioned API endpoints. This fits the repo’s Worker responsibilities (credit accounting + chain event ingestion) and extends it with reporting/observability data.

Changes:

Add backend/src/analytics.ts KV-backed aggregation querying Blockscout (Celo/Base) + Superfluid subgraph, with daily + global windows.
Expose analytics via GET /v1/analytics and a recompute endpoint POST /v1/analytics/refresh; also run aggregation from the scheduled cron.
Add env/wrangler config for Blockscout URLs + Channels contract address, update docs, and add analytics-focused tests.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
backend/wrangler.toml	Adds Blockscout + Channels env vars; changes cron to run every 6 hours.
backend/test/worker.test.ts	Adds tests around analytics aggregation logic and KV persistence behavior.
backend/src/worker.ts	Wires new `/v1/analytics` routes and runs analytics aggregation in the scheduled handler.
backend/src/logging.ts	Adjusts log emission behavior (notably affects `logInfo`).
backend/src/env.ts	Adds env bindings for Blockscout URLs + Channels address to the Worker config surface.
backend/src/analytics.ts	New module implementing aggregation, KV storage, and explorer/subgraph fetching utilities.
backend/README.md	Documents analytics endpoints and required config.

Comments suppressed due to low confidence (3)

backend/src/analytics.ts:138

These logInfo calls use free-text event names (with punctuation/ellipses), which is inconsistent with the rest of the codebase’s structured dot-separated events and makes log querying harder. Consider using stable event IDs for start/end of base metrics collection.

  logInfo("getting base metrics....");
  const baseMetrics = await collectBaseDayMetrics(cfg, dayWindow, aggregate, knownBuyers);
  logInfo("got base metrics....");

backend/src/analytics.ts:141

This log event string has typos (dialy reocrd) and uses a free-text name. Prefer a stable, dot-separated event ID (and include details as structured data).

  logInfo("building dialy reocrd....");

backend/src/analytics.ts:368

This log call uses an unstructured, punctuation-heavy event name ("base range:"). For consistency with the rest of the repo, use a stable dot-separated event name and pass the range as structured data.

  logInfo("base range:", range);

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

blueogin

Buyer registry only grows from Celo GdDeposited buyers in collectCeloDayMetrics, then Base events are filtered by knownBuyers in collectBaseDayMetrics. Stream-only users may miss Base credit attribution until they deposit.
Is this intentional?

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

backend/src/worker.ts:224

The first-run analytics backfill runs a potentially unbounded day-by-day loop inside a single scheduled invocation. This can exceed Cloudflare cron execution limits and/or hit Blockscout/Superfluid rate limits, leaving analytics partially backfilled with no clear recovery path. At minimum, add a TODO noting the need to persist a backfill cursor and process a bounded number of days per cron tick (or move the backfill to a manual/admin operation).

        logInfo("cron.analytics.first-run");
        const startDate = new Date("2026-07-02T00:00:00Z");
        // TODO: Persist a backfill cursor and process a bounded number of days per cron tick;
        // this first-run catch-up loop can exceed scheduled execution limits as the date range grows.
        while (startDate < new Date()) {

backend/src/analytics.ts:276

analyticsConfigFromEnv() defaults SUPERFLUID_SUBGRAPH_URL to https://celo-mainnet.subgraph.x.superfluid.dev/, but the rest of the Worker uses https://subgraph-endpoints.superfluid.dev/celo-mainnet/protocol-v1. If SUPERFLUID_SUBGRAPH_URL is not set in the environment, analytics stream collection is likely to query the wrong endpoint and fail.

    superfluidSubgraphUrl: env.SUPERFLUID_SUBGRAPH_URL ?? "https://celo-mainnet.subgraph.x.superfluid.dev/"

backend/src/analytics.ts:143

Log message has a spelling typo ("dialy reocrd"), which makes searching/alerting on logs harder and looks unpolished in production output.

  logInfo("building dialy reocrd....");

backend/README.md:49

Docs say known buyers are learned only from Celo vault buyer fields, but the implementation also learns buyers from Superfluid stream userData (see decodeBuyerFromUserData(...) usage). Update this line so the filtering/learning rules match the code.

- Base usage (`aiCreditsUsedWei`, `uniqueCreditUsers`) is filtered to known buyers only; known buyers are learned from Celo vault `buyer` fields.

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (7)

backend/src/analytics.ts:396

This log call uses an inconsistent event name (contains spaces/colon) compared to the rest of the codebase’s structured log events; it also makes it harder to filter by prefix.

  logInfo("base range:", range);

backend/README.md:51

Docs say refresh always recomputes the current day, but runAnalyticsAggregation() may backfill the next non-finalized day instead (via resolveRunDate). Update this line so consumers don’t assume refresh only affects “today”.

`POST /v1/analytics/refresh` recomputes the current UTC day from midnight to now, overwrites the day snapshot, finalizes any closed days into persisted globals, and returns a summary.

backend/src/worker.ts:8

KVAnalyticsStore is imported but never used in this file, which can cause lint/typecheck failures and adds noise. Remove it from the import list.

import { getAnalyticsWindow, runAnalyticsAggregation, KVAnalyticsStore } from "./analytics.js";

backend/src/analytics.ts:145

These log event names contain typos ("dialy reocrd") and don't follow the structured namespace.action pattern used elsewhere, making logs harder to query/debug.

This issue also appears on line 396 of the same file.

  logInfo("getting base metrics....");
  const baseMetrics = await collectBaseDayMetrics(cfg, dayWindow, aggregate, knownBuyers);
  logInfo("got base metrics....");
  logInfo("building dialy reocrd....");

backend/src/analytics.ts:314

Minor typo in comment: "dont" → "don't" (helps keep the codebase’s comments polished/searchable).

    //dont finalize if a day is missing

backend/README.md:46

Docs say the current day is always recomputed on each refresh, but the aggregator can also run for past (backfill) days depending on finalizedThroughDate. Clarify that the overwritten snapshot is for the day being aggregated.

This issue also appears on line 51 of the same file.

- `daily` — one record per UTC day, newest last; the current day is always recomputed from midnight to now on each refresh and its snapshot is overwritten

backend/README.md:49

Known buyers aren’t only learned from Celo vault buyer fields: the implementation also discovers buyers from Superfluid stream userData and adds them to the registry. Update the docs so the Base filtering behavior matches the code.

- Base usage (`aiCreditsUsedWei`, `uniqueCreditUsers`) is filtered to known buyers only; known buyers are learned from Celo vault `buyer` fields.

blueogin · 2026-07-30T19:06:43Z

+export function resolveRunDate(state: AnalyticsState, requestedDate: Date, runDay: string): string {
+  const requested = dayFromDate(requestedDate);
+  if (state.finalizedThroughDate) {
+    const next = nextDate(state.finalizedThroughDate);
+    if (next && next <= runDay) {
+      return next;
+    }
+  }
+
+  return requested <= runDay ? requested : runDay;
+}


On a fresh KV, finalizedThroughDate is absent, so this returns today instead of the intended backfill start date. Finalization then encounters yesterday as missing and leaves the cursor undefined. Because cron sees that currentDate === today, it exits, and every later invocation repeats today forever. The original July 2 backfill seed has effectively been removed. Please initialize/persist an explicit backfill cursor when no analytics state exists, and add a fresh-KV integration test.

@copilot on first run use 30 days backfill

Implemented in da8eef2: scheduled analytics now seeds and persists a 30-day backfill cursor (analytics:cron:backfill-cursor) on fresh KV and advances it across cron ticks. Added a fresh-KV scheduled integration test to verify the first run backfills and persists cursor progression.

blueogin · 2026-07-30T19:08:09Z

+            flowUpdatedEvents(orderBy: timestamp, orderDirection: desc, first: 1) {
+              userData
+              oldFlowRate


This uses the current stream entity and only its latest update to reconstruct an arbitrary historical day.
That is insufficient if the stream changed more than once during the day or its latest update happened after the backfilled day. For example, rates 2 -> 4 at 06:00 and 4 -> 8 at 10:00 are calculated as 4 for the entire pre-10:00 period. Historical aggregation needs all relevant FlowUpdated events for the target window and the rate active at midnight.

sirpy · 2026-08-03T09:46:12Z

@copilot review latest changes

Copilot · 2026-08-03T09:50:44Z

@copilot review latest changes

Reviewed latest changes and fixed the failing analytics/worker tests in 1a191db (streamPeriods fixtures + fresh-KV backfill cursor expectation). Backend tests and lint now pass.

blueogin · 2026-08-03T13:20:11Z

+      if (!cursorDate) {
+        maxRunsPerTick = ANALYTICS_CRON_BACKFILL_DAYS;
+        cursorDate = dateDaysAgo(runAt, ANALYTICS_CRON_BACKFILL_DAYS);
+        await env.ANTSEED_KV.put(ANALYTICS_CRON_BACKFILL_CURSOR_KEY, cursorDate);


Fresh KV sets maxRunsPerTick = 30, so the first scheduled invocation can issue dozens of Blockscout + Superfluid calls and hit Worker time/subrequest limits. Prefer a small bound per tick (e.g. 2) and rely on the persisted cursor across crons
the cursor already supports that.

should be ok and changed in the futures

blueogin · 2026-08-03T13:21:32Z

+    const json = (await response.json()) as {
+      data?: {
+        streamPeriods?: Array<{
+          sender: { id: string };
+          flowRate: string;
+          startedAtTimestamp: string;
+          stoppedAtTimestamp: string | null;
+          userData: string;
+        }>;
+      };
+    };
+
+    const batch = json.data?.streamPeriods ?? [];


GraphQL { errors: [...] } with no data is treated as empty success. That can write a zero daily snapshot and later finalize it.
Please fail the aggregation on subgraph/explorer logical errors and skip replace/finalize.

thalescb · 2026-08-10T11:41:19Z

@sirpy Hey -- the /v1/analytics endpoint on the deployed Worker (gooddollar-antseed-integration.gooddollar.workers.dev) doesn't seem to be responding yet. Since there's no CD step in the CI workflow, I think the Worker needs a manual wrangler deploy from backend/ now that this is on main.

Also a heads-up: the CI run triggered by the merge (#106) is failing on the Backend job -- might be worth checking before deploying.

Let me know if there's anything blocking the deploy or if I can help.

sirpy added 2 commits July 24, 2026 16:57

add: daily analytics

e6f5b8c

fix: analytics first run

b28cbc3

This was linked to issues Jul 24, 2026

Execution plan for analytics aggregation and endpoint #14

Closed

feat: add analytics #13

Closed

sirpy requested review from blueogin and Copilot July 24, 2026 14:03

Copilot started reviewing on behalf of sirpy July 24, 2026 14:03 View session

Copilot AI reviewed Jul 24, 2026

View reviewed changes

blueogin reviewed Jul 27, 2026

View reviewed changes

Comment thread backend/src/logging.ts

sirpy and others added 5 commits July 28, 2026 16:06

Potential fix for pull request finding

05cdda0

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

f22f737

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

20ce999

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

dec1fb6

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Potential fix for pull request finding

031b4f5

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot started work on behalf of sirpy July 28, 2026 13:12 View session

Add analytics route tests

fc1efb5

Copilot finished work on behalf of sirpy July 28, 2026 13:14

sirpy added 2 commits July 28, 2026 16:22

fix: log info issue

bf50e9d

fix: include buyers from streams

bcac185

sirpy requested review from blueogin and Copilot July 28, 2026 13:23

Copilot started work on behalf of sirpy July 28, 2026 13:23 View session

Copilot started reviewing on behalf of sirpy July 28, 2026 13:23 View session

Add analytics backfill TODO

c0edbf4

Copilot finished work on behalf of sirpy July 28, 2026 13:25

Copilot AI reviewed Jul 28, 2026

View reviewed changes

blueogin requested changes Jul 28, 2026

View reviewed changes

Comment thread backend/src/analytics.ts Outdated

Comment thread backend/src/analytics.ts Outdated

Comment thread backend/src/analytics.ts

fix: analytics code review

f2894c6

sirpy requested a review from blueogin July 29, 2026 10:53

fix: analytics test

5d28cd1

Copilot AI reviewed Jul 29, 2026

View reviewed changes

Comment thread backend/src/worker.ts

Copilot started work on behalf of sirpy July 30, 2026 10:47 View session

Add hourly throttle for analytics refresh endpoint

5bbe333

Copilot finished work on behalf of sirpy July 30, 2026 10:51

thalescb mentioned this pull request Jul 30, 2026

feat: add Antseed AI Credits analytics dashboard GoodDollar/data-team#49

Merged

blueogin requested changes Jul 30, 2026

View reviewed changes

Copilot started work on behalf of sirpy August 3, 2026 09:09 View session

Seed 30-day analytics cron backfill cursor

da8eef2

Copilot finished work on behalf of sirpy August 3, 2026 09:16

sirpy added 2 commits August 3, 2026 12:43

fix: stream periods

eef3608

fix: backfill

22d4775

sirpy requested a review from blueogin August 3, 2026 09:45

Copilot started work on behalf of sirpy August 3, 2026 09:46 View session

test: align analytics fixtures with streamPeriods backfill cursor

1a191db

Copilot finished work on behalf of sirpy August 3, 2026 09:51

blueogin requested changes Aug 3, 2026

View reviewed changes

sirpy added 2 commits August 4, 2026 15:29

fix: throw subgraph error

73ed113

Merge branch 'main' into feat-analytics

aeae2a6

sirpy requested a review from blueogin August 4, 2026 12:35

sirpy added 6 commits August 6, 2026 11:05

fix: stream query userdata

f037496

add: better refresh

9823ea2

add: improve refresh

917516c

fix: base analytics

bcf94be

fix: daily streams

1c98bae

fix: subgraph query

8e989b2

sirpy merged commit 8e989b2 into main Aug 9, 2026
2 of 3 checks passed

sirpy deleted the feat-analytics branch August 9, 2026 07:08

Conversation

sirpy commented Jul 24, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

blueogin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sirpy commented Aug 3, 2026

Uh oh!

Copilot AI commented Aug 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

thalescb commented Aug 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants