Skip to content

feat(insights): paths with start/end points, cleaning rules, groupings, exclusions and actors - #10

Merged
ayushozha merged 2 commits into
mainfrom
ayush/paths-v2
Nov 17, 2026
Merged

ayushozha merged 2 commits into
mainfrom
ayush/paths-v2

Conversation

@ayushozha

Copy link
Copy Markdown
Owner

What and Why

Pulse's paths showed only a one-hop next/previous page from a single URL. This adds PostHog-style multi-step paths: a weighted edge list ready for a Sankey chart, with start and end points, cleaning and grouping, exclusions, and drill-down to the visitors behind any node, edge or drop-off.

Stacked on #9 (base ayush/retention-lifecycle); retarget to main once the stack below merges.

Reference followed: PostHog PathsQuery / PathsFilter (includeEventTypes, startPoint, endPoint, stepLimit, edgeLimit, minEdgeWeight/maxEdgeWeight, localPathCleaningFilters, pathGroupings, excludeEvents) and PathsActorsQuery. The node key format "<step>_<item>" matches PostHog's response, so a Sankey component built for PostHog can render it unchanged.

Changes

  • POST /api/v1/insights/paths
    • Paths are built per session from pageviews (keyed by path) and/or custom events (keyed by event name).
    • Repeated consecutive items collapse into one.
    • start_point starts each path at the first occurrence of that item. end_point keeps the steps leading up to it.
    • Also supports step_limit (2–20), edge_limit, and min_edge_weight / max_edge_weight.
    • path_cleaning_rules ({regex, alias}, applied in order), path_groupings (wildcards such as /blog/*), exclude_events, and global filters with nested groups.
    • Each edge returns source, target, value, and average_conversion_time_secs.
  • POST /api/v1/insights/paths/actors returns the visitors behind a node (path_start_key), an edge (plus path_end_key), or a drop-off (dropoff: true).
  • Path assembly runs in spawn_blocking, with a 2M-event cap that returns 400 when hit.
  • New dependency: regex = "1". It's the standard Rust regex crate, already in Cargo.lock transitively, and its RE2-style syntax matches the ClickHouse/re2 rules PostHog path cleaning uses. Patterns are compiled once per query, and invalid ones return 400.

Reviews and Follow-up

  • Code review: self-reviewed
  • Security review: regex is linear-time (no catastrophic backtracking), the result size is bounded, and person data is gated by identity
  • Documentation: request shape above. Follow-ups:
    • project-level saved path cleaning rules (applied automatically), together with the project settings PR
    • paths before/after a funnel step

Files Changed

  • services/insights/paths.rs (new)
  • services/insights/mod.rs, routes/insights.rs, main.rs
  • Cargo.toml, Cargo.lock

Scope and Non-scope

In scope: the paths insight and its actors. Out of scope: UI, saved project cleaning rules, and the excluded LLM/account features.

Validation

I ran it on Postgres 18 with the seeded dataset, plus extra blog and user pages so the paths vary. Every edge map was compared with an independent Python path builder over the raw rows:

  • default paths, step limit, start point and end point
  • cleaning rule plus wildcard grouping, exclusions, custom events only
  • edge limit and minimum weight
  • edge and drop-off actors

All match.

Paths verification

Testing

cargo test: 118 passed, including 5 new paths tests covering collapsing, exclusions, start/end/step limits, cleaning and grouping, regex escaping and validation.

cargo test paths

Size and Notes

About 540 changed lines. No migrations, and existing endpoints are unchanged.

🤖 Generated with Claude Code

…s, exclusions and actors

POST /api/v1/insights/paths builds per-session paths of pageviews and/or
custom events and returns a weighted edge list (PostHog PathsQuery shape:
"1_/home" -> "2_/pricing"), with step_limit, edge_limit, min/max edge
weight, start_point / end_point, regex path cleaning rules, wildcard
groupings and excluded items. POST /api/v1/insights/paths/actors returns the
visitors behind a node, an edge, or a drop-off.

Adds the `regex` crate (already in Cargo.lock transitively) for RE2-style
cleaning rules, compiled once per query; invalid patterns return 400.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T22:10:54.976840Z 5b24086 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5b24086de4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…es cannot exhaust memory

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ayushozha
ayushozha changed the base branch from ayush/retention-lifecycle to main November 17, 2026 01:13
@ayushozha
ayushozha merged commit b86fe24 into main Nov 17, 2026
0 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant