Where dataprof is going over the next few months

[AndreaBozzo]

This is a short note on where dataprof is heading over the next few months. No hype, just the plan.

Where things stand

dataprof is a Rust core with a Python API. It profiles CSV, JSON, JSONL, and Parquet files, plus in memory data from pandas, polars, and Arrow. You get schema, per column statistics, pattern detection, and quality metrics based on the ISO 8000 and 25012 families.

The CLI was retired in the last version. It was hard to maintain and not good enough to justify the upkeep. The Python package and the Rust library are the supported ways to use dataprof.

0.9.0: make the basics solid

The goal of 0.9.0 is to reduce friction for someone who installs the package and runs their first profile. It is not about adding new surface.

What is planned:

• Python ergonomics cleanup, so the common things feel obvious (Python: ergonomics cleanup — DX polish from post-#302 audit #315).
• One deliberate breaking pass while we are still before 1.0: fixing the confusing flat versus nested quality accessors, and setting a realistic minimum Python version (Python 0.9 breaking pass: DataQualityMetrics flat→nested semantics + requires-python #326).
• A small analyze_structure() for a cheap first look at a file before you pay for a full profile (Expand partial analysis into a configurable lightweight structural profiling surface #319, minimal version only).
• ProfileReport.load() so you can save a report and read it back later without reprofiling (Python: ProfileReport.load(path) — reload a saved report from a file #325).
• Three real world examples, and a README that clearly separates what works in the base install from what needs a source build (Add real-world examples for common data quality scenarios #67, 0.9 README refresh: install → first profile → interpret → export → compare #327).
• A smoke install test so the documented examples actually run in CI (CI: smoke-install test (Python wheel + Rust examples) as 0.9.0 definition-of-done #328).
• Short release notes and a migration guide, since 0.9.0 has at least one breaking change (0.9.0 release notes + migration guide #329).

Report exports already landed recently: to_html, to_markdown, compare, and reloading a report from JSON.

After 0.9.0: dataprof for coding agents

This is a separate track, roughly 0.10. The idea is simple. A lot of people now work through agents like Claude Code, Codex, and Cursor. dataprof should be something an agent can call safely to understand a dataset before it starts changing things.

The order matters here. We start with the output format, not with any AI feature.

• to_llm_context() gives a compact, deterministic summary of a profile that you can paste into any chat or agent. It is useful on its own, so it is being pulled into the 0.9 line (Python: to_llm_context(max_tokens=...) — token-bounded report summary for agents #331).
• Redaction by default. Anything that goes to a model leaves out raw cell values unless you explicitly ask for them, and it never shows values for sensitive patterns like emails, phone numbers, or IDs (Agent-safe output: redact raw sample values by default in agent/MCP surfaces #332).
• An MCP server so those three tools can call dataprof directly (dataprof MCP server (python -m dataprof.mcp, pip install dataprof[mcp]) #330). It is gated by a security checklist (Agent/MCP security hardening checklist (blocker for MCP server) #336): a sandbox root for file access, resource and timeout limits, no network by default, and tests for malicious paths.
• A small knowledge layer, meaning an AGENTS.md snippet, a Cursor rule, and a Claude skill, that teaches an agent the right workflow (Knowledge layer: AGENTS.md block + Cursor rule + Claude Code Skill for dataprof #333), plus one page of docs that ties it together (Docs: "Using dataprof with AI agents" (Claude Code / Codex / Cursor + code-execution pattern) #334).

To be clear about scope: dataprof produces structured signals. It is not trying to be an assistant that gives advice. Reading the numbers is the agent's job, or yours.

What is not happening soon, on purpose

• No new file formats yet, such as Avro, ORC, or Delta. Install, examples, clear errors, and a stable API matter more for adoption right now.
• No streaming runtime yet. Continuous streams, sliding windows, alerting, and quality deltas are parked for 0.10 or later. 0.9 ships one runnable streaming example at most.
• No CLI comeback unless it can be genuinely good. One more surface to maintain is not worth it otherwise.

Feedback welcome

If you use dataprof, the most useful things to hear are: which format or workflow felt rough, where the Python API did something you did not expect, and whether the quality scores matched your own read of the data.

Milestones for reference:

• 0.9.0: https://github.com/AndreaBozzo/dataprof/milestone/6
• Agent and LLM Integration: https://github.com/AndreaBozzo/dataprof/milestone/7