OpenDataFramework

Host a Data Framework tool chain

Open Data Framework

A complete, containerized, open-source data engineering and analytics platform that you can stand up with a single docker compose up. It demonstrates a full modern data stack — from simulated data generation through ingestion, orchestration, storage, transformation, machine learning, and BI dashboards.

Built as a modular reference architecture, the stack runs entirely through Docker Compose and is designed to be easy to run locally and extend.

---

Table of Contents

• Architecture
• What's inside
• Data flow
• Requirements
• Getting started
• Access the services
• One-time setup (Superset)
• Dagster / other pipelines
• Configuration
• Security
• Limitations & notes
• Contributing
• License

---

Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                        DATA SIMULATION                                  │
│   pos-db (Postgres)  ◄──  mock-pos-generator (Faker POS sales)          │
│        │                                                               │
│        └──►  mock-api  (context / competitor pricing API)              │
└───────────────────────────────┬─────────────────────────────────────────┘
                                ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                        INGESTION (Meltano)                              │
│   tap-postgres  ─┐                                                     │
│   tap-custom-api ─┴──►  target-trino  (Bronze layer)                    │
└───────────────────────────────┬─────────────────────────────────────────┘
                                ▼
┌─────────────────────────────────────────────────────────────────────────┐
│              STORAGE · CATALOG · QUERY                                  │
│   MinIO  (S3 object storage: warehouse + mlflow buckets)                │
│   Nessie + metastore-db  (table catalog / versioning)                   │
│   Trino  (distributed SQL engine over the data)                        │
└───────────────────────────────┬─────────────────────────────────────────┘
                                ▼
┌─────────────────────────────────────────────────────────────────────────┐
│            TRANSFORM (dbt-trino) · ORCHESTRATION (Prefect)              │
│   Bronze ─► Silver (slv_pos_transactions)                               │
│   Silver ─► Gold   (gld_sales_performance, gld_forecast_training)       │
└───────────────────────────────┬─────────────────────────────────────────┘
                                ▼
┌─────────────────────────────────────────────────────────────────────────┐
│            CONSUME · ML · OBSERVE                                        │
│   Superset (dashboards) · Jupyter (notebooks)                           │
│   MLflow (experiment tracking + model registry)                         │
│   Jaeger  (distributed tracing)                                         │
└─────────────────────────────────────────────────────────────────────────┘

---

What's inside

Category	Tool	Role	Host port
Source	pos-db (PostgreSQL)	Transactional database that feeds tap-postgres	5433
Simulation	mock-pos-generator	Generates realistic POS sales into pos-db	—
Source	mock-api (Flask)	Serves market context / competitor pricing for tap-custom-api	5005
Ingestion	Meltano (in Prefect)	EL pipelines (tap-postgres, tap-custom-api → Trino)	—
Orchestration	Prefect	Schedules and runs the EL + dbt + ML flows	4200
Object storage	MinIO	S3-compatible storage (warehouse, mlflow buckets)	19000 / 9001
Table catalog	Nessie	Git-like versioning for Iceberg tables	19120
Metastore	metastore-db (MariaDB)	Metadata store for the catalog	—
Query engine	Trino	Distributed SQL engine over Iceberg tables	8080
Transformation	dbt (dbt-trino)	Silver → Gold transformations stored in the transform/ dir	—
ML tracking	MLflow	Experiment tracking, model registry, artifact store on MinIO	5006
BI / dashboards	Superset	Explore and visualize the Gold layer	8088
Notebooks	Jupyter	Ad-hoc analysis over Trino / MLflow	8888
Observability	Jaeger	Distributed tracing (OTLP) from Trino, Nessie, etc.	16686

---

Data flow

1. Simulate – mock-pos-generator writes random sales to pos-db; mock-api exposes competitor/context data.
2. Extract – Prefect runs the Meltano flows on a schedule (default: every 15 minutes for ingestion).
   • tap-postgres → reads pos_transactions / pos_catalog.
   • tap-custom-api → reads the mock context API.
   • Both load into Trino/Iceberg as the Bronze layer (nessie.bronze.*).
3. Transform – dbt runs the q_commerce project:
   • Silver slv_pos_transactions
   • Gold gld_sales_performance, gld_forecast_training
4. Model – prefect_forecast_modely.py trains an XGBoost restock-forecast model and logs it to MLflow (artifacts stored on MinIO).
5. Visualize / query – Superset builds dashboards on the Gold layer; Jupyter allows ad-hoc exploration.

---

Requirements

• Docker (with Docker Compose v2).
• ~8 GB of free RAM recommended (Trino defaults to a 4 GB heap).

---

Getting started

1. Clone the repository

git clone https://github.com/your-org/OpenDataFramework.git
cd OpenDataFramework/docker-compose

2. Configure the environment

cp .env.example .env
# then edit .env to set your own passwords / secret key

The [REQUIRED] variable is SUPERSET_SECRET_KEY. Generate a strong one with:

python3 -c "import secrets; print(secrets.token_urlsafe(48))"

3. Start the stack

docker compose up -d

First run downloads several images and takes a few minutes. To follow the logs:

docker compose logs -f

4. Create the MLflow database

MLflow stores its metadata in a separate database on prefect-db. Create it once:

docker compose exec prefect-db psql -U prefect -c "CREATE DATABASE mlflow_db;"

If this database doesn't exist, MLflow will fail to start. If you restart the stack from scratch and get an MLflow error, run the command above first.

5. Verify the pipeline

Check that Trino is up:

curl http://localhost:8080/v1/info

Open the Prefect UI (http://localhost:4200) — the ingestion and forecasting flows should be registered and scheduled.

---

Access the services

Service	URL / credentials
Prefect	http://localhost:4200
MinIO	http://localhost:9001 (console) — MINIO_ROOT_USER / MINIO_ROOT_PASSWORD
Trino	http://localhost:8080
Superset	http://localhost:8088 (see setup)
MLflow	http://localhost:5006
Jupyter	http://localhost:8888 — token from JUPYTER_TOKEN
Jaeger	http://localhost:16686
mock-api	http://localhost:5005/api/v1/context

---

One-time setup (Superset)

Superset ships uninitialized. The first time you start the stack, run:

# Within the superset container
docker compose exec superset superset db upgrade
docker compose exec superset superset fab create-admin
docker compose exec superset superset init

# Optional: load example dashboards
docker compose exec superset superset load_examples

# Install the Trino dialect so Superset can query it
docker compose exec --user root superset pip install "trino[sqlalchemy]"

Then restart Superset (docker compose restart superset) and connect it to Trino:

• Host: trino, Port: 8080, Database: nessie.

---

Dagster / other pipelines

The primary orchestrator in this stack is Prefect. The repository also contained an experimental Dagster pipeline (Zonaprop scraping) which has been archived and removed to keep the published project focused. If you need it, it is recoverable from the git history.

---

Configuration

All runtime configuration lives in docker-compose/.env (copy from .env.example). See the .env.example file for the full list and purpose of every variable.

Note on MinIO ↔ Trino credentials: Trino's configuration file (trino_config/catalog/nessie.properties) does not support environment-variable interpolation, so the S3 keys there are static. If you change MINIO_ROOT_USER / MINIO_ROOT_PASSWORD, you must update the matching s3.aws-access-key / s3.aws-secret-key values in that file.

Default development credentials

Component	Username	Password
pos-db	admin	password
MinIO	admin	password
Prefect DB	prefect	prefect

These are development-only defaults meant to make local setup frictionless. Change them before any deployment beyond a local sandbox.

---

Security

See SECURITY.md for the full policy. Highlights:

• Secrets are not committed. Credentials are injected via a git-ignored .env file (a .env.example template is provided).
• The MinIO warehouse bucket is not public. Both warehouse and mlflow buckets are created private on startup.
• Only the intended service ports are exposed to the host; backing stores (Postgres/MinIO API) are reachable only inside the internal Docker network (except the documented UI/API ports).
• Provided credentials are local-development placeholders — rotate them for any shared or production use.

If you discover a security issue, please report it privately (see SECURITY.md).

---

Limitations & notes

• Demo-oriented credentials and scheduling. The stack is a teaching/reference architecture, not a hardened production platform. Treat it as such.
• Prefect bootstrapping. The Prefect container installs dependencies and registers flows inline on startup — a deliberate simplification. In production, split the server from flow deployments and pre-build images.
• Static Trino S3 keys. Documented above; kept static because Trino property files can't interpolate env vars.
• Tracing is wired for Trino/Nessie via Jaeger over OTLP.

---

Contributing

Contributions are welcome. Please read CONTRIBUTING.md before opening an issue or pull request. All contributions are subject to the Apache License 2.0.

---

License

This project is licensed under the Apache License 2.0.

ef2d051 (Initial commit: open data framework)