Issue 40 validate example data

.github/workflows/validate-data.yml

@@ -0,0 +1,24 @@
+name: Validate Example TIDES Data
+
+on:
+  push:
+    paths:
+      - 'data/*/data/*'
+      - 'spec/*'
+  pull_request:
+    paths:
+      - 'data/*/data/*'
+      - 'spec/*'
+  workflow_dispatch:
+  create:
+
+jobs:
+  validate:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v2
+      - name: Validate data
+        uses: frictionlessdata/repository@v2
+        with:
+          packages: "data/*/data/datapackage.json"

.gitignore

@@ -1,8 +1,9 @@
 .DS_Store*
 .vscode*
+/__pycache__/*
 .env
 /venv*
-/site
+/site/*
 # pages that are copied in from main repo
 /docs/CONTRIBUTING.md
 /docs/CODE_OF_CONDUCT.md

README.md

@@ -12,6 +12,38 @@ Human-friendlier documentation is auto-generated and available at:
 - [Architecture](architecture.md)
 - [Table Schemas](tables.md)
 
+## Example Data
+
+[![Example Data](https://github.com/TIDES-transit/TIDES/actions/workflows/validate-data.yaml/badge.svg)](https://repository.frictionlessdata.io/pages/dashboard.html?user=TIDES-transit&repo=TIDES&flow=validate-data)
+
+Example data can be found in the `/data` directory, with one directory for each example.  
+
+Example data in the `/TIDES` subdirectories is validated upon a push action to the main repository according to the `TIDES` schema contained in the respective repository commit.
+
+## Validating TIDES data
+
+The easiest way to validate data to the TIDES specifications is to use the frictionless framework, which can be installed from the command line using:
+
+```sh
+pip install frictionless
+```
+
+### Data Package
+
+To validate a package of TIDES data, you must add a frictionless-compliant [`datapackage.json`](https://specs.frictionlessdata.io/data-package/) alongside your data which describes which files should be validated to which schemas.  Most of this can be copied from [`/data/example/data/datapackage.json`](https://raw.githubusercontent.com/TIDES-transit/TIDES/main/data/example/data/datapackage.json).
+
+Once this is created, mapping the data files to the schema, simply run:
+
+```sh
+frictionless validate datapackage.json
+```
+
+### Specific files
+
+```sh
+frictionless validate datapackage.json
+```
+
 ## Contributing to TIDES
 
 Those who want to help with the development of the TIDES specification should review the guidance in the [CONTRIBUTING.md](CONTRIBUTING.md) file.

contributors.md

@@ -1 +1 @@
-# Contributors to the TIDES Suite
+# Contributors to the TIDES Suite

data/README.md

@@ -0,0 +1,24 @@
+# Data Directory Organization
+
+Each TIDES Data Package example should follow the following directory structure, consistent with the structure of the [Frictionless Data Package specification](https://specs.frictionlessdata.io/data-package/), including:
+
+```
+unique-example-name
+  \data     # data to be validated against the TIDES specification
+  \data\datapackages.json # data package metadata per https://specs.frictionlessdata.io/data-package/
+  \raw      # data which the agency uses to create TIDES data
+  \scripts  # scripts used to transform raw --> TIDES
+```
+
+## Data validation
+
+Data with a valid `datapackage.json` can be easily validated using the [frictionless framework](https://framework.frictionlessdata.io/), which can be installed and invoke as follows:
+
+```bash
+pip install frictionless
+frictionless validate path/to/your/datapackage.json
+```
+
+### Continuous Data Validation
+
+Example data in the `\data` subdirectories is validated upon a push action to the main repository according to the `TIDES` schema posted to the `main` branch.

data/example/README.md

@@ -0,0 +1,16 @@
+# Example TIDES Data Package
+
+Template directory for example scaffolding and helper scripts.
+
+## Scripts for Generating Data
+
+`scripts\create_example.py` has some template code which can help with the following
+
+- `write_schema_examples()`: will generate blank csvs according to the TIDES schema
+- `write_datapackage()` will generate a datapackage.json based on the TIDES schemas and a set of defaults specified in th script.
+
+To run both (note this replaces the existing files in the directory)
+
+```bash
+python data/example/scripts/create_example.py
+```

data/example/data/datapackage.json

@@ -0,0 +1,79 @@
+{
+    "name": "Example TIDES Data Package",
+    "title": "example",
+    "profile": "tabular-data-package",
+    "licenses": [
+        {
+            "name": "Apache-2.0"
+        }
+    ],
+    "contributors": [
+        {
+            "title": "My Name",
+            "email": "me@myself.com"
+        }
+    ],
+    "maintainers": [
+        {
+            "title": "Another Name",
+            "email": "another@myself.com"
+        }
+    ],
+    "resources": [
+        {
+            "name": "devices",
+            "path": "devices.csv",
+            "schema": "https://raw.githubusercontent.com/TIDES-transit/TIDES/main/spec/devices.schema.json"
+        },
+        {
+            "name": "vehicle_locations",
+            "path": "vehicle_locations.csv",
+            "schema": "https://raw.githubusercontent.com/TIDES-transit/TIDES/main/spec/vehicle_locations.schema.json"
+        },
+        {
+            "name": "fare_transactions",
+            "path": "fare_transactions.csv",
+            "schema": "https://raw.githubusercontent.com/TIDES-transit/TIDES/main/spec/fare_transactions.schema.json"
+        },
+        {
+            "name": "train_cars",
+            "path": "train_cars.csv",
+            "schema": "https://raw.githubusercontent.com/TIDES-transit/TIDES/main/spec/train_cars.schema.json"
+        },
+        {
+            "name": "operators",
+            "path": "operators.csv",
+            "schema": "https://raw.githubusercontent.com/TIDES-transit/TIDES/main/spec/operators.schema.json"
+        },
+        {
+            "name": "stop_visits",
+            "path": "stop_visits.csv",
+            "schema": "https://raw.githubusercontent.com/TIDES-transit/TIDES/main/spec/stop_visits.schema.json"
+        },
+        {
+            "name": "vehicle_train_cars",
+            "path": "vehicle_train_cars.csv",
+            "schema": "https://raw.githubusercontent.com/TIDES-transit/TIDES/main/spec/vehicle_train_cars.schema.json"
+        },
+        {
+            "name": "vehicles",
+            "path": "vehicles.csv",
+            "schema": "https://raw.githubusercontent.com/TIDES-transit/TIDES/main/spec/vehicles.schema.json"
+        },
+        {
+            "name": "trips_performed",
+            "path": "trips_performed.csv",
+            "schema": "https://raw.githubusercontent.com/TIDES-transit/TIDES/main/spec/trips_performed.schema.json"
+        },
+        {
+            "name": "station_activities",
+            "path": "station_activities.csv",
+            "schema": "https://raw.githubusercontent.com/TIDES-transit/TIDES/main/spec/station_activities.schema.json"
+        },
+        {
+            "name": "passenger_events",
+            "path": "passenger_events.csv",
+            "schema": "https://raw.githubusercontent.com/TIDES-transit/TIDES/main/spec/passenger_events.schema.json"
+        }
+    ]
+}

data/example/data/devices.csv

@@ -0,0 +1 @@
+device_id,stop_id,vehicle_id,train_car_id,device_type,device_vendor,device_model,location

data/example/data/fare_transactions.csv

@@ -0,0 +1 @@
+transaction_id,date,timestamp,amount,currency_type,fare_action,trip_id_performed,stop_sequence,vehicle_id,device_id,fare_id,stop_id,group_size,media_type,rider_category,fare_product,fare_period,fare_capped,fare_media_id,fare_media_id_purchased,balance

data/example/data/operators.csv

@@ -0,0 +1 @@
+operator_id

data/example/data/passenger_events.csv

@@ -0,0 +1 @@
+passenger_event_id,date,timestamp,trip_id_performed,stop_sequence,event_type,vehicle_id,device_id,train_car_id,stop_id

data/example/data/station_activities.csv

@@ -0,0 +1 @@
+date,stop_id,time_period_start,time_period_end,time_period_category,total_entries,total_exits,number_of_transactions,transaction_revenue_cash,transaction_revenue_smartcard,transaction_revenue_magcard,transaction_revenue_bankcard,transaction_revenue_nfc,transaction_revenue_optical,transaction_revenue_operator,transaction_revenue_other,transaction_count_cash,transaction_count_smartcard,transaction_count_magcard,transaction_count_bankcard,transaction_count_nfc,transaction_count_optical,transaction_count_operator,transaction_count_other,bike_entries,bike_exits,ramp_entries,ramp_exits

data/example/data/stop_visits.csv

@@ -0,0 +1 @@
+date,trip_id_performed,stop_sequence,vehicle_id,dwell,stop_id,checkpoint,schedule_arrival_time,schedule_departure_time,actual_arrival_time,actual_departure_time,distance,boarding_1,alighting_1,boarding_2,alighting_2,load,door_open,door_close,door_status,ramp_deployed_time,ramp_failure,kneel_deployed_time,lift_deployed_time,bike_rack_deployed,bike_load,revenue,number_of_transactions,transaction_revenue_cash,transaction_revenue_smartcard,transaction_revenue_magcard,transaction_revenue_bankcard,transaction_revenue_nfc,transaction_revenue_optical,transaction_revenue_operator,transaction_revenue_other,transaction_count_cash,transaction_count_smartcard,transaction_count_magcard,transaction_count_bankcard,transaction_count_nfc,transaction_count_optical,transaction_count_operator,transaction_count_other,schedule_relationship

data/example/data/train_cars.csv

@@ -0,0 +1 @@
+train_car_id,model_name,facility_name,capacity_seated,wheelchair_capacity,bike_capacity,bike_rack,capacity_standing,train_car_type

data/example/data/trips_performed.csv

@@ -0,0 +1 @@
+date,trip_id_performed,vehicle_id,trip_id_scheduled,route_id,route_type,shape_id,direction_id,operator_id,block_id,trip_start_stop_id,trip_end_stop_id,schedule_trip_start,schedule_trip_end,actual_trip_start,actual_trip_end,in_service,schedule_relationship

data/example/data/vehicle_locations.csv

@@ -0,0 +1 @@
+location_ping_id,date,timestamp,trip_id_performed,stop_sequence,vehicle_id,device_id,stop_id,current_status,latitude,longitude,gps_quality,heading,speed,odometer,schedule_deviation,headway_deviation,in_service,schedule_relationship

data/example/data/vehicle_train_cars.csv

@@ -0,0 +1 @@
+vehicle_id,train_car_id,order,operator_id

data/example/data/vehicles.csv

@@ -0,0 +1 @@
+vehicle_id,vehicle_start,vehicle_end,model_name,facility_name,capacity_seated,wheelchair_capacity,capacity_bike,bike_rack,capacity_standing

data/example/scripts/create_example.py

@@ -0,0 +1,121 @@
+#!/usr/bin/env python3
+
+import glob
+import json
+import os
+import pathlib
+
+
+EXAMPLE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+BASE_REPO_DIR = os.path.dirname(os.path.dirname(EXAMPLE_DIR))
+TIDES_SPEC = os.path.join(BASE_REPO_DIR, "spec")
+SCHEMAS = glob.glob(os.path.join(TIDES_SPEC, "**/*.schema.json"), recursive=True)
+SCHEMAS_LOC = "https://raw.githubusercontent.com/TIDES-transit/TIDES/main/spec/"
+
+# DATAPACKAGE.JSON INFORMATION
+# Per https://specs.frictionlessdata.io/data-package/
+TITLE = "example"
+NAME = "Example TIDES Data Package"
+PROFILE = "tabular-data-package"
+LICENSES = [{"name": "Apache-2.0"}]
+SOURCES = [{"title": "Generated from /scripts/create_example.py"}]
+CONTRIBUTORS = [{"title": "My Name", "email": "me@myself.com"}]
+MAINTAINERS = [{"title": "Another Name", "email": "another@myself.com"}]
+DATAPACKAGE_TEMPLATE = {
+    "name": NAME,
+    "title": TITLE,
+    "profile": PROFILE,
+    "licenses": LICENSES,
+    "contributors": CONTRIBUTORS,
+    "maintainers": MAINTAINERS,
+    "resources": [],
+}
+
+
+def write_schema_examples(
+    out_dir: str,
+    schemas: list = SCHEMAS,
+) -> None:
+    """Write blank csvs to out_dir with headings for for each schema in list.
+
+    Args:
+        out_dir (str): Where blank csvs are written.
+        schemas (list, optional): List of schemas to generate blank csvs for. Defaults to SCHEMAS.
+    """
+    for s in schemas:
+        write_csv_for_schema(s, out_dir)
+
+
+def write_datapackage(
+    out_dir: str,
+    schemas: list = SCHEMAS,
+    template: dict = DATAPACKAGE_TEMPLATE,
+) -> None:
+    """Write a datapackage.json file in Frictionless data-package format based on list of schemas.
+
+    Args:
+        out_dir (str): directory where datapackage.json is written.
+        schemas (list, optional): List of schemas to add to resources list . Defaults to SCHEMAS.
+    """
+    datapackage = template
+    datapackage["resources"] = [schema_to_resources(s) for s in schemas]
+
+    out_filename = os.path.join(out_dir, "datapackage.json")
+    with open(out_filename, "w") as outfile:
+        outfile.write(json.dumps(datapackage, indent=4))
+        print(f"Wrote {out_filename}")
+    json.dumps
+
+
+def schema_to_resources(schema_filename: str) -> dict:
+    """Transform a schema filename into a frictionless resource for listing in datapackage.json
+
+    Args:
+        schema_filename (str): Schema file in frictionless format.
+
+    Returns:
+        dict: object consistent with frictionless data resource specification
+    """
+    schema_filename = pathlib.Path(schema_filename)
+    name = schema_filename.stem.split(".")[0]
+    path = name + ".csv"
+    schema_loc = SCHEMAS_LOC + name + ".schema.json"
+
+    return {"name": name, "path": path, "schema": schema_loc}
+
+
+def write_csv_for_schema(
+    schema_filename: str,
+    out_dir: str,
+) -> None:
+    """Creates blank csvs which comply with a schema.
+
+    Args:
+        schema_filename (str): Filename with the Frictionless data schema
+        out_dir (str): Where the csv will be written
+    """
+    schema = read_schema(schema_filename)
+    fields = [s["name"] for s in schema["fields"]]
+    schema_name = pathlib.Path(schema_filename).stem.split(".")[0]
+    out_filename = os.path.join(out_dir, schema_name + ".csv")
+    with open(out_filename, "w") as outfile:
+        outfile.write(",".join(fields))
+        print(f"Wrote {out_filename}")
+
+
+def read_schema(schema_file: str) -> dict:
+    """
+    Reads in schema from schema json file and returns as dictionary.
+
+    Args:
+        schema_file: File location of the schema json file.
+    Returns: The schema as a dictionary
+    """
+    with open(schema_file, encoding="utf-8") as f:
+        schema = json.load(f)
+    return schema
+
+
+if __name__ == "__main__":
+    write_schema_examples(out_dir=os.path.join(EXAMPLE_DIR, "data"))
+    write_datapackage(out_dir=os.path.join(EXAMPLE_DIR, "data"))

data/example/scripts/requirements.txt

@@ -0,0 +1,2 @@
+frictionless
+frictionless[pandas]

docs/development.md

@@ -0,0 +1,7 @@
+# Development
+
+{{ include_file('CONTRIBUTING.md') }}
+
+{{ include_file('CODE_OF_CONDUCT.md') }}
+
+{{ include_file('contributors.md') }}

docs/document.py

@@ -359,7 +359,3 @@ def repo_to_docs(
 if __name__ == "__main__":
     document_spec()
     document_schemas()
-    repo_to_docs("README.md")
-    repo_to_docs("CONTRIBUTING.md")
-    repo_to_docs("contributors.md")
-    repo_to_docs("CODE_OF_CONDUCT.md")

docs/examples.md

@@ -0,0 +1,9 @@
+# Example Data
+
+Example data can be found in the `/data` directory, with one directory for each example.  
+
+{{ include_file('data/README.md')}}
+
+## Data List
+
+{{ list_examples('data') }}

docs/index.md

@@ -0,0 +1 @@
+{{ include_file('README.md') }}

docs/requirements.txt

@@ -8,4 +8,4 @@ mkdocs-material
 mkdocs-mermaid2-plugin
 mkdocs-redirects
 pandas
-tabulate
+tabulate