Issue 40 validate example data

.github/workflows/validate-data.yml

@@ -0,0 +1,24 @@
+name: Validate Example TIDES Data
+
+on:
+  push:
+    paths:
+      - 'data/*/TIDES/*'
+      - 'spec/*'
+  pull_request:
+    paths:
+      - 'data/*/TIDES/*'
+      - 'spec/*'
+  workflow_dispatch:
+  create:
+
+jobs:
+  validate:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v2
+      - name: Validate data
+        uses: frictionlessdata/repository@v2
+        with:
+          packages: "data/*/TIDES/datapackage.json"

.gitignore

@@ -1,6 +1,13 @@
 .DS_Store*
 .vscode*
+/__pycache__/*
 .env
 /venv*
-/site
-/__pycache__
+/site/*
+# pages that are copied in from main repo
+/docs/CONTRIBUTING.md
+/docs/CODE_OF_CONDUCT.md
+/docs/README.md
+# pages that are generated from templates
+/docs/tables.md
+/docs/architecture.md

.markdownlint.yaml

@@ -8,5 +8,11 @@ MD007:
 # Remove line length limit
 MD013: false
 
+# Remove "fenced code blocks should have language specified"
+MD040: false
+
+# Remove "no inline HTML"
+MD033: false
+
 # Remove "no emphasis as heading"
 MD036: false

README.md

@@ -6,12 +6,43 @@ This repository provides data schemas and tools to support the access, managemen
 
 ## TIDES specification
 
-The TIDES specification is maintained in the `/spec` sub-directory as a series of JSON tables compatible with the [frictionless data](https://specs.frictionlessdata.io/table-schema/) table schema standards.
+The TIDES specification is maintained in the `/spec` sub-directory as a series of JSON tables compatible with the [frictionless data](https://specs.frictionlessdata.io/table-schema/) table schema and [data package](https://specs.frictionlessdata.io/data-package) standards.
 Human-friendlier documentation is auto-generated and available at:
 
 - [Architecture](architecture.md)
 - [Table Schemas](tables.md)
 
+## Example Data
+
+Example data can be found in the `/data` directory, with one directory for each example.  
+
+## Validating TIDES data
+
+TIDES data with a valid [`datapackage.json`](#data-package) can be easily validated using the [frictionless framework](https://framework.frictionlessdata.io/), which can be installed and invoke as follows:
+
+```bash
+pip install frictionless
+frictionless validate path/to/your/datapackage.json
+```
+
+### Data Package
+
+To validate a package of TIDES data, you must add a frictionless-compliant [`datapackage.json`](https://specs.frictionlessdata.io/data-package/) alongside your data which describes which files should be validated to which schemas.  Most of this can be copied from [`/data/template/TIDES/datapackage.json`](https://raw.githubusercontent.com/TIDES-transit/TIDES/main/data/template/TIDES/datapackage.json).
+
+Once this is created, mapping the data files to the schema, simply run:
+
+```sh
+frictionless validate datapackage.json
+```
+
+### Specific files
+
+Specific files can be validated by running the frictionless framework against them and their corresponding schemas as follows:
+
+```sh
+frictionless validate vehicles.csv https://raw.githubusercontent.com/TIDES-transit/TIDES/main/spec/schema.vehicles.json
+```
+
 ## Contributing to TIDES
 
 Those who want to help with the development of the TIDES specification should review the guidance in <CONTRIBUTING.md>.

contributors.md

@@ -1 +1 @@
-# Contributors
+# Contributors

docs/index.md

@@ -1 +1,3 @@
-{{ include_file('README.md', downshift_h1= False) }}
+# TIDES Transit Specification Suite
+
+{{ include_file('README.md', start_line = 2, downshift_h1= False) }}

docs/samples.md

@@ -0,0 +1,9 @@
+# Sample Data
+
+Sample data can be found in the `/samples` directory, with one directory for each data sample.  
+
+{{ include_file('samples/README.md')}}
+
+## Data List
+
+{{ list_samples('samples') }}

main.py

@@ -17,6 +17,7 @@
     "tables.md": "tables",
 }
 
+
 def define_env(env):
     """
     This is the hook for defining variables, macros and filters
@@ -26,15 +27,46 @@ def define_env(env):
     """
 
     @env.macro
-    def include_file(filename: str, downshift_h1 = True, start_line: int = 0, end_line: int = None):
+    def list_samples(data_dir: str) -> str:
+        """Outputs a simple list of the directories in a folder in markdown.
+        Args:
+            data_dir (str):directory to search in
+        Returns:
+            str: markdown-formatted list
+        """
+
+        fields = ["Sample", "Agency", "Resources", "Vendors"]
+        df = pd.DataFrame([], columns=fields)
+
+        data_dir = os.path.join(env.project_dir, data_dir)
+        samples = [
+            d for d in os.listdir(data_dir) if os.path.isdir(os.path.join(data_dir, d))
+        ]
+
+        for s in samples:
+            df = pd.concat(
+                [
+                    df,
+                    pd.DataFrame(
+                        sample_metadata_as_dict(os.path.join(data_dir, s)), index=[0]
+                    ),
+                ]
+            )
+        return df.to_markdown(index=False)
+
+    @env.macro
+    def include_file(
+        filename: str, downshift_h1=True, start_line: int = 0, end_line: int = None
+    ):
         """
         Include a file, optionally indicating start_line and end_line.
 
         Will create redirects if specified in FIND_REPLACE in main.py.
 
         args:
             filename: file to include, relative to the top directory of the documentation project.
-            downshift_h1: If true, will downshift headings by 1 if h1 heading found. Defaults to True. 
+            downshift_h1: If true, will downshift headings by 1 if h1 heading found.
+                Defaults to True.
             start_line (Optional): if included, will start including the file from this line
                 (indexed to 0)
             end_line (Optional): if included, will stop including at this line (indexed to 0)
@@ -53,20 +85,19 @@ def include_file(filename: str, downshift_h1 = True, start_line: int = 0, end_li
             4: re.compile(r"(#{4}\s)(.*)"),
             5: re.compile(r"(#{5}\s)(.*)"),
         }
-        print(f"???before downshifting! {full_filename}")
+
         if md_heading_re[1].search(content) and downshift_h1:
-            print("!!!downshifting!")
-            content = re.sub(md_heading_re[5],r'#\1\2',content)
-            content = re.sub(md_heading_re[4],r'#\1\2',content)
-            content = re.sub(md_heading_re[3],r'#\1\2',content)
-            content = re.sub(md_heading_re[2],r'#\1\2',content)
-            content = re.sub(md_heading_re[1],r'#\1\2',content)
+            content = re.sub(md_heading_re[5], r"#\1\2", content)
+            content = re.sub(md_heading_re[4], r"#\1\2", content)
+            content = re.sub(md_heading_re[3], r"#\1\2", content)
+            content = re.sub(md_heading_re[2], r"#\1\2", content)
+            content = re.sub(md_heading_re[1], r"#\1\2", content)
 
         _filenamebase = env.page.file.url
         for _find, _replace in FIND_REPLACE.items():
             if _filenamebase in _replace:
                 _replace = _replace.replace(_filenamebase, "")
-                
+
             content = content.replace(_find, _replace)
         return content
 
@@ -95,7 +126,7 @@ def frictionless_spec(
             columns=["fullpath", "fullpath_schema", "path", "schema", "name"]
         ).reset_index()
         spec_df["name"] = spec_df["name"].apply(
-            lambda x: f"[`{x}`](tables.md#{x})".replace("_","-")
+            lambda x: f"[`{x}`](tables.md#{x})".replace("_", "-")
         )
 
         return spec_df.to_markdown(index=False)
@@ -133,6 +164,40 @@ def frictionless_schemas(
 TABLE_TYPES = ["Event", "Summary", "Supporting"]
 
 
+def sample_metadata_as_dict(dir: str) -> dict:
+    """Reads datapackage and translates key metadata to dictionary.
+
+    Args:
+        dir (str): fully qualified directory for sample data
+
+    Returns:
+        dict: dictionary with data pacakge name, agency, resources and vendors.
+    """
+    dp_filename = os.path.join(dir, "TIDES", "datapackage.json")
+    if not os.path.isfile(dp_filename):
+        raise ValueError(f"Can't find datapackage file:\n   {dp_filename}.")
+    with open(dp_filename, "r") as dp_file:
+        dp = json.loads(dp_file.read())
+
+        _vendors = [
+            s.get("vendor", None) for r in dp["resources"] for s in r.get("sources", [])
+        ]
+        _vendors = list(set(_vendors) - set([None]))
+
+        return {
+            "Sample": _to_sample_readme_link(dp["name"], dir),
+            "Agency": dp["agency"],
+            "Resources": "<ul><li>`"
+            + "`</li><li>`".join([r["name"] for r in dp["resources"]])
+            + "`</li></ul>",
+            "Vendors": "<ul><li>`" + "`</li><li>`".join(_vendors) + "`</li></ul>",
+        }
+
+
+def _to_sample_readme_link(sample_name, folder_dir):
+    return f"[{sample_name.capitalize()}]({folder_dir}/README.md)"
+
+
 def _document_frictionless_schema(schema_filename: str) -> dict:
     """Documents a single frictionless schema.
 

mkdocs.yml

@@ -34,8 +34,19 @@ plugins:
         theme: |
             ^(window.matchMedia && window.matchMedia('(prefers-color-scheme: dark)').matches) ? 'dark' : 'light'
         securityLevel: 'loose'
+  - mike:
+      version_selector: true
+  - redirects:
+      redirect_maps: #original relative to /docs : redirect target
+        '../CODE_OF_CONDUCT.md': 'CODE_OF_CONDUCT.md'
+        '../CONTRIBUTING.md': 'CONTRIBUTING.md'
+        '../contributors.md': 'contributors.md'
   - search
 
+extra:
+  version:
+    provider: mike
+
 watch:
   - CONTRIBUTING.md
   - CODE_OF_CONDUCT.md
@@ -72,4 +83,5 @@ nav:
   - Home: index.md
   - Architecture: architecture.md
   - Table Schemas: tables.md
+  - Sample Data: samples.md
   - Development: development.md

requirements.txt

@@ -1,2 +1,2 @@
+-r docs/requirements.txt
 pre-commit
--r docs/requirements.txt

samples/README.md

@@ -0,0 +1,83 @@
+# Data Directory Organization
+
+Each TIDES Data Package example should follow the following directory structure, consistent with the structure of the [Frictionless Data Package specification](https://specs.frictionlessdata.io/data-package/), including:
+
+```
+unique-example-name
+  \TIDES     # data to be validated against the TIDES specification
+  \TIDES\datapackages.json # data package metadata per https://specs.frictionlessdata.io/data-package/
+  \raw      # data which the agency uses to create TIDES data
+  \scripts  # scripts used to transform raw --> TIDES
+```
+
+## Adding Examples
+
+We encourage the addition of examples, but please follow the following guidelines:
+
+1. *No large files* This isn't the place to store your data, rather to document some minimal examples.  The recommended size is 100-1000 records per file, more if absolutely required to reproduce an issue with the spec.  All individual files should be well under 50 MB.  
+2. *Include Metadata* as specified in [`datapackage.json`](#data-package).
+3. *Include a README.md* in the base folder of your example with an overview so that it can be included in the documentation.
+
+## Data Package
+
+TIDES data packages must include a [`datapackage.json`](https://specs.frictionlessdata.io/data-package/).  Key information to include in [`datapackage.json`](https://specs.frictionlessdata.io/data-package/) includes:
+
+| **Field** | **Description** | **Required** |
+| --------- | --------------- | ------------ |
+| `title` | A human-readable title. | Required |
+| `name` | Identifier string as a URL-friendly slug. | Required |
+| `description` | Short description of data package. | Recommended |
+| `agency` | Transit agency name. | Recommended |
+| `ntd_id` | ID for the National Transit Database. | Recommended |
+| `profile` | Should be `tabular-data-package` | Required |
+| `licenses` | Should be `[{"name": "Apache-2.0"}]` to be consistent with this repository | Required |
+| `contributors` | Array of data contributors `[{"title": "My Name", "github": "my_handle", "email": "me@myself.com"}]` | Recommended |
+| `maintainers` |  Array of data maintainers `[{"title": "My Name", "github": "my_handle", "email": "me@myself.com"}]` | Recommended |
+| `resources` | Array of data files included in your package, formated as a [`tabular-data-resource`](#data-resource)| Required |
+
+### Template
+
+A `datapackage.json` template is available at [`/data/template/TIDES/datapackage.json`](https://raw.githubusercontent.com/TIDES-transit/TIDES/main/data/template/TIDES/datapackage.json).
+
+Once `datapackage.json` is created for your data, you can easily conduct [data validation](#data-validation) using a variet of tools.
+
+### Data Resource
+
+Key fields for each [`tabular-data-resource`](https://specs.frictionlessdata.io/tabular-data-resource/) are as follows:
+
+| **Field** | **Description** | **Required** |
+| --------- | --------------- | ------------ |
+| `name` | Short sluggable name used to refer to data in this file. | Required |
+| `path` | Path of the data resource file relative to the `datapackage.json` | Required |
+| `schema` | Data schema to use to valdiate the data resource to | Required |
+| `sources` | Array of data sources formatted as a [`source`](#data-source) | Recommended |
+
+### Data Source
+
+| **Field** | **Description** | **Required** |
+| --------- | --------------- | ------------ |
+| `title` | Description of the data source. | Required |
+| `component` | What technology component was used to generate this data (directly or indirectly)? Examples include `AVL`, `APC`, `AFC`, etc.  | Recommended |
+| `product` | What product was used to generate this data (directly or indirectly)? | Recommended |
+| `vendor` | What company makes this product? | Recommended |
+
+## Data validation
+
+Data with a valid [`datapackage.json`](#data-package) can be easily validated using the [frictionless framework](https://framework.frictionlessdata.io/), which can be installed and invoke as follows:
+
+```bash
+pip install frictionless
+frictionless validate path/to/your/datapackage.json
+```
+
+### Specific files
+
+Specific files can be validated by running the frictionless framework against them and their corresponding schemas as follows:
+
+```sh
+frictionless validate vehicles.csv https://raw.githubusercontent.com/TIDES-transit/TIDES/main/spec/schema.vehicles.json
+```
+
+### Continuous Data Validation
+
+Example data in the `\TIDES` subdirectories is validated upon a push action to the main repository according to the `TIDES` schema posted to the `main` branch.

samples/template/README.md

@@ -0,0 +1,16 @@
+# Example TIDES Data Package
+
+Template directory for example scaffolding and helper scripts.
+
+## Scripts for Generating Template Data
+
+`scripts\create_template.py` has some template code which can help with the following
+
+- `write_schema_examples()`: will generate blank csvs according to the TIDES schema
+- `write_datapackage()` will generate a datapackage.json based on the TIDES schemas and a set of defaults specified in th script.
+
+To run both (note this replaces the existing files in the directory)
+
+```bash
+python samples/template/scripts/create_template.py
+```