Skip to content

Add documentation of sample data structure #100

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
e-lo wants to merge 25 commits into from
Closed

Add documentation of sample data structure #100

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
a24f119
Merge content from main repo using Macro include_files
e-lo Sep 26, 2022
74b02d9
[ actually include the macros ]
e-lo Sep 26, 2022
9db30a9
Add Examples
e-lo Sep 28, 2022
7fdddfe
first hack at creating random example data
e-lo Sep 28, 2022
dba9e26
Initial example data.
e-lo Oct 7, 2022
eaa009a
Update Based on Comments
e-lo Oct 7, 2022
af15c9a
update validate-data workflow to match datapackage.json
e-lo Oct 7, 2022
c0acbd1
Merge branch 'main' into issue-40-validate-example-data
e-lo Oct 7, 2022
bcc5a0f
Merge upstream into issue-40-validate-example-data
e-lo Oct 7, 2022
3532b15
Clean up merge fails
e-lo Oct 7, 2022
53a05b9
Responses to comments
e-lo Oct 7, 2022
3aadfea
Fix validate CLI call
e-lo Oct 10, 2022
9661da0
Responses to comments
e-lo Oct 10, 2022
5332619
Removed pandas from markdown table writing
e-lo Oct 10, 2022
e05cd3c
pre-commit
e-lo Oct 10, 2022
053e986
Delete validate-data.yml
e-lo Jan 7, 2023
7981308
Remove template directory
e-lo Jan 7, 2023
b6c52e5
remove template folder
e-lo Jan 7, 2023
20b1a97
Update sample creation instructions to remove template and CI refs
e-lo Jan 7, 2023
36bb261
fix typo; add --schema-sync
e-lo Jan 30, 2023
67015af
fix schema filename
e-lo Jan 30, 2023
be1d112
Addressed Joey's comments
e-lo Jan 30, 2023
e64f03d
support top-level sources
e-lo Mar 27, 2023
3f1a357
responded to definition consistency requests
e-lo Mar 27, 2023
a8565b9
Merge branch 'main' into pr1-sample-data
e-lo Jun 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Sample data can be found in the `/samples` directory, with one directory for eac

## Validating TIDES data

TIDES data with a valid [`datapackage.json`](#data-package) can be easily validated using the [frictionless framework](https://framework.frictionlessdata.io/), which can be installed and invoke as follows:
TIDES data with a valid [`datapackage.json`](#data-package) can be easily validated using the [frictionless framework](https://framework.frictionlessdata.io/), which can be installed and invoked as follows:

```bash
pip install frictionless
Expand All @@ -40,7 +40,7 @@ frictionless validate datapackage.json
Specific files can be validated by running the frictionless framework against them and their corresponding schemas as follows:

```sh
frictionless validate vehicles.csv --schema https://raw.githubusercontent.com/TIDES-transit/TIDES/main/spec/schema.vehicles.json
frictionless validate vehicles.csv --schema https://raw.githubusercontent.com/TIDES-transit/TIDES/main/spec/vehicles.schema.json --schema-sync
```

## Contributing to TIDES
Expand Down
2 changes: 2 additions & 0 deletions main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,8 @@ def datapackage_to_row(dir: str, fields: list) -> str:
_vendors = [
Comment thread
e-lo marked this conversation as resolved.
s.get("vendor", None) for r in dp["resources"] for s in r.get("sources", [])
]
_vendors += dp.get("vendor",[])

_vendors = list(set(_vendors) - set([None]))

_md_cells = {
Expand Down
11 changes: 6 additions & 5 deletions samples/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ unique-example-name

We encourage the addition of examples, but please follow the following guidelines:

1. *No large files* This isn't the place to store your data, rather to document some minimal examples. The recommended size is 100-1000 records per file, more if absolutely required to reproduce an issue with the spec. All individual files should be well under 50 MB.
1. *No large files* This isn't the place to store your data, rather to document some minimal examples. The recommended size is 100-1000 records per file, more if absolutely required to reproduce an issue with the spec. The whole data package should be well under 50 MB.
2. *Include Metadata* as specified in [`datapackage.json`](#data-package).
3. *Include a README.md* in the base folder of your example with an overview so that it can be included in the documentation.

Expand All @@ -25,7 +25,7 @@ TIDES data packages must include a [`datapackage.json`](https://specs.frictionle
| **Field** | **Description** | **Required** |
| --------- | --------------- | ------------ |
| `title` | A human-readable title. | Required |
| `name` | Identifier string as a URL-friendly slug. | Required |
| `name` | Short identifier string as a [URL-friendly slug](https://en.wikipedia.org/wiki/Clean_URL#Slug). | Recommended |

@botanize botanize Jan 30, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same question as line 44 below, plus these descriptions should match

@e-lo e-lo Mar 27, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made more consistent.

| `description` | Short description of data package. | Recommended |
| `agency` | Transit agency name. | Recommended |
| `ntd_id` | ID for the National Transit Database. | Recommended |
Expand All @@ -41,7 +41,7 @@ Key fields for each [`tabular-data-resource`](https://specs.frictionlessdata.io/

| **Field** | **Description** | **Required** |
| --------- | --------------- | ------------ |
| `name` | Short sluggable name used to refer to data in this file. | Required |
| `name` | Short [sluggable](https://en.wikipedia.org/wiki/Clean_URL#Slug) name used to refer to data in this file. `name` must be unique within this datapackage. | Required |

@botanize botanize Jan 30, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this sluggable, or does it need to be a slug?

@e-lo e-lo Mar 27, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be sluggable.

| `path` | Path of the data resource file relative to the `datapackage.json` | Required |
| `schema` | Data schema to use to valdiate the data resource to | Required |
| `sources` | Array of data sources formatted as a [`source`](#data-source) | Recommended |
Expand All @@ -53,11 +53,12 @@ Key fields for each [`tabular-data-resource`](https://specs.frictionlessdata.io/
| `title` | Description of the data source. | Required |
| `component` | What technology component was used to generate this data (directly or indirectly)? Examples include `AVL`, `APC`, `AFC`, etc. | Recommended |
| `product` | What product was used to generate this data (directly or indirectly)? | Recommended |
Comment thread
e-lo marked this conversation as resolved.
| `product_version` | Describe the version of the product was used. | Optional |
| `vendor` | What company makes this product? | Recommended |

## Data validation

Data with a valid [`datapackage.json`](#data-package) can be easily validated using the [frictionless framework](https://framework.frictionlessdata.io/), which can be installed and invoke as follows:
Data with a valid [`datapackage.json`](#data-package) can be easily validated using the [frictionless framework](https://framework.frictionlessdata.io/), which can be installed and invoked as follows:

```bash
pip install frictionless
Expand All @@ -69,6 +70,6 @@ frictionless validate path/to/your/datapackage.json
Specific files can be validated by running the frictionless framework against them and their corresponding schemas as follows:

```sh
frictionless validate vehicles.csv --schema https://raw.githubusercontent.com/TIDES-transit/TIDES/main/spec/vehicles.schema.json
frictionless validate vehicles.csv --schema https://raw.githubusercontent.com/TIDES-transit/TIDES/main/spec/vehicles.schema.json --schema-sync
```