Update handling of vehicles and consists - #272
Conversation
remove inadvertent constraints
Remove service_date from key, required fields Make type required
|
Documentation available at: |
|
Data Validation Report
|
|
Data Validation Report
|
|
Data Validation Report
|
|
@lauriemerrell does Option 1 in #272 (comment) resolve your question regarding foreign keys between operational tables and |
|
Given discussion in #273, there is an open question of how to represent attributes in Do we want to leave this detail for a future minor, non-breaking version?
There is also the question of how to handle the existing attribute fields from
I see three options: Option 1: Temporarily remove these fields from the spec. Leave all attributes out of the new Option 2: Include these existing fields as attributes in Option 1A: Remove the fields, with the recommendation to current users of these fields that they include them as local extensions to the My preference is Option 1A. Remove these fields for now, and re-add them with a future minor release that fully defines the approach to vehicle_group attributes. |
|
@jlstpaul -- yes, I guess Option 1 works. I guess it's like My concern was more that I was wondering if there are cases where in practice the operational system will emit the unique |
|
@jabhij appreciate the thorough review on #251. If you have bandwidth for one more before the window closes tomorrow, #272 is the one that could most use another set of eyes. It restructures vehicles into Any level of review helps, and a comment with a score (Accepted / Accepted with minor changes / Substantially revised) counts toward its community review. |
|
QQ: Could we make the identifier semantics of the operational I understand the intent behind using That makes For example, an end user should be able to determine unambiguously whether:
This seems particularly important for those agencies building relational data models on top of TIDES. |
|
Suggestion: Can we add a concrete example showing the expected values of |
|
Good questions, and they land on ground this thread has been chewing on for a while. The intended join path is what John laid out in #272 (comment), his example with the bus, two LRVs, and consist CON0099 covers the physical-vehicle vs. consist situation you're describing. Short version, And yes, that makes the operational field broader than a plain physical vehicle ID. Downstream users shouldn't have to figure that out on their own. I agree the worked example belongs in the documentation. @jlstpaul what do you think about lifting your example into the table docs as part of this PR, or into the implementation documentation alongside the trip_stop_sequence example we moved there? |
|
@chrisyamas Thanks for the clarification. That makes the intended relationship much clearer, especially the distinction between Additionally, I can add a worked example to make the intended relationship very clear. Sounds good? |
|
Workd Example: Physical vehicle → Consist → CON0099 does not exist in The operational join can be shown as: and, depending on the assignment type: I believe showing both cases would help prevent downstream users from assuming that every operational |
|
QQ: How should downstream users resolve the applicable I understand from the discussion that operational records should resolve through For example: An operational record containing Should downstream users use (thinking from a data modeling perspective) |
|
Re: polymorphic On following on the discussion between @lauriemerrell and @jabhij about Proposal: redefine "vehicle" as the operating unit, not the physical unit.
Why this helps: the join Trade-off, to be upfront about it: this does mean another rename on an already-breaking PR, and every bus-only agency ends up with a trivially one-row |
|
@bijustrada360 if vehicle_id refers to the operating (logical) unit, what do I do with consists that report at the physical unit level? For example, in a vehicle_locations table I've got individual vehicles in consists reporting their own locations (3 vehicles in a consist, three distinct locations), and I'd also like to aggregate those individual vehicle locations to a consist location (e.g., location of front vehicle). Should I treat those train-car vehicle_locations as tide formatted but non-compliant because they're using a vehicle_id that refers to a physical unit, not a logical one? |
|
Picking up on @jabhij's #272 (comment), Maybe I'm misunderstanding, but from John's comment on this issue, either:
In either case it seems like you're likely to need to use additional join clauses to get information about the actual vehicle. -- NOTE: this won't work if both service_date and either of assignment_start_time and assignment_end_time are null
select
*
from vehicle_locations
inner join vehicle_assignments
on
locations.vehicle_id = vehicle_assignments.vehicle_id and
(
(
locations.event_timestamp between
vehicle_assignments.assignment_start_time and
assignment_end_time
) or
(
locations.service_date = vehicle_assignments.service_date
)
) And if both the service_date and the assignment start/end times are null, you're still out of luck. I think that means you are pretty much forced to supply either service_date or assignment start/end times for all vehicles, making all three of those fields conditionally required. This is obviously undesirable for service date because it requires a new set of rows for each bus on each date! But I don't particularly like conditionally required fields in these situations. An alternative is to remove Example:
Now joins are |
|
Good catch @botanize I missed that; this is a real gap in the "vehicle = logical unit" framing as I described it, not just a semantics question. Given per-car GPS is the common case for most rail systems, Proposal: Add
We'd deliberately make One thing I would recommend leaving out of scope: aggregating per-car locations into a single "consist location" (e.g., front-of-train). That's a derived value, and computing it correctly requires orientation/direction-of-travel data that already lives in Net effect: this is a contained change - it only touches |
|
I want to start by saying I hate to stir the pot this late in the review window, but on a second look getting this table (vehicle_assignment) right matters more than getting it merged on schedule, this is exactly the kind of structural decision that's expensive to walk back once agencies build against it. So its a detailed response please bear with me. Proposal: restructure
|
| Field | Type | Required |
|---|---|---|
vehicle_id |
string → vehicles.vehicle_id |
yes |
block_id |
string → block reference (TIDES already has block_id on trips_performed, following GTFS's block definition) |
yes |
service_date |
date | yes |
timestamp |
datetime | yes |
assignment_status |
enum: assigned / unassigned |
yes |
Primary key: (vehicle_id, timestamp, block_id).
block_id is included in the key, not just carried as data, because a vehicle finishing one block and starting another can plausibly log both events at the same timestamp (e.g. an instantaneous relief/swap) — (vehicle_id, timestamp) alone would collide in that case, while adding block_id correctly treats them as two distinct rows.
service_date is not part of the key it's carried on every row for direct, unambiguous joinability to trips_performed on (service_date, block_id), without downstream consumers having to re-derive it themselves.
Why this resolves the current issues
- No optional fields, no sentinel values. Every field is mandatory and every row is a real, meaningful timestamp, no placeholder dates (
1900-01-01/2999-12-31) needed for open-ended cases, and no ambiguity about which rows a join will or won't match. - Deterministic resolution. "What block is vehicle X on at time T" becomes an as-of lookup, most recent row for that
vehicle_idat or before T (e.g.ROW_NUMBER() OVER (PARTITION BY vehicle_id ORDER BY timestamp DESC)) rather than aBETWEENagainst an interval with nullable bounds. - Handles vehicle swaps naturally. A vehicle that breaks down mid-block and is replaced produces an
unassignedrow for the failed vehicle and anassignedrow for its replacement, sameblock_idandservice_date, near-simultaneous or identical timestamps, no special-casing needed. - Unambiguous block resolution. Since
block_idalone can repeat across service days, carryingservice_dateon every row (fixed at pullout, per above) removes any guesswork about which occurrence of a block an assignment row refers to. - Drops fields that were compensating for the interval design's gaps:
vehicle_assignment_id(a surrogate key whose own current description concedes it's typically redundant withvehicle_id),type(vehicle-vs-consist distinction, orthogonal to this table's purpose),vehicle_group_id(resolvable viavehicle_id → vehicles → vehicle_groups, doesn't need to be duplicated here), andoperational_status.
Validation rule worth stating explicitly
For a given vehicle_id, rows should strictly alternate assigned → unassigned → assigned → unassigned... in timestamp order. Two consecutive assigned rows (or two consecutive unassigned rows) for the same vehicle with no opposite-status row between them indicates malformed data; this is a cheap, concrete integrity check implementers can run against the table.
Known limitation: near-duplicate events
The key and alternation rule above catch structural problems, but they can't catch a subtler case: two rows with the same vehicle_id, block_id, and assignment_status, at different timestamps, are structurally valid either way, as a legitimate re-assignment to the same block later in the day, or as an accidental duplicate emitted by an upstream AVL/CAD system (dropped-ack retry, batch reprocessing, etc.). No schema constraint can distinguish these; it's a semantic question the data alone can't answer.
This ambiguity is compounded by the fact that not all agencies operate the same way. Some agencies do run the same vehicle on the same block twice in a service day (e.g. interlined or split work), so a second assigned row for an identical (vehicle_id, block_id) pair later in the day is entirely legitimate for them. But an agency whose operating model never re-assigns the same vehicle to the same block within a service day would see that exact same pattern, same vehicle, same block, later timestamp and have good reason to treat it as a duplicate or ingestion error rather than a real event. Whether a repeated (vehicle_id, block_id) pair on the same service_date is legitimate or a data quality problem depends on the agency's own operating model, not on anything visible in the row data itself. I dont believe the schema should try to encode this, a hard constraint against repeats would silently break agencies who do legitimately reuse vehicle/block pairs. I'd recommend the spec leave this as agency-specific validation guidance: agencies should document, and their own downstream tooling should encode, whether repeated vehicle/block pairs within a service day are expected or should be flagged.
Enforcing the alternation rule
The alternation rule above can't be expressed as a JSON Table Schema constraint - frictionless/Table Schema only supports single-row constraints (required, unique, enum, foreign keys), with no mechanism for cross-row sequencing checks. I'd recommend documenting this as a normative rule plus a recommended validation query, so implementers have something concrete to run rather than just a sentence to interpret:
WITH ordered AS (
SELECT
vehicle_id,
timestamp,
block_id,
assignment_status,
LAG(assignment_status) OVER (
PARTITION BY vehicle_id ORDER BY timestamp
) AS prev_status
FROM vehicle_assignments
)
SELECT *
FROM ordered
WHERE assignment_status = prev_statusAny row returned here is a violation, two consecutive same-status events for the same vehicle with nothing alternating between them.
Rail note: vehicle swaps vs. composition changes
This design already supports a block being handed off between different vehicles mid-service, e.g. a stalled train pulled from service and a replacement train dispatched to finish its block using the same swap pattern described above (an unassigned row for the outgoing vehicle, an assigned row for the incoming one, same block_id and service_date). This works cleanly because vehicle_id here refers to the logical operating unit (a full train/consist), not a physical car, per the earlier redefinition in this thread.
This is distinct from a consist recomposition cars added to or removed from a train that keeps the same vehicle_id throughout. That's not an assignment-level event at all; it's tracked entirely in vehicle_composition/consist_vehicles, independent of vehicle_assignments.
Summary: this design handles both transit modes
Buses and trains are handled by the same four fields and the same rules, with no mode-specific branching anywhere in the schema:
- Bus:
vehicle_id= the bus itself. Oneassigned/unassignedpair per piece of work; a mechanical failure mid-block is handled by the vehicle-swap pattern (outgoing busunassigned, replacement busassigned, sameblock_id). - Rail:
vehicle_id= the train/consist (per the logical-unit redefinition upthread). The exact same swap pattern covers a stalled train being pulled and a replacement train picking up its block. Changes to a train's physical composition (cars added/removed) never touch this table at all, that's entirelyvehicle_composition's job, andvehicle_assignmentsdoesn't need to know or care how many cars are behind a givenvehicle_id.
The only thing that differs between modes is what vehicle_id resolves to underneath, a single physical unit for a bus, a composed set of units for a train, and that distinction is handled entirely upstream, in vehicles/vehicle_composition. vehicle_assignments itself stays mode-agnostic throughout, which is what lets a single schema, a single key, and a single validation rule serve both without exceptions.
Scope note on operational_status
This table would only contain rows for vehicles actively available for assignment (i.e., roadworthy). Vehicle condition/readiness (e.g. under maintenance, out of service) is a separate concern from work assignment and arguably belongs in its own table rather than as a field here.
Hope I've covered every possible critical scenario, but if not do let me know.
|
I think this discussion has surfaced an important temporal integrity issue with I agree that the physical-unit vs. logical/consist distinction needs to be handled, but before changing the terminology/model more broadly, I think we should make sure the assignment model provides a deterministic way to resolve an operational record to the correct assignment. With Option 1, If an operational record has Could we define a normative rule for this? I think this is important to resolve before v2.0 because otherwise two downstream implementations could legitimately resolve the same operational record to different assignments. |
|
QQ: Could we reconcile the The discussion describes Option 1 as having Could the schema and foreign-reference definition be updated to reflect the agreed Option 1 behavior? |
|
@jabhij both of these are already resolved by the event-log restructuring proposed above ( ##272 (comment)), rather than needing a new normative rule layered onto the current interval design: On your first question (deterministic resolution): the event-log model doesn't need a rule mandating service_date or a valid interval be populated, because there's no interval at all, every row is a single, mandatory, real timestamp (assigned/unassigned). Resolving "what's BUS001's assignment at time T" becomes an as-of lookup (most recent row at or before T), which is deterministic by construction rather than by a normative rule that has to be enforced separately. So rather than defining a rule for the current schema, I'd suggest we settle whether the event-log restructuring itself is the right direction, it removes the class of problem you're flagging rather than patching it. On your second question (reconciling vehicle_id's foreign-key definition with the consist case): this is also resolved upstream by the "vehicle = logical operating unit" redefinition from earlier in this thread ( ##272 (comment)). Under that model, a consist is a vehicle_id with a normal row in vehicles, there's no more type=consist case where vehicle_id doesn't resolve to vehicles.vehicle_id, so the foreign-key definition doesn't need special-casing at all. Given the review window closes today, wanted to flag directly so we're not tracking two parallel fixes to the same underlying issue. |
@bijustrada360 By requiring |
|
@gabriel-korbato I think this is a good example of where precise transit-domain definitions matter a lot for getting this schema right, and it's worth summarizing the rules explicitly here since the audience following this thread may be missing that context. Worth noting upfront: these are conventions informally followed by most leading service planning/scheduling systems and CAD/AVL vendors. I say informally because there's no formal standard defining them, they're industry practice rather than a documented spec.
Given that, whether a block appears in
So |
Unfortunately, legitimate vehicle movement without a block_id is fairly common in our system. It's not unusual in certain circumstances for a vehicle to fail to pickup a duty/block assignment and yet, the operator continues to do the intended work. |
|
@bijustrada360 I agree that leading scheduling and CAD/AVL systems follow the convention of having blocks, but TIDES is meant to capture a variety of scenarios, including of smaller operations that don't have a leading CAD/AVL system. For example, we have worked with systems that are operated more loosely and don't have a proper CAD/AVL system, just GPS tracking attached to the vehicle. In some of these agencies there is a rough service plan without blocks, and certainly nothing about a schedule comes in through the AVL data... this must be figured out when the data are processed. It's worth preserving the ability to describe a physical observable fact (e.g. vehicle v operated a trip on route r, direction d, starting at time t, and the observed times at each stop) even when there is no associated schedule. I've also seen what @botanize reports about buses not having a block because the driver doesn't properly log on at more than one agency. |
@botanize, thanks for the real-world example, before we build this into the standard, I want to understand the nature of the gap a bit better, since it changes how we should handle it. Is a failed block pickup, where the operator proceeds with the intended work anyway, a limitation specific to the systems you're currently working with, and are there any plans on your end to close that gap? Asking because TIDES is meant to be a standard that works above system limitations rather than baking in permanent workarounds for gaps that could reasonably be resolved at the system level. That said, regardless of the answer, I think there's a solution that lets I've recommended this exact pattern to an agency before; they were seeing real downstream problems from unresolved block assignments (issues with their CAD/AVL, trip prediction algorithm, and radio communications), and the reserved-range approach resolved it without requiring any change to their core assignment model. So even if this does turn out to be something inherent that every implementation has to handle, I think we can document this reserved-range convention as the recommended pattern rather than changing the schema itself. |
TIDES is meant to be flexible enough to work with many different types of systems, and we do not want to require that an agency have a schedule to be able to use TIDES to represent their operations. Another point is that TIDES data is meant to be used at various steps in the data lifecycle, so we would want to be able to represent blockless movements if blocks are missing in the raw AVL data. Later that data might get processed and matched to a schedule, at which point most trips could be assigned trips and blocks (if there is a schedule). The TIDES format is not meant to be used only for the final/processed/clean data; it must also accommodate common imperfections in raw data to allow adopters to easily convert vendor-formats into TIDES without having to include complex cleansing processes. |
That wasn't my understanding of TIDES's intent, apologies, and thanks for the correction. That's an important distinction: Revised proposal:
Limitation worth flagging: two genuinely identical rows, same |
7 participants
Pull Request
Summary
This PR updates the handling of vehicles and vehicles assignments in TIDES as follows
vehicle_groupstablevehiclestable to represent the static/slowly changing list of physical vehicles (buses and train cars)vehicle_assignmentstable to represent the vehicles (buses and train cars) and consists that are available to be assigned each date.train_carsandvehicle_train_carstables and add a newconsist_vehiclestable to document that cars that are assigned to each consist (with their order and orientation)vehicle_directionfield to the operational event and summary data tables to represent the direction that a train is moving during operation.This is a breaking change.
Resolves #269
Schema Tables / Terminology
vehicle_groups are collections of vehicles that share the same characteristics. A vehicle group may represent a group of individual vehicles (e.g., buses of the same type) or planned consist types (e.g., two-car trains of type 3 LRVs)
vehiclesvehicles are single physical vehicles, a bus or a train car. A vehicle has a start date and end date when it is in service, along with unique identifying information such as the label on the vehicle and license plate. Other attributes are referenced from
vehicle_groupsvehicle_assignments are the daily record of either a single vehicle (such as a bus or train car) or a consist (i.e., a train) that can have data associated with it. This allows for the fact that rail operations systems may associate data with either an entire train or each individual car of a train. There is some flexibility in whether and how the
vehicle_assignmentstable is used. In the simplest context, it is and optional table. And in some contexts, vehicle assignments may stretch over multiple days instead of being for a single date.consist_vehicles are the individual train cars that make up a consist. The cars of the consists refer to vehicle records and have both an order and an orientation.
Files Changed
Removed:
spec/train_cars.schema.jsonspec/vehicle_train_cars.schema.jsonsamples/template/TIDES/train_cars.csvsamples/template/TIDES/vehicle_train_cars.csvNew:
spec/vehicle_groups.schema.json- new table schemaspec/vehicle_assignments.schema.json- new table schemaspec/consists_vehicles.schema.json- new table schemasamples/template/TIDES/vehicle_groups.csv- template csv filesamples/template/TIDES/vehicle_assignments.csv- template csv filesamples/template/TIDES/consist_vehicles.csv- template csv fileModified:
spec/vehicles.schema.jsonspec/tides-schema.jsopnsamples/template/TIDES/vehicles.csvspec/tides-datapackage-profile.jsonsamples/template/TIDES/datapackage.jsonCHANGELOG.md- documented new table_- See related discussion in #269 and discussion document Vehicles and Crew in GTFS / TODS / TIDES
Usage Notes
Outstanding Tasks
vehicle_groupstable. This could be a subset of the fields proposed in Addition of vehicles.txt to GTFS schedule google/transit#636Review checklist
Per change management policy, the following must be met before feature branch changes can merge to
developbranch:Community review status (updated August 27, 2026)
How community review works: Per the TIDES Change Management Policy, a proposal moves forward once at least three TIDES Contributors outside the originating working group publicly comment with a score: Accepted, Accepted with minor changes, or Substantially revised. A few sentences with your read of the proposal is a complete review.
Originating working group (July 1, 2026 vehicles session): Christopher Yamas, John Levin, Laurie Merrell, Gabriel Sánchez Martínez, Joey Reid. Their work is reflected in the proposal itself; community review comes from Contributors beyond this group.
Review so far: @doconnoronca and @bijustrada360 have contributed to the design discussion, and the join-semantics and worked-example questions raised this week are being worked through in the comments. Scored reviews from Contributors are welcome to complete community review.
Community review remains open, and reviews are welcome while we finalize v2.0. If you have not yet weighed in, a short comment closing with where you land (Accepted / Accepted with minor changes / Substantially revised) is all we need.