Enhance Traffic Object to clarify how bytes have been counted (packet header/payload)

[cei1219]

Related Issue:

Closes #1678

Overview

This PR addresses #1678, Network Traffic 'bytes' - What does this include?. Reviewers are encouraged to read the issue as it provides background of the problem we are solving.

We deliberately constrained this first enhancement to the most urgent problem, one that is real today. Existing telemetry sources send byte counters which have been counted in different ways. A consumer of bytes, bytes_in, bytes_out or bytes_missed has no way to determine what was counted, so traffic counters from different sources cannot be compared reliably.

This PR provides that clarity by (a) defining, in their descriptions, what the existing byte counters measure, and (b) adding payload counters, defined the same way, to support the sources that count only the payload.

We stopped short of a less urgent use case: a single telemetry source providing several sets of counters, each measured at a different layer. We know this use case exists (IPFIX, for example, can export both L2 and L3 octet counters for the same flow), but it has not yet been seen in the wild as an urgent problem. If it becomes important to support, a later change could allow, say, a Network Activity event to hold multiple traffic objects, one per layer. We will cross that bridge if we come to it.

Description of changes:

This PR defines what the byte counters on the network_traffic object count, and adds payload counters beside them.

The four existing byte counters now say what they count. Each description ends with the same sentence: Counts bytes in Layer 3 (Network Layer) packets, including IP header and payload.

attribute	caption	type	requirement
bytes	Total Bytes	long_t	recommended
bytes_in	Bytes In	long_t	optional
bytes_out	Bytes Out	long_t	optional
bytes_missed	Bytes Missed	long_t	optional

Three new optional counters report the payload. Each description ends with: Counts bytes in the Layer 4 (Transport Layer) payload, excluding headers (such as TCP, UDP, etc.).

attribute	caption	type	requirement
payload_bytes	Total Payload Bytes	long_t	optional
payload_bytes_in	Payload Bytes In	long_t	optional
payload_bytes_out	Payload Bytes Out	long_t	optional

The payload_bytes* names mirror the existing bytes, bytes_in and bytes_out.

The three payload counters are defined in dictionary.json. The byte counter clarifications are object-level descriptions in objects/network_traffic.json, because the dictionary bytes attribute is also used by query_info, where it means the size of a query result.

These definitions match how known producers count today. NetFlow, IPFIX, and AWS VPC, Azure and OCI flow logs count the Layer 3 packet. GCP and NVM flow logs count the Layer 4 payload. IPFIX and Zeek connection logs report both, so they can fill both sets of counters.

No class changes are required, since traffic is already present on every Network Activity class.

Files touched: dictionary.json, objects/network_traffic.json, CHANGELOG.md.

Confirmations

1. Unreleased entry added to CHANGELOG.md, referencing this PR.
2. Contribution guidelines followed. Every commit carries a DCO sign-off.
3. Validated locally:
   • python -m ocsf_validator . passes
   • ocsf-schema-compiler . compiles with zero errors and zero warnings, in both default and browser mode
   • python -m ocsf.validate.compatibility against main passes all five checks
   • a local ocsf-server boots against the compiled schema, serves HTTP 200 and logs no warnings or errors
4. PR title matches the description.
5. Captions and descriptions follow the normative and neutral style.

[davemcatcisco]

See my naming comment above.

[cei1219]

My reply above ;)

[davemcatcisco]

See my naming comment above.

[cei1219]

My reply above ;)

[davemcatcisco]

The word packet is correct at L2 but for consistency with the L2 and L4 descriptions it migth be better to use the generic OSI term protocol data unit (PDU).

[cei1219]

My reply above ;)

[github-actions]

Schema Description Review

Automated suggestions for improving description clarity for LLM consumption. These are advisory — not required changes.

Previous Review Status

• ✅ packet_layer_id: no longer in the PR, so nothing is left to review.
• ✅ payload_bytes, payload_bytes_in and payload_bytes_out: the descriptions no longer depend on a configurable header layer. Each now states a fixed scope: the Layer 4 payload, excluding TCP/UDP headers. The bytes* attributes likewise state Layer 3 packets, including the IP header and payload.

✅ No description issues found — descriptions look clear for LLM consumption.

[pagbabian-splunk]

Unfortunately when we say "packet" it is reserved for the Network layer in the OSI model:

Where Packets Live in the OSI Model

Data changes its form and name as it moves down through the seven layers:
• Layer 7: Application – Data / User Request (e.g., HTTP, FTP, SMTP)
• Layer 6: Presentation – Formatted / Encrypted Data
• Layer 5: Session – Session Control Data
• Layer 4: Transport – Segments (TCP) or Datagrams (UDP)
• Layer 3: Network – Packets (adds source and destination IP addresses)
• Layer 2: Data Link – Frames (adds MAC addresses and error-checking)
• Layer 1: Physical – Bits (raw electrical, optical, or radio signals)

Interestingly, the current object network_traffic is defined correctly for the Network Layer of OSI - as packets are only defined at that layer, and the bytes would include the IP headers. So, we are asking for additional OSI layers in a class that so far is defined by one of those layers.

If we are to add the enum for OSI Layer type, the packet counts would always refer to L3 Network, because that is how packets are defined on OSI. However, the payload_bytes payload_bytes_in payload_bytes_out would be overloaded to be the value at the particular layer minus all header bytes. Hence the payload or data transferred is independent of the OSI layer. Payload bytes, or data, never includes the header bytes. This begs the question, do we need the enum at all since we generally know the protocol (connection.protocol_name/num).

This picture from Cisco illustrates it well:

Layer 2 is a problem. Total bytes would be the sum of the bytes of the frames and all the packets, the trailing bytes being the IP layer bytes. Given that L2 frames don't survive intact across a connection (each link replaces those frames), and are generally not output by Zeek, Suricata or flow logs, the bytes bytes_in bytes_out` would always mean L3 packet bytes regardless of the enum. Conveniently, that is how they are defined today, if not explicitly (and they should be made explicit).

Also, the object name is Network Traffic and is the practical measure of the total bytes transferred.

One twist, which would not be explicitly captured is when the traffic traverses a proxy or a NAT and headers are re-written because of termination at those points. In those cases, the exact IP header bytes are not transferred across the termination point, but assuming IP header size is constant, the network layer bytes counters would still be the same.

[cei1219]

Unfortunately when we say "packet" it is reserved for the Network layer in the OSI model:

Where Packets Live in the OSI Model

Data changes its form and name as it moves down through the seven layers: • Layer 7: Application – Data / User Request (e.g., HTTP, FTP, SMTP) • Layer 6: Presentation – Formatted / Encrypted Data • Layer 5: Session – Session Control Data • Layer 4: Transport – Segments (TCP) or Datagrams (UDP) • Layer 3: Network – Packets (adds source and destination IP addresses) • Layer 2: Data Link – Frames (adds MAC addresses and error-checking) • Layer 1: Physical – Bits (raw electrical, optical, or radio signals)

Interestingly, the current object network_traffic is defined correctly for the Network Layer of OSI - as packets are only defined at that layer, and the bytes would include the IP headers. So, we are asking for additional OSI layers in a class that so far is defined by one of those layers.

If we are to add the enum for OSI Layer type, the packet counts would always refer to L3 Network, because that is how packets are defined on OSI. However, the payload_bytes payload_bytes_in payload_bytes_out would be overloaded to be the value at the particular layer minus all header bytes. Hence the payload or data transferred is independent of the OSI layer. Payload bytes, or data, never includes the header bytes. This begs the question, do we need the enum at all since we generally know the protocol (connection.protocol_name/num).

This picture from Cisco illustrates it well:

Layer 2 is a problem. Total bytes would be the sum of the bytes of the frames and all the packets, the trailing bytes being the IP layer bytes. Given that L2 frames don't survive intact across a connection (each link replaces those frames), and are generally not output by Zeek, Suricata or flow logs, the `bytes` `bytes_in` bytes_out` would always mean L3 packet bytes regardless of the enum. Conveniently, that is how they are defined today, if not explicitly (and they should be made explicit).

Also, the object name is Network Traffic and is the practical measure of the total bytes transferred.

One twist, which would not be explicitly captured is when the traffic traverses a proxy or a NAT and headers are re-written because of termination at those points. In those cases, the exact IP header bytes are not transferred across the termination point, but assuming IP header size is constant, the network layer bytes counters would still be the same.

Yes, we still need the enum. Why? Isn't the protocol number sufficient? No, because even though the protocol was known (such as TCP or UDP), that does not (necessarily) imply how the bytes were measured.

[cei1219]

"The total number of payload bytes, summed across all packets transferred in both directions (sum of payload_bytes_in and payload_bytes_out). Payload excludes the headers of the layer identified by packet_layer_id."

I agree, we can improve our wording (for packet and payload), to address the question of whether counters refer to individual packets or to packets over time. I think this is a nice optimization but it does go beyond the scope of this ticket (it's true for the existing counters). @pagbabian-splunk if you'd like me to adjust descriptions of all counters in our traffic object to reflect the fact that exporters often accumulate counters over a period of time, I'm happy to do that.

[cei1219]

@pagbabian-splunk - please let me know if my changes reflect what was discussed in today's Network Activities meeting. If they do not, it was not intentional and I will address.

[cei1219]

@pagbabian-splunk @Aniak5 @davemcatcisco - My Ai agent noticed that we missed one (current) use case:

Cisco's FTD Firewall Connection Logs, essentially Flow Logs, byte counters capture L2 Packet (headers plus payload)

This was actually in my issue, but it was a sentence added to the end (not a bullet w/ bold text) so I missed it.

So this means we have not (fully) addressed all use cases. Just FYI.

#1678

[pagbabian-splunk]

Cisco's FTD Firewall Connection Logs, essentially Flow Logs, byte counters capture L2 Packet (headers plus payload)

This is why we need a new object for the Data Link layer (not overload the Network layer which is the current network_traffic object). We would have byte counters (we can reuse our bytes et al attributes) and we would have a frames counter, just like we have a packets counter for the network_traffic object. That is, assuming FTD or other products give us frame counters.

[pagbabian-splunk]

After more discussion at Maintainers, the term "data" seems to be more correct for the transport layer payload, since the term "payload" is interpreted relative to the OSI layer, while "data" is the same at any layer. So we might consider using the term data_bytes et al, rather than payload_bytes when we enhance the network_traffic object to accommodate the Zeke example.

One other discussion item is ICMP: technically also at the network layer, but defined as a "message" rather than "data" it includes the IP header but does not have separate transport layer headers. It is still part of the IP "datagram."

Per Stevens, the ICMP message includes an ICMP header, the IP header of the datagram that generated an error, and a UDP header, for example if the message was a UDP port unreachable error.