-
Notifications
You must be signed in to change notification settings - Fork 247
DOCS-1738 - Document 256KB log message size support #6881
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 4 commits
e9d6ef3
1835d53
c8ee1d1
798c981
4a6dd93
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| --- | ||
| title: Increased Maximum Log Message Size to 256KB (Search) | ||
| image: https://assets-www.sumologic.com/company-logos/_800x418_crop_center-center_82_none/SumoLogic_Preview_600x600.jpg?mtime=1617040082 | ||
| keywords: | ||
| - log message size | ||
| - large messages | ||
| - ingestion | ||
| hide_table_of_contents: true | ||
| --- | ||
|
|
||
| We're excited to announce that Sumo Logic now supports a maximum log message size of **256KB**, up from 64KB, so large single-line logs are ingested without being split as often. | ||
|
|
||
| **Sumo Logic platform limitations:** | ||
| * LogCompare, LogReduce, Cloud SIEM parsing, and Field Extraction Rules still process messages at 64KB. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is this Field Extraction Rule limitation accurate? This makes it sound like FER will only work if the log is <64kb. I thought the limit was in the volume extracted? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. FER can work with 256KB logs, it just cannot extract cumulative 64kb values for all fields
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good catch — the bullet was misleading. Per lei-sumo's clarification (FER works with 256KB logs, it's the cumulative extracted-field size that's capped at 64KB, not the message itself), removed this summarized bullet entirely rather than try to compress it further. See |
||
|
|
||
| **Downstream webhook connection limitations:** | ||
| * Slack (40,000 characters) and Jira Cloud (32,767 characters) enforce their own limits, which larger messages are more likely to reach. | ||
|
|
||
| [Learn more](/docs/search/get-started-with-search/search-basics/search-large-messages/). | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. from a discussion with @kevin-sumo, we suggest to skip the above limitation paragraphs, but just mention to find limitations using this link.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done — removed both summarized limitation sections and replaced with a link to the Known limitations section of the main doc. See |
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -81,6 +81,10 @@ You need the **Manage connections** [role capability](/docs/manage/users-roles | |
| 1. To test the connection, click **Test Alert**. If successful, you'll see a `200 OK` response message. | ||
| 1. Click **Save**. | ||
|
|
||
| ## Known limitations | ||
|
|
||
| The Jira Cloud issue description field has a hard limit of 32,767 characters, enforced at the database level. If a payload variable such as `{{ResultsJson}}` pushes the **Issue Description** past this limit, Jira Cloud returns a `400 Bad Request` error and the issue is not created. To avoid this, reference individual result fields (for example, `{{ResultsJson.fieldname}}`) instead of the full `{{ResultsJson}}` object, or otherwise keep the payload within the limit. | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @JV0812 We did a fix for this, the behavior is that Issue description will be truncated at 32KB and issue will still be created. Could you update this accordingly? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It is also truncated with "…", similar as the slack webhook.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Updated — Known limitations now says the description is truncated with "..." and the issue is still created, matching the Slack webhook behavior, instead of a 400 Bad Request failure. See |
||
|
|
||
| ## Create a Scheduled Search | ||
|
|
||
| Scheduled searches are saved searches that run automatically at specified intervals. When a scheduled search is configured to send an alert, it can be sent to another tool using a webhook connection. | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,23 +1,45 @@ | ||
| --- | ||
| id: search-large-messages | ||
| title: Search Large Messages | ||
| description: When collecting log messages or event logs that are larger than 64KB in size, Sumo Logic slices the messages into a stream of smaller message chunks. | ||
| description: When collecting log messages or event logs that are larger than 256KB in size, Sumo Logic slices the messages into a stream of smaller message chunks. | ||
| --- | ||
|
|
||
|
|
||
|
|
||
| When collecting log messages or event logs that are larger than 64KB in size, Sumo Logic slices the messages into a stream of smaller message chunks. | ||
| When collecting log messages or event logs that are larger than 256KB in size, Sumo Logic slices the messages into a stream of smaller message chunks. | ||
|
|
||
| Chunks are ideally created at a line break depending on the Source type, protocol, and size of the message. Each section of the large messages is annotated with metadata to keep the message in order when viewing or searching the log. | ||
|
|
||
| See [Collecting Multiline Logs](/docs/send-data/reference-information/collect-multiline-logs.md) for information on caveats and your configuration options. | ||
|
|
||
| For additional information on collecting large messages review the relevant [Source's documentation](/docs/send-data/choose-collector-source). | ||
|
|
||
| Some Sumo Logic platform features handle messages larger than 64KB differently, and alert notifications sent to downstream tools like Slack and Jira Cloud can hit character limits of their own. See [Known limitations](#known-limitations) for details. | ||
|
|
||
| ## Query to Identify Truncated Logs | ||
|
|
||
| We have a [metadata](built-in-metadata.md) tag called `_size`. The `_size` metadata tag provides the size of the log message in bytes. A log is truncated at the size of 64K or 65536 bytes. | ||
| We have a [metadata](built-in-metadata.md) tag called `_size`. The `_size` metadata tag provides the size of the log message in bytes. A log is truncated at the size of 256K or 262144 bytes. | ||
|
|
||
| To find truncated logs you can reference the `_size` metadata tag, an example would be using the where operator. | ||
|
|
||
| `| where _size = 65536` | ||
| `| where _size = 262144` | ||
|
|
||
| ## Known limitations | ||
|
|
||
| Keep the following limitations in mind when you work with messages larger than 64KB. | ||
|
|
||
| ### Sumo Logic platform limitations | ||
|
|
||
| These features process large messages differently within Sumo Logic itself: | ||
|
|
||
| - **LogCompare and LogReduce**. These operators truncate raw 256KB messages to 64KB before matching and grouping the logs into signatures, so content beyond 64KB is not considered. This can also affect response time when you run them against large messages. Learn more in [LogReduce](/docs/search/behavior-insights/logreduce/) and [LogCompare](/docs/search/behavior-insights/logcompare/). | ||
| - **Log Search messages table**. The messages table displays up to 64KB of a message, even after you expand it. To view a complete message larger than 64KB, use the [Log Message Inspector](/docs/search/get-started-with-search/search-page/log-message-inspector). | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. After this feature is released, the message table will display messages up to approximately 25 KB. @JV0812, could you please update this? Also, please add a note that messages larger than 25 KB containing JSON values as strings will not be rendered as JSON fields by default in the table. To view them as structured JSON, users will need to open the message in the Message Explorer.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Updated — the messages table limit is now documented as 25,000 characters (GA behavior), and added a note that JSON values in messages beyond that size aren't rendered as JSON fields by default in the table; the Log Message Inspector shows the full message and structured JSON. See |
||
| - **Cloud SIEM**. Parsing and mapping might not process messages larger than 64KB correctly. | ||
| - **Field Extraction Rules**. The cumulative size of all fields extracted by a rule for a message is limited to 64KB, regardless of the message size. Learn more in [Field Extraction Rule limitations](/docs/manage/field-extractions/create-field-extraction-rule/#limitations). | ||
|
|
||
| ### Downstream webhook connection limitations | ||
|
|
||
| Alert notifications sent to these tools can also be affected by character limits that are fixed by the tool itself, independent of Sumo Logic's message size. Larger messages make these limits more likely to be reached: | ||
|
|
||
| - **Slack webhook connections**. Slack has a hard limit of 40,000 characters per message. Content beyond this limit, such as a large `{{ResultsJson}}` value, is truncated with "…" in the notification. Learn more in [Known limitations](/docs/alerts/webhook-connections/slack/#known-limitations). | ||
| - **Jira Cloud webhook connections**. The Jira Cloud issue description field has a hard limit of 32,767 characters. A payload variable such as `{{ResultsJson}}` that exceeds this limit returns a `400 Bad Request` error. Learn more in [Known limitations](/docs/alerts/webhook-connections/jira-cloud/#known-limitations). | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Need to update this as well: issue can be created with a truncated payload.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Updated — this bullet now says the Jira Cloud payload is truncated with "..." and the issue is still created, instead of returning a 400 Bad Request. See |
||
Uh oh!
There was an error while loading. Please reload this page.