-
Notifications
You must be signed in to change notification settings - Fork 247
DOCS-1738 - Document 256KB log message size support #6881
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
e9d6ef3
1835d53
c8ee1d1
798c981
4a6dd93
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| --- | ||
| title: Increased Maximum Log Message Size to 256KB (Search) | ||
| image: https://assets-www.sumologic.com/company-logos/_800x418_crop_center-center_82_none/SumoLogic_Preview_600x600.jpg?mtime=1617040082 | ||
| keywords: | ||
| - log message size | ||
| - large messages | ||
| - ingestion | ||
| hide_table_of_contents: true | ||
| --- | ||
|
|
||
| We're excited to announce that Sumo Logic now supports a maximum log message size of **256KB**, up from 64KB, so large single-line logs are ingested without being split as often. | ||
|
|
||
| Some Sumo Logic platform features and downstream webhook connections handle messages larger than 64KB differently. See [Known limitations](/docs/search/get-started-with-search/search-basics/search-large-messages/#known-limitations) for details. | ||
|
|
||
| [Learn more](/docs/search/get-started-with-search/search-basics/search-large-messages/). | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. from a discussion with @kevin-sumo, we suggest to skip the above limitation paragraphs, but just mention to find limitations using this link.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done — removed both summarized limitation sections and replaced with a link to the Known limitations section of the main doc. See |
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,23 +1,45 @@ | ||
| --- | ||
| id: search-large-messages | ||
| title: Search Large Messages | ||
| description: When collecting log messages or event logs that are larger than 64KB in size, Sumo Logic slices the messages into a stream of smaller message chunks. | ||
| description: When collecting log messages or event logs that are larger than 256KB in size, Sumo Logic slices the messages into a stream of smaller message chunks. | ||
| --- | ||
|
|
||
|
|
||
|
|
||
| When collecting log messages or event logs that are larger than 64KB in size, Sumo Logic slices the messages into a stream of smaller message chunks. | ||
| When collecting log messages or event logs that are larger than 256KB in size, Sumo Logic slices the messages into a stream of smaller message chunks. | ||
|
|
||
| Chunks are ideally created at a line break depending on the Source type, protocol, and size of the message. Each section of the large messages is annotated with metadata to keep the message in order when viewing or searching the log. | ||
|
|
||
| See [Collecting Multiline Logs](/docs/send-data/reference-information/collect-multiline-logs.md) for information on caveats and your configuration options. | ||
|
|
||
| For additional information on collecting large messages review the relevant [Source's documentation](/docs/send-data/choose-collector-source). | ||
|
|
||
| Some Sumo Logic platform features handle messages larger than 64KB differently, and alert notifications sent to downstream tools like Slack and Jira Cloud can hit character limits of their own. See [Known limitations](#known-limitations) for details. | ||
|
|
||
| ## Query to Identify Truncated Logs | ||
|
|
||
| We have a [metadata](built-in-metadata.md) tag called `_size`. The `_size` metadata tag provides the size of the log message in bytes. A log is truncated at the size of 64K or 65536 bytes. | ||
| We have a [metadata](built-in-metadata.md) tag called `_size`. The `_size` metadata tag provides the size of the log message in bytes. A log is truncated at the size of 256K or 262144 bytes. | ||
|
|
||
| To find truncated logs you can reference the `_size` metadata tag, an example would be using the where operator. | ||
|
|
||
| `| where _size = 65536` | ||
| `| where _size = 262144` | ||
|
|
||
| ## Known limitations | ||
|
|
||
| Keep the following limitations in mind when you work with messages larger than 64KB. | ||
|
|
||
| ### Sumo Logic platform limitations | ||
|
|
||
| These features process large messages differently within Sumo Logic itself: | ||
|
|
||
| - **LogCompare and LogReduce**. These operators truncate raw 256KB messages to 64KB before matching and grouping the logs into signatures, so content beyond 64KB is not considered. This can also affect response time when you run them against large messages. Learn more in [LogReduce](/docs/search/behavior-insights/logreduce/) and [LogCompare](/docs/search/behavior-insights/logcompare/). | ||
| - **Log Search messages table**. The messages table displays up to 25,000 characters of a message, even after you expand it. If a message larger than 25,000 characters contains JSON values as strings, those values are not rendered as JSON fields by default in the table. To view a complete message, or to view large JSON values as structured JSON, use the [Log Message Inspector](/docs/search/get-started-with-search/search-page/log-message-inspector). | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @AyanGhatak @ssharma-sumo can you review and confirm the accuracy of this updated paragraph? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes.. it looks good |
||
| - **Cloud SIEM**. Parsing and mapping might not process messages larger than 64KB correctly. | ||
| - **Field Extraction Rules**. The cumulative size of all fields extracted by a rule for a message is limited to 64KB, regardless of the message size. Learn more in [Field Extraction Rule limitations](/docs/manage/field-extractions/create-field-extraction-rule/#limitations). | ||
|
|
||
| ### Downstream webhook connection limitations | ||
|
|
||
| Alert notifications sent to these tools can also be affected by character limits that are fixed by the tool itself, independent of Sumo Logic's message size. Larger messages make these limits more likely to be reached: | ||
|
|
||
| - **Slack webhook connections**. Slack has a hard limit of 40,000 characters per message. Content beyond this limit, such as a large `{{ResultsJson}}` value, is truncated with "…" in the notification. Learn more in [Known limitations](/docs/alerts/webhook-connections/slack/#known-limitations). | ||
| - **Jira Cloud webhook connections**. The Jira Cloud issue description field has a hard limit of 32,767 characters. Content beyond this limit, such as a large `{{ResultsJson}}` value, is truncated with "…" and the issue is still created. Learn more in [Known limitations](/docs/alerts/webhook-connections/jira-cloud/#known-limitations). | ||
Uh oh!
There was an error while loading. Please reload this page.