From 271678291dffd6d0a264e9acfc642632eff8cdeb Mon Sep 17 00:00:00 2001 From: Simon Schneider Date: Mon, 31 Aug 2026 08:17:11 +0200 Subject: [PATCH 1/2] [akamai] Fix ingest pipeline failures and malformed geo codes in siem MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three defects in the `siem` ingest pipeline caused Akamai security events to be indexed as `event.kind: pipeline_error` with all parsed fields lost, or to carry malformed values. All three are reproducible with `POST _ingest/pipeline/_simulate`. 1. `script_base64_decode_attackData_rule_573db939` threw a NullPointerException in two cases: - `ctx.akamai.siem.rules = rules_array` assumed `akamai.siem` existed, but that object is only created by the KV processors for `requestHeaders` / `responseHeaders`. Events with attack rules but no HTTP headers (header logging is optional in the Akamai security configuration, and an empty string is stripped earlier by `remove_http_message_request_headers_non_kv`) failed with "cannot access method/field [siem] from a null def reference". - `ctx.json.attackData[key].length` assumed all seven `rule*` fields were present. Akamai only sends the ones that apply, so a missing key failed with "cannot access method/field [length] from a null def reference". Because the script runs before every `attackData`, `botData`, `clientData` and `userRiskData` rename and before `event.category` / `event.kind`, the whole tail of the pipeline was skipped for these events. 2. `set_source_geo_region_iso_code_3a81fa3d` ran after the rename that removes `json.geo.country`, so whenever GeoIP could not resolve the client address the field rendered as `-NY` instead of `US-NY`. With no geo data at all the value rendered as `-`, which is not empty and therefore not caught by `ignore_empty_value` — every such event gained a synthetic `source.geo` object holding only `region_iso_code: "-"`. The processor now runs before the rename and requires both parts. 3. `set_client_10dffeb7` copied `client` from `source` without a guard. Events without `attackData.clientIP` have no `source` object, and `copy_from` fails on a missing path. This was masked by defect 2, which incidentally created a `source` object; fixing 2 alone would have turned it into a live failure. Adds four pipeline test documents covering each path: attack rules without headers, a missing `rule*` key, an unresolvable client address with an Akamai geo block, and an event with no client IP. The two pre-existing expected documents are unchanged. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014AUSnTxmZK2NCk1JuyyoiH --- packages/akamai/changelog.yml | 13 +- .../test/pipeline/test-placeholder-json.log | 4 + .../test-placeholder-json.log-expected.json | 376 ++++++++++++++++++ .../elasticsearch/ingest_pipeline/default.yml | 44 +- .../akamai/data_stream/siem/fields/fields.yml | 4 +- .../_dev/test/policy/test-default.yml | 1 - .../_dev/test/policy/test-no-cursor.yml | 2 - packages/akamai/manifest.yml | 2 +- 8 files changed, 428 insertions(+), 18 deletions(-) diff --git a/packages/akamai/changelog.yml b/packages/akamai/changelog.yml index 02fa3d99add..83876c7c538 100644 --- a/packages/akamai/changelog.yml +++ b/packages/akamai/changelog.yml @@ -1,4 +1,15 @@ # newer versions go on top +- version: "4.1.1" + changes: + - description: Fix ingest pipeline failures on events that carry attack rules but no HTTP headers, or only a subset of the `attackData.rule*` fields. + type: bugfix + link: https://github.com/elastic/integrations/pull/99999 + - description: Fix malformed `source.geo.region_iso_code` values such as `-NY` or `-` on events whose client address GeoIP cannot resolve. + type: bugfix + link: https://github.com/elastic/integrations/pull/99999 + - description: Do not copy the `client` fields for events that have no client IP. + type: bugfix + link: https://github.com/elastic/integrations/pull/99999 - version: "4.1.0" changes: - description: Add tags to ingest pipeline processors. @@ -53,7 +64,7 @@ link: https://github.com/elastic/integrations/pull/16225 - version: "3.0.0" changes: - - description: | + - description: | Migrated SIEM data stream from HTTPJSON to CEL. type: enhancement link: https://github.com/elastic/integrations/pull/15713 diff --git a/packages/akamai/data_stream/siem/_dev/test/pipeline/test-placeholder-json.log b/packages/akamai/data_stream/siem/_dev/test/pipeline/test-placeholder-json.log index 73b9d2de9a2..cffed15df81 100644 --- a/packages/akamai/data_stream/siem/_dev/test/pipeline/test-placeholder-json.log +++ b/packages/akamai/data_stream/siem/_dev/test/pipeline/test-placeholder-json.log @@ -1,2 +1,6 @@ {"attackData":{"appliedAction":"tarpit","clientIP":"198.51.100.1","configId":"67217","policyId":"PNWD_110088","ruleActions":"bW9uaXRvcg%3d%3d%3bbW9uaXRvcg%3d%3d%3bbW9uaXRvcg%3d%3d%3bbW9uaXRvcg%3d%3d%3bbW9uaXRvcg%3d%3d%3bdGFycGl0","ruleData":"%3b%3b%3b%3b%3bQm90X0E5RDg0MDJFQ0NGQjY3N0NCMjBEMDlBRUNDODk5MkFE","ruleMessages":"TWlzc2luZyBDb29raWUgSGVhZGVy%3bTm9uLVBlcnNpc3RlbnQgSFRUUCBDb25uZWN0aW9u%3bQ2hyb21lIFNpZ25hdHVyZSBBbm9tYWx5%3bU2VjLUZldGNoLUhlYWRlciBNaXNzaW5n%3bTG93IEhlYWRlciBDb3VudA%3d%3d%3bVW5rbm93biBCb3RzIChCcm93c2VyIEltcGVyc29uYXRvcik%3d","ruleSelectors":"%3b%3b%3b%3b%3b","ruleTags":"QUtBTUFJL0JPVC9CSUQ%3d%3bQUtBTUFJL0JPVC9CSUQ%3d%3bQUtBTUFJL0JPVC9CSUQ%3d%3bQUtBTUFJL0JPVC9CSUQ%3d%3bQUtBTUFJL0JPVC9CSUQ%3d%3bQUtBTUFJL0JPVC9CSUQ%3d","ruleVersions":"MQ%3d%3d%3bMQ%3d%3d%3bMQ%3d%3d%3bMQ%3d%3d%3bMQ%3d%3d%3bMQ%3d%3d","rules":"MzkwNDAwNg%3d%3d%3bMzkwNDAwNw%3d%3d%3bMzkwNDAyMA%3d%3d%3bMzkwNDA1Mg%3d%3d%3bMzkwNDA1Mw%3d%3d%3bQk9ULUJST1dTRVItSU1QRVJTT05BVE9S"},"format":"json","geo":{"asn":"28573","city":"SOROCABA","continent":"SA","country":"BR","regionCode":"SP"},"httpMessage":{"bytes":"0","host":"vinrcl.safercar.gov","method":"GET","path":"/vin/","port":"443","protocol":"HTTP/1.1","query":"vin=1G1FD3DSXP0118247","requestHeaders":"Host%3a%20vinrcl.safercar.gov%0d%0aUser-Agent%3a%20Mozilla%2f5.0%20(Windows%20NT%2010.0%3b%20Win64%3b%20x64)%20AppleWebKit%2f537.36%20(KHTML,%20like%20Gecko)%20Chrome%2f112.0.0.0%20Safari%2f537.36%0d%0aAccept%3a%20text%2fhtml,application%2fxhtml+xml,application%2fxml%3bq%3d0.9,image%2favif,image%2fwebp,image%2fapng,%2f%3bq%3d0.8,application%2fsigned-exchange%3bv%3db3%3bq%3d0.7%0d%0aAccept-Language%3a%20zh-CN,zh%3bq%3d0.9,en%3bq%3d0.8%0d%0aCache-Control%3a%20no-cache%0d%0aConnection%3a%20close%0d%0aAccept-Encoding%3a%20gzip%0d%0a","requestId":"f3fe4c34","responseHeaders":"{p}","start":"1762365006","status":"200","tls":"tls1.3"},"identity":{"ja4":"t13d131000_f57a46bbacb6_e7c285222651","tlsFingerprintV2":"c6a4ce37a8dc4153","tlsFingerprintV3":"3~de293936a8dc4153"},"type":"akamai_siem","version":"1.0"} { "type": "akamai_siem", "format": "json", "version": "1.0", "attackData": { "configId": "12345", "policyId": "qik1_3333", "clientIP": "89.160.20.156", "appliedAction": "monitor", "rules": "dfafdafdsf%3b", "ruleVersions": "%3b", "ruleMessages": "testtest%3d%3b", "ruleTags": "testtest%3d%3d%3b", "ruleData": "testtest%3d%3d%3b", "ruleSelectors": "%3b", "ruleActions": "testabc%3d%3d%3b", "apiId": "API_12345", "apiKey": "test123" }, "httpMessage": { "requestId": "abc123", "start": "1775068082", "protocol": "HTTP/1.1", "tls": "tls1.2", "method": "POST", "host": "www.example.com", "port": "443", "path": "/extranet-api/login/2.0", "requestHeaders": "Accept%3a%20*%2f*%0d%0aContent-Type%3a%20application%2fjson%0d%0aHost%3a%20www.example.com%0d%0aContent-Length%3a%2084%0d%0aExpect%3a%20100-continue%0d%0aremove-dup-edge-ctrl-headers-rollout-enabled%3a%201%0d%0a", "status": "200", "bytes": "1778", "responseHeaders": "Content-Type%3a%20application%2fjson%0d%0aContent-Disposition%3a%20inline%3bfilename%3df.txt%0d%0aExpect-CT%3a%20max-age%3d3600%0d%0aX-Frame-Options%3a%20DENY%0d%0aX-Frame-Options%3a%20SAMEORIGIN%0d%0aX-XSS-Protection%3a%201%3b%20mode%3dblock%0d%0aX-Forwarded-Proto%3a%20https%0d%0aX-Content-Type%3a%20nosniff%0d%0aContent-Security-Policy%3a%20default-src%20%27self%27%3b%20script-src%20%27self%27%3b%20style-src%20%27self%27%3b%20img-src%20%27self%27%3b%20font-src%20%27self%27%3b%20connect-src%20%27self%27%3b%20frame-src%20%27self%27%3b%0d%0a%09%20%20%20%20%20%20%20%20%20%20%20%20object-src%20%27self%27%3b%20base-uri%20%27self%27%3b%20form-action%20%27self%27%3b%20frame-ancestors%20%27self%27%3b%20upgrade-insecure-requests%0d%0aVary%3a%20origin%0d%0aVary%3a%20access-control-request-method%0d%0aVary%3a%20access-control-request-headers%0d%0aExpires%3a%20Wed,%2001%20Apr%202026%2018%3a28%3a02%20GMT%0d%0aCache-Control%3a%20max-age%3d0,%20no-cache,%20no-store%0d%0aDate%3a%20Wed,%2001%20Apr%202026%2018%3a28%3a02%20GMT%0d%0aContent-Length%3a%20417%0d%0aConnection%3a%20keep-alive%0d%0aSet-Cookie%3a%20HttpOnly%3b%20Secure%0d%0aStrict-Transport-Security%3a%20max-age%3d31536000%20%3b%20includeSubDomains%20%3b%20preload%0d%0aStrict-Transport-Security%3a%20max-age%3d31536000%20%3b%20includeSubDomains%20%3b%20preload%0d%0aSet-Cookie%3a%20_abck%3dtest123456%7e-1%7etest123+LjM%2ftest123%2fy%2ftest123%2ftest123+test123%2ftest123+test123+test123+test123%2fabc12%2fabc123%2fabc123%2fG1+abc123+abc123%3d%3d%7e-1%7e-1%7e-1%7e-1%7e-1%3b%20Domain%3d.example.com%3b%20Path%3d%2f%3b%20Expires%3dThu,%2001%20Apr%202027%2018%3a28%3a02%20GMT%3b%20Max-Age%3d31536000%3b%20Secure%0d%0aSet-Cookie%3a%20bm_sz%3dexample12345%7etest123+PjM%2fexample12345+abc123%2frjs+GVttH5f+%2fabc123%2fabc123%2ftLFHt29E+dQj7rjYd1V7q%2fabc123+abc123%2fabc123+evOFr9vXY%2fMN3g%3d%7eabc123%7eabc123%3b%20Domain%3d.example.com%3b%20Path%3d%2f%3b%20Expires%3dWed,%2001%20Apr%202026%2022%3a28%3a02%20GMT%3b%20Max-Age%3d14400%0d%0a" }, "geo": { "continent": "AS", "country": "IN", "city": "VADODARA", "regionCode": "GJ", "asn": "45117" }, "identity": { "tlsFingerprintV2": "1234567890", "tlsFingerprintV3": "3~1234567" } } +{"attackData":{"appliedAction":"deny","clientIP":"198.51.100.23","configId":"67217","policyId":"PNWD_110088","ruleActions":"ZGVueQ%3d%3d","ruleData":"YWxlcnQo","ruleMessages":"Q3Jvc3Mtc2l0ZSBTY3JpcHRpbmcgKFhTUykgQXR0YWNr","ruleSelectors":"QVJHUzph","ruleTags":"V0VCX0FUVEFDSy9YU1M%3d","ruleVersions":"NA%3d%3d","rules":"OTUwMDA0"},"format":"json","geo":{"asn":"64501","city":"AMSTERDAM","continent":"EU","country":"NL","regionCode":"NH"},"httpMessage":{"bytes":"512","host":"www.example.com","method":"POST","path":"/login","port":"443","protocol":"HTTP/1.1","requestId":"9a1c2f4b","start":"1700000000","status":"403","tls":"TLSv1.3"},"type":"akamai_siem","version":"1.0"} +{"attackData":{"appliedAction":"monitor","clientIP":"203.0.113.5","configId":"12345","policyId":"qik1_3333","ruleActions":"bW9uaXRvcg%3d%3d","ruleData":"Y3VybA%3d%3d","ruleMessages":"UmVxdWVzdCBJbmRpY2F0ZXMgYW4gYXV0b21hdGVkIHByb2dyYW0%3d","ruleTags":"QVVUT01BVElPTi9NSVND","rules":"OTkwMDEx"},"format":"json","geo":{"asn":"64502","city":"MADRID","continent":"EU","country":"ES","regionCode":"M"},"httpMessage":{"bytes":"1024","host":"www.example.com","method":"GET","path":"/api/v1/status","port":"443","protocol":"HTTP/2","requestId":"7d0e51aa","requestHeaders":"User-Agent%3a%20curl%2f8.4.0%0d%0aAccept%3a%20*%2f*%0d%0a","responseHeaders":"Server%3a%20AkamaiGHost%0d%0a","start":"1700000060","status":"200","tls":"TLSv1.3"},"type":"akamai_siem","version":"1.0"} +{"attackData":{"appliedAction":"alert","clientIP":"10.0.0.1","configId":"67217","policyId":"PNWD_110088"},"format":"json","geo":{"asn":"64501","city":"NEWYORK","continent":"NA","country":"US","regionCode":"NY"},"httpMessage":{"bytes":"256","host":"www.example.com","method":"GET","path":"/","port":"443","protocol":"HTTP/1.1","requestId":"3f5b8c10","requestHeaders":"User-Agent%3a%20Mozilla%2f5.0%0d%0a","start":"1700000120","status":"200","tls":"TLSv1.2"},"type":"akamai_siem","version":"1.0"} +{"attackData":{"configId":"67217","policyId":"PNWD_110088"},"format":"json","httpMessage":{"bytes":"0","host":"www.example.com","method":"GET","path":"/health","port":"443","protocol":"HTTP/1.1","requestId":"c48d9e02","requestHeaders":"User-Agent%3a%20Mozilla%2f5.0%0d%0a","start":"1700000180","status":"500"},"type":"akamai_siem","version":"1.0"} diff --git a/packages/akamai/data_stream/siem/_dev/test/pipeline/test-placeholder-json.log-expected.json b/packages/akamai/data_stream/siem/_dev/test/pipeline/test-placeholder-json.log-expected.json index df9854f6624..fc54513e319 100644 --- a/packages/akamai/data_stream/siem/_dev/test/pipeline/test-placeholder-json.log-expected.json +++ b/packages/akamai/data_stream/siem/_dev/test/pipeline/test-placeholder-json.log-expected.json @@ -315,6 +315,382 @@ "path": "/extranet-api/login/2.0", "port": 443 } + }, + { + "@timestamp": "2023-11-14T22:13:20.000Z", + "akamai": { + "siem": { + "config_id": "67217", + "policy_id": "PNWD_110088", + "rule_actions": [ + "deny" + ], + "rule_tags": [ + "web_attack/xss" + ], + "rules": [ + { + "ruleActions": "deny", + "ruleData": "alert(", + "ruleMessages": "Cross-site Scripting (XSS) Attack", + "ruleSelectors": "ARGS:a", + "ruleTags": "WEB_ATTACK/XSS", + "ruleVersions": "4", + "rules": "950004" + } + ] + } + }, + "client": { + "address": "198.51.100.23", + "as": { + "number": 64501, + "organization": { + "name": "Documentation ASN" + } + }, + "geo": { + "city_name": "Amsterdam", + "continent_name": "Europe", + "country_iso_code": "NL", + "country_name": "Netherlands", + "location": { + "lat": 52.37404, + "lon": 4.88969 + }, + "region_iso_code": "NL-NH", + "region_name": "North Holland" + }, + "ip": "198.51.100.23" + }, + "ecs": { + "version": "8.11.0" + }, + "event": { + "category": [ + "network" + ], + "id": "9a1c2f4b", + "kind": "event", + "original": "{\"attackData\":{\"appliedAction\":\"deny\",\"clientIP\":\"198.51.100.23\",\"configId\":\"67217\",\"policyId\":\"PNWD_110088\",\"ruleActions\":\"ZGVueQ%3d%3d\",\"ruleData\":\"YWxlcnQo\",\"ruleMessages\":\"Q3Jvc3Mtc2l0ZSBTY3JpcHRpbmcgKFhTUykgQXR0YWNr\",\"ruleSelectors\":\"QVJHUzph\",\"ruleTags\":\"V0VCX0FUVEFDSy9YU1M%3d\",\"ruleVersions\":\"NA%3d%3d\",\"rules\":\"OTUwMDA0\"},\"format\":\"json\",\"geo\":{\"asn\":\"64501\",\"city\":\"AMSTERDAM\",\"continent\":\"EU\",\"country\":\"NL\",\"regionCode\":\"NH\"},\"httpMessage\":{\"bytes\":\"512\",\"host\":\"www.example.com\",\"method\":\"POST\",\"path\":\"/login\",\"port\":\"443\",\"protocol\":\"HTTP/1.1\",\"requestId\":\"9a1c2f4b\",\"start\":\"1700000000\",\"status\":\"403\",\"tls\":\"TLSv1.3\"},\"type\":\"akamai_siem\",\"version\":\"1.0\"}", + "start": "2023-11-14T22:13:20.000Z" + }, + "http": { + "request": { + "id": "9a1c2f4b", + "method": "POST" + }, + "response": { + "bytes": 512, + "status_code": 403 + }, + "version": "1.1" + }, + "network": { + "protocol": "http", + "transport": "tcp" + }, + "observer": { + "type": "proxy", + "vendor": "akamai" + }, + "related": { + "ip": [ + "198.51.100.23" + ] + }, + "source": { + "address": "198.51.100.23", + "as": { + "number": 64501, + "organization": { + "name": "Documentation ASN" + } + }, + "geo": { + "city_name": "Amsterdam", + "continent_name": "Europe", + "country_iso_code": "NL", + "country_name": "Netherlands", + "location": { + "lat": 52.37404, + "lon": 4.88969 + }, + "region_iso_code": "NL-NH", + "region_name": "North Holland" + }, + "ip": "198.51.100.23" + }, + "tls": { + "version": "1.3", + "version_protocol": "tls" + }, + "url": { + "domain": "www.example.com", + "full": "www.example.com/login", + "path": "/login", + "port": 443 + } + }, + { + "@timestamp": "2023-11-14T22:14:20.000Z", + "akamai": { + "siem": { + "config_id": "12345", + "policy_id": "qik1_3333", + "request": { + "headers": { + "Accept": "*/*", + "User-Agent": "curl/8.4.0" + } + }, + "response": { + "headers": { + "Server": "AkamaiGHost" + } + }, + "rule_actions": [ + "monitor" + ], + "rule_tags": [ + "automation/misc" + ], + "rules": [ + { + "ruleActions": "monitor", + "ruleData": "curl", + "ruleMessages": "Request Indicates an automated program", + "ruleTags": "AUTOMATION/MISC", + "rules": "990011" + } + ] + } + }, + "client": { + "address": "203.0.113.5", + "as": { + "number": 64502, + "organization": { + "name": "Documentation ASN" + } + }, + "geo": { + "city_name": "Madrid", + "continent_name": "Europe", + "country_iso_code": "ES", + "country_name": "Spain", + "location": { + "lat": 40.41639, + "lon": -3.7025 + }, + "region_iso_code": "ES-M", + "region_name": "Madrid" + }, + "ip": "203.0.113.5" + }, + "ecs": { + "version": "8.11.0" + }, + "event": { + "category": [ + "network" + ], + "id": "7d0e51aa", + "kind": "event", + "original": "{\"attackData\":{\"appliedAction\":\"monitor\",\"clientIP\":\"203.0.113.5\",\"configId\":\"12345\",\"policyId\":\"qik1_3333\",\"ruleActions\":\"bW9uaXRvcg%3d%3d\",\"ruleData\":\"Y3VybA%3d%3d\",\"ruleMessages\":\"UmVxdWVzdCBJbmRpY2F0ZXMgYW4gYXV0b21hdGVkIHByb2dyYW0%3d\",\"ruleTags\":\"QVVUT01BVElPTi9NSVND\",\"rules\":\"OTkwMDEx\"},\"format\":\"json\",\"geo\":{\"asn\":\"64502\",\"city\":\"MADRID\",\"continent\":\"EU\",\"country\":\"ES\",\"regionCode\":\"M\"},\"httpMessage\":{\"bytes\":\"1024\",\"host\":\"www.example.com\",\"method\":\"GET\",\"path\":\"/api/v1/status\",\"port\":\"443\",\"protocol\":\"HTTP/2\",\"requestId\":\"7d0e51aa\",\"requestHeaders\":\"User-Agent%3a%20curl%2f8.4.0%0d%0aAccept%3a%20*%2f*%0d%0a\",\"responseHeaders\":\"Server%3a%20AkamaiGHost%0d%0a\",\"start\":\"1700000060\",\"status\":\"200\",\"tls\":\"TLSv1.3\"},\"type\":\"akamai_siem\",\"version\":\"1.0\"}", + "start": "2023-11-14T22:14:20.000Z" + }, + "http": { + "request": { + "id": "7d0e51aa", + "method": "GET" + }, + "response": { + "bytes": 1024, + "status_code": 200 + }, + "version": "2" + }, + "network": { + "protocol": "http", + "transport": "tcp" + }, + "observer": { + "type": "proxy", + "vendor": "akamai" + }, + "related": { + "ip": [ + "203.0.113.5" + ] + }, + "source": { + "address": "203.0.113.5", + "as": { + "number": 64502, + "organization": { + "name": "Documentation ASN" + } + }, + "geo": { + "city_name": "Madrid", + "continent_name": "Europe", + "country_iso_code": "ES", + "country_name": "Spain", + "location": { + "lat": 40.41639, + "lon": -3.7025 + }, + "region_iso_code": "ES-M", + "region_name": "Madrid" + }, + "ip": "203.0.113.5" + }, + "tls": { + "version": "1.3", + "version_protocol": "tls" + }, + "url": { + "domain": "www.example.com", + "full": "www.example.com/api/v1/status", + "path": "/api/v1/status", + "port": 443 + } + }, + { + "@timestamp": "2023-11-14T22:15:20.000Z", + "akamai": { + "siem": { + "config_id": "67217", + "policy_id": "PNWD_110088", + "request": { + "headers": { + "User-Agent": "Mozilla/5.0" + } + } + } + }, + "client": { + "address": "10.0.0.1", + "as": { + "number": 64501 + }, + "geo": { + "city_name": "NEWYORK", + "country_iso_code": "US", + "region_iso_code": "US-NY" + }, + "ip": "10.0.0.1" + }, + "ecs": { + "version": "8.11.0" + }, + "event": { + "category": [ + "network" + ], + "id": "3f5b8c10", + "kind": "event", + "original": "{\"attackData\":{\"appliedAction\":\"alert\",\"clientIP\":\"10.0.0.1\",\"configId\":\"67217\",\"policyId\":\"PNWD_110088\"},\"format\":\"json\",\"geo\":{\"asn\":\"64501\",\"city\":\"NEWYORK\",\"continent\":\"NA\",\"country\":\"US\",\"regionCode\":\"NY\"},\"httpMessage\":{\"bytes\":\"256\",\"host\":\"www.example.com\",\"method\":\"GET\",\"path\":\"/\",\"port\":\"443\",\"protocol\":\"HTTP/1.1\",\"requestId\":\"3f5b8c10\",\"requestHeaders\":\"User-Agent%3a%20Mozilla%2f5.0%0d%0a\",\"start\":\"1700000120\",\"status\":\"200\",\"tls\":\"TLSv1.2\"},\"type\":\"akamai_siem\",\"version\":\"1.0\"}", + "start": "2023-11-14T22:15:20.000Z" + }, + "http": { + "request": { + "id": "3f5b8c10", + "method": "GET" + }, + "response": { + "bytes": 256, + "status_code": 200 + }, + "version": "1.1" + }, + "network": { + "protocol": "http", + "transport": "tcp" + }, + "observer": { + "type": "proxy", + "vendor": "akamai" + }, + "related": { + "ip": [ + "10.0.0.1" + ] + }, + "source": { + "address": "10.0.0.1", + "as": { + "number": 64501 + }, + "geo": { + "city_name": "NEWYORK", + "country_iso_code": "US", + "region_iso_code": "US-NY" + }, + "ip": "10.0.0.1" + }, + "tls": { + "version": "1.2", + "version_protocol": "tls" + }, + "url": { + "domain": "www.example.com", + "full": "www.example.com/", + "path": "/", + "port": 443 + } + }, + { + "@timestamp": "2023-11-14T22:16:20.000Z", + "akamai": { + "siem": { + "config_id": "67217", + "policy_id": "PNWD_110088", + "request": { + "headers": { + "User-Agent": "Mozilla/5.0" + } + } + } + }, + "ecs": { + "version": "8.11.0" + }, + "event": { + "category": [ + "network" + ], + "id": "c48d9e02", + "kind": "event", + "original": "{\"attackData\":{\"configId\":\"67217\",\"policyId\":\"PNWD_110088\"},\"format\":\"json\",\"httpMessage\":{\"bytes\":\"0\",\"host\":\"www.example.com\",\"method\":\"GET\",\"path\":\"/health\",\"port\":\"443\",\"protocol\":\"HTTP/1.1\",\"requestId\":\"c48d9e02\",\"requestHeaders\":\"User-Agent%3a%20Mozilla%2f5.0%0d%0a\",\"start\":\"1700000180\",\"status\":\"500\"},\"type\":\"akamai_siem\",\"version\":\"1.0\"}", + "start": "2023-11-14T22:16:20.000Z" + }, + "http": { + "request": { + "id": "c48d9e02", + "method": "GET" + }, + "response": { + "bytes": 0, + "status_code": 500 + }, + "version": "1.1" + }, + "network": { + "protocol": "http", + "transport": "tcp" + }, + "observer": { + "type": "proxy", + "vendor": "akamai" + }, + "url": { + "domain": "www.example.com", + "full": "www.example.com/health", + "path": "/health", + "port": 443 + } } ] } diff --git a/packages/akamai/data_stream/siem/elasticsearch/ingest_pipeline/default.yml b/packages/akamai/data_stream/siem/elasticsearch/ingest_pipeline/default.yml index 806c1bcb72c..8ae3ee0284a 100644 --- a/packages/akamai/data_stream/siem/elasticsearch/ingest_pipeline/default.yml +++ b/packages/akamai/data_stream/siem/elasticsearch/ingest_pipeline/default.yml @@ -261,18 +261,26 @@ processors: field: source.ip target_field: source.geo ignore_missing: true + - set: + tag: set_source_geo_region_iso_code_3a81fa3d + field: source.geo.region_iso_code + description: >- + Must run before the `json.geo.country` rename below: that rename removes the + field whenever GeoIP did not resolve the address, which would leave this + template rendering a malformed code such as `-NY`. Both parts are required + because a value of `-` is not empty, so `ignore_empty_value` does not catch it. + value: '{{{json.geo.country}}}-{{{json.geo.regionCode}}}' + ignore_empty_value: true + if: >- + ctx.source?.geo?.region_iso_code == null && ctx.json.geo?.country != null && + ctx.json.geo.country != '' && ctx.json.geo?.regionCode != null && + ctx.json.geo.regionCode != '' - rename: tag: rename_json_geo_country_to_source_geo_country_iso_code_66520e30 field: json.geo.country target_field: source.geo.country_iso_code ignore_missing: true if: ctx.source?.geo?.country_iso_code == null - - set: - tag: set_source_geo_region_iso_code_3a81fa3d - field: source.geo.region_iso_code - value: "{{{json.geo.country}}}-{{{json.geo.regionCode}}}" - ignore_empty_value: true - if: ctx.source?.geo?.region_iso_code == null - rename: tag: rename_json_geo_city_to_source_geo_city_name_b38d8cc7 field: json.geo.city @@ -392,7 +400,11 @@ processors: ignore_missing: true - script: lang: painless - description: Base64 Decode the json.attackData.rule* fields + description: >- + Base64 Decode the json.attackData.rule* fields. Akamai only sends the rule* + arrays that apply to the event, and `akamai.siem` exists at this point only if + the request or response headers were present, so both the per-key lookup and + the target parents are created defensively. tag: script_base64_decode_attackData_rule_573db939 params: items: @@ -411,8 +423,9 @@ processors: for (def i = 0; i < ctx.json.attackData.rules.length; i++) { HashMap map = new HashMap(); for (String key: params.items) { - if (i < ctx.json.attackData[key].length) { - String data = ctx.json.attackData[key][i].replace(" ", ""); + def values = ctx.json.attackData[key]; + if (values instanceof List && i < values.length) { + String data = values[i].replace(" ", ""); try { String value = data.decodeBase64(); map.put(key, value); @@ -437,6 +450,12 @@ processors: } rules_array.add(map); } + if (ctx.akamai == null) { + ctx.akamai = new HashMap(); + } + if (ctx.akamai.siem == null) { + ctx.akamai.siem = new HashMap(); + } ctx.akamai.siem.rules = rules_array; ctx._rule_actions = rule_actions; ctx._rule_tags = rule_tags; @@ -633,12 +652,17 @@ processors: - append: tag: append_related_ip_feeba47d field: related.ip - value: "{{{source.ip}}}" + value: '{{{source.ip}}}' allow_duplicates: false + if: ctx.source?.ip != null - set: tag: set_client_10dffeb7 field: client copy_from: source + description: >- + Events without an `attackData.clientIP` have no `source` object at all, and + `copy_from` fails on a missing source path. + if: ctx.source != null - set: tag: set_event_category_dbab8a4e field: event.category diff --git a/packages/akamai/data_stream/siem/fields/fields.yml b/packages/akamai/data_stream/siem/fields/fields.yml index fa3666a6d68..eabfd0ae80d 100644 --- a/packages/akamai/data_stream/siem/fields/fields.yml +++ b/packages/akamai/data_stream/siem/fields/fields.yml @@ -19,6 +19,7 @@ type: nested description: > Rules triggered by this request + - name: rules.ruleVersions type: keyword description: Versions of rules triggered for this request. @@ -111,15 +112,12 @@ - name: ja4 type: keyword description: A JA4 TLS client fingerprint value. - - name: tls_fingerprint_v2 type: keyword description: A client TLS fingerprint V2 value. - - name: tls_fingerprint_v3 type: keyword description: A client TLS fingerprint V3 value. - - name: user_risk.uuid type: keyword description: > diff --git a/packages/akamai/data_stream/siem_otel/_dev/test/policy/test-default.yml b/packages/akamai/data_stream/siem_otel/_dev/test/policy/test-default.yml index 46584c2545f..7e66546de7f 100644 --- a/packages/akamai/data_stream/siem_otel/_dev/test/policy/test-default.yml +++ b/packages/akamai/data_stream/siem_otel/_dev/test/policy/test-default.yml @@ -5,4 +5,3 @@ data_stream: client_secret: cs-example access_token: at-example config_ids: "123456" - diff --git a/packages/akamai/data_stream/siem_otel/_dev/test/policy/test-no-cursor.yml b/packages/akamai/data_stream/siem_otel/_dev/test/policy/test-no-cursor.yml index 2dd75a97935..5411db1732a 100644 --- a/packages/akamai/data_stream/siem_otel/_dev/test/policy/test-no-cursor.yml +++ b/packages/akamai/data_stream/siem_otel/_dev/test/policy/test-no-cursor.yml @@ -6,5 +6,3 @@ data_stream: access_token: at-example config_ids: "123456" persist_cursor: false - - diff --git a/packages/akamai/manifest.yml b/packages/akamai/manifest.yml index 5addcca4206..3a2a971189e 100644 --- a/packages/akamai/manifest.yml +++ b/packages/akamai/manifest.yml @@ -1,6 +1,6 @@ name: akamai title: Akamai -version: "4.1.0" +version: "4.1.1" description: Collect logs from Akamai with Elastic Agent. type: integration format_version: "3.6.0" From 30740ea65deedee7091f293e1deb36c2e04485d3 Mon Sep 17 00:00:00 2001 From: Simon Schneider Date: Mon, 31 Aug 2026 08:18:16 +0200 Subject: [PATCH 2/2] [akamai] Point changelog links at the real PR Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014AUSnTxmZK2NCk1JuyyoiH --- packages/akamai/changelog.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/packages/akamai/changelog.yml b/packages/akamai/changelog.yml index 83876c7c538..524a795b4ef 100644 --- a/packages/akamai/changelog.yml +++ b/packages/akamai/changelog.yml @@ -3,13 +3,13 @@ changes: - description: Fix ingest pipeline failures on events that carry attack rules but no HTTP headers, or only a subset of the `attackData.rule*` fields. type: bugfix - link: https://github.com/elastic/integrations/pull/99999 + link: https://github.com/elastic/integrations/pull/20969 - description: Fix malformed `source.geo.region_iso_code` values such as `-NY` or `-` on events whose client address GeoIP cannot resolve. type: bugfix - link: https://github.com/elastic/integrations/pull/99999 + link: https://github.com/elastic/integrations/pull/20969 - description: Do not copy the `client` fields for events that have no client IP. type: bugfix - link: https://github.com/elastic/integrations/pull/99999 + link: https://github.com/elastic/integrations/pull/20969 - version: "4.1.0" changes: - description: Add tags to ingest pipeline processors.