diff --git a/doc/userguide/rules/datasets.rst b/doc/userguide/rules/datasets.rst index b174922f76da..40dde13e1858 100644 --- a/doc/userguide/rules/datasets.rst +++ b/doc/userguide/rules/datasets.rst @@ -79,12 +79,12 @@ Syntax:: dataset:,,; dataset:, \ - [, type , save , load , state , memcap , hashsize + [, type , save , load , state , memcap , hashsize , format , context_key , value_key , array_key , - remove_key, match subdomain]; + remove_key, match subdomain, mask ]; type - the data type: string, md5, sha256, ipv4, ip + the data type: string, md5, sha256, ipv4, ip, cidr load file name for load the data when Suricata starts up state @@ -92,6 +92,22 @@ state save advanced option to set the file name for saving the in-memory data when Suricata exits. +mask + CIDR datasets only, ``set`` and ``unset`` commands only. Apply a prefix + length to the matched address before adding or removing it, so that the + dataset operates on /N prefixes rather than individual hosts. The value may + be expressed as a decimal integer (``mask 24``), a hex prefix length + (``mask 0x18``), or an IPv4 bitmask (``mask 0xffffff00`` — equivalent to + /24). Valid range: 1–32 for IPv4 prefix lengths and IPv4 bitmasks, 1–128 + for IPv6 prefix lengths. A mask of 0, which would match every address, + is rejected at rule load. The same mask value must be used on both ``set`` + and ``unset`` to ensure the correct prefix is removed. + + ``ip.src`` and ``ip.dst`` fire on both IPv4 and IPv6 packets. A mask + greater than 32 applied to a packet whose address is IPv4 is rejected + at match time and the ``set`` (or ``unset``) does not fire on that + packet. Use two separate rules if the mask semantics for IPv4 and + IPv6 need to differ. memcap maximum memory limit for the respective dataset hashsize @@ -169,6 +185,107 @@ keywords. Keep in mind the cost of additional keywords though e.g. in the second example rule above, negative performance impact can be expected due to ``pcrexform``. +CIDR datasets +~~~~~~~~~~~~~ + +The ``cidr`` dataset type stores IPv4 and IPv6 network ranges and checks +whether a given IP address falls within any of them. It is backed by +radix trees and reads the raw address from the ``ip.src`` or ``ip.dst`` +sticky buffer. + +Dataset file format — one CIDR block per line:: + + # IPv4 ranges + 192.168.0.0/16 + 10.0.0.0/8 + 172.16.0.0/12 + + # Single IPv4 host (equivalent to /32) + 8.8.8.8 + + # IPv6 ranges + fc00::/7 + 2001:db8::/32 + + # Single IPv6 host (equivalent to /128) + 2001:4860:4860::8888 + +Example rules: + +.. container:: example-rule + + alert ip any any -> any any (msg:"Traffic from RFC1918 range"; ip.src; dataset:isset,rfc1918, type cidr, load rfc1918.lst; sid:1; rev:1;) + +.. container:: example-rule + + alert ip any any -> any any (msg:"Traffic to non-allowlisted destination"; ip.dst; dataset:isnotset,allowed-nets, type cidr, load allowed-nets.lst; sid:2; rev:1;) + +``ip.src`` and ``ip.dst`` provide the raw address bytes (4 for IPv4, +16 for IPv6). The CIDR dataset selects the tree from the address +length. + +The ``set`` command adds the matched host address as a /32 (IPv4) or +/128 (IPv6) host entry. Combined with ``mask``, it adds the containing +subnet prefix instead. For example, to build a set of seen /24 networks: + +.. container:: example-rule + + alert ip any any -> any any (msg:"New /24 network seen"; ip.src; dataset:set,seen-nets,type cidr,mask 24; sid:3; rev:1;) + +The ``unset`` command removes an entry that was previously added via ``set`` +or loaded from a file. Without ``mask``, it removes the exact host entry +(/32 or /128). With ``mask N``, it applies the same prefix length as the +corresponding ``set`` rule and removes the /N netblock — use the same mask +value on both ``set`` and ``unset`` to correctly undo the addition. Subnet +entries loaded from a file (e.g. ``192.168.0.0/16``) can be removed via the +unix socket ``dataset-remove`` command. + +``isset`` and ``isnotset`` perform a longest-prefix (best-match) lookup: +the raw host address is matched against every stored prefix and the +most-specific covering entry wins. A source address of ``192.168.1.5`` +matches a stored ``192.168.1.0/24`` directly. ``mask`` is not valid on these commands because the +radix tree already handles the host-to-prefix matching. + +.. note:: ``save`` and ``state`` are not supported for CIDR datasets; only ``load`` is available. Specifying either will cause rule loading to fail. + +.. note:: Reputation values (``datarep``) are not supported for CIDR datasets. + +.. note:: + + ``memcap`` on a CIDR dataset is approximate. Each stored prefix + counts as ~128 bytes for IPv4 and ~192 bytes for IPv6; the radix + tree's internal split nodes are not counted, so real memory use + will be somewhat higher than the tracked total. Once the tracked + total for a family exceeds ``memcap``, further ``set`` operations + for that family are rejected and one warning is logged per family. + A ``memcap`` of 0 disables the cap. Rules that use ``dataset:set`` + on ``ip.src`` or ``ip.dst`` grow the tree in response to network + traffic, so keep ``memcap`` tight when the source is untrusted. + +.. note:: + + A ``dataset:set`` rule on ``ip.src`` or ``ip.dst`` exposed to + attacker-controlled unique sources at multi-Gbps rates can also + saturate the per-family write lock across worker threads. Narrow + the rule with specific ports, protocols, or flow states so the + write path only runs on packets that need to update the set. + +.. note:: + + For deployments that treat the CIDR set as an admin-managed + allow/deny list, use ``dataset:isset`` or ``dataset:isnotset`` in + rules and update the set at runtime through the unix socket + (``dataset-add`` / ``dataset-remove`` for single entries, or + ``dataset-add-batch`` for bulk imports). The packet path takes + only the read lock; admin writes take the write lock for the + length of one tree walk per update, blocking concurrent readers + during that window. Trickle updates are invisible; bulk imports + of thousands of entries can block the packet path for the + duration of the import. Prefer ``dataset-add-batch`` for bulk + loads -- it takes one write lock per address family for the + whole batch instead of one per entry -- and schedule such + imports during quiet windows. + datarep ~~~~~~~ @@ -260,7 +377,7 @@ Syntax:: set name Name of an already defined dataset type - Data type: string, md5, sha256, ipv4, ip + Data type: string, md5, sha256, ipv4, ip, cidr data Data to add in serialized form (base64 for string, hex notation for md5/sha256, string representation for ipv4/ip) @@ -268,6 +385,53 @@ Example adding 'google.com' to set 'myset':: dataset-add myset string Z29vZ2xlLmNvbQ== +dataset-add-batch +~~~~~~~~~~~~~~~~~ + +Unix Socket command to add many values to a set in one call. The command +takes a JSON array of values instead of a single ``data`` argument. + +For CIDR datasets, the whole batch is inserted under one write lock per +address family instead of one lock per entry, which is much cheaper when +loading thousands of prefixes at once. For hash-based dataset types the +per-entry lock is already fine-grained; batching only saves the +per-entry unix-socket round trip. + +Arguments (JSON):: + + { + "command": "dataset-add-batch", + "arguments": { + "setname": "", + "settype": "string|md5|sha256|ipv4|ip|cidr", + "values": ["value1", "value2", ...] + } + } + +The reply reports how each value was handled:: + + { + "return": "OK", + "message": { + "added": , /* new entries inserted */ + "existed": , /* value already present, skipped */ + "failed": , /* malformed or insert failure */ + "rejected_memcap": /* CIDR only: skipped by memcap */ + } + } + +The batch is not atomic: a malformed or memcap-rejected value is +counted in its own bucket while the rest of the batch proceeds. For +CIDR sets, the write lock is held for the whole batch, so a bulk +import of thousands of entries can briefly block the packet path. +Schedule bulk imports during quiet windows. + +Example adding three CIDR blocks to set 'blocklist':: + + {"command": "dataset-add-batch", + "arguments": {"setname": "blocklist", "settype": "cidr", + "values": ["10.0.0.0/8", "192.168.0.0/16", "fc00::/7"]}} + dataset-remove ~~~~~~~~~~~~~~ @@ -281,7 +445,7 @@ Syntax:: set name Name of an already defined dataset type - Data type: string, md5, sha256, ipv4, ip + Data type: string, md5, sha256, ipv4, ip, cidr data Data to remove in serialized form (base64 for string, hex notation for md5/sha256, string representation for ipv4/ip) @@ -298,7 +462,7 @@ Syntax:: set name Name of an already defined dataset type - Data type: string, md5, sha256, ipv4, ip + Data type: string, md5, sha256, ipv4, ip, cidr dataset-lookup ~~~~~~~~~~~~~~ @@ -312,7 +476,7 @@ Syntax:: set name Name of an already defined dataset type - Data type: string, md5, sha256, ipv4, ip + Data type: string, md5, sha256, ipv4, ip, cidr data Data to test in serialized form (base64 for string, hex notation for md5/sha256, string notation for ipv4/ip) @@ -342,7 +506,7 @@ Syntax:: set name Name of an already defined dataset type - Data type: string, md5, sha256, ipv4, ip + Data type: string, md5, sha256, ipv4, ip, cidr data Data to add in serialized form (base64 for string, hex notation for md5/sha256, string representation for ipv4/ip) @@ -369,6 +533,9 @@ ipv4 in the file as string ip in the file as string, it can be IPv6 or IPv4 address (standard notation or IPv4 in IPv6 one) +cidr + in the file as a CIDR block in standard notation (e.g. ``192.168.0.0/16``, ``fc00::/7``); + a bare address without a prefix length is treated as a host route (``/32`` or ``/128``) dataset diff --git a/rust/src/detect/datasets.rs b/rust/src/detect/datasets.rs index f99002cf4518..91bc98ca9bac 100644 --- a/rust/src/detect/datasets.rs +++ b/rust/src/detect/datasets.rs @@ -36,10 +36,19 @@ pub enum DatasetType { DSSha256, DSIpv4, DSIpv6, + DSCIDR, } use suricata_sys::sys::{Dataset, SCDatasetAdd, SCDatasetAddwRep}; +/* DatasetAddCIDRString is not in suricata_sys::sys because the bindgen + * allowlist is 'SC.*'. Renaming to SCDatasetAddCIDRString and adding + * datasets-cidr.h to bindgen.h would let it migrate to a sys import; + * done as a follow-up so this commit doesn't need a bindgen rerun. */ +extern "C" { + fn DatasetAddCIDRString(set: *mut Dataset, cidr_str: *const c_char) -> i32; +} + #[no_mangle] pub unsafe extern "C" fn ParseDatasets( set: &mut Dataset, name: *const c_char, fname: *const c_char, fmode: *const c_char, @@ -51,6 +60,7 @@ pub unsafe extern "C" fn ParseDatasets( let filename = Path::new(file_string); let mut no_rep = false; let mut with_rep = false; + let mut cidr_rep_warned = false; let lines = match read_or_create_file(filename, mode) { Ok(fp) => fp, Err(_) => return -1, @@ -111,6 +121,19 @@ pub unsafe extern "C" fn ParseDatasets( continue; } } + DatasetType::DSCIDR => { + if !no_rep && !cidr_rep_warned { + SCLogWarning!( + "Reputation values not supported for CIDR dataset {} in {}, ignoring", + set_name, + filename.display() + ); + cidr_rep_warned = true; + } + if process_cidr_set(set, v, set_name, filename) == -1 { + continue; + } + } } } @@ -264,6 +287,33 @@ unsafe fn process_ipv6_set( 0 } +unsafe fn process_cidr_set( + set: &mut Dataset, v: Vec<&str>, set_name: &str, filename: &Path, +) -> i32 { + // Add one CIDR entry to the radix tree. Format: "192.168.1.0/24" or + // "2001:db8::/32" or a bare host address. Reputation values are dropped + // here; the caller emits a single warning per (set, file) if any of the + // lines in the file carried a reputation column. + let cidr_str = v[0]; + let c_str = match std::ffi::CString::new(cidr_str) { + Ok(s) => s, + Err(_) => { + SCLogError!("Failed to convert CIDR string to C string: {}", cidr_str); + return -1; + } + }; + if DatasetAddCIDRString(set, c_str.as_ptr()) < 0 { + SCLogError!( + "Failed to add CIDR '{}' to dataset {} from {}", + cidr_str, + set_name, + filename.display() + ); + return -1; + } + 0 +} + fn read_or_create_file

(filename: P, fmode: &str) -> io::Result>> where P: AsRef, diff --git a/src/Makefile.am b/src/Makefile.am index cea03c08a8c5..59292ba28a5e 100755 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -56,6 +56,7 @@ noinst_HEADERS = \ conf-yaml-loader.h \ conf.h \ counters.h \ + datasets-cidr.h \ datasets-context-json.h \ datasets-ipv4.h \ datasets-ipv6.h \ @@ -633,6 +634,7 @@ libsuricata_c_a_SOURCES = \ conf-yaml-loader.c \ conf.c \ counters.c \ + datasets-cidr.c \ datasets-context-json.c \ datasets-ipv4.c \ datasets-ipv6.c \ diff --git a/src/datasets-cidr.c b/src/datasets-cidr.c new file mode 100644 index 000000000000..3f574fc2639e --- /dev/null +++ b/src/datasets-cidr.c @@ -0,0 +1,634 @@ +/* Copyright (C) 2026 Open Information Security Foundation + * + * You can copy, redistribute or modify this Program under the terms of + * the GNU General Public License version 2 as published by the Free + * Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * version 2 along with this program; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA + * 02110-1301, USA. + */ + +/** + * \file + * + * \author Jeff Lucovsky + */ + +#include "suricata-common.h" +#include "datasets-cidr.h" +#include "util-ip.h" + +/* Radix tree configs: no user-data free callback, no debug print callback. */ +static SCRadix4Config radix4_cfg = { 0 }; +static SCRadix6Config radix6_cfg = { 0 }; + +/** + * \internal + * \brief Reserve one entry against a per-family memcap. + * + * Called while the family write lock is held. Bumps `bytes` on success. + * On memcap exhaustion, bumps `rejected` and emits a WARNING on the + * first rejection and again every million rejections, so operators + * see continued pressure rather than a single lost initial warning. + * + * \retval 0 reservation ok; caller should insert + * \retval -2 memcap exhausted; caller should reject with SC_ENOENT-equivalent + */ +static inline int cidr_reserve_entry(_Atomic(uint64_t) * bytes, _Atomic(uint64_t) * rejected, + uint64_t memcap, uint64_t entry_bytes, const char *set_name, const char *family_str) +{ + if (memcap != 0) { + uint64_t cur = atomic_load(bytes); + /* Written as "entry_bytes > memcap - cur" to avoid wrapping when + * memcap is close to UINT64_MAX. cur <= memcap holds because we + * only enter this branch after prior successful reserves. */ + if (cur > memcap || entry_bytes > memcap - cur) { + uint64_t r = atomic_fetch_add(rejected, 1) + 1; + if (r == 1 || (r % 1000000) == 0) { + SCLogWarning("CIDR dataset '%s' memcap reached (%s); rejected %" PRIu64 + " new-entry insert(s) so far", + set_name ? set_name : "(unknown)", family_str, r); + } + return -2; + } + } + atomic_fetch_add(bytes, entry_bytes); + return 0; +} + +/** + * \internal + * \brief Release one entry's byte accounting (opposite of cidr_reserve_entry). + */ +static inline void cidr_release_entry(_Atomic(uint64_t) * bytes, uint64_t entry_bytes) +{ + uint64_t cur = atomic_load(bytes); + if (cur >= entry_bytes) + atomic_fetch_sub(bytes, entry_bytes); +} + +/** + * \brief Parse a CIDR notation string into a binary address and prefix length. + * Does not log on failure -- the caller reports one error after trying + * both address families. + * \param cidr_str Input string, e.g. "192.168.1.0/24" or "2001:db8::/32" + * \param af Address family: AF_INET or AF_INET6 + * \param addr_out Receives the parsed address bytes + * \param mask_out Receives the prefix length + * \param max_mask Maximum valid prefix length (32 for IPv4, 128 for IPv6) + * \retval 0 on success + * \retval -1 on parse error + */ +static int ParseCIDRString( + const char *cidr_str, int af, void *addr_out, int *mask_out, int max_mask) +{ + char ip_copy[256]; + if (strlcpy(ip_copy, cidr_str, sizeof(ip_copy)) >= sizeof(ip_copy)) + return -1; + + int mask = max_mask; + char *slash = strchr(ip_copy, '/'); + if (slash != NULL) { + *slash = '\0'; + const char *prefix_str = slash + 1; + if (*prefix_str == '\0' || *prefix_str == '-') + return -1; + char *endptr = NULL; + errno = 0; + long m = strtol(prefix_str, &endptr, 10); + if (errno != 0 || endptr == prefix_str || *endptr != '\0' || m < 0 || m > max_mask) + return -1; + mask = (int)m; + } + + if (inet_pton(af, ip_copy, addr_out) != 1) + return -1; + + *mask_out = mask; + return 0; +} + +/** + * \brief Parse a CIDR string and dispatch by address family. + * \retval 1 = IPv4, 2 = IPv6, -1 = parse failed + * On success, exactly one of v4/mask4 or v6/mask6 is populated. + */ +static int ParseCIDRAny(const char *cidr_str, struct in_addr *v4, int *mask4, struct in6_addr *v6, + int *mask6) +{ + if (ParseCIDRString(cidr_str, AF_INET, v4, mask4, 32) == 0) + return 1; + if (ParseCIDRString(cidr_str, AF_INET6, v6, mask6, 128) == 0) + return 2; + return -1; +} + +/** + * \brief Retrieve the CIDRType from a Dataset, validating type and initialization + * \param set The dataset + * \retval CIDRType* on success + * \retval NULL if set is NULL, wrong type, or uninitialized + */ +static inline CIDRType *CIDRFromDataset(const Dataset *set) +{ + if (set == NULL || set->type != DATASET_TYPE_CIDR || set->cidr_data == NULL) + return NULL; + return set->cidr_data; +} + +CIDRType *CIDRNew(uint64_t memcap) +{ + CIDRType *cidr = SCCalloc(1, sizeof(*cidr)); + if (cidr == NULL) + return NULL; + cidr->ipv4.tree = SCRadix4TreeInitialize(); + cidr->ipv4.memcap = memcap; + SC_ATOMIC_INIT(cidr->ipv4.bytes); + SC_ATOMIC_INIT(cidr->ipv4.rejected); + SCRWLockInit(&cidr->ipv4.lock, NULL); + cidr->ipv6.tree = SCRadix6TreeInitialize(); + cidr->ipv6.memcap = memcap; + SC_ATOMIC_INIT(cidr->ipv6.bytes); + SC_ATOMIC_INIT(cidr->ipv6.rejected); + SCRWLockInit(&cidr->ipv6.lock, NULL); + return cidr; +} + +void CIDRFree(CIDRType *cidr) +{ + if (cidr == NULL) + return; + SCRadix4TreeRelease(&cidr->ipv4.tree, &radix4_cfg); + SCRWLockDestroy(&cidr->ipv4.lock); + SCRadix6TreeRelease(&cidr->ipv6.tree, &radix6_cfg); + SCRWLockDestroy(&cidr->ipv6.lock); + SCFree(cidr); +} + +void CIDRClear(CIDRType *cidr) +{ + if (cidr == NULL) + return; + /* Hold both write locks for the whole operation so an observer never + * sees "one family empty, the other full". Order ipv4-before-ipv6 is + * fixed to avoid any future deadlock if another path ever takes both. */ + SCRWLockWRLock(&cidr->ipv4.lock); + SCRWLockWRLock(&cidr->ipv6.lock); + + SCRadix4TreeRelease(&cidr->ipv4.tree, &radix4_cfg); + cidr->ipv4.tree = SCRadix4TreeInitialize(); + SC_ATOMIC_SET(cidr->ipv4.bytes, 0); + SC_ATOMIC_SET(cidr->ipv4.rejected, 0); + + SCRadix6TreeRelease(&cidr->ipv6.tree, &radix6_cfg); + cidr->ipv6.tree = SCRadix6TreeInitialize(); + SC_ATOMIC_SET(cidr->ipv6.bytes, 0); + SC_ATOMIC_SET(cidr->ipv6.rejected, 0); + + SCRWLockUnlock(&cidr->ipv6.lock); + SCRWLockUnlock(&cidr->ipv4.lock); +} + +/** + * \brief Add a raw-byte IPv4 network entry, enforcing the tree's memcap. + * \param ipv4 The IPv4 tree wrapper (memcap read from ipv4->memcap) + * \param set_name Dataset name for the memcap warning log + * \param addr 4-byte IPv4 address (host bits may be set; will be masked) + * \param prefix Prefix length (0..32; 32 = host route) + * \retval 1 if a new entry was inserted + * \retval 0 if the entry was already present + * \retval -1 on parameter error + * \retval -2 on memcap exhaustion + */ +int CIDRAddIPv4Netblock( + CIDRIPv4Type *ipv4, const char *set_name, const uint8_t *addr, uint8_t prefix) +{ + if (ipv4 == NULL || addr == NULL || prefix == 0 || prefix > 32) + return -1; + + uint8_t masked[4]; + memcpy(masked, addr, 4); + if (prefix < 32) + MaskIPNetblock(masked, prefix, 32); + + /* Fast path: read-locked probe. Under a spoofed-source flood most + * arrivals re-hit an entry already in the tree; serving those without + * touching the write lock lets readers scale across cores. */ + SCRWLockRDLock(&ipv4->lock); + void *user_data = NULL; + SCRadix4Node *existing = SCRadix4TreeFindNetblock(&ipv4->tree, masked, prefix, &user_data); + SCRWLockUnlock(&ipv4->lock); + if (existing != NULL) + return 0; + + /* Slow path: writer contention. Re-check under the write lock in case + * a concurrent writer inserted the same entry between our probe and + * the wrlock. */ + SCRWLockWRLock(&ipv4->lock); + existing = SCRadix4TreeFindNetblock(&ipv4->tree, masked, prefix, &user_data); + int rc; + if (existing != NULL) { + rc = 0; + } else if (cidr_reserve_entry(&ipv4->bytes_sc_atomic__, &ipv4->rejected_sc_atomic__, + ipv4->memcap, CIDR_IPV4_ENTRY_BYTES, set_name, "IPv4") != 0) { + rc = -2; + } else if (SCRadix4AddKeyIPV4Netblock(&ipv4->tree, &radix4_cfg, masked, prefix, NULL) == + NULL) { + cidr_release_entry(&ipv4->bytes_sc_atomic__, CIDR_IPV4_ENTRY_BYTES); + rc = -1; + } else { + rc = 1; + } + SCRWLockUnlock(&ipv4->lock); + return rc; +} + +/** + * \brief Add a raw-byte IPv6 network entry, enforcing the tree's memcap. + * \param ipv6 The IPv6 tree wrapper (memcap read from ipv6->memcap) + * \param set_name Dataset name for the memcap warning log + * \param addr 16-byte IPv6 address (host bits may be set; will be masked) + * \param prefix Prefix length (0..128; 128 = host route) + * \retval 1 if a new entry was inserted + * \retval 0 if the entry was already present + * \retval -1 on parameter error + * \retval -2 on memcap exhaustion + */ +int CIDRAddIPv6Netblock( + CIDRIPv6Type *ipv6, const char *set_name, const uint8_t *addr, uint8_t prefix) +{ + if (ipv6 == NULL || addr == NULL || prefix == 0 || prefix > 128) + return -1; + + uint8_t masked[16]; + memcpy(masked, addr, 16); + if (prefix < 128) + MaskIPNetblock(masked, prefix, 128); + + SCRWLockRDLock(&ipv6->lock); + void *user_data = NULL; + SCRadix6Node *existing = SCRadix6TreeFindNetblock(&ipv6->tree, masked, prefix, &user_data); + SCRWLockUnlock(&ipv6->lock); + if (existing != NULL) + return 0; + + SCRWLockWRLock(&ipv6->lock); + existing = SCRadix6TreeFindNetblock(&ipv6->tree, masked, prefix, &user_data); + int rc; + if (existing != NULL) { + rc = 0; + } else if (cidr_reserve_entry(&ipv6->bytes_sc_atomic__, &ipv6->rejected_sc_atomic__, + ipv6->memcap, CIDR_IPV6_ENTRY_BYTES, set_name, "IPv6") != 0) { + rc = -2; + } else if (SCRadix6AddKeyIPV6Netblock(&ipv6->tree, &radix6_cfg, masked, prefix, NULL) == + NULL) { + cidr_release_entry(&ipv6->bytes_sc_atomic__, CIDR_IPV6_ENTRY_BYTES); + rc = -1; + } else { + rc = 1; + } + SCRWLockUnlock(&ipv6->lock); + return rc; +} + +/** + * \brief Check if an IPv4 address falls within any CIDR in the tree + * \param ipv4 The IPv4 tree node + * \param addr IPv4 address (4 bytes) + * \retval true if address is covered by a stored prefix + */ +bool CIDRLookupIPv4(CIDRIPv4Type *ipv4, const uint8_t *addr) +{ + if (ipv4 == NULL || addr == NULL) + return false; + + SCRWLockRDLock(&ipv4->lock); + void *user_data = NULL; /* required by API, unused */ + SCRadix4Node *node = SCRadix4TreeFindBestMatch(&ipv4->tree, addr, &user_data); + bool found = (node != NULL); + SCRWLockUnlock(&ipv4->lock); + return found; +} + +/** + * \brief Check if an IPv6 address falls within any CIDR in the tree + * \param ipv6 The IPv6 tree node + * \param addr IPv6 address (16 bytes) + * \retval true if address is covered by a stored prefix + */ +bool CIDRLookupIPv6(CIDRIPv6Type *ipv6, const uint8_t *addr) +{ + if (ipv6 == NULL || addr == NULL) + return false; + + SCRWLockRDLock(&ipv6->lock); + void *user_data = NULL; /* required by API, unused */ + SCRadix6Node *node = SCRadix6TreeFindBestMatch(&ipv6->tree, addr, &user_data); + bool found = (node != NULL); + SCRWLockUnlock(&ipv6->lock); + return found; +} + +/** + * \brief Remove an exact netblock IPv4 entry if present, under a single write lock + * \param ipv4 The IPv4 tree node + * \param addr IPv4 address (4 bytes) + * \param prefix Prefix length (1-32) + * \retval true if the entry was found and removed + * \retval false if the entry was not present + */ +bool CIDRRemoveIPv4Netblock(CIDRIPv4Type *ipv4, const uint8_t *addr, uint8_t prefix) +{ + if (ipv4 == NULL || addr == NULL) + return false; + + /* The tree stores masked network addresses; mask the query address to match. */ + uint8_t masked[4]; + memcpy(masked, addr, 4); + if (prefix < 32) + MaskIPNetblock(masked, prefix, 32); + + /* Remove is admin/rule-driven, not per-packet, so it does not use the + * rdlock-first probe that Add uses. Go straight to the write lock. */ + SCRWLockWRLock(&ipv4->lock); + void *user_data = NULL; + SCRadix4Node *node = SCRadix4TreeFindNetblock(&ipv4->tree, masked, prefix, &user_data); + bool found = (node != NULL); + if (found) { + SCRadix4RemoveKeyIPV4Netblock(&ipv4->tree, &radix4_cfg, masked, prefix); + cidr_release_entry(&ipv4->bytes_sc_atomic__, CIDR_IPV4_ENTRY_BYTES); + } + SCRWLockUnlock(&ipv4->lock); + return found; +} + +/** + * \brief Remove an exact netblock IPv6 entry if present, under a single write lock + * \param ipv6 The IPv6 tree node + * \param addr IPv6 address (16 bytes) + * \param prefix Prefix length (1-128) + * \retval true if the entry was found and removed + * \retval false if the entry was not present + */ +bool CIDRRemoveIPv6Netblock(CIDRIPv6Type *ipv6, const uint8_t *addr, uint8_t prefix) +{ + if (ipv6 == NULL || addr == NULL) + return false; + + /* The tree stores masked network addresses; mask the query address to match. */ + uint8_t masked[16]; + memcpy(masked, addr, 16); + if (prefix < 128) + MaskIPNetblock(masked, prefix, 128); + + /* See CIDRRemoveIPv4Netblock for why there's no rdlock probe here. */ + SCRWLockWRLock(&ipv6->lock); + void *user_data = NULL; + SCRadix6Node *node = SCRadix6TreeFindNetblock(&ipv6->tree, masked, prefix, &user_data); + bool found = (node != NULL); + if (found) { + SCRadix6RemoveKeyIPV6Netblock(&ipv6->tree, &radix6_cfg, masked, prefix); + cidr_release_entry(&ipv6->bytes_sc_atomic__, CIDR_IPV6_ENTRY_BYTES); + } + SCRWLockUnlock(&ipv6->lock); + return found; +} + +/** + * \brief Add a CIDR string to a dataset + * \param set The dataset + * \param cidr_str CIDR string (e.g., "192.168.1.0/24" or "2001:db8::/32") + * \retval 1 on success (added or already present) + * \retval -1 on error + */ +int DatasetAddCIDRString(Dataset *set, const char *cidr_str) +{ + if (cidr_str == NULL) + return -1; + CIDRType *cidr = CIDRFromDataset(set); + if (cidr == NULL) + return -1; + + struct in_addr v4; + struct in6_addr v6; + int mask4 = 0, mask6 = 0; + int r; + switch (ParseCIDRAny(cidr_str, &v4, &mask4, &v6, &mask6)) { + case 1: + r = CIDRAddIPv4Netblock( + &cidr->ipv4, set->name, (const uint8_t *)&v4.s_addr, (uint8_t)mask4); + return r >= 0 ? r : -1; + case 2: + r = CIDRAddIPv6Netblock( + &cidr->ipv6, set->name, (const uint8_t *)&v6.s6_addr, (uint8_t)mask6); + return r >= 0 ? r : -1; + } + SCLogDebug("invalid CIDR address format: %s", cidr_str); + return -1; +} + +/** + * \brief Remove a CIDR string from a dataset + * \param set The dataset + * \param cidr_str CIDR string (e.g., "192.168.1.0/24" or "2001:db8::/32") + * \retval 1 if removed, 0 if not present + * \retval -1 on error + */ +int DatasetRemoveCIDRString(Dataset *set, const char *cidr_str) +{ + if (cidr_str == NULL) + return -1; + CIDRType *cidr = CIDRFromDataset(set); + if (cidr == NULL) + return -1; + + struct in_addr v4; + struct in6_addr v6; + int mask4 = 0, mask6 = 0; + switch (ParseCIDRAny(cidr_str, &v4, &mask4, &v6, &mask6)) { + case 1: + return CIDRRemoveIPv4Netblock( + &cidr->ipv4, (const uint8_t *)&v4.s_addr, (uint8_t)mask4) + ? 1 + : 0; + case 2: + return CIDRRemoveIPv6Netblock( + &cidr->ipv6, (const uint8_t *)&v6.s6_addr, (uint8_t)mask6) + ? 1 + : 0; + } + SCLogDebug("invalid CIDR address format: %s", cidr_str); + return -1; +} + +/** + * \brief Look up an IP address in a CIDR dataset + * \param set The dataset + * \param ip_str IP string (e.g., "192.168.1.5" or "2001:db8::1") + * \retval 1 if found + * \retval 0 if not found + * \retval -1 on error + */ +int DatasetLookupCIDRString(Dataset *set, const char *ip_str) +{ + if (ip_str == NULL || *ip_str == '\0') + return -1; + CIDRType *cidr = CIDRFromDataset(set); + if (cidr == NULL) + return -1; + + struct in_addr v4; + if (inet_pton(AF_INET, ip_str, &v4) == 1) + return CIDRLookupIPv4(&cidr->ipv4, (uint8_t *)&v4.s_addr) ? 1 : 0; + + struct in6_addr v6; + if (inet_pton(AF_INET6, ip_str, &v6) == 1) + return CIDRLookupIPv6(&cidr->ipv6, (uint8_t *)&v6.s6_addr) ? 1 : 0; + + SCLogDebug("invalid IP address format: %s", ip_str); + return -1; +} + +/* Per-entry parsed buckets used only inside DatasetAddCIDRBatch. */ +typedef struct { + uint8_t addr[4]; + uint8_t prefix; +} CIDRBatchV4; + +typedef struct { + uint8_t addr[16]; + uint8_t prefix; +} CIDRBatchV6; + +int DatasetAddCIDRBatch( + Dataset *set, const char **cidr_strs, size_t n, CIDRBatchResult *out) +{ + if (out == NULL) + return -1; + memset(out, 0, sizeof(*out)); + + CIDRType *cidr = CIDRFromDataset(set); + if (cidr == NULL) + return -1; + if (n == 0) + return 0; + + /* Two-pass parse. Pass 1 classifies each input to size the per-family + * buckets exactly; homogeneous v4-only or v6-only batches then use + * O(n * sizeof one-struct) memory instead of O(n * sum-of-both). + * Cost: each valid string is parsed twice. Bulk import isn't parse- + * bound (the wrlock hold dominates), so trading CPU for peak memory + * is the right call here. */ + size_t n4 = 0, n6 = 0; + for (size_t i = 0; i < n; i++) { + if (cidr_strs[i] == NULL) + continue; + struct in_addr in4; + struct in6_addr in6; + int mask; + if (ParseCIDRString(cidr_strs[i], AF_INET, &in4, &mask, 32) == 0) + n4++; + else if (ParseCIDRString(cidr_strs[i], AF_INET6, &in6, &mask, 128) == 0) + n6++; + } + + CIDRBatchV4 *v4 = (n4 > 0) ? SCCalloc(n4, sizeof(*v4)) : NULL; + CIDRBatchV6 *v6 = (n6 > 0) ? SCCalloc(n6, sizeof(*v6)) : NULL; + if ((n4 > 0 && v4 == NULL) || (n6 > 0 && v6 == NULL)) { + SCFree(v4); + SCFree(v6); + return -1; + } + + /* Pass 2: re-parse and fill. Anything that failed pass 1 fails again + * here and increments out->failed. */ + size_t j4 = 0, j6 = 0; + for (size_t i = 0; i < n; i++) { + if (cidr_strs[i] == NULL) { + out->failed++; + continue; + } + struct in_addr in4; + struct in6_addr in6; + int mask; + if (ParseCIDRString(cidr_strs[i], AF_INET, &in4, &mask, 32) == 0) { + memcpy(v4[j4].addr, &in4.s_addr, 4); + v4[j4].prefix = (uint8_t)mask; + if (v4[j4].prefix < 32) + MaskIPNetblock(v4[j4].addr, v4[j4].prefix, 32); + j4++; + } else if (ParseCIDRString(cidr_strs[i], AF_INET6, &in6, &mask, 128) == 0) { + memcpy(v6[j6].addr, &in6.s6_addr, 16); + v6[j6].prefix = (uint8_t)mask; + if (v6[j6].prefix < 128) + MaskIPNetblock(v6[j6].addr, v6[j6].prefix, 128); + j6++; + } else { + out->failed++; + } + } + + /* Phase 2: one wrlock per non-empty family for the whole batch. */ + if (n4 > 0) { + SCRWLockWRLock(&cidr->ipv4.lock); + for (size_t i = 0; i < n4; i++) { + void *ud = NULL; + if (SCRadix4TreeFindNetblock( + &cidr->ipv4.tree, v4[i].addr, v4[i].prefix, &ud) != NULL) { + out->existed++; + continue; + } + if (cidr_reserve_entry(&cidr->ipv4.bytes_sc_atomic__, + &cidr->ipv4.rejected_sc_atomic__, cidr->ipv4.memcap, + CIDR_IPV4_ENTRY_BYTES, set->name, "IPv4") != 0) { + out->rejected_memcap++; + continue; + } + if (SCRadix4AddKeyIPV4Netblock(&cidr->ipv4.tree, &radix4_cfg, v4[i].addr, + v4[i].prefix, NULL) == NULL) { + cidr_release_entry(&cidr->ipv4.bytes_sc_atomic__, CIDR_IPV4_ENTRY_BYTES); + out->failed++; + } else { + out->added++; + } + } + SCRWLockUnlock(&cidr->ipv4.lock); + } + + if (n6 > 0) { + SCRWLockWRLock(&cidr->ipv6.lock); + for (size_t i = 0; i < n6; i++) { + void *ud = NULL; + if (SCRadix6TreeFindNetblock( + &cidr->ipv6.tree, v6[i].addr, v6[i].prefix, &ud) != NULL) { + out->existed++; + continue; + } + if (cidr_reserve_entry(&cidr->ipv6.bytes_sc_atomic__, + &cidr->ipv6.rejected_sc_atomic__, cidr->ipv6.memcap, + CIDR_IPV6_ENTRY_BYTES, set->name, "IPv6") != 0) { + out->rejected_memcap++; + continue; + } + if (SCRadix6AddKeyIPV6Netblock(&cidr->ipv6.tree, &radix6_cfg, v6[i].addr, + v6[i].prefix, NULL) == NULL) { + cidr_release_entry(&cidr->ipv6.bytes_sc_atomic__, CIDR_IPV6_ENTRY_BYTES); + out->failed++; + } else { + out->added++; + } + } + SCRWLockUnlock(&cidr->ipv6.lock); + } + + SCFree(v4); + SCFree(v6); + return 0; +} diff --git a/src/datasets-cidr.h b/src/datasets-cidr.h new file mode 100644 index 000000000000..4de68f55205c --- /dev/null +++ b/src/datasets-cidr.h @@ -0,0 +1,99 @@ +/* Copyright (C) 2026 Open Information Security Foundation + * + * You can copy, redistribute or modify this Program under the terms of + * the GNU General Public License version 2 as published by the Free + * Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * version 2 along with this program; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA + * 02110-1301, USA. + */ + +/** + * \file + * + * \author Jeff Lucovsky + */ + +#ifndef SURICATA_DATASETS_CIDR_H +#define SURICATA_DATASETS_CIDR_H + +#include "datasets.h" +#include "util-radix4-tree.h" +#include "util-radix6-tree.h" +#include "threads.h" + +typedef struct CIDRIPv4Type { + SCRadix4Tree tree; + SCRWLock lock; + /* Approximate bytes consumed by stored entries, for memcap. + * Atomic so future stats consumers can sample without contending the lock. */ + SC_ATOMIC_DECLARE(uint64_t, bytes); + /* Cumulative count of new-entry inserts rejected because memcap was + * reached. Used to re-emit a periodic WARNING so operators see + * continued pressure instead of just the first-hit warning. */ + SC_ATOMIC_DECLARE(uint64_t, rejected); + uint64_t memcap; /* 0 = unlimited; set at CIDRNew, immutable after */ +} CIDRIPv4Type; + +typedef struct CIDRIPv6Type { + SCRadix6Tree tree; + SCRWLock lock; + SC_ATOMIC_DECLARE(uint64_t, bytes); + SC_ATOMIC_DECLARE(uint64_t, rejected); + uint64_t memcap; +} CIDRIPv6Type; + +typedef struct CIDRType { + CIDRIPv4Type ipv4; + CIDRIPv6Type ipv6; +} CIDRType; + +/* Per-entry byte estimate used to check memcap. Node size varies with + * tree topology (internal vs leaf nodes, alignment), and internal split + * nodes are not counted here, so real memory use is higher than the + * tracked total. Treat these as approximate. */ +#define CIDR_IPV4_ENTRY_BYTES 128 +#define CIDR_IPV6_ENTRY_BYTES 192 + +CIDRType *CIDRNew(uint64_t memcap); +void CIDRFree(CIDRType *cidr); +void CIDRClear(CIDRType *cidr); + +/* Raw-byte add helpers. Enforce the memcap configured at CIDRNew time. + * Return 1 if a new entry was added, 0 if it was already present, + * -1 on parameter error, -2 on memcap exhaustion. */ +int CIDRAddIPv4Netblock( + CIDRIPv4Type *ipv4, const char *set_name, const uint8_t *addr, uint8_t prefix); +int CIDRAddIPv6Netblock( + CIDRIPv6Type *ipv6, const char *set_name, const uint8_t *addr, uint8_t prefix); + +bool CIDRLookupIPv4(CIDRIPv4Type *ipv4, const uint8_t *addr); +bool CIDRLookupIPv6(CIDRIPv6Type *ipv6, const uint8_t *addr); +bool CIDRRemoveIPv4Netblock(CIDRIPv4Type *ipv4, const uint8_t *addr, uint8_t prefix); +bool CIDRRemoveIPv6Netblock(CIDRIPv6Type *ipv6, const uint8_t *addr, uint8_t prefix); + +int DatasetAddCIDRString(Dataset *set, const char *cidr_str); +int DatasetRemoveCIDRString(Dataset *set, const char *cidr_str); +int DatasetLookupCIDRString(Dataset *set, const char *cidr_str); + +typedef struct CIDRBatchResult_ { + uint32_t added; /* new entries inserted */ + uint32_t existed; /* already present, skipped */ + uint32_t rejected_memcap; /* skipped because family memcap exhausted */ + uint32_t failed; /* malformed CIDR or radix insert failure */ +} CIDRBatchResult; + +/* Add a batch of CIDR strings under one write lock per address family. + * cidr_strs may contain NULLs; those count as failed. Returns 0 on success + * (out is populated), -1 on setup error (wrong set type or allocation). */ +int DatasetAddCIDRBatch( + Dataset *set, const char **cidr_strs, size_t n, CIDRBatchResult *out); + +#endif /* SURICATA_DATASETS_CIDR_H */ diff --git a/src/datasets-context-json.c b/src/datasets-context-json.c index c15d1d9035d4..e91159f60168 100644 --- a/src/datasets-context-json.c +++ b/src/datasets-context-json.c @@ -724,6 +724,9 @@ Dataset *DatajsonGet(const char *name, enum DatasetTypes type, const char *load, if (DatajsonLoadIPv6(set, json_key_value, json_array_key, format) < 0) goto out_err; break; + case DATASET_TYPE_CIDR: + SCLogError("CIDR datasets are not supported in JSON context (%s)", name); + goto out_err; } SCLogDebug( @@ -881,6 +884,11 @@ DataJsonResultType DatajsonLookup(Dataset *set, const uint8_t *data, const uint3 return DatajsonLookupIPv4(set, data, data_len); case DATASET_TYPE_IPV6: return DatajsonLookupIPv6(set, data, data_len); + case DATASET_TYPE_CIDR: + /* CIDR + JSON is rejected at DatajsonGet; unreachable in practice. + * Log if we somehow get here so the operator sees the mismatch. */ + SCLogWarning("JSON lookup called on CIDR dataset '%s' (unsupported)", set->name); + return rrep; default: break; } @@ -961,6 +969,9 @@ int DatajsonAddSerialized(Dataset *set, const char *value, const char *json) ret = DatajsonAdd(set, (uint8_t *)&in6.s6_addr, SC_IPV6_LEN, &jvalue); break; } + case DATASET_TYPE_CIDR: + SCLogError("CIDR datasets are not supported in JSON context (%s)", set->name); + goto operror; } SCFree(jvalue.value); return ret; diff --git a/src/datasets.c b/src/datasets.c index 6ab64527d4e2..22a147f886fc 100644 --- a/src/datasets.c +++ b/src/datasets.c @@ -31,6 +31,7 @@ #include "datasets-ipv6.h" #include "datasets-md5.h" #include "datasets-sha256.h" +#include "datasets-cidr.h" #include "datasets-reputation.h" #include "datasets-context-json.h" #include "util-conf.h" @@ -73,16 +74,23 @@ enum DatasetTypes DatasetGetTypeFromString(const char *s) return DATASET_TYPE_IPV4; if (strcasecmp("ip", s) == 0) return DATASET_TYPE_IPV6; + if (strcasecmp("cidr", s) == 0) + return DATASET_TYPE_CIDR; return DATASET_TYPE_NOTSET; } int DatasetAppendSet(Dataset *set) { + if (set->type == DATASET_TYPE_CIDR) { + SCLogDebug("set %p/%s type CIDR load %s", set, set->name, set->load); + set->next = sets; + sets = set; + return 0; + } if (set->hash == NULL) { return -1; } - if (SC_ATOMIC_GET(set->hash->memcap_reached)) { SCLogError("dataset too large for set memcap"); return -1; @@ -93,8 +101,6 @@ int DatasetAppendSet(Dataset *set) set->next = sets; sets = set; - - /* hash size accounting */ DatasetUpdateHashsize(set->name, set->hash->config.hash_size); return 0; } @@ -277,6 +283,35 @@ static int DatasetLoadString(Dataset *set) return 0; } +static int DatasetLoadCIDR(Dataset *set, uint64_t memcap) +{ + CIDRType *cidr = CIDRNew(memcap); + if (cidr == NULL) { + SCLogError("Failed to allocate CIDR dataset"); + return -1; + } + set->cidr_data = cidr; + + if (strlen(set->load) == 0) + return 0; + + SCLogConfig("dataset: %s loading from '%s'", set->name, set->load); + + const char *fopen_mode = "r"; + if (strlen(set->save) > 0 && strcmp(set->save, set->load) == 0) { + fopen_mode = "a+"; + } + + int retval = ParseDatasets(set, set->name, set->load, fopen_mode, DSCIDR); + if (retval == -2) { + FatalErrorOnInit("dataset %s could not be processed", set->name); + } else if (retval == -1) { + return -1; + } + + return 0; +} + extern bool g_system; enum DatasetGetPathType { @@ -510,6 +545,10 @@ Dataset *DatasetGet(const char *name, enum DatasetTypes type, const char *save, if (DatasetLoadIPv6(set) < 0) goto out_err; break; + case DATASET_TYPE_CIDR: + if (DatasetLoadCIDR(set, memcap) < 0) + goto out_err; + break; } if (DatasetAppendSet(set) < 0) { @@ -520,7 +559,9 @@ Dataset *DatasetGet(const char *name, enum DatasetTypes type, const char *save, DatasetUnlock(); return set; out_err: - if (set->hash) { + if (set->type == DATASET_TYPE_CIDR) { + CIDRFree(set->cidr_data); + } else if (set->hash) { THashShutdown(set->hash); } SCFree(set); @@ -552,7 +593,7 @@ void DatasetReload(void) continue; } set->hidden = true; - if (dataset_max_total_hashsize > 0) { + if (dataset_max_total_hashsize > 0 && set->type != DATASET_TYPE_CIDR) { DEBUG_VALIDATE_BUG_ON(set->hash->config.hash_size > dataset_used_hashsize); dataset_used_hashsize -= set->hash->config.hash_size; } @@ -581,7 +622,11 @@ void DatasetPostReloadCleanup(void) } else { sets = next; } - THashShutdown(cur->hash); + if (cur->type == DATASET_TYPE_CIDR) { + CIDRFree(cur->cidr_data); + } else { + THashShutdown(cur->hash); + } SCFree(cur); cur = next; } @@ -758,6 +803,26 @@ int DatasetsInit(void) } SCLogDebug("dataset %s: id %u type %s", set_name, dset->id, set_type->val); dset->from_yaml = true; + + } else if (strcmp(set_type->val, "cidr") == 0) { + /* CIDR sets do not persist across restart; save/state + * would silently drop the tree on shutdown. Reject at + * YAML parse to match the rule-load-time check in + * DetectDatasetSetup. */ + if (strlen(save) > 0) { + FatalErrorOnInit( + "save/state is not supported for CIDR dataset '%s'", set_name); + continue; + } + Dataset *dset = DatasetGet(set_name, DATASET_TYPE_CIDR, save, load, + memcap > 0 ? memcap : default_memcap, + hashsize > 0 ? hashsize : default_hashsize); + if (dset == NULL) { + FatalErrorOnInit("failed to setup dataset for %s", set_name); + continue; + } + SCLogDebug("dataset %s: id %u type %s", set_name, dset->id, set_type->val); + dset->from_yaml = true; } list_pos++; @@ -775,7 +840,11 @@ void DatasetsDestroy(void) while (set) { SCLogDebug("destroying set %s", set->name); Dataset *next = set->next; - THashShutdown(set->hash); + if (set->type == DATASET_TYPE_CIDR) { + CIDRFree(set->cidr_data); + } else { + THashShutdown(set->hash); + } SCFree(set); set = next; } @@ -784,6 +853,18 @@ void DatasetsDestroy(void) SCLogDebug("destroying datasets done: %p", sets); } +int DatasetClear(Dataset *set) +{ + if (set == NULL) + return -1; + if (set->type == DATASET_TYPE_CIDR) { + CIDRClear(set->cidr_data); + return 0; + } + THashCleanup(set->hash); + return 0; +} + static int SaveCallback(void *ctx, const uint8_t *data, const uint32_t data_len) { FILE *fp = ctx; @@ -876,6 +957,15 @@ void DatasetsSave(void) case DATASET_TYPE_IPV6: THashWalk(set->hash, IPv6AsAscii, SaveCallback, fp); break; + case DATASET_TYPE_CIDR: + /* Should not happen: DetectDatasetSetup and the YAML init + * both reject save/state for CIDR sets, so set->save should + * be empty here and we should not have reached the switch. + * Guarded loudly in DEBUG_VALIDATION builds; production + * gets a debug trace. */ + DEBUG_VALIDATE_BUG_ON("saving CIDR dataset should be rejected at load"); + SCLogDebug("saving CIDR dataset %s is not implemented", set->name); + break; } fclose(fp); @@ -1088,6 +1178,29 @@ static DataRepResultType DatasetLookupSha256wRep(Dataset *set, return rrep; } +/* DatasetLookupCIDR and DatasetAddCIDR expose the 5/17-byte encoded + * form as a generic entry point for SCDatasetAdd / DatasetLookup. The + * per-packet hot path in DetectDatasetCIDRMatch calls the CIDR helpers + * directly and bypasses this encoding. These functions exist so + * external plugin authors calling SCDatasetAdd on a CIDR-typed set + * still have a working path. */ +static int DatasetLookupCIDR(Dataset *set, const uint8_t *data, const uint32_t data_len) +{ + if (set == NULL || set->cidr_data == NULL) + return -1; + + CIDRType *cidr = set->cidr_data; + + /* 5-byte: IPv4 address + prefix byte (prefix ignored for best-match lookup). + * 17-byte: IPv6 address + prefix byte (same). */ + if (data_len == 5) { + return CIDRLookupIPv4(&cidr->ipv4, data) ? 1 : 0; + } else if (data_len == 17) { + return CIDRLookupIPv6(&cidr->ipv6, data) ? 1 : 0; + } + return -1; +} + /** * \brief see if \a data is part of the set * \param set dataset @@ -1113,6 +1226,8 @@ int DatasetLookup(Dataset *set, const uint8_t *data, const uint32_t data_len) return DatasetLookupIPv4(set, data, data_len); case DATASET_TYPE_IPV6: return DatasetLookupIPv6(set, data, data_len); + case DATASET_TYPE_CIDR: + return DatasetLookupCIDR(set, data, data_len); } return -1; } @@ -1135,6 +1250,13 @@ DataRepResultType DatasetLookupwRep(Dataset *set, const uint8_t *data, const uin return DatasetLookupIPv4wRep(set, data, data_len, rep); case DATASET_TYPE_IPV6: return DatasetLookupIPv6wRep(set, data, data_len, rep); + case DATASET_TYPE_CIDR: + /* This path is guarded by detect-datarep parse-time rejection of + * `type cidr`. If we get here despite that guard (bypass path or + * future caller) we can't warn per-packet without flooding the + * log; leave a debug trace instead. */ + SCLogDebug("datarep lookup not supported for CIDR dataset '%s'", set->name); + return rrep; } return rrep; } @@ -1331,6 +1453,28 @@ static int DatasetAddSha256(Dataset *set, const uint8_t *data, const uint32_t da return -1; } +static int DatasetAddCIDR(Dataset *set, const uint8_t *data, const uint32_t data_len) +{ + if (set == NULL || set->cidr_data == NULL) + return -1; + + CIDRType *cidr = set->cidr_data; + if (data_len == 5) { + uint8_t prefix = data[4]; + if (prefix > 32) + return -2; + int r = CIDRAddIPv4Netblock(&cidr->ipv4, set->name, data, prefix); + return r >= 0 ? r : -1; + } else if (data_len == 17) { + uint8_t prefix = data[16]; + if (prefix > 128) + return -2; + int r = CIDRAddIPv6Netblock(&cidr->ipv6, set->name, data, prefix); + return r >= 0 ? r : -1; + } + return -2; +} + int SCDatasetAdd(Dataset *set, const uint8_t *data, const uint32_t data_len) { if (set == NULL) @@ -1347,6 +1491,8 @@ int SCDatasetAdd(Dataset *set, const uint8_t *data, const uint32_t data_len) return DatasetAddIPv4(set, data, data_len); case DATASET_TYPE_IPV6: return DatasetAddIPv6(set, data, data_len); + case DATASET_TYPE_CIDR: + return DatasetAddCIDR(set, data, data_len); } return -1; } @@ -1368,15 +1514,20 @@ int SCDatasetAddwRep( return DatasetAddIPv4wRep(set, data, data_len, rep); case DATASET_TYPE_IPV6: return DatasetAddIPv6wRep(set, data, data_len, rep); + case DATASET_TYPE_CIDR: + SCLogError("datarep is not supported for CIDR datasets (set '%s')", set->name); + return -1; } return -1; } typedef int (*DatasetOpFunc)(Dataset *set, const uint8_t *data, const uint32_t data_len); +typedef int (*DatasetOpCIDRFunc)(Dataset *set, const char *cidr_str); + static int DatasetOpSerialized(Dataset *set, const char *string, DatasetOpFunc DatasetOpString, DatasetOpFunc DatasetOpMd5, DatasetOpFunc DatasetOpSha256, DatasetOpFunc DatasetOpIPv4, - DatasetOpFunc DatasetOpIPv6) + DatasetOpFunc DatasetOpIPv6, DatasetOpCIDRFunc DatasetOpCIDR) { if (set == NULL) return -1; @@ -1415,6 +1566,12 @@ static int DatasetOpSerialized(Dataset *set, const char *string, DatasetOpFunc D return -2; return DatasetOpSha256(set, hash, 32); } + case DATASET_TYPE_CIDR: + if (DatasetOpCIDR != NULL) { + return DatasetOpCIDR(set, string); + } + SCLogWarning("serialized CIDR operation not available for set %s", set->name); + return -1; case DATASET_TYPE_IPV4: { struct in_addr in; if (inet_pton(AF_INET, string, &in) != 1) @@ -1442,7 +1599,7 @@ static int DatasetOpSerialized(Dataset *set, const char *string, DatasetOpFunc D int DatasetAddSerialized(Dataset *set, const char *string) { return DatasetOpSerialized(set, string, DatasetAddString, DatasetAddMd5, DatasetAddSha256, - DatasetAddIPv4, DatasetAddIPv6); + DatasetAddIPv4, DatasetAddIPv6, DatasetAddCIDRString); } /** \brief add serialized data to set @@ -1454,7 +1611,7 @@ int DatasetAddSerialized(Dataset *set, const char *string) int DatasetLookupSerialized(Dataset *set, const char *string) { return DatasetOpSerialized(set, string, DatasetLookupString, DatasetLookupMd5, - DatasetLookupSha256, DatasetLookupIPv4, DatasetLookupIPv6); + DatasetLookupSha256, DatasetLookupIPv4, DatasetLookupIPv6, DatasetLookupCIDRString); } /** @@ -1523,6 +1680,33 @@ static int DatasetRemoveSha256(Dataset *set, const uint8_t *data, const uint32_t return THashRemoveFromHash(set->hash, &lookup); } +/** \brief Remove an entry from a CIDR dataset using binary address bytes. + * 4 bytes = IPv4 /32, 5 bytes = IPv4 with prefix, 16 bytes = IPv6 /128, + * 17 bytes = IPv6 with prefix. Only removes entries that exist at the + * exact prefix length specified. + * \retval 1 removed + * \retval 0 not present + * \retval -1 error + * \retval -2 bad data length */ +static int DatasetRemoveCIDR(Dataset *set, const uint8_t *data, const uint32_t data_len) +{ + if (set == NULL || set->cidr_data == NULL) + return -1; + CIDRType *cidr = set->cidr_data; + if (data_len == 5) { + uint8_t prefix = data[4]; + if (prefix > 32) + return -2; + return CIDRRemoveIPv4Netblock(&cidr->ipv4, data, prefix) ? 1 : 0; + } else if (data_len == 17) { + uint8_t prefix = data[16]; + if (prefix > 128) + return -2; + return CIDRRemoveIPv6Netblock(&cidr->ipv6, data, prefix) ? 1 : 0; + } + return -2; +} + /** \brief remove serialized data from set * \retval int 1 removed * \retval int 0 found but busy (not removed) @@ -1531,7 +1715,7 @@ static int DatasetRemoveSha256(Dataset *set, const uint8_t *data, const uint32_t int DatasetRemoveSerialized(Dataset *set, const char *string) { return DatasetOpSerialized(set, string, DatasetRemoveString, DatasetRemoveMd5, - DatasetRemoveSha256, DatasetRemoveIPv4, DatasetRemoveIPv6); + DatasetRemoveSha256, DatasetRemoveIPv4, DatasetRemoveIPv6, DatasetRemoveCIDRString); } int DatasetRemove(Dataset *set, const uint8_t *data, const uint32_t data_len) @@ -1550,6 +1734,8 @@ int DatasetRemove(Dataset *set, const uint8_t *data, const uint32_t data_len) return DatasetRemoveIPv4(set, data, data_len); case DATASET_TYPE_IPV6: return DatasetRemoveIPv6(set, data, data_len); + case DATASET_TYPE_CIDR: + return DatasetRemoveCIDR(set, data, data_len); } return -1; } diff --git a/src/datasets.h b/src/datasets.h index 452dcaed392a..d4705301604e 100644 --- a/src/datasets.h +++ b/src/datasets.h @@ -49,6 +49,7 @@ enum DatasetTypes { DATASET_TYPE_SHA256, DATASET_TYPE_IPV4, DATASET_TYPE_IPV6, + DATASET_TYPE_CIDR, }; #define DATASET_NAME_MAX_LEN 63 @@ -60,6 +61,7 @@ typedef struct Dataset { bool hidden; /* Mark the old sets hidden in case of reload */ bool remove_key; /* Mark that value key should be removed from extra data */ THashTableContext *hash; + struct CIDRType *cidr_data; /* Radix tree wrapper for DATASET_TYPE_CIDR; NULL otherwise */ char load[PATH_MAX]; char save[PATH_MAX]; @@ -78,6 +80,7 @@ Dataset *DatasetGet(const char *name, enum DatasetTypes type, const char *save, uint64_t memcap, uint32_t hashsize); int DatasetGetOrCreate(const char *name, enum DatasetTypes type, const char *save, const char *load, uint64_t *memcap, uint32_t *hashsize, Dataset **ret_set); +int DatasetClear(Dataset *set); int DatasetRemove(Dataset *set, const uint8_t *data, const uint32_t data_len); int DatasetLookup(Dataset *set, const uint8_t *data, const uint32_t data_len); DataRepResultType DatasetLookupwRep(Dataset *set, const uint8_t *data, const uint32_t data_len, diff --git a/src/detect-datarep.c b/src/detect-datarep.c index bae9c62751af..4ec5459982d8 100644 --- a/src/detect-datarep.c +++ b/src/detect-datarep.c @@ -167,6 +167,9 @@ static int DetectDatarepParse(const char *str, char *cmd, int cmd_len, char *nam *type = DATASET_TYPE_IPV6; } else if (strcmp(val, "ipv6") == 0) { *type = DATASET_TYPE_IPV6; + } else if (strcmp(val, "cidr") == 0) { + SCLogError("datarep does not support type cidr"); + return -1; } else { SCLogDebug("bad type %s", val); return -1; diff --git a/src/detect-dataset.c b/src/detect-dataset.c index 4dabbd6c2c80..bdd4fa2bd9b5 100644 --- a/src/detect-dataset.c +++ b/src/detect-dataset.c @@ -28,6 +28,7 @@ #include "detect.h" #include "threads.h" #include "datasets.h" +#include "datasets-cidr.h" #include "datasets-context-json.h" #include "detect-dataset.h" @@ -158,13 +159,56 @@ static int DatasetLookupSubdomain(Dataset *set, const uint8_t *data, const uint3 return 0; } +/** + * \brief CIDR fast path: dispatch straight to CIDR helpers without building + * the 5/17-byte encoded buffer required by the generic entry points. + */ +static int DetectDatasetCIDRMatch( + const DetectDatasetData *sd, const uint8_t *data, const uint32_t data_len) +{ + if (data_len != 4 && data_len != 16) + return 0; + CIDRType *cidr = sd->set->cidr_data; + if (cidr == NULL) + return 0; + + switch (sd->cmd) { + case DETECT_DATASET_CMD_ISSET: { + bool found = (data_len == 4) ? CIDRLookupIPv4(&cidr->ipv4, data) + : CIDRLookupIPv6(&cidr->ipv6, data); + return found ? 1 : 0; + } + case DETECT_DATASET_CMD_ISNOTSET: { + bool found = (data_len == 4) ? CIDRLookupIPv4(&cidr->ipv4, data) + : CIDRLookupIPv6(&cidr->ipv6, data); + return found ? 0 : 1; + } + case DETECT_DATASET_CMD_SET: { + uint8_t prefix = sd->mask ? sd->mask : (data_len == 4 ? 32 : 128); + int r = (data_len == 4) + ? CIDRAddIPv4Netblock(&cidr->ipv4, sd->set->name, data, prefix) + : CIDRAddIPv6Netblock(&cidr->ipv6, sd->set->name, data, prefix); + return r == 1 ? 1 : 0; + } + case DETECT_DATASET_CMD_UNSET: { + uint8_t prefix = sd->mask ? sd->mask : (data_len == 4 ? 32 : 128); + bool removed = (data_len == 4) + ? CIDRRemoveIPv4Netblock(&cidr->ipv4, data, prefix) + : CIDRRemoveIPv6Netblock(&cidr->ipv6, data, prefix); + return removed ? 1 : 0; + } + default: + DEBUG_VALIDATE_BUG_ON("unknown dataset command"); + } + return 0; +} + /* 1 match 0 no match */ -int DetectDatasetBufferMatch(DetectEngineThreadCtx *det_ctx, - const DetectDatasetData *sd, - const uint8_t *data, const uint32_t data_len) +int DetectDatasetBufferMatch(DetectEngineThreadCtx *det_ctx, const DetectDatasetData *sd, + const uint8_t *data, const uint32_t data_len) { if (data == NULL || data_len == 0) return 0; @@ -173,9 +217,12 @@ int DetectDatasetBufferMatch(DetectEngineThreadCtx *det_ctx, return DetectDatajsonBufferMatch(det_ctx, sd, data, data_len); } + if (sd->set->type == DATASET_TYPE_CIDR) { + return DetectDatasetCIDRMatch(sd, data, data_len); + } + switch (sd->cmd) { case DETECT_DATASET_CMD_ISSET: { - //PrintRawDataFp(stdout, data, data_len); int r = DatasetLookup(sd->set, data, data_len); if (r != 1 && sd->match_subdomain) { r = DatasetLookupSubdomain(sd->set, data, data_len); @@ -186,7 +233,6 @@ int DetectDatasetBufferMatch(DetectEngineThreadCtx *det_ctx, break; } case DETECT_DATASET_CMD_ISNOTSET: { - //PrintRawDataFp(stdout, data, data_len); int r = DatasetLookup(sd->set, data, data_len); if (r != 1 && sd->match_subdomain) { r = DatasetLookupSubdomain(sd->set, data, data_len); @@ -197,7 +243,6 @@ int DetectDatasetBufferMatch(DetectEngineThreadCtx *det_ctx, break; } case DETECT_DATASET_CMD_SET: { - //PrintRawDataFp(stdout, data, data_len); int r = SCDatasetAdd(sd->set, data, data_len); if (r == 1) return 1; @@ -219,7 +264,7 @@ static int DetectDatasetParse(const char *str, char *cmd, int cmd_len, char *nam enum DatasetTypes *type, char *load, size_t load_size, char *save, size_t save_size, uint64_t *memcap, uint32_t *hashsize, DatasetFormats *format, char *value_key, size_t value_key_size, char *array_key, size_t array_key_size, char *enrichment_key, - size_t enrichment_key_size, bool *remove_key, bool *match_subdomain) + size_t enrichment_key_size, bool *remove_key, bool *match_subdomain, uint8_t *mask) { bool cmd_set = false; bool name_set = false; @@ -293,6 +338,8 @@ static int DetectDatasetParse(const char *str, char *cmd, int cmd_len, char *nam *type = DATASET_TYPE_IPV6; } else if (strcmp(val, "ip") == 0) { *type = DATASET_TYPE_IPV6; + } else if (strcmp(val, "cidr") == 0) { + *type = DATASET_TYPE_CIDR; } else { SCLogError("bad type %s", val); return -1; @@ -383,6 +430,43 @@ static int DetectDatasetParse(const char *str, char *cmd, int cmd_len, char *nam *hashsize = 0; } } + if (strcmp(key, "mask") == 0) { + if (val[0] == '-') { + SCLogError("invalid mask value '%s': must not be negative", val); + return -1; + } + char *endptr; + errno = 0; + unsigned long m = strtoul(val, &endptr, 0); + if (errno != 0 || endptr == val || *endptr != '\0') { + SCLogError("invalid mask value '%s'", val); + return -1; + } + uint8_t prefix; + if (m <= 128) { + /* direct prefix length: 24, 0x18, etc. */ + prefix = (uint8_t)m; + } else if (m <= 0xffffffffUL) { + /* IPv4 bitmask notation, e.g. 0xffffff00 == /24. + * Must be a contiguous run of leading 1-bits. */ + uint32_t bm = (uint32_t)m; + uint32_t inv = ~bm; + if (inv != 0 && (inv & (inv + 1)) != 0) { + SCLogError("invalid mask value '%s': not a contiguous IPv4 netmask", val); + return -1; + } + prefix = (uint8_t)__builtin_popcount(bm); + } else { + SCLogError( + "invalid mask value '%s': must be 1-128 or a valid IPv4 netmask", val); + return -1; + } + if (prefix == 0) { + SCLogError("invalid mask value '%s': prefix length 0 is not allowed", val); + return -1; + } + *mask = prefix; + } } SCLogDebug("key: %s, value: %s", key, val); @@ -532,6 +616,7 @@ int DetectDatasetSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawst char enrichment_key[SIG_JSON_CONTENT_KEY_LEN] = ""; bool remove_key = false; bool match_subdomain = false; + uint8_t mask = 0; if (DetectBufferGetActiveList(de_ctx, s) == -1) { SCLogError("datasets are only supported for sticky buffers"); @@ -544,10 +629,10 @@ int DetectDatasetSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawst SCReturnInt(-1); } - if (!DetectDatasetParse(rawstr, cmd_str, sizeof(cmd_str), name, sizeof(name), &type, load, + if (DetectDatasetParse(rawstr, cmd_str, sizeof(cmd_str), name, sizeof(name), &type, load, sizeof(load), save, sizeof(save), &memcap, &hashsize, &format, value_key, sizeof(value_key), array_key, sizeof(array_key), enrichment_key, - sizeof(enrichment_key), &remove_key, &match_subdomain)) { + sizeof(enrichment_key), &remove_key, &match_subdomain, &mask) != 1) { return -1; } @@ -583,6 +668,32 @@ int DetectDatasetSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawst } } + if (type == DATASET_TYPE_CIDR) { + if (strlen(save) != 0) { + SCLogError("save/state is not supported for CIDR datasets"); + return -1; + } + if (mask > 0 && cmd != DETECT_DATASET_CMD_SET && cmd != DETECT_DATASET_CMD_UNSET) { + SCLogError("mask is only supported for CIDR datasets with 'set' and 'unset' commands"); + return -1; + } + /* CIDR datasets read raw address bytes from ip.src or ip.dst. + * Any other sticky buffer (http.uri, dns.query, ...) delivers + * arbitrary-length content that the match path can't turn into + * an IPv4 or IPv6 address, so the rule would silently never + * match (or, for isnotset, would silently invert). Reject at + * load time with a clear error. */ + int ip_src_id = DetectBufferTypeGetByName("ip.src"); + int ip_dst_id = DetectBufferTypeGetByName("ip.dst"); + if (list != ip_src_id && list != ip_dst_id) { + SCLogError("CIDR datasets require the ip.src or ip.dst sticky buffer"); + return -1; + } + } else if (mask > 0) { + SCLogError("mask is only supported for CIDR datasets"); + return -1; + } + if ((format == DATASET_FORMAT_JSON) || (format == DATASET_FORMAT_NDJSON)) { if (strlen(save) != 0) { SCLogError("json format is not supported with 'save' or 'state' option"); @@ -647,6 +758,7 @@ int DetectDatasetSetup (DetectEngineCtx *de_ctx, Signature *s, const char *rawst cd->set = set; cd->cmd = cmd; + cd->mask = mask; cd->format = format; cd->match_subdomain = match_subdomain; if ((format == DATASET_FORMAT_JSON) || (format == DATASET_FORMAT_NDJSON)) { diff --git a/src/detect-dataset.h b/src/detect-dataset.h index 7022b6084ff4..dc07c3c405c1 100644 --- a/src/detect-dataset.h +++ b/src/detect-dataset.h @@ -30,6 +30,7 @@ typedef struct DetectDatasetData_ { Dataset *set; uint8_t cmd; + uint8_t mask; /* CIDR prefix length for 'set'/'unset' with mask option; 0 = no mask */ bool match_subdomain; DatasetFormats format; DataJsonType json; diff --git a/src/runmode-unix-socket.c b/src/runmode-unix-socket.c index 706a35b7e6e7..453af294f2d7 100644 --- a/src/runmode-unix-socket.c +++ b/src/runmode-unix-socket.c @@ -55,6 +55,7 @@ #include "conf-yaml-loader.h" #include "datasets.h" +#include "datasets-cidr.h" #include "datasets-context-json.h" #include "runmode-unix-socket.h" @@ -660,6 +661,120 @@ TmEcode UnixSocketDatasetAdd(json_t *cmd, json_t* answer, void *data) } } +/** + * Add many values to a dataset in one call. + * + * Arguments: setname (string), settype (string), values (array of strings). + * + * CIDR datasets take a batch fast path that holds the per-family write + * lock once for the whole batch instead of once per entry. Other types + * loop through DatasetAddSerialized; the win there is fewer unix-socket + * round trips, not lock consolidation. + */ +TmEcode UnixSocketDatasetAddBatch(json_t *cmd, json_t *answer, void *data) +{ + json_t *narg = json_object_get(cmd, "setname"); + if (!json_is_string(narg)) { + json_object_set_new(answer, "message", json_string("setname is not a string")); + return TM_ECODE_FAILED; + } + const char *set_name = json_string_value(narg); + + json_t *targ = json_object_get(cmd, "settype"); + if (!json_is_string(targ)) { + json_object_set_new(answer, "message", json_string("settype is not a string")); + return TM_ECODE_FAILED; + } + const char *type = json_string_value(targ); + + json_t *values = json_object_get(cmd, "values"); + if (!json_is_array(values)) { + json_object_set_new( + answer, "message", json_string("values must be a JSON array of strings")); + return TM_ECODE_FAILED; + } + size_t n = json_array_size(values); + + /* Cap batch size so a malformed control-plane request cannot request + * an arbitrarily large transient allocation or wrap the size_t product + * used by SCCalloc below. Admin socket only, but 1M entries is well + * beyond any legitimate bulk import. */ +#define DATASET_BATCH_MAX 1000000 + if (n > DATASET_BATCH_MAX) { + json_object_set_new(answer, "message", + json_string("values array exceeds per-call limit of 1000000 entries")); + return TM_ECODE_FAILED; + } + + enum DatasetTypes t = DatasetGetTypeFromString(type); + if (t == DATASET_TYPE_NOTSET) { + json_object_set_new(answer, "message", json_string("unknown settype")); + return TM_ECODE_FAILED; + } + + Dataset *set = DatasetFind(set_name, t); + if (set == NULL) { + json_object_set_new(answer, "message", json_string("set not found or wrong type")); + return TM_ECODE_FAILED; + } + + uint32_t added = 0, existed = 0, failed = 0, rejected_memcap = 0; + + /* Empty array is not an error; just return zeroed counters. */ + if (n == 0) + goto reply; + + if (set->type == DATASET_TYPE_CIDR) { + /* SCCalloc's checked multiplication guards against a size_t + * overflow on 32-bit builds. */ + const char **strs = SCCalloc(n, sizeof(*strs)); + if (strs == NULL) { + json_object_set_new(answer, "message", json_string("allocation failed")); + return TM_ECODE_FAILED; + } + for (size_t i = 0; i < n; i++) { + json_t *v = json_array_get(values, i); + strs[i] = json_is_string(v) ? json_string_value(v) : NULL; + } + CIDRBatchResult r; + int rc = DatasetAddCIDRBatch(set, strs, n, &r); + SCFree(strs); + if (rc < 0) { + json_object_set_new(answer, "message", json_string("batch add failed")); + return TM_ECODE_FAILED; + } + added = r.added; + existed = r.existed; + rejected_memcap = r.rejected_memcap; + failed = r.failed; + } else { + for (size_t i = 0; i < n; i++) { + json_t *v = json_array_get(values, i); + if (!json_is_string(v)) { + failed++; + continue; + } + int rc = DatasetAddSerialized(set, json_string_value(v)); + if (rc == 1) + added++; + else if (rc == 0) + existed++; + else + failed++; + } + } + +reply:; + json_t *msg = json_object(); + json_object_set_new(msg, "added", json_integer(added)); + json_object_set_new(msg, "existed", json_integer(existed)); + json_object_set_new(msg, "failed", json_integer(failed)); + if (set->type == DATASET_TYPE_CIDR) + json_object_set_new(msg, "rejected_memcap", json_integer(rejected_memcap)); + json_object_set_new(answer, "message", msg); + return TM_ECODE_OK; +} + TmEcode UnixSocketDatasetRemove(json_t *cmd, json_t* answer, void *data) { /* 1 get dataset name */ @@ -752,7 +867,10 @@ TmEcode UnixSocketDatasetClear(json_t *cmd, json_t *answer, void *data) return TM_ECODE_FAILED; } - THashCleanup(set->hash); + if (DatasetClear(set) != 0) { + json_object_set_new(answer, "message", json_string("failed to clear dataset")); + return TM_ECODE_FAILED; + } json_object_set_new(answer, "message", json_string("dataset cleared")); return TM_ECODE_OK; diff --git a/src/runmode-unix-socket.h b/src/runmode-unix-socket.h index dfd76b985d67..8d8b43ae5089 100644 --- a/src/runmode-unix-socket.h +++ b/src/runmode-unix-socket.h @@ -34,6 +34,7 @@ float MemcapsGetPressure(void); #ifdef BUILD_UNIX_SOCKET TmEcode UnixSocketDatasetAdd(json_t *cmd, json_t* answer, void *data); +TmEcode UnixSocketDatasetAddBatch(json_t *cmd, json_t *answer, void *data); TmEcode UnixSocketDatasetRemove(json_t *cmd, json_t* answer, void *data); TmEcode UnixSocketDatasetDump(json_t *cmd, json_t *answer, void *data); TmEcode UnixSocketDatasetClear(json_t *cmd, json_t *answer, void *data); diff --git a/src/unix-manager.c b/src/unix-manager.c index 6885d205348d..8c9140f01952 100644 --- a/src/unix-manager.c +++ b/src/unix-manager.c @@ -1113,6 +1113,8 @@ int UnixManagerInit(void) UnixManagerRegisterCommand("memcap-list", UnixSocketShowAllMemcap, NULL, 0); UnixManagerRegisterCommand("dataset-add", UnixSocketDatasetAdd, &command, UNIX_CMD_TAKE_ARGS); + UnixManagerRegisterCommand( + "dataset-add-batch", UnixSocketDatasetAddBatch, &command, UNIX_CMD_TAKE_ARGS); UnixManagerRegisterCommand("dataset-remove", UnixSocketDatasetRemove, &command, UNIX_CMD_TAKE_ARGS); UnixManagerRegisterCommand( "dataset-add-json", UnixSocketDatajsonAdd, &command, UNIX_CMD_TAKE_ARGS); diff --git a/src/util-lua-dataset.c b/src/util-lua-dataset.c index e9e3fe9e7f56..f22dedbf21d4 100644 --- a/src/util-lua-dataset.c +++ b/src/util-lua-dataset.c @@ -33,6 +33,7 @@ #include "util-debug.h" #include "datasets.h" +#include "datasets-cidr.h" struct LuaDataset { Dataset *set; @@ -63,7 +64,7 @@ static int LuaDatasetGetRef(lua_State *luastate) LUA_ERROR("null string"); } - Dataset *dataset = DatasetFind(name, DATASET_TYPE_STRING); + Dataset *dataset = DatasetSearchByName(name); if (dataset == NULL) { LUA_ERROR("dataset not found"); } @@ -92,7 +93,14 @@ static int LuaDatasetAdd(lua_State *luastate) uint32_t str_len = lua_tonumber(luastate, 3); - int r = SCDatasetAdd(s->set, (const uint8_t *)str, str_len); + int r; + if (s->set != NULL && s->set->type == DATASET_TYPE_CIDR) { + /* CIDR sets store binary IPs; treat the Lua string as a CIDR text + * value and dispatch through the string helper. */ + r = DatasetAddCIDRString(s->set, (const char *)str); + } else { + r = SCDatasetAdd(s->set, (const uint8_t *)str, str_len); + } /* return value through luastate, as a luanumber */ lua_pushnumber(luastate, (lua_Number)r); SCLogDebug("add:end");