Mapping multiples into OCSF attributes

[rstillio]

Ok Mappers and Maintainers, seeking your expertise on this one.

In our efforts to map multiple security controls to OCSF we're running into a couple of cases where a field in the origin event contains multiples, an array of 2 or more values, and the best/most logical OCSF mapping object has an attribute that does not allow for arrays.

Does the OCSF community have a formal opinion or informal guidance on how to handle:
1. Representing sets of things (IP addresses, email addresses, or other like strings) within or across the categories of classes?
2. Doing so while retaining directionality context if/when present (e.g., src/dst in network context, to/from in email context, etc.)?

One recommendation has been to map only the first value in the array. This is undesirable for a number of reasons.

Another recommendation has been to break each value out and produce it's own "finding" object, in a one-to-many fashion. This is also problematic and undesirable for a lot of reasons.

Additionally, this issue could arise across any OCSF class of objects, and is not limited to just email or network, though I present those as present-day issues we need to solve for ASAP. It seems we need a more universal way to preserve these values when the origin provides sets of things for a reason. Which has me wondering if a "Multiples" profile might be an approach to consider.
In the short run, we will more than likely have to solve for this in our private schema extension (for now) unless and until the project can find a way to solve for this.

And to be clear and avoid confusion, this post is NOT talking about how to construct an array of objects. It's about cases involving multiples of attribute values in any OCSF attribute that is not an array.

Thanks in advance,
Ryan

An Email Oriented Example

Per RFC-5322, both the Reply-To and From fields can contain multiples (e.g., when an email is sent to/from a mailist, or on behalf someone, like an admin). Currently OCSF attributes reply_to and from are not arrays.

[
    {
        "From": "Ryan <ryan@foo.com>, support@foo.com",
        "Sender": "Ryan <ryan@foo.com>",
        "To": "john.doe@customer.com",
        "Reply-To": "Ryan <ryan@foo.com>, support@foo.com",
        "Subject": "Re: how on earth are we going to map multiples into OCSF?"
    }
]

A Network Oriented Example

We have analytics tools that ingest time-series network-centric data (think netflow, firewall, certain cloud data sets) and produce outputs (called observations) based on aggregations and analytics done over time. A resulting observation can contain a set of network entities involved the observation (i.e., an array of IP addresses, or hostnames, or countries) as well as relevant orientation and directional context that must be preserved (internal/external, source/destination, etc.).

Sample 1
Name: External Country Set Deviation
Purpose: Notify when a tracked device has a deviation in the countries it communicates with.

[
    {
        "id": 246181174,
        "time": "2023-10-11T18:30:00Z",
        "creation_time": "2023-10-11 18:30:00+00:00",
        "source": 531074,
        "sha1": "b196a6e3334477a0b09736343968ba94adf3084c",
        "baseobservation_ptr": 246181174,
        "similarity": 0.125,
        "nr_countries": 7,
        "external_ips": "170.64.142.241, 112.54.91.123, 95.229.149.224, 119.201.206.141, 194.26.29.16, 114.35.130.101, 45.120.227.190, 211.219.250.35, 175.182.237.141",
        "country_codes": "AU, CN, IT, KR, RU, TW, VN",
        "observation_type": "country_set_deviation_v1"
    }
]

In the above example, we can't take just the first value of each array without diluting the value and purpose of the entire event. You also can't split the arrays up and produce individual OCSF findings out of this because of value association / relationship requirements.

Sample 2
Name: Excessive Connections to Network Printers
Purpose: Notify when a tracked device exhibits an excessive number of connections to one or more network printers.

[
    {
        "id": 218382461,
        "alert_set": [97669],
        "time": "2023-06-17T07:50:00Z",
        "creation_time": "2023-06-17T07:50:00Z",
        "source": 550622,
        "observation_name": "Excessive Connections to Network Printers",
        "resource_name": "excessive_connections_to_network_printers_v1",
        "printer_ips": "192.168.27.184, 192.168.37.51, 192.168.42.64, 192.168.145.76",
        "printer_connection_count": 629,
        "connected_devices": ""
    }
]

Sample 3
Name: Geographic Watchlist Observation
Purpose: Notify when a tracked device exhibits connections to one or more IP address destinations associated with a geographic area on watchlist observation.

[
    {
        "id": 244639504,
        "time": "2023-10-01T01:02:32Z",
        "creation_time": "2023-10-01 01:02:32+00:00",
        "source": 270800,
        "sha1": "f7920278749b78c4970aa77f2e0af914c130a916",
        "baseobservation_ptr": 244639504,
        "external_ips": "52.82.211.189, 52.82.211.199, 52.82.211.200",
        "country_code": "CN",
        "observation_type": "geographic_watchlist_observation_v2"
    }
]

I could go on, but you get the idea. Recently I saw a 3rd flavor of example log involving a network scanning and discovery device that produced an output containing an array of IPv4 CIDR blocks representing all networks it had discovered in that most recent discovery job. I don't have an example of it to add to this post, but it further supports the point that this "how to handle mapping multiples into attributes" could exist across nearly all OCSF classes and object types.

[pagbabian-splunk]

@rstillio if I understand your comment that "And to be clear and avoid confusion, this post is NOT talking about how to construct an array of objects. It's about cases involving multiples of attribute values in any OCSF attribute that is not an array."

Rather than adding something like external_ips as an array of IP Address (or more generally ips as an array) you want a general way to make any attribute an array? e.g. ip to ips, email to emails etc.?

Or do you not want to use an indexed array at all, but introduce a new construct, like an unordered set?

[rstillio]

@pagbabian-splunk thank you for calling this out, the point you make about an unordered set is an important distinction.

To answer your first question, no. I don't think the ask here is for a capability to arbitrarily make any attribute an array.

In the examples I provide, the fields do all contain unordered sets, they appear to be neither indexed or ordered. I say appear because in the field, mappers have little to no access to the origin vendor source code or documentation. It's possible this circumstance could exist with indexed and ordered values too, but for the sake of this GH Discussion and to stay anchored to the examples I provided above, let's solve for unordered sets. Either way, I don't think it changes the question we're needing to solve for though it may change how it gets solved.

A minor point of clarification, I am not suggesting creating an OCSF attribute called external_ips or printer_ips, the examples I provided above are raw examples from the origin product.

There will be cases like the examples I provided where a source vendor sends an unordered set of values and that set is what has to get mapped into an OCSF attribute somewhere. We are not at liberty to change what the origin sends us, or refactor it in any way. Assume there is no option to go to the origin vendor and require they make changes to their product output to adhere to any current OCSF schema (despite that being a truly valiant and noble quest). We must, as of this hour, take the event as-is and map it. Final wrinkle, I think I've demonstrated with a few examples how this circumstance could occur in and across any OCSF category or class of objects.

What were your thoughts around introducing a new construct of unordered sets?

[cei1219]

Thanks for writing this up, Ryan.

The problem I have seen arises when trying to describe a Network Activity which is summarized (involving dozens, hundreds, or even thousands of network flows).

One Network Activity today can capture at most one flow (one 5-tuple - 2 IPs, 2 Ports, 1 Protocol). This is because each Network Activity holds one Network Endpoint per end of the flow. And a Network Endpoint describes only one Endpoint (not a Subnet)

But when describing a suspicious Network Activity involving a subnet (such as a scanner - one IP scanning hundreds of thousands of IPs in one subnet), it would be nearly impossible to describe this behavior using the current Network Activity object. One would need hundreds or thousands of objects, when a single object could have (mostly) described the lot.

Open to suggestions on how to handle this. One simple way would be to invent a Network Endpoints (plural) object, and allow a Network Activity to describe multiple Network Endpoints (i.e. a subnet, or other)

Does this help?

[pagbabian-splunk]

I had this topic on the agenda last week for the weekly call, but this additional discussion will help especially the examples, so I will add it again to this week's agenda, higher in the list - hopefully @rstillio you can join to make the case better than I can. I have some thoughts, but our only aggregate construct today is the array, which while ordered, can be considered unordered by description of a particular array attribute (frankly, many of them today are inherently unordered). We have another convention, which was introduced in 1.1 as _list suffixes for arrays (i.e. some arrays are plural attribute names, and now some are _list which bugs me a little bit). We can emphasize an unordered set via a _set suffix convention, however the underlying construct at the moment is still an array. It still requires calling out the specific types in the list/set/array as we don't have a construct for heterogeneous aggregates.

[mavam]

This is really interesting problem. Maybe we can solve it algebraically.

Imagine we have separate event space that consists of aggregated events. For example:

This is fully composable: by standardizing the set of aggregation functions, we implicitly define this "lifted" space in a way that people can write detections against it. This would require that OCSF puts in the sweat to define the groupings and aggregations for every event class.

[cei1219]

In the context of a Network Activity, where an aggregation of network flows is what we are describing, it may be good to consider how IP Addresses are described in aggregate.

One very common way is via a subnet mask (coupled with an IP address). This can describe many hundreds of IP addresses easily.

Another way is via a list of IP Addresses.

In both cases, we must realize that a flow (or set of flows) has 2 ends. It has 2 sets of IP addresses. They are not disjoint things.

Example: I observed a host scanning an IP subnet (hundreds of IPs, all belonging to this subnet). In this example, it is important to describe (a) the host doing the scanning, and (b) the target audience being scanned. The first (a) is a single IP address, and (b) is a subnet. This is the network activity of interest.

It would not be useful to simply provide the single IP (w/o the target audience)
It would not be useful to simply provide the subnet (w/o the host doing the scanning)

The network activity has 2 sides, and it would be useful to describe then in the form of a network activity.

I provide this example because (for IP Addresses) it is much more efficient to use a subnet mask than to enumerate all the IPs in a subnet. Know what I mean?

[mavam]

Your example makes sense and is actually compatible the the approach I outlined.

From an OCSF perspective, the question is how much standardization of aggregations (if any) should be there. It's always possible to create custom aggregations outside the spec for individual needs, with your tool of choice. There is never a one-size-fits all solution, but perhaps a 80% one that's worth considering.

Whether the individual 10.0.0.1, 10.0.0.2, 10.0.0.3 aggregate into [10.0.0.1, 10.0.0.2, 10.0.0.3] or in to 10.0.0.0/30 would be up to the user. However, subnet aggregations get lossy when you need exact IP sets. For your scanner example, a "lifted" event could be, say, Network Scan, that has single IP as source and a list of IPs as destination. A Port Scan would have a list of numbers as target.

There are countless examples for meaningful, aggregate events. I bucketized them as "lifted", which in the simplest form, is just a concatenation of all of them. Performing sensible aggregations according to the desired event semantics is where the effort lies. Then standardizing this even harder.