Implementation – Overview – steps 0-6

IBM Resilient SOAR, MITRE ATT&CK, Recorded Future CTI, IBM QRadar SIEM, QRadar Network Insights (QNI)

By Rukhsar Khan on Monday, December 17, 2018

This feed details on how we implemented our solution draft. As mentioned earlier, our core implementation of the MITRE ATT&CK framework is performed in the IBM Resilient SOAR platform. In order to converge Security Operations and DFIR in this single platform we have defined two stages, namely Stage 1 Analysis and Stage 2 Analysis. Stage 1 Analysis corresponds to a Secops Level1 and Level2 team whereas Stage 2 Analysis applies to a DFIR team. See figure 1.13.

Stage 1 Analysis - L1/L2 (Secops):
  • Recorded Future Threat Intelligence (steps 0.2, 5)
  • SIEM Qradar events - Wincollect - Sysmon (steps 2-6)
  • Qradar Network Insights (QNI) Flows (steps 7-10)
  • CB Response (Carbon Black)
  • A10 SSL interception proxy
  • Cuckoo Sandbox
Stage 2 Analysis - L3 (DFIR):
  • Qradar Incident Forensics (QRIF), PCAP
  • Google Rekall Agent Server
  • Volatility
  • Plaso
  • Log2timeline, etc.

Figure 1.13: Solution draft continued

Only the items of the Stage 1 Analysis that we have coloured red are the ones we have integrated bi-directionally with IBM Resilient as part of this blog feed. These are Recorded Future Threat Intelligence, IBM QRadar SIEM with Wincollect and Sysmon for the endpoint sensing and IBM QRadar Network Insights (QNI) for creating network flows (Internet Protocol Flow Information Export (IPFIX)). Our integrations are all based on REST APIs. Every item has some defined steps. These are the steps that we will demonstrate on the following diagrams. Other system and tool integrations will follow later.

Find Recorded Future matched destination IPs and load CTI – steps 0-6

Figure 1.14: Steps 0-6 – Functions diagram

Figure 1.14 is visualizing how we are initially detecting destination IP addresses that have matched against the Recorded Future (RF) CTI and what actions we are taking in order to get context. As a prerequisite for this solution draft to work properly we are first loading the complete MITRE ATT&CK Matrix for Enterprise into a Postgres database named test. To accomplish this we have written a script that leverages the MITRE TAXII server (step 0.1). Secondly we are integrating QRadar with Recorded Future by taking advantage of the available Recorded Future app. This app can be configured in a way that it loads RF provided Risklists into QRadar Reference Sets (step 0.2). By defining an RF risk score we were able to limit the amount of objects – IP addresses, hash values, domain names, etc. – that are loading into these reference sets. E.g. we configured 65 which corresponds to a high risk score. We also specified a time interval for updating the reference sets hourly.

Within QRadar we have created rules that point to the reference sets. If any local-to-remote network traffic that matches against an IP address object in a reference set has been identified, the rule fires an alarm which is called offense in the QRadar terminology. An offense contains all the related event1 and flow data that has been recorded according to the configured rule specification. An offense also includes a list of source and destination IP addresses that have been identified as part of the offense. We are escalating this offense automatically into Resilient which is represented as a new incident (step 1).

The offense escalation process is only taking high-level information from the QRadar offense into the Resilient incident. If the analyst needs to enrich the original incident by additional information he can simply take advantage of Resilient’s automation feature that re-queries the QRadar Ariel2 database and loads the additional information into the incident.

From the list of all the suspicious destination IP addresses that have been detected by QRadar as part of an offense we want to load the RF matched destination IP addresses into the Resilient incident. For this we are re-querying QRadar from within Resilient (steps 2-3) and loading the retrieved RF matched destination IP addresses into the incident's Artifacts (step 4) tab. An artifact in the Resilient context is an IOC. We have configured to trigger another automation that accomplishes an IP lookup for every newly created RF matched IP address artifact (step 5) in order to load the detailed CTI with all the IOC context in our test database (step 6).

The Postgres test database schema can be seen in figure 1.15. We are storing the MITRE provided STIX Domain Objects of a specific type into its own table. Attack Pattern SDOs are for example stored in the mitre_techniques table. As you can see we are simply storing the STIX2 JSON strings into a corresponding column named techniques and the data type for this column is jsonb. Jsonb is a fairly new data type which allows to store a complete JSON string without breaking it down into individual columns. PostgreSQL allows to query in jsonb strings in order to find JSON keys and values. A special kind of query syntax is required for this which we will introduce at some later point in time. Also, we are leveraging Psycopg3, a PosgreSQL database adapter for Python, in order to query the Postgres database from our Python scripts.

Figure 1.15: Step 0.1 – Postgres test DB – MITRE TAXII server load

Figure 1.16 is further visualizing the loading of objects by the RF app provided risklists into QRadar reference sets. Reference sets have a Name, Type, Number of Elements and Associated Rules column. The associated rule is also shown on the right side of this illustration.

Figure 1.16: Step 0.2 – Qradar reference set load

The QRadar offense escalation into Resilient is further illustrated in figure 1.17. As you can see, the offense has a unique number which in our example is 9697. The source IP – or potential victim IP – address is There are a total of 18 remote destination IP addresses, 84 events and 82 flows recorded as part of this offense. Once this offense is escalated into Resilient, either automatically or manually by klicking on the Send to Resilient button on the right top corner of the offense, this offense becomes an incident in the Resilient SOAR platform. When the incident is created, the name and description of the incident are derived from a field mapping configuration in the QRadar Resilient app. This app is installed on QRadar and is required to escalate QRadar offenses to Resilient.

Figure 1.17: Step 1 – Qradar offense escalation

In order to trigger the re-query from Resilient to QRadar we have provided the RECORDED FUTURE: Destination IP match from QRadar re-query action which constitutes step 2 and can be seen in figure 1.18.

Figure 1.18: Step 2 – RECORDED FUTURE: Destination IP match...

This action is provisioned by a menu-item rule, also known as a semi-automation. A semi-automation requires user intervention to trigger the automation. In contrast to this a full automation is provisioned by an automatic rule that specifies one or more match conditions. If the match condition is true, the automation is triggered without user intervention. Figure 1.19 is further elaborating on steps 2-3. On the left side of this illustration we can see that the rule RECORDED FUTURE: Destination IP match from QRadar re-query is calling a workflow. The workflow has only a single function. Functions have a pre-process script, a function processor and a post-process script.

Figure 1.19: Steps 2-3 – Rule / Workflow

The pre-process script defines an input variable name (inputs.ariel_sql_query in our example) and a corresponding value which in our case is a select statement for querying the QRadar Ariel database. Our select statement is simply searching for all destination IP addresses that are part of the incident’s offense ID ( AND have a qidname equal to “Recorded Future IP match”. Qidname is the name of the QRadar event, so we are matching only on events that have matched against the RF CTI. The query time range is further limited to the discovered date of the incident until now. Now is an Ariel Query Language function that specifies the current time. This input parameter is now taken into a function processor. A function processor is a Python script that can provide 3rd party system APIs and is able to integrate with these. In our example we have a function processor with the QRadar Python APIs. We are using this function processor to process our input parameter (select statement) and provide us with a corresponding result set. The result set is formatted as a JSON string and is eventually taken as input into our post-process script. The post-process script in our example is simply iterating over the result set and is creating an IP address artifact with the description “Provided by RECORDED FUTURE TI match" for every included destination IP address (see figure 1.20). Resilient workflows, pre- and post-process scripts, function processors, the REST API and much more are described in detail under the IBM Resilient Developer portal.

Figure 1.20: Step 4 – Result

Next, an automatic rule is triggered as part of step 5 if the following match condition is true: If an Artifact is created AND the Type is equal to IP Address AND its Description is equal to “Provided by RECORDED FUTURE TI match”. As this condition is true in our example, a JSON message is sent to a queue named RF_lookup on the Resilient message bus that triggers an action processor. See figure 1.21.

Figure 1.21: Step 5 – Automatic rule – RECORDED FUTURE: TI lookup...

In contrast to a function processor that is used when you want to write changes back into the Resilient platform efficiently, an action processor can be used when you need to trigger a script but do not necessarily need to write changes back into Resilient. As step 5 doesn’t require this, we have chosen to call an action processor script by this rule. This script performs an IP lookup operation against the Recorded Future Connect API4 in order to search for the detailed CTI for the two RF matched destination IP addresses and and loads the result set into the Postgres test database (step 6). Figure 1.22 shows how we are storing the two result JSON strings in the ip_lookup column of the table stix.

Figure 1.22: Step 6 – Postgres test DB – RECORDED FUTURE TI lookup

Figure 1.23 is displaying some details on the stored CTI for entity (IOC). As you can see, the evidenceDetails key has a list of multiple rules that are providing intelligence around the IOC. Later in our solution draft we will be concentrating on the key relatedIpAddress because we want to verify the local relevance of all the provided related IP addresses for this IOC.

Figure 1.23: CTI – RECORDED FUTURE example JSON

1Event data is equivalent to log data in the QRadar nomenclature

2Ariel is the name of the QRadar log/event database