Dealing with a great amount of data can be time consuming, thus using Python can be very powerful to help analysts sort information and extract the most relevant data for their investigation. The open-source tools library, MSTICPy, for example, is a Python tool dedicated to threat intelligence. It aims to help threat analysts acquire, enrich, analyze, and visualize data.
This blog provides a workflow for deeper data analysis and visualization using Python, as well as for extraction and analysis of indicators of compromise (IOCs) using MSTICPy. Data sets from the February 2022 leak of data from the ransomware-as-a-service (RaaS) coordinated operation called "Conti" is used as case study.
An interactive Jupyter notebook with related data is also available for analysts interested to do further data exploration.
This research aims to provide a view into research methodology that may help other analysts apply Python to threat intelligence. Analysts can reuse the code and continue to explore the extracted information. Additionally, it offers an out-of-the-box methodology for analyzing chat logs, extracting IOCs, and improving threat intelligence and defense process using Python.
On February 28, 2022, a Twitter account named @ContiLeaks (allegedly a Ukrainian researcher) began posting leaked Conti data on Twitter. The leaked data sets, which were posted in a span of several months, consisted of chat logs, source codes, and backend applications.
For this research, we focused our analysis on the chat logs, which revealed crucial information about the Conti groupโs operating methods, infrastructure, and organizational structure.
The leaked chat logs are written in the Russian language. To make the analysis more accessible, we adopted the methodology published here and translated the logs to English.
The chat logs revealed that the Conti group uses the messaging application Jabber to communicate among members. Since raw Jabber logs are saved using a file per day, they can be compiled in one JSON file so they can easily be manipulated with Python. Once the data is merged, they can be translated using the deep translator library. After the logs are translated and loaded into a new file, itโs then possible to load the data into a dataframe for manipulation and exploration:
df = pd.read_json(codecs.open('translated_Log2.json', 'r', 'utf-8'))
Figure 1. Translated logs
Russian slang words not properly translated by the automated process can be translated by creating a dictionary. A dictionary off a list proposed here was used in this case to correctly translate the slang:
Figure 2. Translating slang
One way to get insights from chat logs is to see its timeline and check the number of discussions per day. The Bokeh library can be used to build an interactive diagram and explore the loaded dataframe.
Figure 3. Python code for exploring discussions
Using the data from Conti chat logs generates the following diagram, which shows the volume of Jabber discussions over time:
Figure 4. Volume of discussions over time
Visualizing the data as a timeline shows some peaks of activity that align to certain events. In the case of the Conti leaks, for example:
Itโs interesting that no peak in chat activity was observed within the Conti group after the first leak, which could indicate that the breach was ignored or not known by the group at that time.
When analyzing chat logs, identifying the number of users and analyzing the most active ones can provide insight into the size of the group and roles of users within it. Using Python, the list of users can be extracted and saved in a text file:
Figure 5. Extracting list of users
Running the script above using the Conti chat logs yielded a list of 346 unique accounts. This list can then be used to create a graph and show which users sent the most messages.
Figure 6. Creating a graph for users with most messages Figure 7. Most active users in the Conti chat logs
Based on the graph, the users named defender, stern, driver, bio, and mango have the largest number of discussions. Checkpoint published extensive research on the structure of the organization and correlated the user discussions with several roles and services like human resources, coders, crypters, offensive team, SysAdmins, and more.
Another way to analyze chat log data is to visualize the usersโ connection. This can be done by creating a dynamic network graph that can highlight the connections between users. The Barnes Hut algorithm and the Pyvis library can be used to visualize this data.
Figure 8. Creation a dynamic graph
Dynamic visualization shows a graphical overview of the network and allows zooming into the network to closely analyze the connections within. Bigger points represent the most active users, and itโs possible to highlight a user to analyze their connections. Additionally, the hovering tool shows which other users a specific user had conversations with.
Figure 9. Conti user network overview Figure 10. Connections to user โSternโ
Since reading data sets can be time-consuming, a simple search engine can be built to search for specific strings in the chat logs or to filter for topics of interest. For the Conti leak data, examples of these include Bitcoin, usernames, malware names, exploits, and CVEs, to name a few.
The following code snippet provides a simple search engine using the TextSearch library:
Figure 11. Search engine using Python
Besides processing chat logs to analyze user activity and connections, Python can also be used to extract and analyze threat intelligence. This section shows how the MSTICPy library can be used to extract IOCs and how it can be used for additional threat hunting and intelligence.
MSTICPy is a Python library used for threat investigation and threat hunting. The library can connect to several threat intelligence providers, as well as Microsoft tools like Microsoft Sentinel. It can be used to query logs and to enrich data. Itโs particularly convenient for analyzing IOCs and adding more threat contextualization.
After installing MSTICPy, the first thing to do is to initialize the notebook. This allows the loading of several modules that can be used to extract and enrich the data. External resources like VirusTotal or OTX can also be added by configuring msticpyconfig.yaml and adding the API keys.
The IoCExtract module from MSTICPy offers a convenient way to extract IOCs using predefined regex. The code automatically extracts IOCs such as DNS, URLs, IP addresses, and hashes and then reports them in a new dataframe.
Figure 12. Passing the dataframe to the module for extraction Figure 13. Sample of extracted IOCs
A regex can be added to filter specific IOCs from those extracted by the IOC extraction module by default. For example, the regex below extracts Bitcoin addresses from the Conti chat logs:
Figure 14. Extracting Bitcoin addresses and adding regex Figure 15. Sample of extracted Bitcoin addresses
After extracting IOCs, the dataframe can be cleaned to remove false positives as well as duplicate data. The final dataframe from the processed Conti chat logs contains the following unique IOC count, (these IOCs require additional analysis as not all of them are considered malicious):
URL | DNS | IPV4 | Bitcoin | MD5 | SHA-256 |
---|---|---|---|---|---|
1,137 | 474 | 317 | 175 | 106 | 16 |
The threat intel lookup module TILookup in MSTICPy can be used to get more information on IOCs such as IP addresses. In the case of the Conti leak, 317 unique IP addresses were identified. Not all these IOCs are malicious but can reveal more relevant information.
The configuration file can be specified to load the TILookup module, along with other threat intelligence providers such as VirusTotal, GreyNoise, and OTX.
Figure 16. Threat intel provider within MSTICPy
Running the module generates a new dataframe with more context for every IP address provided.
Figure 17. Sample of IP addresses enriched with additional info
The module also allows to request information for a single observable.
Figure 18. Extracting information for a single observable Figure 19. Additional threat context for one IP address
The browser provided by MSTICPy can also be used to explore the IOCs previously enriched. The interactive Jupyter notebook includes this view of the IOCs.
Figure 20. IOC browser provided by MSTICPy
In addition, MSTICPy has an embedded module that looks up the geolocation of IP addresses using Maxmind, which can be used to create a map of the IP addresses previously extracted.
Figure 21. Generating the IP geolocation map Figure 22. Geolocation of IPs extracted from the Conti leaks
Extracted URLs from IOC lists can provide details about targets, tools used to exchange information, and the infrastructure used to deploy attacks. A total of 1,137 unique URLs were extracted from the Conti leak dataset, but not all of them are usable for threat intelligence. The following code snippet shows how to filter for URLs.
Figure 23. Filtering the IOCs for URLs Figure 24. Sample of URLs extracted
A filter can be created to get details on executables, DLLs, ZIP files, and other files related to the extracted URLs. This can provide interesting insights and can be extracted for further research.
Figure 25. Filtering URLs for specific file formats Figure 26. Sample of URLs delivering extracted file
Using the same technique for filtering, .onion URLs can also be identified from the URL list. This proved particularly useful in this case, since the Conti group used the Tor network for some of their infrastructure.
Figure 27. Sample of extracted .onion URLs
The use of the pivot function within the MSTICPy library allows enrichment of data and discovery of additional infrastructure and IOC. This is particularly useful for threat intelligence and threat actor tracking. The next sections demonstrate the use of the VirusTotal module VTlookupV3 in MSTICPy to obtain intelligence about an IP address extracted from the Conti leak dataset that was used to deliver additional malware.
The following code initiates the _VTlookupV3 _in MSTICPy:
Figure 28. Configuring the VirusTotal module in MSTICPy
The VirusTotal module can be used to get data related to a particular IOC. The code below searches for files downloaded from a particular IP address from the Conti leak dataset:
Figure 29. Getting files downloaded from one IP address
The results show that the IP address 109[.]230[.]199[.]73 delivers several strains of malware.
Figure 30. Hashes related to IP 109[.]230[.]199[.]73
The VirusTotal module can then be used to pivot and extract more information about these hashes. The table below shows information about the first hash on the list:****
** ** | Attributes**** |
---|---|
authentihash | 0d10a35c1bed8d5a4516a2e704d43f10d47ffd2aabd9ce9e04fb3446f62168bf creation_date |
The results indicate that the hash is a Cobalt Strike loader, which means that Conti affiliates also use the penetration testing tool as part of their infrastructure during their operation.
In addition, the VirusTotal module can also provide details such as detection rate, type, description, and other information related to the hashes. The code snippet below generates the list of domains to which the hashes connect to.
Figure 31. Getting contacted domains Figure 32. Additional domains retrieved from previously extracted hashes
Doing this kind of analysis on the Conti leak data or similar data sets can lead to the discovery of possibly related domains that were not in the initial data sets.
This blog outlines how Python can be used to find valuable threat intelligence from data sets such as chat logs. It also presents details on how processing data using the MSTICPy library can be useful for enriching and hunting within environments, as well as collecting additional threat context. The interactive notebook provides additional code snippets that can also be used to continue log exploration.
The types of information extracted in this blog provides insights into the various elements of the criminal ecosystem that were coordinating their activities. Threat intelligence from research like this informs products and services like Microsoft 365 Defender, translating knowledge into real-world protection for customers. More importantly, the methodology described in this blog can be adapted to specific threat intelligence services, and the broader community is invited to use it for further analysis, enrichment of data, and intelligence sharing for the benefit of all.
**Thomas Roccia **Microsoft 365 Defender Research Team
The post Using Python to unearth a goldmine of threat intelligence from leaked chat logs appeared first on Microsoft Security Blog.