Parsing Nessus v2 XML reports with python

Upd. This is an updated post from 2017. The original script worked pretty well for me until the most recent moment when I needed to get compliance data from Nessus scan reports, and it failed. So I researched how this information is stored in a file, changed my script a bit, and now I want to share it with you.

Previous post about Nessus v2 reports I was writing mainly about the format itself. Now let’s see how you can parse them with Python.

Please don’t work with XML documents the same way you process text files. I adore bash scripting and awk, but that’s an awful idea to use it for XML parsing. In Python you can do it much easier and the script will work much faster. I will use lxml library for this.

So, let’s assume that we have Nessus xml report. We could get it using Nessus API (upd. API is not officially supported in Nessus Professional since version 7) or SecurityCenter API. First of all, we need to read content of the file.

#!/usr/bin/python

f = open('scanreport.nessus', 'r')
xml_content = f.read()
f.close()

Now I want to make a dict of vulnerabilities from this xml file. The key of this dict will have structure “host|plugin_id|port”. So, vulnerabilities["host|plugin_id|port"] will return me a dict with all parameters of vulnerability (Nessus plugin). Moreover, I want to see there not only information about particular plugin, but also information about the host: os, network interfaces, MAC, scan configuration. It won’t be an optimal way of storing the scan data, but it will make processing much easier, because you will see the context of any vulnerability.

If we look at the Nessus XML report structure we see that actual results are in Report section:

And in Report section will be ReportHost blocks containing some information about the host in HostProperties and some information about vulnerabilities in several ReportItem blocks.

So, I should get to the ReportItem, read all the data I need to produce a key, initialize vulnerability structure with this key and add all data from the HostProperties.

Here is the code:

#!/usr/bin/python
from lxml import etree
def get_vulners_from_xml(xml_content):
    vulnerabilities = dict()
    single_params = ["agent", "cvss3_base_score", "cvss3_temporal_score", "cvss3_temporal_vector", "cvss3_vector",
                     "cvss_base_score", "cvss_temporal_score", "cvss_temporal_vector", "cvss_vector", "description",
                     "exploit_available", "exploitability_ease", "exploited_by_nessus", "fname", "in_the_news",
                     "patch_publication_date", "plugin_modification_date", "plugin_name", "plugin_publication_date",
                     "plugin_type", "script_version", "see_also", "solution", "synopsis", "vuln_publication_date",
                     "compliance",
                     "{http://www.nessus.org/cm}compliance-check-id",
                     "{http://www.nessus.org/cm}compliance-check-name",
                     "{http://www.nessus.org/cm}audit-file",
                     "{http://www.nessus.org/cm}compliance-info",
                     "{http://www.nessus.org/cm}compliance-result",
                     "{http://www.nessus.org/cm}compliance-see-also"]
    p = etree.XMLParser(huge_tree=True)
    root = etree.fromstring(text=xml_content, parser=p)
    for block in root:
        if block.tag == "Report":
            for report_host in block:
                host_properties_dict = dict()
                for report_item in report_host:
                    if report_item.tag == "HostProperties":
                        for host_properties in report_item:
                            host_properties_dict[host_properties.attrib['name']] = host_properties.text
                for report_item in report_host:
                    if 'pluginName' in report_item.attrib:
                        vulner_struct = dict()
                        vulner_struct['port'] = report_item.attrib['port']
                        vulner_struct['pluginName'] = report_item.attrib['pluginName']
                        vulner_struct['pluginFamily'] = report_item.attrib['pluginFamily']
                        vulner_struct['pluginID'] = report_item.attrib['pluginID']
                        vulner_struct['svc_name'] = report_item.attrib['svc_name']
                        vulner_struct['protocol'] = report_item.attrib['protocol']
                        vulner_struct['severity'] = report_item.attrib['severity']
                        for param in report_item:
                            if param.tag == "risk_factor":
                                risk_factor = param.text
                                vulner_struct['host'] = report_host.attrib['name']
                                vulner_struct['riskFactor'] = risk_factor
                            elif param.tag == "plugin_output":
                                if not "plugin_output" in vulner_struct:
                                    vulner_struct["plugin_output"] = list()
                                if not param.text in vulner_struct["plugin_output"]:
                                    vulner_struct["plugin_output"].append(param.text)
                            else:
                                if not param.tag in single_params:
                                    if not param.tag in vulner_struct:
                                        vulner_struct[param.tag] = list()
                                    if not isinstance(vulner_struct[param.tag], list):
                                        vulner_struct[param.tag] = [vulner_struct[param.tag]]
                                    if not param.text in vulner_struct[param.tag]:
                                        vulner_struct[param.tag].append(param.text)
                                else:
                                    vulner_struct[param.tag] = param.text
                        for param in host_properties_dict:
                            vulner_struct[param] = host_properties_dict[param]
                        compliance_check_id = ""
                        if 'compliance' in vulner_struct:
                            if vulner_struct['compliance'] == 'true':
                                compliance_check_id = vulner_struct['{http://www.nessus.org/cm}compliance-check-id']
                        if compliance_check_id == "":
                            vulner_id = vulner_struct['host'] + "|" + vulner_struct['port'] + "|" + \
                                        vulner_struct['protocol'] + "|" + vulner_struct['pluginID']
                        else:
                            vulner_id = vulner_struct['host'] + "|" + vulner_struct['port'] + "|" + \
                                        vulner_struct['protocol'] + "|" + vulner_struct['pluginID'] + "|" + \
                                        compliance_check_id
                        if not vulner_id in vulnerabilities:
                            vulnerabilities[vulner_id] = vulner_struct
    return(vulnerabilities)
file_path = "scanreport.nessus"
f = open(file_path, 'r')
xml_content = f.read()
f.close()
vulners = get_vulners_from_xml(xml_content)

As you can see in the code, I get the root of XML document, then I check it’s child blocks in cycle. I can read block tag name (tag), attributes (attrib dict) and text in the block. If we are in block with tag name “Report” I make two new cycles: first one to initialize the host_properties_dict with information about the host, the second one to produce the key (vulner_id), to add information about each plugin to the vulnerability dictionary and copy parameters from host_properties_dict to the vulnerability dictionary.

upd1. What about plugin_output? Is it possible for plugin to have several outputs? Some Nessus plugins have complicated output, for example Service Detection (22964):

In XML it looks like several absolutely simmilar ReportItems (the same port, svc_name, protocol, severity, pluginID, pluginName and pluginFamily) in ReportHost. So, it’s easier to think that it’s actually the same ReportItem, but with a list ofplugin_outputs.

upd2. Changed sets to lists, because it’s hard to export dict with sets to json

To see the results in pretty print form:

import pprint                   
ids = vulnerabilities.keys()
pp = pprint.PrettyPrinter(indent=4)
pp.pprint(vulnerabilities["117.121.13.14|445|tcp|73570"]) 

Output:

{   'Credentialed_Scan': 'true',
    'HOST_END': 'Thu Dec 29 12:13:17 2016',
    'HOST_START': 'Thu Dec 29 12:03:53 2016',
    'LastAuthenticatedResults': '1483002797',
**    'bid': ['66920'],**
    'bios-uuid': '155C0A00-5BCB-11D9-8E9C-5404A6BFD7AB',
_    'cpe': 'cpe:/o:microsoft:windows',
    'cpe-0': 'cpe:/o:microsoft:windows_8_1::gold',
    'cpe-1': 'cpe:/a:wireshark:wireshark:1.12.6 -> Wireshark 1.12.6',
    'cpe-10': 'cpe:/a:oracle:jre:1.8.0:update112',
    'cpe-11': 'cpe:/a:oracle:jre:1.8.0:update112',
    'cpe-12': 'cpe:/a:videolan:vlc_media_player:2.2.4',
    'cpe-2': 'cpe:/a:adobe:acrobat_reader:11.0.18.21',
    'cpe-3': 'cpe:/a:adobe:adobe_air:23.0.0',
    'cpe-4': 'cpe:/a:adobe:flash_player:24.0.0.186',
    'cpe-5': 'cpe:/a:adobe:flash_player:23.0.0.205',
    'cpe-6': 'cpe:/a:opera:opera_browser:42.0',
    'cpe-7': 'cpe:/a:microsoft:ie:11.0.9600.18538',
    'cpe-8': 'cpe:/a:mozilla:firefox:50.1.0',
    'cpe-9': 'cpe:/a:oracle:jre:1.7.0:update51',_
**    'cve': ['CVE-2014-2428'],
    'cvss_base_score': '10.0',
    'cvss_temporal_score': '9.0',
    'cvss_temporal_vector': 'CVSS2#E:POC/RL:U/RC:ND',**   **    'cvss_vector': 'CVSS2#AV:N/AC:L/Au:N/C:C/I:C/A:C',**   **    'description': 'The version of Oracle (formerly Sun) Java SE or Java for Business installed on the remote host is earlier than 8 Update 5, 7 Update 55, 6 Update 75, or 5 Update 65.  It is, therefore, potentially affected by security issues in the following components :\n\n  - 2D\n  - AWT\n  - Deployment\n  - Hotspot\n  - JAX-WS\n  - JAXB\n  - JAXP\n  - JNDI\n  - JavaFX\n  - Javadoc\n  - Libraries\n  - Scripting\n  - Security\n  - Sound',**   **    'exploit_available': 'true',
    'exploitability_ease': 'Exploits are available',**   **    'fname': 'oracle_java_cpu_apr_2014.nasl',**
    'host': '117.121.13.14',
    'host-fqdn': 'user3273c3.corporation.com',
    'host-ip': '117.121.13.14',
    'hostname': 'USER3273C3',
    'local-checks-proto': 'smb',
    'mac-address': '54:15:A6:BF:18:AB',
    'netbios-name': 'USER3273C3',
    'netstat-established-tcp4-0': '117.121.13.14:135-117.121.13.11:37440',
...
    'netstat-established-tcp4-20': '127.0.0.1:61179-127.0.0.1:8191',
    'netstat-listen-tcp4-0': '0.0.0.0:135',
...
    'netstat-listen-tcp6-38': '[::1]:30523',
    'netstat-listen-udp4-39': '0.0.0.0:123',
...
    'netstat-listen-udp6-94': '[::]:15000',
    'operating-system': 'Microsoft Windows 8.1 Pro',
_    'os': 'windows',_
**    'osvdb': ['105899'],**
    'patch-summary-cve-num-6b52ad5d58bdbf4d9ae49271ad3ae15f': '30',
    'patch-summary-cve-num-6baf8f308323d224984bd832e424c820': '202',
    'patch-summary-cve-num-7b052ee02101353b71c18c498e71a2b7': '18',
    'patch-summary-cve-num-c78dfc9faff6e53e76ec9d3b3fa9a0d2': '26',
    'patch-summary-cve-num-eb2933669984591a7b9aa0b30d7ed02b': '43',
    'patch-summary-cve-num-fec1888dda0cda68ca7a95076d7f0cde': '30',
    'patch-summary-cves-6b52ad5d58bdbf4d9ae49271ad3ae15f': 'CVE-2016-0602, ... CVE-2010-5298',
    'patch-summary-cves-6baf8f308323d224984bd832e424c820': 'CVE-2016-5597, ... CVE-2015-7830',
    'patch-summary-cves-fec1888dda0cda68ca7a95076d7f0cde': 'CVE-2015-8104, ... CVE-2010-5298',
    'patch-summary-total-cves': '333',
    'patch-summary-txt-6b52ad5d58bdbf4d9ae49271ad3ae15f': 'Oracle VM VirtualBox < 4.3.36 / 5.0.14 Multiple Vulnerabilities (January 2016 CPU): Upgrade to Oracle VM VirtualBox version 4.3.36 / 5.0.14 or later as referenced in the January 2016 Oracle Critical Patch Update advisory.',
    'patch-summary-txt-6baf8f308323d224984bd832e424c820': 'Oracle Java SE Multiple Vulnerabilities (October 2016 CPU): Upgrade to Oracle JDK / JRE 8 Update 111 / 7 Update 121 / 6 Update 131 or later. If necessary, remove any affected versions.\n\nNote that an Extended Support contract with Oracle is needed to obtain JDK / JRE 6 Update 95 or later.',
    'patch-summary-txt-7b052ee02101353b71c18c498e71a2b7': 'WinSCP 5.x < 5.5.5 Multiple Vulnerabilities: Upgrade to WinSCP version 5.5.5 or later.',
    'patch-summary-txt-c78dfc9faff6e53e76ec9d3b3fa9a0d2': 'Adobe Flash Player <= 23.0.0.207 Multiple Vulnerabilities (APSB16-39): Upgrade to Adobe Flash Player version 24.0.0.186 or later.',
    'patch-summary-txt-eb2933669984591a7b9aa0b30d7ed02b': 'Wireshark 1.12.x < 1.12.13 Multiple DoS: Upgrade to Wireshark version 1.12.13 or later.',
    'patch-summary-txt-fec1888dda0cda68ca7a95076d7f0cde': 'Oracle VM VirtualBox < 4.0.36 / 4.1.44 / 4.2.36 / 4.3.34 / 5.0.10 Multiple Vulnerabilities (January 2016 CPU): Upgrade to Oracle VM VirtualBox version 4.0.36 / 4.1.44 / 4.2.36 / 4.3.34 / 5.0.10 or later as referenced in the January 2016 Oracle Critical Patch Update advisory.',
    'patch_publication_date': '2014/04/15',
**    'pluginFamily': 'Windows',
    'pluginID': '73570',
    'pluginName': 'Oracle Java SE Multiple Vulnerabilities (April 2014 CPU)',
    'plugin_modification_date': '2016/05/20',
    'plugin_name': 'Oracle Java SE Multiple Vulnerabilities (April 2014 CPU)',
    'plugin_output': ['\nThe following vulnerable instance of Java is installed on the\nremote host :\n\n  Path              : C:\\Program Files\\Java\\jdk1.7.0_51\\jre\n  Installed version : 1.7.0_51\n  Fixed version     : 1.5.0_65 / 1.6.0_75 / 1.7.0_55 / 1.8.0_5\n'],
    'plugin_publication_date': '2014/04/16',**
    'plugin_type': 'local',
    'policy-used': 'CorporationCred',
    'port': '445',
    'protocol': 'tcp',
**    'riskFactor': 'Critical',
    'script_version': '$Revision: 1.13 **

It is a typical Java vulnerability. All important plugin data is in bold here, including CVE references CVSS (base and temporal) and actual plugin output. You can see here host information: OS type and version, CPEs, patches, vulnerabilities, network configuration. You can also see that scan was authenticated, what policy (CorporationCred), transport (smb) and credentials (scan-windows) were used.

What’s next? You can make JSON and easily export this vulnerability structure to Splunk SIEM (“Export anything to Splunk with HTTP Event Collector”). Or you can process it by your own python scripts. For example, to find all critical vulnerabilities with public exploits and Network access vector:

for vulner_id in vulnerabilities:
    if "riskFactor" in vulnerabilities[vulner_id].keys() and "cvss_vector" in vulnerabilities[vulner_id].keys() and "exploit_available" in vulnerabilities[vulner_id].keys():
        if vulnerabilities[vulner_id]["riskFactor"] == "Critical" and "AV:N" in vulnerabilities[vulner_id]["cvss_vector"] and  vulnerabilities[vulner_id]["exploit_available"] == "true":
            print(vulner_id + " " + vulnerabilities[vulner_id]["plugin_name"])

Output:

117.121.11.51|74011|445 Adobe Acrobat < 10.1.10 / 11.0.07 Multiple Vulnerabilities (APSB14-15) 117.121.31.33|84824|445 Oracle Java SE Multiple Vulnerabilities (July 2015 CPU) (Bar Mitzvah) 117.121.21.10|84824|445 Oracle Java SE Multiple Vulnerabilities (July 2015 CPU) (Bar Mitzvah) 117.121.13.33|73570|445 Oracle Java SE Multiple Vulnerabilities (April 2014 CPU) 

**Upd3.**And what about compliance?

• The good news is that it is stored in the same ReportItem structures

• The bad news is that all of these ReportItem structures have the same host name, port, protocol and pluginID

  <ReportItem port=“0” svc_name=“general” protocol=“tcp” severity=“3” pluginID=“64455” pluginName=“VMware vCenter/vSphere Compliance Checks” pluginFamily=“Policy Compliance”>
  <compliance>true</compliance>
  <fname>vmware_compliance_check.nbin</fname>
  <plugin_modification_date>2019/09/19</plugin_modification_date>
  <plugin_name>VMware vCenter/vSphere Compliance Checks</plugin_name>
  <plugin_publication_date>2013/04/08</plugin_publication_date>
  <plugin_type>local</plugin_type>
  <risk_factor>None</risk_factor>
  <script_version>$Revision: 1.123 $</script_version>
  <cm:compliance-check-name>8.7.1 Ensure VIX messages from the VM are disabled</cm:compliance-check-name>
  <description>“8.7.1 Ensure VIX messages from the VM are disabled” : [FAILED] The VIX API is a library for writing scripts and programs to manipulate virtual machines…
  Rationale
  […]
  Remote value:
  […]
  Policy value:
  […]
  Solution :
  […]
  See Also :
  […]
  Reference(s) :
  […]</description>
  <cm:compliance-check-id>b83400214bc03e82cfb069c597ed1871</cm:compliance-check-id>
  <cm:compliance-actual-value>…</cm:compliance-actual-value>
  <cm:compliance-policy-value>[…]</cm:compliance-policy-value>
  <cm:compliance-info>[…]</cm:compliance-info>
  <cm:compliance-result>FAILED</cm:compliance-result>
  <cm:compliance-reference>800-53|CM-7,800-171|3.4.6,800-171|3.4.7,CSF|PR.IP-1,CSF|PR.PT-3,ITSG-33|CM-7,NIAv2|SS15a,SWIFT-CSCv1|2.3,LEVEL|2S</cm:compliance-reference>
  <cm:compliance-solution>[…]</cm:compliance-solution>
  <cm:compliance-see-also>https://workbench.cisecurity.org/files/2168&lt;/cm:compliance-see-also&gt;
  </ReportItem>

That is why, to differentiate them, I needed to add some unique identifier to the key - compliance-check-id.

How to get all IDs for vulnerabilities and compliance checks

f = open(file_path, 'r')
xml_content = f.read()
f.close()

vulners = get_vulners_from_xml(xml_content)

for vulner_id in vulners:
    print(vulner_id + " - " + vulners[vulner_id]['plugin_name'])

Output:

localhost|0|tcp|117887 - Local Checks Enabled
localhost|0|tcp|110095 - Authentication Success
localhost|0|tcp|19506 - Nessus Scan Information
localhost|0|tcp|21157|ed50607fc9a75055e838584afa805a3c - Unix Compliance Checks
localhost|0|tcp|21157|8677fb78800ea4c64abcac174ac5d975 - Unix Compliance Checks
...

How to list all compliance checks and statuses

for vulner in vulners:
    if "compliance" in vulners[vulner]:
        if vulners[vulner]['compliance'] == 'true':
            print(vulners[vulner]['host'] + "|" +
                  vulners[vulner]['{http://www.nessus.org/cm}compliance-check-name'] + "|" +
                  vulners[vulner]['{http://www.nessus.org/cm}compliance-result'] )

Output:

localhost|5.6 Ensure access to the su command is restricted - pam_wheel.so|FAILED
localhost|5.4.4 Ensure default user umask is 027 or more restrictive - /etc/bashrc|FAILED
localhost|5.4.2 Ensure system accounts are non-login|FAILED
localhost|5.4.1.4 Ensure inactive password lock is 30 days or less|FAILED
... 

How to get all parameters of vulnerability

print(vulners["localhost|0|tcp|128372"])

Output:

{'port': '0', 'pluginName': 'CentOS 7 : curl (CESA-2019:2181)', 'pluginFamily': 'CentOS Local Security Checks', 'pluginID': '128372', 'svc_name': 'general', 'protocol': 'tcp', 'severity': '2', 'cpe': 'cpe:/o:linux:linux_kernel', 'cve': ['CVE-2018-16842'], 'cvss3_base_score': '9.1', 'cvss3_temporal_score': '7.9', 'cvss3_temporal_vector': 'CVSS:3.0/E:U/RL:O/RC:C', 'cvss3_vector': 'CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:H', 'cvss_base_score': '6.4', 'cvss_score_source': ['CVE-2018-16842'], 'cvss_temporal_score': '4.7', 'cvss_temporal_vector': 'CVSS2#E:U/RL:OF/RC:C', 'cvss_vector': 'CVSS2#AV:N/AC:L/Au:N/C:P/I:N/A:P', 'description': 'An update for curl is now available for Red Hat Enterprise Linux 7.\n\nRed...

How to get all parameters of compliance check

print(vulners["localhost|0|tcp|21157|2273cd619431d601a4cb1f7c59d57f97"])

Output:

{'port': '0', 'pluginName': 'Unix Compliance Checks', 'pluginFamily': 'Policy Compliance', 'pluginID': '21157', 'svc_name': 'general', 'protocol': 'tcp', 'severity': '0', 'agent': 'unix', 'compliance': 'true', 'fname': 'unix_compliance_check.nbin', 'plugin_modification_date': '2019/12/17', 'plugin_name': 'Unix Compliance Checks', 'plugin_publication_date': '2006/03/27', 'plugin_type': 'local', 'host': 'localhost', 'riskFactor': 'None', 'script_version': '$Revision: 1.441 $', '{http://www.nessus.org/cm}compliance-check-name': '4.2.2.4 Ensure syslog-ng is configured to send logs to a remote log host - log src', 'description': '"4.2.2.4 Ensure syslog-ng is configured to send logs to a remote log host - log src" : [PASSED] The \'syslog-ng\' utility supports the ability to send logs it gathers to a remote log host or to receive messages from remote hosts, reducing administrative…