This API Best Practices Series is designed for Qualys customer programmers or stakeholders with a general knowledge of programming who want to implement best practices for improving the development, design, and performance of their programs that use the Qualys API. For non-customers, the Qualys API demonstrates our commitment to interoperability with the enterprise IT security stack.
Qualys Web Application Scanning (WAS) enables organizations to assess, track, and drive remediation for web application vulnerabilities, helping them secure their web applications and keep the attackers at bay.
If you are a programmer, your enterprise can derive superlative benefits from the WAS data from the step-by-step guide provided in this blog post.
Part 5 of this series aims to obtain WAS Web Applications, Web Application Details, Web Application Findings, and the WAS Catalog in both compressed JavaScript Object Notation (JSON) form and into the latest timestamped, point-in-time SQLite database.
But, before we get to it, let’s quickly look at the most sought-after capabilities of the Qualys Web Application Scanning (WAS):
We are also providing a blueprint of the example code called QualysETL, which is an opensource code under Apache 2 License for your organization to develop with:
We will provide more details along with live examples to help you effectively extract, transform, load, and distribute Qualys WAS data to the cloud database.
The Qualys API is a key component in our API-first model. Since the founding of Qualys in 1999, a rich set of Qualys APIs has been available and continually optimized. As a result, programmers at Qualys customer organizations have automated processing Qualys in new ways, increasing their return on investment (ROI) and improving overall mean-time-to-remediate (MTTR).
The diagram below depicts QualysETL as a workflow, resulting in compressed JSON Data and an SQLite Database. The JSON and SQLite data can be used on your desktop for analysis or as a continuous live data feed to update your corporate data store.
Even with all these advances in our API, some enterprise customers continue to experience suboptimal performance in automation. What are the inherent automation challenges to Extract, Transform, and Load (ETL) Qualys data?
And what do we mean by ETL?
With any API, there are inherent automation challenges. Some of these are:
In the Example JSON Output image below, we have noted some key fields, including:
You will want to transform the JSON data and prepare the data for ingestion into a database for future correlations with other corporate data sources. The QualysETL blueprint of example code can help you with that objective.
For more information about our JSON Fields in Qualys WAS, please refer to the WAS API Appendix.
QualysETL is a blueprint that can be used as a starting point to develop your ETL automation. The image below shows QualysETL workflow, which includes the processes to:
In the diagram, we show the base schema for the following tables:
These tables are automatically created through QualysETL of WAS and will grow over time as Qualys keeps adding new capabilities. See the WAS API Guide for a complete list of available fields.
With WAS data prepared for use, you would want to distribute it for usage by your corporation. For example, you can distribute a timestamped version of the SQLite Database into an Amazon Web Services Relational Database Service or an AWS (Amazon Web Services) S3 Bucket.
Further, you create the SQLite database available locally for analysts so they can process and report on vulnerabilities in your organization using their desktop tool of choice.
Data usage flexibility is achieved at this point.
Contact your Qualys Technical Account Manager and arrange a meeting with David Gregory, Principal Solutions Architect of API, if you are interested in automation to inject data directly into your downstream database from QualysETL.
The WAS Activity Diagram below depicts QualysETL pagination to obtain Qualys WAS data and simultaneously load WAS data into an SQL Database. The activities include:
API Call Recommendations:****
In the following four examples, we will:
In the first example below, we use Postman to count web applications using the “/qps/rest/3.0/count/was/webapp” endpoint. "https://{{base_url}}/" is your subscription API Server URL from Platform Identification.
In the request's body is the ServiceRequest payload, which includes the last scan date, limiting the count to Web Apps scanned after January 1st, 2023. The resulting ServiceResponse includes a count of sixteen web applications.
If you are new to Postman, check the Qualys API Fundamentals Training to learn more about Postman and how it can be developed with Qualys API.
In the second example, the “/qps/rest/3.0/search/was/webapp” endpoint is used to download web applications, filtered using lastScan.date to limit download of Web Apps to those scanned after January 1st, 2023. Notice that the hasMoreRecords field is true, and the lastId is set to a number. Use the lastId in your next API call to get the next batch of hosts by updating the filter criteria “id” to the value of lastId.
In the third example, The /qps/rest/3.0/get/was/webapp/[id] endpoint is used to download web application details using the unique id for each Web App. Extended Web App configuration details are included with this API call.
In the fourth example, the “/qps/rest/3.0/get/search/was/finding” endpoint is used to download web application findings using the unique ID for each Web App. Details of vulnerability findings for the application are included in the data. See the end of this presentation for links to details describing each field.
You can develop your own integration with the WAS API or leverage the QualysETL Blueprint of open-source Python code to download all your WAS Data with a single command!
To install QualysETL, we recommend you provision a secure, patched, up-to-date virtual machine instance of Ubuntu 22.04 with internet connectivity. When this step is completed, you can log into your Ubuntu instance and follow along through this video to install the application and run your first ETL.
The instructions are located on pypi.org at: <https://pypi.org/project/qualysetl/>
Next, you can run your own SQL queries to analyze the data and tune the application to meet your needs.
At this point, you should be able to query your WAS SQLite database. Below is an example SQL query to find data of severity type 4 and finding Type Qualys.
Leverage QualysETL as a blueprint of example code to produce a current WAS SQLite Database, ready for analysis or distribution.
For questions, existing Qualys customers can schedule time through their Technical Account Manager to meet with our solutions architects for help. Non-customers can request access to the Qualys API or QualysETL as a free trial of Qualys WAS to learn more about their full capabilities.
This blog series lets you create a unified view of Qualys data across our cloud services at the Qualys customer organizations. It includes the Qualys VMDR (Vulnerability Management, Detection and Response), CSAM (Cyber Security Asset Management), WAS (Web Application Scanning), and PCRS (Policy Compliance Reporting Service).
The API Best Practices Series will continue to expand over the coming months to cover other key aspects of the Qualys API, with each presentation building on the previous one and, in the aggregate, providing an overall best practice view of the Qualys API.