Payload detection WAF challenge

One of the things that we implemented early on in our lab is an semi-automated process of collecting some new payloads/exploits/bypass techniques from the public feeds (including Twitter) and checking whether our current WAF solutions can detect it.

Introduction

What if we can test their essential capability of payload detection with the same set of tests? Let’s call it a payload detection challenge. After all, we need a WAF and this is an important procedure that would be useful not only for our evaluation but pretty much everybody else who’s picking the right WAF.

It turns out there are tools out there that help to do so. No surprise, a lot of them are developed by WAF vendors - so we didn’t blindly take them and get the results WAF vendors want us to have. No,  we made a few tweaks to make them more fair. Trial versions from marketplaces and here we are getting results for our payload challenge.

Test candidates

So whom are we going to test? Since we didn’t have a lot of time we took an open-source mod_security and a few other vendors who offered free trials from AWS Marketplace. So the final tests were performed for the following WAF solutions:

• AWS WAF
• Imperva SecureSphere WAF
• F5 Advanced WAF
• Wallarm WAF
• ModSecurity WAF

Tools to test WAFs

There are few projects on the market that can help evaluate if WAF does a good job on the payload detection. Surprise-surprise, a lot of them are actually developed by the WAF vendors.

1. WAF Testing Framework (WTF) is an old project by Imperva. The tool is able to measure the response of the WAF to each one of the requests and display a chart that includes information on False Negatives as well as False Positives. Unfortunately the project is not maintained anymore so it’s really hard to find even its binary.
2. Framework for Testing WAFs (FTW) is a project with a similar name that was created by researchers from ModSecurity and Fastly to help provide rigorous tests for WAF rules.  It uses the OWASP Core Ruleset V3 as a baseline to test rules on a WAF. Each rule from the ruleset is loaded into a YAML file that issues HTTP requests that will trigger these rules. This is a good project to test Ruleset but isn’t useful when you want to test any WAF.
3. GoTestWAF is a relatively new project that is actively maintained by another WAF vendor Wallarm (Full disclaimer: They are our friends). GoTestWAF uses YAML to define multiple tests using different variations of payloads, bypass techniques, encoding and also payload insertion points (including JSON structures emulating API calls).

There might be something else as well (please give us a note if you know good alternatives) but we decided to choose GoTestWAF. First, it’s actively developed. Second, it’s maintained by Wallarm who are our friends. And third, a very cool concept behind this, explained in the next section. The tool is extendable. So in addition to the default payload, we plugged in a library of the payloads that we collected from Twitter.

How GoTestWAF works?

GoTestWAF generates requests with predefined, basic payloads as well as attacks specific to different APIs (REST, SOAP, XMLRPC). Afterwards, it sends them to the application and analyzes the responses to generate a detailed report in the console output or as a PDF.

The main idea is to encode and place attack payloads in different parts of an HTTP request: its body, headers, URL parameters, etc. To simplify things, authors implemented the following logic:

Payload → Encoder → Placeholder

It means that every payload sample (malicious attack sample such as an XSS string like <script>alert(1)</script>) will be first encoded in some way then placed into an HTTP request. There is also an option to use a plain encoder that keeps the string as-is.

To make tests readable, GoTestWAF uses YAML DSL. Here is an example of SQL Injection payloads:

payload:
  - '"union select -7431.1, name, @aaa from u_base--w-'
  - "'or 123.22=123.22"
  - "' waitfor delay '00:00:10'--"
  - "')) or pg_sleep(5)--"
encoder:
  - Base64Flat
  - Url
placeholder:
  - UrlPath
  - UrlParam
  - JsonBody
  - Header

As a result of the permutation of each 4 payloads, 2 encoders, and 4 placeholders, this test will send 4*2*4=32 requests.

Note:

Be careful with the YAML. If you need to send a few binary payloads, then it is best to use !!binary attribute on the payload fields.

Test methodology

The goal of the testing process is to find how well different WAFs can block popular malicious attacks in default WAF configurations.

For every tested WAF solution (besides AWS WAF), we deployed an AWS EC2 instance using a vendor-provided AMI published in the AWS Marketplace. We used standard deployment procedures published by WAF vendors. We documented in detail the deployment and configuration process for each tested product.

We also deployed a simple NGINX web server to simulate a protected web application. Note: we didn’t try to check how good WAF is in detecting attacks against some application-specific issues or use application response in the benchmarks. The idea is simple: to check malicious payload detection in HTTP/HTTPS requests to web apps and APIs (meaning payloads can be encoded and inserted somewhere within JSON/XML/etc structure).

Using all preconfigured attack simulations (test cases) from the GoTestWAF tool, we tested how well each of the WAFs were able to protect the simple web application.

GoTestWAF test cases cover the following OWASP Top 10 scenarios:

1. Mail Injection;
2. Cross-site scripting;
3. SQL Injection;
4. NoSQL Injection;
5. Path Traversal;
6. RCE;
7. XML Injection;
8. LDAP Injection;
9. Server-Side Template Injection;
10. Server-Side Includes.

During a test run, the GoTestWAF tool sent about 170 requests covering the mentioned attack types. Before sending simulated attacks we verified that each deployed WAF product was working as expected and properly passed regular web requests to the origin web server.

For every tested WAF, we performed three runs of the GoTestWAF tool. After each run, we collected reported scores for each attack category listed above and calculated an average score for each test run.

The score report for an attack type shows what percent of potentially malicious requests were successfully blocked by the tested WAF. For example, result 27/28 and score 0.96 mean that the test tool sent 28 malicious requests, from which 27 requests were blocked by the WAF resulting in a 96% WAF protection efficiency score.

We intentionally used the default settings. Probably with some extra tuning you can get a higher score for the vendor. In our next research paper, we will try to put a reasonable effort to fine-tune the configuration of the tested products.

AWS WAF

AWS WAF

For the test, we used a simple architecture: one test web server and one AWS load balancer with an attached AWS WAF service. The configuration was deployed in us-west-1 AWS region (N. California)

Deployment of test web server instance

Deployment

It was deployed a single r1.micro instance using standard Ubuntu 18.04 LTS instance. We configured the network firewall to allow incoming requests to SSH (22/TCP) and HTTP (80/TCP) ports:

After the instance was started, we installed the standard “nginx” package from the official Ubuntu repository. When accessed from a browser the instance showed the default NGINX homepage:

It was deployed a single r1.micro instance using standard Ubuntu 18.04 LTS instance. We configured the network firewall to allow incoming requests to SSH (22/TCP) and HTTP (80/TCP) ports:

After the instance was started, we installed the standard “nginx” package from the official Ubuntu repository. When accessed from a browser the instance showed the default NGINX homepage:

Creation of AWS load balancer

We selected the HTTP/HTTPS load balancer:

Selecting HTTP/HTTPS load balancer

We selected two Availability Zones, including the one used to start the test web server instance:

The HTTP service (port 80 / TCP) was enabled in the security group in use.

On the Register Targets step, we selected the instance of the test web server:

Selecting instance

After creating the load balancer instance, we got the DNS name associated with it (something like vs-test-LB-1-626278186.us-west-1.elb.amazonaws.com). When accessing the DNS name from a browser, we received the default NGINX home page.

WAF configuration

Finally, enable the WAF service

On the page https://console.aws.amazon.com/wafv2/homev2, we clicked the "Create web ACL" link.
After that, we selected "Regional resources (Application Load Balancer and API Gateway)" and the region where we deployed the ALB instance. Next, in the "Associated AWS resources" section, we selected the ALB instance.

WAF rules selection page:

Default set of rules offered by AWS

With the default settings, AWS WAD uses the following rulesets:

• Core rule set
• Known bad inputs
• Linux operating system
• PHP application
• POSIX operating system
• SQL database

The used Web ACL rule capacity was 1500/1500 WCUs.

Verifying the AWS WAF functionality

Using a normal HTTP request, we tested the whole system to ensure it was working properly:

curl  -I 'http://vs-test-LB-1-626278186.us-west-1.elb.amazonaws.com/?q=test'
HTTP/1.1 200 OK

Using a known malicious request we verified that the WAF is capable to detect basic security attacks:

curl  -I 'http://vs-test-LB-1-626278186.us-west-1.elb.amazonaws.com/?q=<script>alert(1)</script>'
HTTP/1.1 403 Forbidden

Collection of test results

On the next stage, we used a GoTestWAF tool to run WAF penetration tests:

docker run gotestwaf --url=http://vs-test-LB-1-626278186.us-west-1.elb.amazonaws.com/

owasp sst-injection 0/20  (0.00)
owasp xml-injection 6/12  (50.00)
owasp xss-scripting 6/28  (21.43)
owasp ldap-injection  0/8 (0.00)
owasp mail-injection  0/12  (0.00)
owasp shell-injection 1/8 (12.50)
owasp sql-injection 4/32  (12.50)
owasp nosql-injection 0/18  (0.00)
owasp path-traversal  3/24  (12.50)
owasp ss-include  0/20  (0.00)
owasp-api graphql 0/1 (0.00)
owasp-api rest  0/2 (0.00)
owasp-api soap  2/2 (100.00)
community community-sqli  14/48 (29.17)
community community-xss 107/304 (35.20)
community community-xxe 2/4 (50.00)
community community-lfi 1/6 (16.67)
community community-rce 2/42  (4.76)
false-pos texts 1/8 (12.50)

WAF score: 18.80%
450 bypasses in 599 tests / 19 test cases

Imperva SecureSphere WAF

We have deployed two Imperva WAF components available in the AWS Marketplace:

• SecureSphere WAF Management for AWS (On-Demand) - the management console component
• SecureSphere WAF AV1000 Gateway for AWS (On-Demand) - the WAF gateway node

Imperva management host

On the product page of “SecureSphere WAF Management for AWS (On-Demand),” we clicked on "Continue to Subscribe" and activated a trial 30 day license. On the product Marketplace page (“Usage Information” section), we found links to the product deployment documentation (in PDF format) and the CloudFormation template.

It was used the page, “https://cloud-template-tool.imperva.com,” to create a deployment template; we were required to select an AWS zone, instance type, etc. and entered the following:

Create Imperva deployment template

It was used the resulting CloudFormation template to deploy an AWS stack:

The specified “Secure Password” parameter was later used to connect a WAF gateway node. Once the deployment was completed, we were able to connect to the running management instance using the URL: https://_IP_:8083 (the IP address of the management console EC2 instance was taken from the AWS console).

Deployment of WAF Gateway node

Deployment

Once the management console was up and running, we deployed a WAF gateway node using the product “SecureSphere WAF AV1000 Gateway for AWS (On-Demand)”. Using the same approach as the management mode, we found the product documentation and used this link: “https://cloud-template-tool.imperva.com,” to create a proper CloudFront deployment template.

In the template, we specified the IP address of the management host:

It was deployed the new CloudFormation stack in the same AWS region where we deployed the WAF management host:

Once the management console was up and running, we deployed a WAF gateway node using the product “SecureSphere WAF AV1000 Gateway for AWS (On-Demand)”. Using the same approach as the management mode, we found the product documentation and used this link: “https://cloud-template-tool.imperva.com,” to create a proper CloudFront deployment template.

In the template, we specified the IP address of the management host:

It was deployed the new CloudFormation stack in the same AWS region where we deployed the WAF management host:

WAF configuration

In the management console under "SETUP> Gateways", it was confirmed that the system is reporting a new WAF gateway:

After that, a new server group was created in the "SETUP -> Sites -> Default Site" section. Inside the new group, we changed the WAF mode of operation to Active (this means that WAF should block detected attacks):

A new "Site" object was created inside the new server group, and on the "Definitions" tab of this object, the HTTP response code and the blocking page were changed:

On the "Reverse Proxy" tab, rules were configured to point to the test web server:

Verifying the WAF gateway node functionality

Using the IP address of the WAF gateway node, we sent a normal test request to confirm that the WAF can properly handle requests:

curl  -I 'http://54.183.131.246/?q=test'
HTTP/1.1 200 OK

Using a simulated malicious request, we confirmed that the WAF is functioning and can block web attacks.

curl  -I 'http://54.183.131.246/?q=<script>alert(1)</script>'
HTTP/1.1 403 Forbidden

Additional reading about SecureSphere

We found that the following page provides good coverage of the SecureSphere WAF product.

Testing the WAF gateway node

It was executed the mentioned GoTestWAF testing tool and pointed it to the WAF gateway node:

docker run gotestwaf --url=http://54.183.131.246/

community community-xss 209/304 (68.75)
community community-xxe 0/4 (0.00)
community community-lfi 2/6 (33.33)
community community-rce 0/42  (0.00)
community community-sqli  14/48 (29.17)
false-pos texts 6/8 (75.00)
owasp ldap-injection  0/8 (0.00)
owasp nosql-injection 0/18  (0.00)
owasp ss-include  1/20  (5.00)
owasp xss-scripting 8/28  (28.57)
owasp xml-injection 3/12  (25.00)
owasp mail-injection  0/12  (0.00)
owasp path-traversal  9/24  (37.50)
owasp shell-injection 4/8 (50.00)
owasp sql-injection 8/32  (25.00)
owasp sst-injection 4/20  (20.00)
owasp-api graphql 0/1 (0.00)
owasp-api rest  2/2 (100.00)
owasp-api soap  0/2 (0.00)

WAF score: 26.17%
329 bypasses in 599 tests / 19 test cases

F5 Advanced WAF

In the Usage Instruction block, there is a link to a video describing the deployment process using CloudFormation, and the process looked pretty straight forward.

We tried to follow the instructions and used this CloudFormation template. While trying to deploy the CloudFormation stack, we have received the following error:

“Failed trying to describe product codes for snapshot '3940739910' (Service: AmazonEC2; Status Code: 400; Error Code: InvalidSnapshot.NotFound; Request ID: 9f7950c2-ab45-4f8d-9f7b-7be486c0e90b)

The following resource(s) failed to create: [Bigip1subnet1Az1SelfEipAddress, ExternalDefaultRoute, Az1ApplicationSubnetRouteTableAssociation, Az1ExternalSubnetRouteTableAssociation, Bigip1subnet1Az1Interface, ApplicationDefaultRoute, bigipServiceDiscoveryProfile, Webserver]. Rollback requested by user.”

A quick Internet search showed that the problem was already reported in GitHub (GitHub issue https://github.com/F5Networks/f5-aws-cloudformation/issues/89) by other users and monitored by F5 using internal Jira ticket #1751. The GitHub issue is still open.

Wallarm WAF

To test Wallarm’s WAF solution we used their AWS Marketplace offering: “Wallarm Node (AI-based NG-WAF instance),” and followed pretty straight-forward instructions published on the documentation website.

WAF node deployment

Deployment

First, we registered a free trial account using the Wallarm console UI. After that, we added a user with a “Deploy” role while registering a new WAF node with the Wallarm Cloud backend. Just like the following step, we included our deployment location, US-WEST-1 AWS region (N. California), and created a new t2.micro EC2 instance using the provided Wallarm Node AMI.

During the Security Group configuration phase, the default firewall configuration offered by AWS was accepted.

On the freshly started server, we followed the provided instructions to connect the node to the Wallarm Cloud using command “/usr/share/wallarm-common/addcloudnode -H us1.api.wallarm.com”.
It was also configured the pre-installed NGINX instance to enable the Wallarm WAF in blocking mode. In addition, we configured the NGINX to forward all filtered requests to the test web server, which we previously deployed while testing the AWS WAF solution.

server {       listen 80;       listen [::]:80 ipv6only=on;       # the domains for which traffic is processed       server_name example.com;        server_name www.example.com;       # turn on the monitoring mode of traffic processing       wallarm_mode block;        # wallarm_instance 1;       location / {         # setting the address for request forwarding         proxy_pass http://18.144.79.112;          proxy_set_header Host $host;         proxy_set_header X-Real-IP $remote_addr;         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;       }     }

After completing the configuration process, the NGINX process was restarted using the "systemctl restart nginx" command and began testing the WAF node.

Verifying the WAF gateway functionality

Using the IP address of the WAF node, we sent a normal test request to confirm that the WAF can properly handle requests:

curl  -I 'http://18.144.72.45/?q=test'
HTTP/1.1 200 OK

Using a simulated malicious request, we confirmed that the WAF is functioning and can block web attacks.

curl  -I 'http://18.144.72.45/?q=<script>alert(1)</script>'
HTTP/1.1 403 Forbidden

Testing the WAF node

We executed the mentioned GoTestWAF testing tool and pointed it to the Wallarm WAF node:

docker run gotestwaf --url=http://18.144.72.45/

community  community-sqli  30/48 (62.50)
community community-xss 244/304 (80.26)
community community-xxe 4/4 (100.00)
community community-lfi 4/6 (66.67)
community community-rce 4/42  (9.52)
false-pos texts 1/8 (12.50)
owasp mail-injection  8/12  (66.67)
owasp shell-injection 8/8 (100.00)
owasp ss-include  20/20 (100.00)
owasp sst-injection 16/20 (80.00)
owasp ldap-injection  8/8 (100.00)
owasp nosql-injection 18/18 (100.00)
owasp path-traversal  24/24 (100.00)
owasp sql-injection 32/32 (100.00)
owasp xml-injection 12/12 (100.00)
owasp xss-scripting 28/28 (100.00)
owasp-api graphql 1/1 (100.00)
owasp-api rest  2/2 (100.00)
owasp-api soap  2/2 (100.00)

WAF score: 83.06%
133 bypasses in 599 tests / 19 test cases

ModSecurity WAF

For ModSecurity WAF, their popular OWASP ModSecurity Core Rule Set was tested.

WAF node deployment

Deployment

For the testing, we deployed one t2.small EC2 instance in US-WEST-1 AWS region using official Ubuntu 18.04 LTS image, added necessary tools like Git and Docker, and built and started a ModSecurity Docker container offered by Trustwave:

git clone https://github.com/SpiderLabs/owasp-modsecurity-crs cd owasp-modsecurity-crs/util/docker docker build -t modsec_crs --file Dockerfile-3.0-nginx . docker run --rm -p 8080:80 -e PARANOIA=1 modsec_crs

We used PARANOIA level “1” to increase the level of WAF protection (read more about it here). Later we added results for PARANOIA levels “2”and “3”.

Verifying the WAF node functionality

Using the IP address of the WAF node, we sent a normal test request to confirm that the WAF can properly handle requests:

curl  -I 'http://18.144.71.24/?q=test'
HTTP/1.1 200 OK

Using a simulated malicious request, we confirmed that the WAF is functioning and can block web attacks.

curl  -I 'http://18.144.71.24/?q=<script>alert(1)</script>'
HTTP/1.1 403 Forbidden

Testing the WAF node

It was executed the GoTestWAF tool and pointed to the test WAF node with PARANOIA level “1”:

docker run gotestwaf --url=http://18.144.71.24/

community community-lfi 4/6 (66.67)
community community-rce 6/42  (14.29)
community community-sqli  34/48 (70.83)
community community-xss 279/304 (91.78)
community community-xxe 4/4 (100.00)
false-pos texts 7/8 (87.50)
owasp ldap-injection  1/8 (12.50)
owasp shell-injection 3/8 (37.50)
owasp sql-injection 8/32  (25.00)
owasp ss-include  5/20  (25.00)
owasp sst-injection 5/20  (25.00)
owasp mail-injection  3/12  (25.00)
owasp nosql-injection 0/18  (0.00)
owasp path-traversal  8/24  (33.33)
owasp xml-injection 12/12 (100.00)
owasp xss-scripting 9/28  (32.14)
owasp-api graphql 1/1 (100.00)
owasp-api rest  2/2 (100.00)
owasp-api soap  0/2 (0.00)

WAF score: 49.82%
208 bypasses in 599 tests / 19 test cases

PARANOIA level “2”:

docker run gotestwaf --url=http://18.144.71.24/

community community-lfi 6/6 (100.0)
community community-rce 32/42 (76.19)
community community-sqli  48/48 (100.00)
community community-xss 302/304 (99.34)
community community-xxe 4/4 (100.00)
false-pos texts 8/8 (100.00)
owasp ldap-injection  8/8 (100.00)
owasp shell-injection 4/8 (50.00)
owasp sql-injection 8/32 (25.00)
owasp ss-include  5/20  (25.00)
owasp sst-injection 6/20 (30.00)
owasp mail-injection 6/12 (50.00)
owasp nosql-injection 6/18 (33.33)
owasp path-traversal 9/24 (37.50)
owasp xml-injection 12/12 (100.00)
owasp xss-scripting 9/28  (32.14)
owasp-api graphql 1/1 (100.00)
owasp-api rest  2/2 (100.00)
owasp-api soap  0/2 (0.00)

WAF score: 63.61%
127 bypasses in 599 tests / 19 test cases

PARANOIA level “3”

docker run gotestwaf --url=http://18.144.71.24/

community community-lfi 6/6 (100.0)
community community-rce 42/42 (100.00)
community community-sqli  48/48 (100.00)
community community-xss 302/304 (99.34)
community community-xxe 4/4 (100.00)
false-pos texts 8/8 (100.00)
owasp ldap-injection  8/8 (100.00)
owasp shell-injection 4/8 (50.00)
owasp sql-injection 8/32 (25.00)
owasp ss-include  6/20  (30.00)
owasp sst-injection  7/20  (35.00)
owasp mail-injection 6/12 (50.00)
owasp nosql-injection 6/18 (33.33)
owasp path-traversal 10/24 (41.67)
owasp xml-injection 12/12 (100.00)
owasp xss-scripting 9/28  (32.14)
owasp-api graphql 1/1 (100.00)
owasp-api rest  2/2 (100.00)
owasp-api soap  0/2 (0.00)

WAF score: 65.6%
114 bypasses in 599 tests / 19 test cases

Executive summary

The results of performed tests are formulated in the following table.

Table 1. Overall WAF scores (higher scope reflects higher level of WAF protection).

Testing so many WAF's turned out to be a non-trivial task, but more fun and intriguing. Perhaps some of the tested solutions are already used by our readers, or if you did not find your WAF in the article, you can test it with a similar method!

PDF format:

Payload-detection-WAF-challengeDownload