Multiple Cisco CSS / ACE Client Certificate And HTTP Header Manipulation Vulnerabilities

Type packetstorm
Reporter George D. Gal
Modified 2010-07-03T00:00:00


Virtual Security Research, LLC.  
Security Advisory  
Advisory Name: Multiple Cisco CSS / ACE Client Certificate and HTTP Header  
Manipulation Vulnerabilities  
Release Date: 2010-07-02  
Application: Cisco Content Services Switch (CSS) / ACE Products  
Versions: Cisco CSS 11500 -  
Cisco ACE 4710 - Version A3(2.5) [build 3.0(0)A3(2.5)  
(Other versions may be affected)  
Severity: High (in specific configurations)  
Author: George D. Gal <ggal (a) vsecurity . com>  
Vendor Status: Cisco CSS vulnerability remains unpatched, workarounds  
Cisco ACE workarounds available  
CVE Candidate: CVE-2010-1575 - Certificate Spoofing Flaw  
CVE-2010-1576 - HTTP Request Parsing Flaw  
Product Description  
>From [1]:  
"The Cisco CSS 11500 Series Content Services Switch is a high-performance,  
high-availability modular architecture for Web infrastructures. As the  
premiere switch for the Cisco Web Network Services Software, the Cisco  
CSS 11500 Series helps businesses to build global Web networks  
optimized for content delivery and e-commerce. By activating HTTP  
headers, the CSS 11500 Series helps to ensure availability, optimize  
utilization, reduce latency, increase scalability, and enhance security  
for Websites, server farms, cache clusters, and firewall systems."  
>From [2]:  
"Cisco(R) ACE Application Control Engine application switches represent  
the state of the art in next-generation application switches for  
increasing the availability, performance, and security of data center  
The Cisco ACE family of application switches includes the Cisco ACE  
Service Module for the Cisco Catalyst(R) 6500 Series Switches and Cisco  
7600 Series Routers, as well as the Cisco ACE 4710 Appliance in a  
standalone form factor for discrete data center deployments."  
Vulnerability Overview  
On June 4th 2009, VSR identified multiple weaknesses in the Cisco CSS  
11500's handling of HTTP header interpretation and client-side SSL  
certificates. Individually, these issues may be considered minor, but  
combined they could allow for the compromise of an application that  
relies on a vulnerable CSS to assist in authenticating clients. If  
successfully exploited, an attacker could spoof another application  
user's identity without possession of the victim's client certificate.  
Additionally, due to the fact that the Cisco CSS product has been  
effectively superceded by the Cisco ACE, VSR has also identified  
similar issues to those described below in the ACE in particular  
These issues may affect any CSS installation, but would have the  
greatest impact on deployments that have the following feature enabled  
in the configuration:  
ssl-server <context> http-header client-cert  
Similarly, on the Cisco ACE, these issues may manifest themselves when  
using a policy map with a class-default class, as shown below:  
policy-map type loadbalance first-match SLB-VIP-REDIRECT  
class class-default  
serverfarm TEST-FARM  
insert-http X-SRC-IP header-value "%is"  
Issue 1: Weak Enforcement of Authority in HTTP Certificate Headers  
Cisco Bug Id - CSCSZ04690  
Affects - Cisco CSS  
The first weakness affecting the Cisco CSS is that, in a typical client  
certificate configuration, HTTP clients may confuse web applications by  
injecting their own certificate headers. When utilizing the CSS to  
terminate SSL communications, SSL client certificates are first  
authenticated by the CSS. From there, the CSS will normally pass the  
client's identity to the back-end web server in the form of several HTTP  
headers as shown below:  
ClientCert-Subject: XXX  
ClientCert-Subject-CN: XXX  
ClientCert-Fingerprint: XXX  
ClientCert-Subject-CN: XXX  
ClientCert-Issuer-CN: XXX  
ClientCert-Certificate-Version: XXX  
ClientCert-Serial-Number: XXX  
ClientCert-Data-Signature-Algorithm: XXX  
ClientCert-Subject: XXX  
ClientCert-Issuer: XXX  
ClientCert-Not-Before: XXX  
ClientCert-Not-After: XXX  
ClientCert-Public-Key-Algorithm: XXX  
ClientCert-RSA-Modulus-Size: XXX  
ClientCert-RSA-Modulus: XXX  
ClientCert-RSA-Exponent: XXX  
ClientCert-X509v3-Subject-Key-Identifier: XXX  
ClientCert-X509v3-Authority-Key-Identifier: XXX  
ClientCert-Signature-Algorithm: XXX  
ClientCert-Signature: XXX  
However, there is no attempt by the CSS to prevent clients from  
supplying their own ClientCert-* headers. Depending on how application  
developers handle multiple copies of these headers, an attacker may be  
able to impersonate other users.  
For example, assuming that a back-end web application simply trusts  
the user identity supplied by the CSS in the ClientCert-Subject-CN  
header and userX wants to impersonate userY, he may simply insert  
the following HTTP header(s) in the HTTP request issued to the  
ClientCert-Subject-CN: CN=userY  
ClientCert-Subject: C=US, ST=MA, L=Boston, O=xxx, OU=xxx, CN=userY  
Upon injecting the attacker-supplied HTTP headers the application  
would receive an HTTP request similar to that shown below:  
POST /targetapp HTTP/1.1  
Content-Type: text/xml; charset=utf-8  
ClientCert-Subject: C=US, ST=MA, L=Boston, O=xxx, OU=xxx, CN=userY  
ClientCert-Subject-CN: CN=userY  
Content-Length: 1024  
ClientCert-Subject: C=US, ST=MA, L=Boston, O=xxx, OU=xxx, CN=userX  
ClientCert-Subject-CN: CN=userX  
ClientCert-Fingerprint: XXX  
ClientCert-Subject-CN: XXX  
ClientCert-Issuer-CN: XXX  
ClientCert-Certificate-Version: XXX  
ClientCert-Serial-Number: XXX  
ClientCert-Data-Signature-Algorithm: XXX  
ClientCert-Subject: XXX  
ClientCert-Issuer: XXX  
ClientCert-Not-Before: XXX  
ClientCert-Not-After: XXX  
ClientCert-Public-Key-Algorithm: XXX  
ClientCert-RSA-Modulus-Size: XXX  
ClientCert-RSA-Modulus: XXX  
ClientCert-RSA-Exponent: XXX  
ClientCert-X509v3-Subject-Key-Identifier: XXX  
ClientCert-X509v3-Authority-Key-Identifier: XXX  
ClientCert-Signature-Algorithm: XXX  
ClientCert-Signature: XXX  
Since existing ClientCert-* headers are left intact, application  
developers are expected to trust only the last instance of a given  
certificate header. This approach is clearly prone to error if  
application developers do not carefully test this attack scenario.  
An alternative approach to securing these headers can be achieved  
through an optional configuration where the CSS places an additional  
prefix string on the inserted certificate headers [4]. For instance, a  
server administrator could select a random header prefix through a  
command such as:  
ssl-server <context> http-header prefix "<random_prefix>"  
This would cause the new certificate headers to be included with the  
form: <random_prefix>-ClientCert-*  
So long as an attacker could not discover this random prefix, then there  
would be no way to spoof these headers. However, this solution is far  
from ideal, since there may be several ways for an attacker to obtain a  
copy of these headers. Examples include TRACE/TRACK requests being  
honored by the back-end web servers or debugging components in web  
applications which echo client headers.  
Issue 2: Lack of HTTP Request Validation  
Cisco Bug Id - CSCTA04885  
Affects - Cisco CSS & ACE  
A second weakness that manifests itself on the CSS and ACE through  
different interpretation of HTTP newline sequences between the content  
switch and commonly used web servers. RFC 2616 [3] defines the US ASCII  
carriage return/line feed (CRLF) sequence as the end-of-line marker for  
protocol elements (excluding the entity-body). Indeed, the CSS and ACE  
appear to adhere relatively closely to this requirement.  
Popular web servers, however, permit various permutations of the CRLF  
sequence as end-of-line markers, including: LF, CR, and LFCR. This  
difference in interpretation could lead to serious consequences if the  
device relies on any of these headers and end-of-line delimiters or  
makes changes to any of them. This has been well documented in the past  
in relation to HTTP request smuggling and related attacks [5].  
It is difficult to consider either of these two issues, taken alone, as  
extremely critical flaws. In the case of certificate header handling,  
there are clearly work-arounds for header spoofing attacks, even if they  
are error-prone or imperfect. For HTTP newline interpretation, it is  
difficult to fault Cisco for adhering strictly to the RFC. However, in  
combining these issues in typical deployment scenarios the end result  
could be catastrophic for an application using a CSS and relying on  
client certificates for user identification and authorization.  
During testing, VSR found that use of invalid newline sequences caused  
the CSS to fail to insert it's own ClientCert-* headers, though the  
back-end Apache web server accepted these newline sequences. This  
clearly defeats the approach that some application developers might take  
in trying to rely only on the last set of certificate headers.   
The following simple HTTP request demonstrates how a client could cause  
the CSS to omit its insertion of the ClientCert-* headers:  
GET /protected_resource HTTP/1.0\x0a  
ClientCert-Subject: C=US, ST=MA, L=Boston, O=xxx, OU=xxx, CN=userX\x0a  
ClientCert-Subject-CN: CN=userX\x0a\x0a  
However, various combinations of end-of-line delimiters should be  
possible. In fact, Cisco describes their logic for the fix as follows:  
"The HTTP request detection will attempt to determine the end-of-line  
marker following the HTTP/1.x line. If it consists of a single LF,  
then the logic will attempt to identify a LFLF."  
Unfortunately, this logic may be problematic when attempting to parse  
HTTP requests consisting of various combinations of end-of-line markers  
within the same HTTP request. For instance, many web servers accept  
requests with mixed newline sequences such as:  
GET /protected_resource HTTP/1.0\x0d\x0a  
ClientCert-Subject: C=US, ST=MA, L=Boston, O=xxx, OU=xxx, CN=userX\x0a  
ClientCert-Subject-CN: CN=userX\x0a\x0a  
While testing a Cisco ACE appliance, VSR was able to cause rules that  
normally add or remove headers in HTTP requests to be omitted when  
the class-default class was used in a policy map as opposed to a layer 7  
class for http communications. It is possible that if a class-default  
class were used for client certificate processing on the Cisco ACE  
Module, that client certificate headers could also possibly be omitted,  
allowing an attacker to insert their own headers used to set the  
authenticated user context.  
Cisco's recommendation to utilize a layer 7 policy class map as shown  
below appears to mitigate this issue, preventing malformed or unexpected  
end-of-line delimiters:  
class-map type http loadbalance match-any http-cm  
2 match http url .*  
policy-map type loadbalance http first-match http-pm  
class http-cm  
serverfarm TEST-FARM  
insert-http X-SRC-IP header-value "%is"  
Recommendations for Cisco and Competing Vendors  
VSR recommends that Cisco (and any other vendors who provide similar  
products) implement more stringent request validation and/or corrections  
when receiving requests which do not utilize HTTP-compliant newlines.  
While failing to validate requests forwarded to back-end servers is a  
reasonable approach when requests are not interpreted or modified, VSR  
believes that some responsibility for correctness should be assigned  
when HTTP requests are modified in transit.  
Three primary approaches are possible for Cisco CSS/ACE devices and  
similar load balancers upon receiving invalid newlines (CR which lack a  
trailing LF and LFs which lack a preceeding CR):  
* Requests with invalid newlines could be rejected outright  
* Invalid newlines could be stripped prior to header interpretation or  
* Invalid newlines could be converted to valid newlines prior to  
header interpretation or modification  
Each of these approaches carries its own set of security, speed and  
compatibility risks which should be weighed carefully. However, given  
that load balancers are border devices which commonly terminate SSL,  
some level of validation should be implemented to help protect internal  
In addition to addressing newline issues, Cisco CSS devices should  
provide better tools and documentation on securing certificate headers.  
VSR recommends one or both of the following approaches be adopted:  
* The CSS and ACE should strip any client certificate headers provided  
by clients prior to adding new ones. Note that this approach cannot  
be guaranteed to work if CR/LF ambiguities are not first corrected.  
Note: Cisco has added a new command to the CSS firmware, currently  
only available by TAC to address this issue via the following command:  
ssl pre-remove-http-hdr  
* The CSS and ACE should require device administrators to specify  
a random header prefix when configuring client certificates, but  
provide a mechanism for existing deployments to disable or opt  
out of a secure by default configuration to support exisitng or  
legacy applications. One way to accomplish this would be to prevent  
the client-cert header insertion configuration from taking effect  
until a device administrator has configured the header prefix using  
the following command syntax:  
ssl-server <context> http-header prefix "<random_prefix>"  
This <random_prefix> parameter should be documented as  
"password" or "key" and users should be urged to select an  
unpredictable one, known only by the CSS / ACE administrators  
and the back-end applications.  
Recommendations for Web Server Vendors  
Web server vendors should provide options for requiring strict HTTP  
compliance in HTTP headers. That is, instances of bare CR or bare LF  
characters should not be accepted as delimiters of HTTP headers. These  
should either be ignored (and considered a part of the HTTP header  
content) or requests containing bare CR and/or bare LF characters should  
be rejected. Eventually, such a configuration option should be enabled  
by default.  
It is impossible to predict the number of combinations of HTTP proxies,  
load balancers and web servers which, in combination, could allow for  
serious vulnerabilities along the lines of HTTP request smuggling, or in  
this case, certificate header spoofing. Web server vendors should stop  
accepting non-compliant requests in order to help head off future  
Recommendations for Customers  
There are currently no known workarounds for the end-of-line marker  
parsing vulnerability on the Cisco CSS. On the Cisco ACE customers are  
encouraged to utilize a layer 7 class map rather than simply using a  
VIP redirect.  
In order to mitigate the risk of the ClientCert header insertion  
that may lead to user impersonation VSR recommends the use of the  
following CSS/ACE command:  
ssl-server <context> http-header prefix "<random_prefix>"  
Where <random_prefix> is a difficult to guess value that is  
comprised of a combination of 3 of the 4 following character classes:  
lowercase alphabetic, uppercase alphabetic, numeric and special  
characters. The value should consist of a minimum of 8 characters,  
decreasing the likelihood of successful brute-force attacks.  
VSR strongly recommends disabling the TRACE/TRACK HTTP methods on the  
target web/application server and periodically changing the random  
Other work arounds are possible if application administrators/developers  
are able to perform additional validation of user identities. One  
possibility would be to simply disable SSL termination at the Cisco CSS,  
instead performing certificate validation at the application server  
level. However, this may not be feasible in many environments.  
As an alternative, user certificate headers could undergo a second round  
of validation at the application server level. For instance, randomized  
user IDs or user-specific secret values could be embedded in user  
certificate headers which would be passed along by the CSS to  
applications. Once received, application servers would validate this  
additional parameter to help prevent header spoofing in a user-specific  
way. However, while this could prevent blind certificate spoofing, an  
attacker who could perform man-in-the-middle attacks would likely be  
able to obtain certificate headers of other users and subsequently  
bypass this protection on the server side.  
Versions Affected  
The described end-of-line interpretation behavior exists in all versions  
of the Cisco CSS and Cisco ACE (Application Control Engine) and has  
only been partially updated in version and ACE A2(3.0).  
Cisco has stated that the new behavior to address this issue in the CSS  
applicance is to look for the terminator of LFLF if the separator that  
follows the HTTP/1.X is a single LF. However if it is a CRLF pair the  
CSS will only search for CRLF as end-of-line markers, however it does  
not currently address situations where mixed end-of-line markers are  
The client supplied header insertion vulnerability continues to affect  
all versions of the Cisco CSS and ACE (Application Control Engine),  
however a mitigation exists and is described above.  
Vendor Response  
The following timeline details Cisco's response to the reported issue:  
2009-06-05 VSR submitted a security bug report to Cisco PSIRT  
2009-06-06 Cisco confirmed receipt of bug report  
2009-07-02 Cisco acknowledged the presence of VSR submitted  
2009-08-04 Cisco confirmed release plans for end-of-line marker  
parsing vulnerability  
2009-10-14 Cisco provided update on defect notes and remediation  
2010-03-11 VSR reviewed the Cisco release notes on potential CSS bug fix  
and provided Cisco with notice indicating that these  
fixes are inadequate  
2010-03-12 Cisco confirmed receipt of correspondence  
2010-04-07 Conference call between VSR and Cisco to discuss security  
ramifications and understand implementation specifics of Cisco  
2010-05-21 VSR performed testing against Cisco ACE 4710 in a lab  
end-of-line parsing issue in default class map configuration  
2010-05-26 Cisco provided guidance on utilizing a layer 7 class map to  
address issue on Cisco ACE  
2010-06-28 VSR performed verification of layer 7 class map in a lab  
verifying correct device behavior  
2010-06-29 Cisco and VSR begin coordinating advisory release  
2010-07-01 Cisco was provided a draft advisory  
2010-07-02 VSR advisory released  
Common Vulnerabilities and Exposures (CVE) Information  
The Common Vulnerabilities and Exposures (CVE) project has assigned  
the numbers CVE-2010-1575 and CVE-2010-1576 to these issues. These  
are candidates for inclusion in the CVE list (,  
which standardizes names for security problems.  
Thanks to Cisco for response and cooperation, Tim Morgan for assistance in  
attempting to verify this issue on other competing load-balancers, and  
Ken Pierce for providing a lab environment to conduct Cisco ACE testing.  
1. Cisco CSS 11500 Series Content Services Switches  
2. Cisco ACE Application Control Engine Application Switches  
3. RFC 2616: Hypertext Transfer Protocol -- HTTP/1.1  
4. Configuring SSL Termination:  
Adding a Prefix to the Fields Inserted in the HTTP Header  
5. HTTP Request Smuggling  
Copyright 2010 Virtual Security Research, LLC. All rights reserved.