Details of our SE-2014-02 project have been released to the public. A technical writeup and accompanying Proof of Concept codes can be found at the following location:
In case of Google App Engine for Java, its first layer of defense was built around a Java VM sandbox. Google decided to implement an additional security layer (sandbox) on top of it. As a result, several custom security measures were integrated into a Java VM runtime. This in particular concerns the Class Sweeper, of which goal was to verify and transform untrusted user code into a corresponding, safer representation. The API Interception and Interjection mechanism was meant to enforce proper security checks in runtime for security sensitive Java SE API calls. Finally, the JRE Class Whitelisting was supposed to limit the scope of Java classes visible to user applications.
Unfortunately, the custom security layer implemented by Google turned out to be vulnerable to multiple security weaknesses. Some of them were instances of known vulnerabilities published in the past. This in particular concerns the issues disclosed as part of SE-2012-01 research affecting Java SE implementation from Oracle and IBM (Issues 4, 9 and 31). This also concerns several security vulnerabilities that made it possible to break Oracle Java Cloud Service (Issues 1 and 4). The ability to break GAE with the use of a prior research indicates that it was either ignored or simply never taken into account.
The majority of the flaws discovered in GAE were related to either Reflection API or Class Loaders. These were rather simple issues, which should have been caught during a security review process preceding a release of GAE software (or any major update to it). Reflection API and Class Loaders are fairly complex and security sensitive components of a Java VM. A lot of expert knowledge and a deep understanding of their operation is usually required prior to introducing any changes to them without jeopardizing the security of a JVM. Regardless of that, Google decided to "reimplement" Java Reflection API, through the GAE interception layer. This was the same API that caused so much trouble for Oracle in the recent years and that was responsible for dozens of security issues in Java. The company also allowed for a creation of arbitrary user provided Class Loaders in GAE, which immediately created a need to protect the environment from these objects.
As a result, a security model of a standard JRE was weakened (Issue 17-19). Arbitrary vulnerabilities were introduced on top of the implementation of JRE API calls (Issues 3 and 5), which GAE Java API interception model aimed to protect. These vulnerabilities constituted the same violations of Java Secure Coding Guidelines  of which Oracle has been usually accused of. Many vulnerabilities had its origin in an incomplete interception of Java SE 7 method handles API (Issues 7, 9, 11, 15, 16), Class Loader's operation (Issue 13) or mitigations aimed at making user Class Loaders less privileged than usual (Issues 8, 10, 12, 14).
During our correspondence with Google, the company often emphasized that we only broke the first layer of defense and that it considered the remaining, lower sandboxing layers sufficiently robust. This could explain why the environment of a cloud computing platform from Google ran on a 1+ year old Java runtime (Issue 21). However, the amount and nature of information leaked by the first sandboxing layer along with the company's preference not to have the details of the next sandboxing layers published seemed to contradict the confidence expressed regarding their robustness.
It should be also mentioned that regardless of a successful detection of our activity in GAE, Google's ability to detect attacks in the environment was not perfect. Our activity raised an alarm 2 years after an initial GAE security sandbox compromise. It was likely detected because we decided to launch more aggressive (more visible / risky) tests and did not follow our usual, low-profile pattern of activity.
Google is a specific software vendor that serves hundreds of millions of users on a daily basis through its custom services. In most cases, the architecture and implementation details of these services are not known due to their server-side nature. As a result, the ability to discover security issues in these services could be quite challenging. Without any doubt security of Google services is not less important than discovering vulnerabilities in a client / server side software of other big software vendors. That thought alone should catch attention of Google itself. At the end of a day, it might turn out that it would be of a more benefit to the company and users of its services to have Google security personnel to be more focused on its own products instead of the products of the competition . The case of Google App Engine for Java shows that this might actually make sense and that there are still places for improvement in Google's own yard.
Best Regards, Adam Gowdiak
Security Explorations http://www.security-explorations.com "We bring security research to the new level"
References:  Secure Coding Guidelines for Java SE http://www.oracle.com/technetwork/java/seccodeguide-139067.html  Project Zero http://googleprojectzero.blogspot.com/