A heap based buffer overflow vulnerability present in KeyView PDF filter as used by Domino can lead to remote arbitrary code execution.
KeyView 10.16 as used by IBM Domino 9.0.1
While parsing an ID description of a PDF trailer, no checks are being made regarding its length. The ID, which is usually an ASCII representation of hexadecimal value, is converted into it's byte array equivalent without size checks leading to a straight forward heap overflow which can be abused to overwrite sensitive data structures or heap metadata and subvert the process execution flow to arbitrary code execution.
Shortened test case triggering the vulnerability can be summarized as follows:
/Root 4 0 R
/ID[<AAAAAAAAAAAAA...(a large number of hex characters)...41414141
When initializing the parser, a large chunk of memory is allocated in
PDFAllocateContext function of
pdfsr.so library (132648 bytes in size to be precise). An offset into this large context memory chunk is used as a start of the buffer that ends up being overflown. The overflow happens in the following code:
.text:B79DC7DA mov [ebp+var_15], al 
.text:B79DC7DD lea eax, [edx+1] 
.text:B79DC7E0 mov [ebp+var_F0], eax 
.text:B79DC7E6 movzx eax, byte ptr [edx+1]
.text:B79DC7EA mov [ebp+var_14], al  .text:B79DC7ED lea eax, [edx+2] .text:B79DC7F0 mov [ebp+var_F0], eax .text:B79DC7F6 mov [ebp+var_13], 0 .text:B79DC7FA mov dword ptr [esp+0Ch], 0 ; group .text:B79DC802 mov dword ptr [esp+8], 10h ; base .text:B79DC80A mov dword ptr [esp+4], 0 ; endptr .text:B79DC812 lea eax, [ebp+var_15] .text:B79DC815 mov [esp], eax ; nptr .text:B79DC818 call ___strtol_internal  .text:B79DC81D mov edx, [ebp+counter] .text:B79DC823 mov [edx+edi+11D8h], al  .text:B79DC82A mov edx, [ebp+var_F0]
.text:B79DC830 cmp esi, edx  .text:B79DC832 jbe loc_B79DCB16
.text:B79DC838 add [ebp+counter], 1 .text:B79DC83F movzx eax, byte ptr [edx] .text:B79DC842 cmp al, '>'  .text:B79DC844 jnz short loc_B79 ```
At the beginning of the above code, at ,
al contains the first character of the ID hex string which is saved in
var_15, at , and  the second character is saved in
var_14 which, along with var_15, makes up a string that is passed to
strtol at . Function
strtol is called with base set to 0x10, converting from hex string into bytes. At  the returned value is written to the buffer. This is where the out of bounds write occurs when the counters are past the boundary of the buffer. At  the current string counter is checked against the file end,
esi points to the end of file. If the check at  passes, another check is made at  which is looking for a terminating
> character. No buffer length checks are being made.
This buffer overflow can be abused to overwrite other parts of the context structure or even, further up the heap, the heap metadata thus leading to process crash and possible arbitrary code execution.
The supplied test case is crafted in such way that the buffer overflow will overwrite a pointer on the heap which is later dereferenced during heap cleanup. The process crashes while calling free on an invalid pointer. Further control over data overwritten in the context structure can lead to arbitrary code execution.
Detection of PDF files specifically crafted to trigger this vulnerability can be based on the abnormally large length of the ID value in the PDF trailer.
The vulnerability can be triggered with the supplied test case in the
filter standalone KeyView binary shipped with IBM Domino, or by sending it as an attachment with an email to a Domino mail server.