A stack overflow leading to a crash due to unbounded recusive function call is present in the PDF file format parsing code of the IX SDK.
Oracle Outside In IX sdk 8.5.1
While parsing a malformed PDF file which contains a reference to the Root element with malformed or missing an xref table a recursive call to a function is made each time with the same parameters eventualy leading to a crash due to process stack exhaustion.
Technical information below:
During a call to VwStreamOpen function in libvs_pdf.so library, code dealing with Root element is reached (image base is at 0xB74BF000):
.text:B74ED100 lea ebp, [esp+6BCh+var_BC]
.text:B74ED108 mov ecx, 8
.text:B74ED10D xor eax, eax
.text:B74ED10F mov edi, ebp
.text:B74ED111 rep stosd
.text:B74ED113 lea ecx, [esp+6BCh+var_34]
.text:B74ED11A mov eax, [esp+6BCh+arg_10]
.text:B74ED121 mov [esp+6BCh+s], eax
.text:B74ED124 lea edx, (aRoot - 0B74F6998h)[ebx] ; "Root"
.text:B74ED12A mov eax, esi
.text:B74ED12C call sub_B74D653E
.text:B74ED131 mov edx, eax
.text:B74ED133 test ax, ax
.text:B74ED136 jnz loc_B74E
sub_B74D653E in turn calls a function
sub_B74D5EEC in which the unbounded recursive call can happen:
.text:B74D6095 lea edx, [esp+5ACh+var_14]
.text:B74D609C lea eax, [esp+5ACh+var_C0]
.text:B74D60A3 mov ecx, ebp
.text:B74D60A5 call sub_B74D5EEC
.text:B74D60AA test ax, ax
.text:B74D60AD jnz short loc_B74
The supplied minimized testcase triggers the recursive call and leads to a crash due to stack exhaustion. The sample program
ixsample supplied with the SDK can be used to reproduce the crash.