Network Security Blog

The ASN.1 vulnerability that has been patched by Microsoft’s latest download is classified as a heap overflow. What exactly is a ‘heap overflow’ you might ask? Here’s an article that will hopefully explain it to you

-Heap Overflows-

The extended entry contains the body of an email sent to the CISSP mailing list by Bill Royds. This is a slightly simpler and more easily read explanation of heap overflows.

The memory that a computer program uses can be allocated in at least 3
different ways. The first is static memory in a data frame that is allocated
at program initiation and only freed when it ends. The second is memory
allocated to each subroutine as it starts and freed when it exists. This is
generally allocated using a stack (like a stack of plates) and in many
architectures (X86 being one) also hold the return address in the caller
program to where the subroutine will return when it completes. The third
memory is a hunk of memory called the heap where dynamic variables created
by the C malloc C++ new routines is stored and which is returned for further
use by the free statement. Ever since the PDP-11 it has been traditional to
put the heap at the bottom of memory available for data and the stack at the
top working downwards. The idea is that it is more flexible to work from
the ends toward the middle rather than giving each kind of memory absolute
fixed bounds.
ASCII ART view
__________Top of memory_________
| stack space |
| ….. |
| ””” |
| heap space |
—— Bottom of data memory— |

So when a subroutine is called on most modern computers, the return
address (and the subroutine parameter addresses) are pushed on the stack,
then a big chunk of stack space is allocated for the local variables in the
subroutine. When the subroutine is working, it uses the local variable space
for things like the buffers where input data is read. Since C uses character
strings that have no bounds, but are just end delimited by a marker in
memory (a byte with value 0), it is possible (if insufficient checks on read
size are made) to read into the input buffer more data than there was space
allocated and overwrite the return address of the subroutine (your classic
buffer overflow) since the data is being read into the buffer from lower
addresses to higher addresses. If the string that is passed into the
subroutine as input is crafted to overwrite the return address with an
address that is in another part of the string, then that part of the string
can contain executable code (shellcode) that can do things that the hacker
wants to do (most often open a Command line shell, hence shell code) and the
program can be take over.
If instead of being read into a local variable on the stack the
programmer allocates memory on the heap (with malloc in C), one can still
overflow this variable (a heap overwrite) , but it is much less likely to
overwrite the return address of the subroutine. There are still some ways to
make this code executable, but they take a much greater knowledge of the
actual logic of the program rather than just knowing that one particular
input statement did not properly handle its input size.
A format string overflow is a variant on either of this overflows where
the overflow is not caused by unchecked input but by not properly checking
the parameters of an output statement. The method for producing formatted
output is C is simply a function that takes a file handle as its first
parameter (which is implicitly SYSOUT in one variant), a character string
containing formatting characters, and a variable number of arguments. If a
programmer forgets to give a formatstring argument and just tries to output
the contents of a variable in that position (fprintf(file,variable) instead
of fprintf(file,”%s”,variable) ,a malicious input that sets the value of
variable can use the C format characters to manipulate memory (and normally
it is stack) creating a vulnerability.