Assembly And The Art Of Debugging

This article is an introduction to understanding Intel’s assembly language while debugging a computer program.

Debugging in assembly can be a daunting task and not every developer likes to make sense of the mnemonics that appear on the screen.

While debugging, one is normally used to viewing local variables, parameters  and syntax highlighted code in the most user friendly manner.  This is the primary reason why many developers dislike debugging in assembly initially.

Why Debug In Assembly?

Debugging in assembly is not an optional skill to have.   Every developer encounters a situation where there is no other alternative other than cracking open the assembly code.  Here are a few reasons why a developer should get their hands dirty with this skill –

1. You have a binary and its corresponding code but the build environment is not functional.  This could be because of missing tools, unavailable compilers or the lack of understanding of complex scripts and steps needed to recreate the binary.
2. You are dealing with a release build only bug and the debug symbols are unavailable or you are dealing with a third party library and don’t have access to its source code.
3. You like to have fun 🙂

What You Need?

1. A debugger that can display assembly language (Visual Studio, WinDbg, gdb, XCode are capable debuggers).
2. A ready reference to the Intel instruction set.  Google works just fine.
3. Patience and maybe a link to this article.  🙂

The key to debugging in assembly is to understand how functions are called, parameters are passed, local variables accessed.  The code flow and logic can be understood by looking up the Intel instruction set .

The diagram below (click for larger image)  is roughly how a stack should look like when a function is called.  Mapping the assembly code to this diagram should simplify the debugging experience.  Note that the stack grows from top to bottom.

Passing Parameters To A Function

Parameters are the first to be pushed onto the stack.  The caller of the function pushes them from right to left.

Cleanup Of Parameters

Parameter cleanup may happen in the function that was called (e.g. stdcall calling convention) or by the caller of the function (e.g. cdecl).  Various kinds of calling conventions of the x86 architecture are explained in detail in this wiki article.   Stack cleanup code right after the function call gives a good idea of  the calling convention being used.

Take note that if cleanup happens in the function being called, variable number of arguments cannot be passed to it.  In contrast, variable number of arguments can be implemented if the parameter cleanup happens outside the function.  This essentially is the main difference between stdcall and cdelc calling conventions on a x86 architecture.

The Function Call

Before control is passed to the function, the return address (where the caller is supposed to resume after the function completes execution) is pushed on the stack.

To recap, parameters are pushed first, followed by the return address.  Later depending on the calling convention of the function, you may see cleanup happening right after the function call or within the function itself.

In the stack diagram above, as the stack grows downwards, the parameters are at the top followed by the return address.

Example

Take a look at the snippet and the equivalent assembly below :

[1]int a = 4;
[2]mov         dword ptr [a],4
[3]
[4]char c = 0;
[5]mov         byte ptr ,0
[6]
[7]c = f(a, 22);
[8]push        16h
[9]  mov         eax,dword ptr [a]
[10]  push        eax
[11]call        f (4111D1h)
[12]add         esp,8 
[13]mov         byte ptr ,al

• In line [8], the constant value 22 (16 hex) is being pushed on the stack.
• In line [9-10] the variable ‘a’ is pushed.
• Line [11] is the function call to f() which implicitly pushes the return address.
• The return value of the function is passed in a register and copied in variable ‘c’ (line[13]).
• In line [12] one can see that 8 bytes are being discarded from the stack as f() uses the cdecl calling convention and cleanup needs to happen in the caller function.

The Function

The function usually starts with a prolog and ends with an epilog.

The prolog is either a simple ENTER instruction or more commonly, it saves the ebp register and copies the stack pointer value in it so as to use it as frame pointer.

push        ebp
mov         ebp,esp

The epilog just reverses what the prolog had done.  It is implented by a LEAVE instruction or is implemented as follows

mov         esp,ebp
pop         ebp

The ret instruction passes control from the function to the return address stored in the stack.

Compilers provide a switch that allows one to omit the frame pointer (fpo or frame pointer omission) which effectively removes the prolog and epilog and uses the ebp register for other optimizations.  It is easy to demarcate function boundaries where the frame pointer is present but one should be aware that the entry and exit points may be absent.

Allocation Of Local Variables

Local variables are allocated on the stack.  The total size of the local variables is computed at compile time and at runtime those many bytes are reserved on the stack.

004113A0  push        ebp
004113A1  mov         ebp,esp
004113A3  sub          esp,0E8h

Note that in the example above, 232 (0xE8) bytes are being reserved for local variables.

The above code will also help in understanding why local variable allocation is much faster than requesting memory from heap.   Allocating local variables requires moving the stack pointer whereas memory heap management is much more complex.

As the return address and local variable area are very close to each other, buffer overflows can be caused if data is written past the the local variable area which can then overwrite the the return address.   When this return address if forced to point to shell code (or to some other code placed intentionally by a hacker), such a buffer overflow is termed as an exploit.

Local Variables And Parameters In Assembly

The most important part in assembly is to be able to identify access to locals and parameters.  The frame pointer that is set in the prolog, acts as the reference pointer using which all variables can be accessed.  If you add to the frame pointer (Remember : P for Plus and P for Parameters), you will be able to access parameters.  If you subtract from the frame pointer, you will be able to access local variables.

mov         byte ptr [ebp-20h],3
mov         byte ptr [ebp+20h],5

The first line above accesses a local variable whereas the second accesses a parameter.

In the case of frame pointer omission, everything is calculated with respect to the stack pointer.  Therefore [esp + 20h] might refer to a local or a parameter depending on where the stack pointer currently points to.  And if say a register is pushed on the stack, the same variable will now be referred using [esp + 24h].  Debugging functions that have optimized out the frame pointer is not that easy as the changes made to the stack pointer need to be constantly tracked.

Conclusion

Debugging in assembly is not only fun but a useful tool to debug difficult problems.  Different debuggers provide different interfaces to interact with the assembly code but under the hood, all programs work alike.   Understanding this is the key to debugging in assembly with ease.