Compiler assisted tracking of hacker assaults

An ever-growing number of cyber attacks are made via network connections on open source software written in C or C++. Such software includes the popular Apache web server, various DHCP servers, etc. These attacks take advantage of flaws inadvertently left in software systems due to a lack of complete testing. Public domain tools such as "gcov" allow a software engineer to assure that each line of code has been executed and tested. These tools operate in a "batch" mode, first collecting statistics, then later displaying the program coverage. This paper presents a new approach to software coverage modifications have been made to the GCC compilers for C and C++, which allow for an execution-time monitoring facility. The program software is compiled with this "instrumentation". As the program executes, information is gathered concerning the execution of the source code. This information can be saved to a file for later processing (as in "gcov") or can be examined while the program executes. This "instrumenting compiler" is used for software which is run in a controlled environment as attacks are made. The call tree and execution trace of the software under test are examined as the hacker assault progresses. This paper outlines the techniques used to modify the internal representations of the GCC compilers to allow this instrumentation. The compiler uses an internal representation called RTX. Additional calls to the instrumentation functions are automatically generated in RTX prior to emitting assembly language output. The paper addresses the techniques for locating the instrumentation points, avoiding problems when software is compiled with optimization, and presents a sample case of open software being instrumented. The latter demonstrates the output formats and shows an example of an attack on an open source program.