Security flaws mitigated by compiler optimizations

An optimizing compiler is one that tries to maximize some attributes of an executable program at the expense of other attributes. Most modern compilers support some sort of optimization. Normally code optimized for performance is the usual preference Unfortunately, sometimes these optimizing techniques can lead to some unexpected security problems Luckily, there are many opportunities for compiler optimizations to actually mitigate security flaws.

mod_rewrite off-by-one

In June 2006, an off-by-one flaw was found in the mod_rewrite module of Apache HTTP server and was assigned CVE-2006-3747. The ability to exploit this flaw depended on stack-layout for a particular compiled version of mod_rewrite, which was, in turn, determined by the optimization used by the compiler. The flaw basically had the ability to potentially overwrite registers which were pushed on the stack.

Consider the following code segment from mod_rewrite:

[modules/mappers/mod_rewrite.c]
static char *escape_absolute_uri(apr_pool_t *p, char *uri, unsigned scheme)
 { 
 char *cp;
. . .
. . .
if (!strncasecmp(uri, "ldap", 4)) { char *token[5]; 

int c = 0; 

token[0] = cp = apr_pstrdup(p, cp); 
while (*cp && c < 5) { 
if (*cp == '?') { 
token[++c] = cp + 1; 
*cp = '\0'; 
 } 
++cp; 
 }

This flaw is triggered when the LDAP scheme is used. Variable token is an array of 5 char pointers. The while loop runs from token[1] to token[5], which triggers this off-by one stack-overwrite security flaw.

For Red Hat Enterprise Linux (RHEL) 3 and 4 (which were both supported at that time), it was determined that this flaw was not exploitable because the compiler inserts several bytes of padding after the vulnerable variable (“token” in this case). The following stack layout should put things in perspective.

On RHEL 3 x86, the stack frame looks like this (note that your offsets will vary based on randomization):

                0xb72fbed1 [b4] tempvar
0xbfffd820:     0x08253eb0 [b8] tempvar
                0xb7097e47 [bc] tempvar
                0xb709261b [C0] tempvar
                0xb709a934 [C4] c
0xbfffd830:     0x08160198 [C8] token[0]
                0x00000000 [CC] token[1]
                0xb712aaec [D0] token[2]
                0xb709a934 [D4] token[3]
0xbfffd840:     0x08255791 [D8] token[4]
                0xb7097e53 [DC] ** unused ** <--- Overwritten 
                0xbfffd868 [E0] ** unused **
                0xb709419d [E4] ** unused **
0xbfffd850:     0x08255791 [E8] ** unused **
                0xb7098010 [EC] ** unused **
                0x00000000 [F0] ** unused **
                0xb709a934 [F4] callee-saved bx
0xbfffd860:     0x00000007 [F8] callee-saved si
                0xb7097e53 [FC] callee-saved edi
                       0xbfffdcd8 [00] ****** ebp points here
                       0xb709154c [04] ****** the return address
0xbfffd870:     0x08253a80 [08] this is where p is
                       0x08255790 [0C] this is where uri is
                      0x00000007 [10] this is the value of scheme

(Some local variables such as cp are stored in registers)

The memory marked ** unused ** can be verified as being unused by disassembling the function and going through the assembly An exploit for this issue can only write to the 4 bytes immediately following token[4] and, in this case, it's pointing to unused memory and the exploit has no effect Therefore Red Hat Enterprise Linux 3 on i386 using Red Hat httpd binaries was not vulnerable to this issue.

In contrast, Fedora Core 4 and 5 builds (both supported at that time) were vulnerable as the compiler version used added no stack padding. For these builds, the pointer being overwritten overwrites a saved register and, unfortunately, one that has possible security consequences. The exploitability depends on the version of gcc compiler used along with the compiler flags enabled

In this case perhaps it was the compiler version which caused the difference in behavior rather than the flags used:

Fedora core 4 version of mod_rewrite was compiled with the following flags:

%__global_cflags    -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fstack-protector --param=ssp-buffer-size=4

Red Hat Enterprise Linux 4 version of mod_rewrite was compiled with the following flags:

%__global_cflags -O2 -g -pipe

As you can see the flags are different, but none of them seem to explain why EL4 added padding and FC4 did not.

The attack, however, only works when mod_rewrite is enabled and a specific style of rewrite rule is used. Also the attacker needed to be able to defeat Fedora Core address space randomization to achieve code execution with this flaw.

kernel: stack-based buffer overflow in chap_server_compute_md5() in iscsi target

A more recent example showed up in 2018 in the Linux kernel Designated CVE-2018-14633, this flaw allowed an unauthenticated, remote attacker to cause a stack buffer overflow and overwrite up to 17 bytes of the stack.

The flaw resides in this code:

[drivers/target/iscsi/iscsi_target_auth.h]
#define CHAP_CHALLENGE_LENGTH  16
#define MD5_SIGNATURE_SIZE     16      /* 16 bytes in a MD5 message digest */
#define MAX_RESPONSE_LENGTH    64      /* sufficient for MD5 */
#define MAX_CHAP_N_SIZE        512

[drivers/target/iscsi/iscsi_target_auth.c]
static int chap_server_compute_md5( ... char *nr_in_ptr, char *nr_out_ptr, ... )
{       ...
        unsigned char client_digest[MD5_SIGNATURE_SIZE];
        unsigned char server_digest[MD5_SIGNATURE_SIZE];
        unsigned char chap_r[MAX_RESPONSE_LENGTH];
        . . .
  unsigned char type, response[MD5_SIGNATURE_SIZE * 2 + 2];
  . . .
        if (extract_param(nr_in_ptr, "CHAP_R", MAX_RESPONSE_LENGTH, chap_r,
            &type) < 0) { ...exit... }
        ...
        chap_string_to_hex(client_digest, chap_r, strlen(chap_r));
        ...
        // int crypto_shash_finup(struct shash_desc *desc, const u8 *data,
        //     unsigned int len, u8 *out)
        // note, server_digest is *out
        ret = crypto_shash_finup(desc, chap->challenge, CHAP_CHALLENGE_LENGTH, server_digest);
        ...
        chap_binaryhex_to_asciihex(response, server_digest, MD5_SIGNATURE_SIZE);

Here chap_string_to_hex(), which basically does hex2bin(), can have a maximum of 64-bytes

input string It converts the input to a 32-byte binary string and writes it, plus a trailing \0, to the 16-bytes on-stack buffer client_digest[], making this a classical buffer overflow At this point, chap_r is attacker-controlled. The overwrite can be 16 bytes + zero byte

In RHEL 7 x86_64, the server_digest[] and the 1st byte of response[] can be overwritten According to the code, server_digest[] and response[] are not used after the overwrite and are filled with correct values later by the crypto_shash_finup() and the chap_binaryhex_to_asciihex() In this case the compiler put server_digest and response next to chap_r, resulting in the overflow not causing any security implications This means the flaw had no impact on RHEL 7 x86_64 systems.

Depending on how the kernel binary is built (e.g. depending on a compiler, compiler flags, and hardware architecture), a compiler may put other local variables or function arguments on the stack after the client_digest. This may lead to different outcomes, like chap_server_compute_md5() erroneously returning with a result of a successful authentication by rewriting auth_ret, thus exposing all the target's content to an attacker. Or, if nr_out_ptr is overwritten, this can damage other kernel memory content via later sprintf() and thus lead to a system crash.

In general, depending on the version of the compiler used and technologies supported by the underlying operating system, various flags can be passed to the compiler to enable different security technologies to be compiled with applications. Most of them insert a small amount of code in the application to enable various checks during runtime and may cause some small performance differences when compiled without these options. Therefore optimization (be it for size or performance) and security usually go in the opposite direction. Fortunately, this doesn’t always have to be the case.