TL;DR: security products attempt to monitor process behavior by hooking Win32 APIs in user-mode. However, as the user-mode component of APIs are loaded and owned by the current process, the process itself can inspect, overwrite or simply just not use them and use its own implementation of the API functionality, to avoid messing with the hooks altogether. High-privileged mechanisms such as kernel callbacks should be used instead/in addition.
Backgound
One way for security products to monitor a process is to apply "hooks" on the Win32 APIs that the process may use, such that if the process calls a hooked win32 function, let's say WriteFile(), then the security product would get alerted.
Where are the APIs functions?
When a process, lets say notepad.exe, wants to use a windows API, lets say WriteFile(), then the process have to load the WriteFile functionality. This is done by loading the windows DLL that contains said functionality. In this case, that DLL is C:\Windows\System32\Kernel32.dll. All this means is that the notepad.exe will get a new module in its process that holds a copy of Kernel32.dll, and therefore also all the functionality that it contains. When we view the modules of nodepad.exe with Process Hacker, it looks like this:
If I then double click on kernel32.dll in the modules list, Process Hacker will show me the details of it, including all the functionality that is available in it. These are listed under Exports:
Here we see all the exported functions, including WriteFile and it's address.
We can look at the instructions that the WriteFile function consists of by loading the kernel32.dll into a disassembler, such as e.g. Ghidra. When Ghidras has analyzed the DLL, then we can browse to the WriteFile function in the Symbol Tree view on the left, under Functions:
Now, Ghidra shows us the assembly/instructions that are dissassembled from the opcodes ff 25 8a c0 05 00 of the WriteFile function in the middle of the screen, as well as an attempt of generating pseudo code to make it easier to read the instructions on the right in the Decompiler view. We see that the WriteFile function is really just a wrapper around some other functionality, apparently in api-ms-win-core-file-l1-1-0.dll.
When notepad.exe loads the kernel32.dll into memory, these opcodes are what's actually loaded (together with a lot of other stuff as well), and this is where API hooking comes in.
Security products may use various techniques of monitoring process creation, and hook specific API functions when they detect that a new process has been launched, such that the security product can monitor the new process' API calls. One way that security products do this is by injecting one of the security product's own DLL into every process, using a technique known as DLL injection. An example of what that could look like, is given in this really nice article: https://ethicalchaos.dev/2020/05/27/lets-create-an-edr-and-bypass-it-part-1/.
NtReadVirtualMemory
The NtReadVirtualMemory API function, exposed from C:\Windows\System32\ntdll.dll, allows processes to read arbitrary memory from other processes (if they have proper privileges, of course). As an example, this function is used under the hood in Task Manager, when it creates minidumps of processes. Doing so is a simple way of stealing credentials from users of a system, by dumping LSASS' memory, and exporting the memory dump for offline analysis with Mimikatz. A proof-of-concept of this attack is shown here: https://www.youtube.com/watch?v=X4YCJYQ-zVg.
Here's a screenshot of what the function look like in Ghidra, when dissassembled. As you can see it is simply a short function that performs a syscall - look at line 11 in the decompiler:
Due to its popularity for attackers, security products tend to hook the NtReadVirtualMemory function, such that they can detect or block attempts to steal the memory of LSASS.
What does a hook look like?
To see what a hook looks like in-action, we can use a debugger to view the raw memory contents of a process that has some of its loaded API functions hooked. We can do that with WinDbg.
The following screenshot shows WinDbg while debugging a malicious application (ConsoleApplication3.exe).
When the executable is first loaded by WinDbg, the list of loaded DLLs is shown.
The list contains a (censored) entry to a non-windows DLL, that is part of a security product, thus indicating that said security product use the technique described earlier to inject its own DLL into the process.
By entering the command "u NtReadVirtualMemory", we ask WinDbg to show us the opcodes+assembly/instruction of the NtReadVirtualMemory function as it appears in memory, highlighted in the red rectangle. The opcodes+assembly/instructions should be identical to the ones in the ntdll.dll file, previously shown with Ghidra.
The opcodes should start with 4c 8b d1 b8 3f 00 00 00 f6 04...
But instead, we see that they are: e9 1b 4d 00 80 00 00 00 f6 04...
We see that the first instruction is actually a jmp (jump) instruction to address 00007ff7c0b31280. This clearly shows us that the NtReadVirtualMemory function has been modified, such that the function is hooked - meaning that the security product now may have the ability to "intercept" any call to the function, where it may alert or block the call altogether. This is because the jump instruction will cause anyone who calls ntReadVirtualMemory to jump into the security products own code, where the detection/blocking logic is implemented.
How can we call NtReadVirtualMemory, without the security product detecting it?
There are different techniques, each with their own pros/cons.
Overwriting the hook
Overwrite the modified bytes (4c 8b d1 b8 3f) with the correct bytes (e9 1b 4d 00 80). This is the technique employed by Dumpert from Outflank (https://github.com/outflanknl/Dumpert). (A slightly modified version of this is https://github.com/CylanceVulnResearch/ReflectiveDLLRefresher that compares all loaded DLLs with their on-disk counterpart to onsure that they havn't been hooked. If they hare hooked, they are automagically unhooked.)
Pros:
Relatively simple and straight forward.
This ensures that any other API that also use NtReadVirtualMemory under the hood, such as the MiniDumpWriteDump function, will also benefit from this bypass technique.
Cons:
Security products can monitor their hooks and check that their hooks are not overwritten. If they detect that their hooks have been tampered with, then they can alert.
Overwriting the loaded module
Overwrite the ~entire loaded ntdll.dll in-memory with the on-disk ntdll.dll: https://www.ired.team/offensive-security/defense-evasion/how-to-unhook-a-dll-using-c++
Pros:
More simple than the above method
This ensures that any other API that also use NtReadVirtualMemory under the hood, such as the MiniDumpWriteDump function, will also benefit from this bypass technique.
No possibility of forgetting a hook in ntdll.dll, since we take the entire thing
Cons:
Security products can monitor their hooks and check that their hooks are not overwritten. If they detect that their hooks have been tampered with, then they can alert.
More noise than previous method, and therefore less stealthy.
Bring your own
Leave the hook in place, and don't attempt to modify it, instead avoid calling the hooked NtReadVirtualMemory function entirely. This can be done in a number of ways:
Using your own NtReadVirtualMemory function that we implement ourselves in our own executables. This is how I did it in MagnusKatz, relevant code here: https://github.com/magnusstubman/MagnusKatz/blob/main/ConsoleApplication3/ConsoleApplication3/ConsoleApplication3.cpp#L188-L230
Here's similar technique, doing pretty much the same thing: https://www.ired.team/offensive-security/defense-evasion/using-syscalls-directly-from-visual-studio-to-bypass-avs-edrs
Pros:
Ensures that the hooks are left in place, and the security product cannot detect any tampering with the hook, since no tampering with the hook takes place
Cons:
Relatively complex and not straight forward to get right. Bringing our own version of NtReadVirtualMemory is hard to get right, as our version has to work perfectly with the exact version of the underlying operating system. Thus, this solution is demanding to get right, if all major versions of Windows should be supported
This technique does not ensure that other API functions also benefit, as they will also have to be re-implemented in our own code to ensure that they use our own version of NtReadVirtualMemory such that they avoid calling the hooked function.
Do your own “loading”
Read the real ntdll.dll from disk, and use the correct bytes from the file dynamically.
Pros:
Ensures that the hooks are left in place, and the security product cannot detect any tampering with the hook, since no tampering with the hook takes place
Will work on most if not all versions of Windows, as the right implementation of NtReadVirtualMemory will be loaded and used correctly.
Simpler to use than the previously described method
Cons:
Security products can monitor when the ntdll.dll file is read, and detect on that. Thus, this technique is slightly less stealthy than the one previously described.
Go around the hook
Try to detect the hooks at runtime, and jump to the real function that follows the security products own logic e.g. the FireWalker technique: https://www.mdsec.co.uk/2020/08/firewalker-a-new-approach-to-generically-bypass-user-space-edr-hooking/
Pros:
Ensures that the hooks are left in place, and the security product cannot detect any tampering with the hook, since no tampering with the hook takes place
Will work on most if not all versions of Windows, as the right implementation of NtReadVirtualMemory will be loaded and used correctly.
Cons:
significant performance slowdown
complex to implement
Conclusion
Thus, user-mode API hooks can effectively be bypassed by a number of techniques. Applying hooks as a security mechanism in processes' own memory space is the equivalent of having security mechanisms in JavaScript executing in the web browser. both are examples of client-side validation, and should only be used for cosmetic reasons, not as a security mechanism itself.
Mitigations
Security products should definitely not rely on user-mode API hooks as a security mechanism, as the hooks will be exposed in the same security context as the processes they are intended to monitor, thus effectiely not providing any security as shown in this post. Instead, high-privileged security mechanisms should be used instead (or in addition), such as e.g. kernel callbacks (which require SYSTEM privileges to defeat, but that's another topic (https://github.com/br-sn/CheekyBlinder))