AV software make use many different systems, some of system hooks, rootkits (i.e. kaspersky) and AV API's form MS for office products.
If you want take a closer look check
clamwin that is an open source AV under GNU license.
About hooking there are some excellent articles here like
EasyHook.