Instead of fighting with Windows to get real time performance - move the problem outside the Windows host.
Get an Arduino Leonardo or low-cost derivative like the Beetle.
http://www.dfrobot.com/index.php?route=product/product&path=35_104&product_id=698[
^]
The Leonardo family plug into a Windows USB port and have a separate RS232 port. You would need to add an RS485 adapter to the Leonardo.
You can write custom code to upload into the Leonardo to speak the Modbus RTU protocol and maintain real time constraints. A plug-in for building Arduino projects from Visual Studio exists. The Windows host can speak to the Leonardo using higher level messaging and relaxed timing constraints. The USB interface emulates a serial port so your user mode Windows programming would be familiar territory.
The Beetle is very cheap.
http://www.dfrobot.com/index.php?route=product/product&path=35_104&product_id=1075[
^]
Good luck.