Then you are doing it wrong!
This code:
string s1 = @"D:\Temp\MyLargeTextFile.txt";
string s2 = @"D:\Temp\mlt.txt";
Stopwatch s = new Stopwatch();
s.Start();
byte[] bytes = File.ReadAllBytes(s1);
for (int i = 0; i < bytes.Length; i++)
{
bytes[i] ^= 0x55;
}
File.WriteAllBytes(s2, bytes);
s.Stop();
Console.WriteLine(s.ElapsedMilliseconds);
takes 74 milliseconds, on my E6700 / 4GB ram, in the debugger, on a 8,651,780 byte text file.
So what are you doing that is so much slower?