The reason it complains it that the bitwise operators are defined for integer and bool only, not byte, so they all get promoted first. Just cast them back, and you'll be fine.
byte b1 = (byte) 0xF0;
byte b2 = 0x0F;
byte b3 = (byte)(b2 << 4);
byte b4 = (byte) (b1 ^ b3);
MessageBox.Show(string.Format("{0}, {1}, {2}, {3}", b1, b2, b3, b4));
^ XORs them together, + adds, | is OR and & is AND.