The overflow in lines:
dec(1) = (Not enc(1)) ^ Hex(89)
...
dec(i) = (enc(i - 1) + Hex("dc")) ^ enc(i)
...
dec(i) = enc(i) ^ dec(2)
because the C Xor operator '^' in VB means power of...
But I think there is also a very basic error: you consider your input stream as a string.
Using System.Text.Encoding.Default you make a conversion based on the current codepage, but if your stream is a
binary one you'll get wrong data (System.Text.Encoding masks MSB bit or alterate other way the values).
You should code your program more or less as:
Public Function DecodeBytePacket(Enc As Byte()) As Byte()
Dim Dec As Byte() = New Byte(Enc.Length) {}
Dec(0) = Enc(0)
Dec(1) = (Not Enc(1) Xor 137)
Dim i As Integer
i = 2
While i < CInt(Dec(0))
Dec(i) = Convert.ToByte(((Convert.ToUInt32(Enc(i - 1)) + 220) Xor Convert.ToUInt32(Enc(i))) And 255)
i = i + 1
End While
Dec(i) = (Enc(i) Xor Dec(2))
DecodeBytePacket = Dec
End Function
[EDIT] reviewed and checked the output of this routine is the same of the C version.