Firstly, UTF-8 is an encoding, but "hexadecimal" and "decimal" aren't: they are numeric formats, which may or may not be contained in your file.
If you mean that your file contains groups of numbers in a text format:
1234,1A7F
5678,00FF
...
And you want to convert them into numeric values that you can manipulate mathematically, then you have to separate each number, and parse it.
For hexadecimal values:
string inp = "1A7F";
int x = Convert.ToInt32(inp, 16);
For decimal values:
string inp = "1234";
int x = int.Parse(inp);
Or
string inp = "1234";
int x;
if (int.TryParse(inp, out x))
{
...
}