How to calculate a the files md5 in c++?

There's an easier alternative than spinning up the full crypto; have a look at the Message Digest functions[^]

typedef struct {
    ULONG i[2];
    ULONG buf[4];
    unsigned char in[64];
    unsigned char digest[16];
} MD5_CTX;

typedef void (_stdcall *MD5Final)(MD5_CTX* context);
typedef void (_stdcall *MD5Init)(MD5_CTX* context);
typedef void (_stdcall *MD5Update)(MD5_CTX* context,unsigned char* input,unsigned int inlen);

Do a LoadLibrary("cryptdll.dll"), GetProcAddress() for each of the three functions, call MD5Init(), MD5Update() as many times required, then MD5Final(), and FreeLibrary()