It seems that you've not fully specified which fields of DEVMODE are valid.
The value you set for flags is rather odd - 0x10 (corresponds to DM_SCALE)
You need to set a bit that indicates each of the valid fields of the structure.
Just change the line:
dm.dmFields = 16;
To
dm.dmFields = DM_BITSPERPEL | DM_PELSWIDTH | DM_PELSHEIGHT;
Oh! Also, do you mean to set a 600x800 screen (as shown in code) or the more likely 800x600? When I tried that, the bits/pixel changed, though the screen res didn't - I also received a success result. Guess the best thing to do is to check that the actual values are the expected ones after an attempt to change the screen settings..
#include <windows.h>
#include <stdio.h>
int main(int argc, char* argv[])
{
DEVMODE dm;
ZeroMemory(&dm, sizeof(dm));
dm.dmSize = sizeof(dm);
int index = 0;
if (0 != EnumDisplaySettings(NULL, ENUM_CURRENT_SETTINGS, &dm))
{
DWORD dwTemp = dm.dmPelsHeight;
dm.dmPelsWidth = 800;
dm.dmPelsHeight= 600;
dm.dmDisplayFlags = DM_DISPLAYFLAGS;
dm.dmBitsPerPel = 16;
dm.dmFields = DM_BITSPERPEL | DM_PELSWIDTH | DM_PELSHEIGHT;
if(DISP_CHANGE_SUCCESSFUL == ChangeDisplaySettings(&dm,0))
{
printf("\nSuccess\n");
}
else
{
printf("\nFailure\n");
}
}
return 0;
}