How do I replace the default right click context menu in a file open dialog box with my own?

I don't have a specific solution but you are on the right track. Ages ago, I made a derived FileOpenDialog class or what ever MFC calls its version and made a custom dialog that included a bitmap preview. That has since been render obsolete but that is what I did for it - added a custom lpfnHook procedure and then I made made message handlers in it to handle things for the previewer. That's one way you can go with it.

I think the recommend method today is to add a shell extension. The open file dialog is calling into the Shell API so it uses the same stuff the Explorer shell does. There are articles here on this topic. I suspect that is it a little more generic than you want go so you have both options. I think.

Yet another option is to make your own fully custom file open dialog and I have done this too. There is a DlgDirList family of functions to help with that. This will give you full control over the right click menu.

Best of luck.