Click here to Skip to main content
15,867,835 members
Articles / Desktop Programming / MFC
Article

The singular non-modality of MFC modal dialogs

Rate me:
Please Sign up or sign in to vote.
4.92/5 (46 votes)
6 Apr 2003Ms-PL9 min read 275.9K   1.9K   52   58
Explains the pseudo modality of CDialog based modal dialogs and a problem with the CDialog::EndDialog implementation

Introduction

For the non-programmer, modal dialog boxes are those that refuse to go away till you dismiss them, which is usually done by clicking on the OK or Cancel buttons. For the programmer (the non-VB ones anyway), modal dialog boxes are those that disable their immediate parent window before they are created, and enable their parent window when they are closed. Thus the essence of a modal dialog is that you cannot do anything to the parent window till you close the modal dialog. Modeless dialogs are comparatively more polite and less fussy in that they do not insist on being dismissed to allow you to access their parent windows. I trust that by now, any infinitesimal doubts any of you might have had regarding the modality of a dialog has been absolutely eradicated.

MFC has a class called CDialog which is a CWnd derived class that is specifically used for the creation and display of dialog boxes on screen - both modal and modeless dialogs are supported. Modeless dialogs are created using Create() and you have to destroy them on your own using

DestroyWindow()
and they behave just like any normal modeless window. Modal dialogs are created and shown using the formidable DoModal() method of the CDialog class. And you close a modal dialog by calling EndDialog() or alternatively by calling OnOK() or OnCancel() both of which internally call EndDialog(). The CDialog::EndDialog method will call the EndDialog Win32 API function which is defined in user32.dll. EndDialog (the API function) will not immediately close the dialog, rather it will set a flag which will instruct the message queue to exit the loop, destroy the dialog window and enable the parent window. So far so good; everything seems so peaceful and serene, and the little birds in the tree across the yard are tweeting in a beautiful voice.

The non-modality factor

The Win32 API supports various dialog box related functions which include functions for creating both modeless and modal dialogs. There are the

CreateDialogXXX
set of functions like CreateDialog,
CreateDialogIndirect
etc. which are used to create modeless dialog boxes and the DialogBoxXXX set of functions like DialogBox, DialogBoxIndirect etc. which create modal dialogs. And as mentioned in the previous section there is also the EndDialog function which is used to end modal dialogs and only modal dialogs. Modeless dialogs must be terminated by calling DestroyWindow directly. The basic reason why you should not attempt to use EndDialog on a modeless dialog is that modeless dialogs do not have their own modal message loop nor do they disable their parent window, which basically nullifies the utility value of using EndDialog on them.

Alright, now this is what will give you a solid kick - MFC CDialog based modal dialogs are not modal dialogs. Just in case you did not read that right the first time, here it goes again - MFC CDialog based modal dialogs are not modal dialogs. I would have said it a third time but the fear of being termed a parrot thwarts me from doing so. Open dlgcore.cpp and examine the source code and you'll see that this astonishing statement is true. The MFC CDialog class uses the

CreateDialogIndirect
API function to create a pseudo-modal dialog and if you look up CreateDialogIndirect on MSDN you'll see that it is used to create a modeless dialog. The MFC command routing mechanism uses a combination of message maps and virtual functions to achieve what it does and a true modal dialog will totally wreck this mechanism because then the modal message loop is controlled outside the scope of the MFC command routing machinery. Thus the developers had no choice but to creat a pseudo-modal modeless dialog and then document it very intensely as a modal dialog.

Basically these are the summarized steps that are done when a pseudo-modal dialog is created via CDialog::DoModal -

  • A boolean flag bEnableParent is set to FALSE
  • If the parent window is enabled, then it is disabled and
    bEnableParent
    
    is set to TRUE
  • The dialog box is created using CreateDialogIndirect
  • A message pump is maintained using CWnd::RunModalLoop
  • Once the modal loop exits (when CDialog::EndDialog is called) the parent window is enabled if bEnableParent is TRUE
  • The dialog window is destroyed by calling DestroyWindow
  • And DoModal returns the argument passed to
    EndDialog

As you can see the developers have taken great pains to ensure that the pseudo-modal dialogs behave exactly as modal dialogs are supposed to. They have even tried to accommodate for unusual scenarios where there might be two modal dialogs both having the same parent - this is where the bEnableParent comes into play. Thus when the second modal dialog comes up, it will not set bEnableParent to TRUE because the parent window is already disabled. Thus when it is dismissed it will not enable the parent window which is exactly what is needed because another dialog modal to the same parent window is still active on screen.

EndDialog - a tiny flaw

As mentioned a couple of times already in this chapter, modal dialogs are dismissed using EndDialog. Let's see what EndDialog looks like (for those interested, it is defined in dlgcore.cpp)

void CDialog::EndDialog(int nResult)
{
    ASSERT(::IsWindow(m_hWnd));

    if (m_nFlags & (WF_MODALLOOP|WF_CONTINUEMODAL))
        EndModalLoop(nResult);

    ::EndDialog(m_hWnd, nResult);
}

CWnd::EndModalLoop is called to exit the message loop maintained by CWnd::RunModalLoop which is quite fine and then the EndDialog API function is called to terminate the modal dialog. This is also fine and Hey, hey, hey, just wait a cotton picking minute there!!!! The EndDialog API function is to be used only to close modal dialogs, but here we are attempting to use it to close a pseudo-modal dialog which is actually nothing but a modeless dialog disguised to reflect a modality it does not truly possess. [stunned silence...] Well, after that initial shock, let's all relax a bit. After all it's not the end of the world and EndDialog is not really a very harmful function call; it will only attempt to end a non-existent modal message loop and will enable the parent window of the dialog window. The former attempt will fail obviously and the latter attempt will just do something which we would have done anyway on our own, because the moment the pseudo-modal pump exits, DoModal will re-enable the parent window. So, is everything well and good, are the birds chirping peacefully once again?

The bug

Remember all that bEnableParent stuff we discussed a couple of paragraphs earlier, and how I mentioned how this flag is used to make sure that when multiple modal dialogs exist together that have the same parent window, this flag prevents the premature enabling of the parent window when one of the modal dialog siblings are dismissed? Well guess what? The careless programmer at Microsoft who coded this particular function had just rendered all those precautionary measures totally null and void. Because now any MFC modal dialog that is closed using EndDialog (which also means OnOK and OnCancel because those functions will internally call EndDialog) will enable it's parent window irrespective of what value bEnableParent holds. This basically means if you have two or more MFC modal dialogs which have the same parent window, then the moment you dismiss any one of those modal dialogs, all the other modal dialogs will lose their modality because now the parent window has been re-enabled.

Steps to reproduce this bug

  • Create an MFC dialog based application
  • Add a new dialog to it and associate a class with it called CChildDialog
  • Set two timers in the OnInitDialog method of the main dialog :-
  • SetTimer(1000,1000,NULL);	
    SetTimer(2000,2000,NULL);
  • Bring up a pseudo-modal dialog each, in the timer handler :-
  • void CModalDemoDlg::OnTimer(UINT nIDEvent)
    {
        KillTimer(nIDEvent);
        CChildDialog dlg(this);
        dlg.DoModal();
    
        CDialog::OnTimer(nIDEvent);
    }
  • Run the program and wait 2 seconds by which time you will have the main dialog and two modal child dialogs
  • Try to access the main dialog and you will be unable to do so because of the presence of the two modal dialogs
  • Now dismiss one of the child dialogs using the OK or the Cancel button
  • Now try to access the main dialog and you'll see to your astonishment that you will be able to do so, despite the presence of a modal dialog on screen

The project I have attached was created using VC++.NET 2003 Final Beta and I apologize to all those of you who do not have that version. But following the above mentioned steps shouldn't take you more than a few minutes at most. What is so amazing is that this bug has gone unnoticed through several versions of MFC. I checked as far back as VC++ 6 and that has the exact same problem too. As far as I see it, all someone's got to do is to comment out or delete the call to the EndDialog API call. I am guessing that what happened was this - the MFC developers were so much used to calling the native API equivalents from their wrapper functions (for example they would call the MessageBox API from inside CWnd::Messagebox) that someone must have automatically typed in that line without thinking and the QA guys must also have overlooked this error. And this problem remained mostly unknown because it was a very rare situation for a program to have a multi-modal window architecture where several modal dialog windows have the same parent window.

Work-around

Okay, so until Microsoft corrects this bug, what we could do (other than having to correct the MFC code and recompile MFC) is to override the OK and Cancel button handlers and to use this code instead of the default :-

{
    if (m_nFlags & (WF_MODALLOOP|WF_CONTINUEMODAL))
        EndModalLoop(IDOK); // or IDCANCEL
}

Notice how we are not calling the base class (if we do then ::EndDialog(...) gets called and all our efforts are wasted). If you want to exit the modal dialog in a place outside the OK/Cancel button handlers you should use the same code except that you might want to return a different value (like perhaps IDYES for example).

Conclusion

For all I know I might be the biggest jack-ass in town and there might be a perfectly legitimate reason for this matter, but something tells me that, that is a very remote contingency. By the way I'd like to thank Shog9 for sending me the VC++ 6 version of dlgcore.cpp at a rather late hour of the night (which to him is the equivalent of noon for most of us). I'd also like to explicitly mention here that this article is by no means an attempt to mock the amazing set of Microsoft programmers who developed the MFC library.

Version History

  • April 5 2003 - Article first published
  • April 7 2003 - Article updated with Work-around to the problem

License

This article, along with any associated source code and files, is licensed under The Microsoft Public License (Ms-PL)


Written By
United States United States
Nish Nishant is a technology enthusiast from Columbus, Ohio. He has over 20 years of software industry experience in various roles including Chief Technology Officer, Senior Solution Architect, Lead Software Architect, Principal Software Engineer, and Engineering/Architecture Team Leader. Nish is a 14-time recipient of the Microsoft Visual C++ MVP Award.

Nish authored C++/CLI in Action for Manning Publications in 2005, and co-authored Extending MFC Applications with the .NET Framework for Addison Wesley in 2003. In addition, he has over 140 published technology articles on CodeProject.com and another 250+ blog articles on his WordPress blog. Nish is experienced in technology leadership, solution architecture, software architecture, cloud development (AWS and Azure), REST services, software engineering best practices, CI/CD, mentoring, and directing all stages of software development.

Nish's Technology Blog : voidnish.wordpress.com

Comments and Discussions

 
AnswerSlight Correction for Modeless CDialog's [modified] Pin
jlddodger8530-Nov-09 11:32
jlddodger8530-Nov-09 11:32 
GeneralAccess violation if parent window is closed before MFC CDialog-based modal dialog is closed Pin
doroboy20-Mar-08 22:54
doroboy20-Mar-08 22:54 
GeneralRe: Access violation if parent window is closed before MFC CDialog-based modal dialog is closed Pin
jlddodger8530-Nov-09 11:34
jlddodger8530-Nov-09 11:34 
GeneralWindow or Dialog with Desktop as parent Pin
Motiram Patil22-Jun-07 3:00
Motiram Patil22-Jun-07 3:00 
Question"Enter" and "ESC" Keys could dismiss a modal dialog, anyway to prevent this? Pin
Sstar7-Aug-06 18:23
Sstar7-Aug-06 18:23 
AnswerRe: "Enter" and "ESC" Keys could dismiss a modal dialog, anyway to prevent this? Pin
lzinggl16-Oct-06 0:55
lzinggl16-Oct-06 0:55 
GeneralThank you !! Pin
Fastfootskater12-Aug-07 20:52
Fastfootskater12-Aug-07 20:52 
GeneralThanks Nish Pin
Paul Hooper12-Jul-06 1:52
Paul Hooper12-Jul-06 1:52 
GeneralC++ version suffers similar issue Pin
ThrashMaster13-Apr-06 10:56
ThrashMaster13-Apr-06 10:56 
GeneralProblem with modal dialog stack Pin
hickory3-Feb-06 4:59
hickory3-Feb-06 4:59 
GeneralRe: Problem with modal dialog stack Pin
hickory16-Feb-06 1:12
hickory16-Feb-06 1:12 
GeneralRe: Problem with modal dialog stack Pin
hickory16-Feb-06 1:18
hickory16-Feb-06 1:18 
GeneralRe: Problem with modal dialog stack Pin
hickory16-Feb-06 8:59
hickory16-Feb-06 8:59 
AnswerRe: Problem with modal dialog stack Pin
abc13a18-Sep-07 21:27
abc13a18-Sep-07 21:27 
GeneralMessage loops Pin
Stephen Hewitt11-Jan-06 12:36
Stephen Hewitt11-Jan-06 12:36 
QuestionHow to enable parent AND one child Pin
Sven Appenrodt23-May-05 23:34
Sven Appenrodt23-May-05 23:34 
GeneralVC6 also needs UpdateData() in OnOK() Pin
bruce2g1-May-05 21:26
bruce2g1-May-05 21:26 
GeneralMultiple Modal Dialogs Pin
r0nen18-Apr-05 8:12
r0nen18-Apr-05 8:12 
GeneralNon-modality of modal dialogs Pin
Yadnesh24-Nov-03 2:04
Yadnesh24-Nov-03 2:04 
QuestionQuestion "DoModal" ? Pin
Petrisor20-Nov-03 5:54
Petrisor20-Nov-03 5:54 
AnswerRe: Question "DoModal" ? Pin
David Crow25-Feb-05 6:15
David Crow25-Feb-05 6:15 
QuestionWhat about scenario with one modal dialog and two frame windows? Pin
domKing25-Oct-03 4:50
domKing25-Oct-03 4:50 
GeneralThe correct way is Pin
JimmyO17-Apr-03 3:47
JimmyO17-Apr-03 3:47 
GeneralRe: The correct way is [Actually no!] Pin
Nish Nishant17-Apr-03 5:47
sitebuilderNish Nishant17-Apr-03 5:47 
Questiony cant we access some of the parent's member variables? Pin
Miguel Lopes14-Apr-03 2:44
Miguel Lopes14-Apr-03 2:44 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.