So you have a main dialog which creates another (modeless) dialog upon a button press (or hopefully activates the modeless dialog if it exists already).
In such cases your main dialog usually contains a member for the second dialog. You can use that member to access the other dialog and it's public members. For access to protected members you can add functions that allow general access (return a pointer to the member) or functions that perform specific actions.
If you refer to a message handler by "afx button function", just send the corresponding message to the window. If you want for example to simulate a button click in the modeless dialog all you need is the
CWnd
of that dialog and the ID of the button:
m_pModelessDialog->SendMessage(
WM_COMMAND,
MAKEWPARAM(ID_OF_BUTTON, BN_CLICKED),
m_pModelessDialog->GetDlgItem(ID_OF_BUTTON)->GetSafeHwnd()
);