Keep in mind that the web is stateless, your popup runs on the client/browser. Invoking a Button will causes a page to do a server postback, thus recreating the page. In other words, your popup will disappear when you trigger a server postback. Bottom line is you can't do server and client operation at the same time.
Now here are your options:
1) You could use your existing popup function and remove the OnClick server event. Then you could write a JavaScript/jQuery function that will do an AJAX requrest to the server to call a server-side method.
2) Get rid of your client-side popup, and instead use a server-side AJAX Dialog, for example the ModalPopupControl from AJAXControlToolkit. That way you can still invoke your OnClick server event and call the ModalPopUpControl from your server code.
3)You can probably do a hack, and that is to add a hiddenfield to mark a page if it postbacks, and decide if you show the popup. Here's a quick example:
<head runat="server">
<title></title>
<link rel="stylesheet" href="http://code.jquery.com/ui/1.10.3/themes/smoothness/jquery-ui.css" />
<script src="http://code.jquery.com/jquery-1.9.1.js"></script>
<script src="http://code.jquery.com/ui/1.10.3/jquery-ui.js"></script>
<script>
$(function () {
$("#dialog").on("dialogbeforeclose", function (event, ui) {
$("#HiddenField1").val("false");
});
if ($("#HiddenField1").val() == "true") {
$("#dialog").dialog({ draggable: false });
}
});
function ShowDialog() {
$("#dialog").dialog({ draggable: false });
$("#HiddenField1").val("true");
}
</script>
</head>
<body>
<form id="form1" runat="server">
<asp:HiddenField ID="HiddenField1" runat="server" Value="false" />
<div id="dialog" title="Basic dialog" style="display:none;" >
<asp:Button ID="Button1" runat="server" Text="I can post back" UseSubmitBehavior="false" OnClick="Button1_Click" />
</div>
<input id="Button2" type="button" value="Show dialog" onclick="ShowDialog()" />
</form>
</body>