You can use javascript to open popup window.it can be a HTML or ASPX page.
In button client click event call the javascript to open the popup window.
here is the Sample code.
<script type="text/javascript">
function openPopup() {
window.open("Default2.aspx", "_blank", "WIDTH=1080,HEIGHT=790,scrollbars=no, menubar=no,resizable=yes,directories=no,location=no");
}
</script>
<asp:imagebutton id="btnpopup" runat="server" onclientclick="return openPopup()" />
If you want to call the Javascript function during your Server Side button click event
protected void Page_Init(object source, System.EventArgs e)
{
btnpopup.Attributes.Add("onclick", "return openPopup();");
}