How to call modal popup from controller without making ajax call from client side in ASP.NET MVC.

You need to take time to understand the web application architecture. You can't call javascript code from your code-behind even though web forms made it look like that is what you were doing. In terms of MVC you don't have any interaction with the client via the controller, that is all done in the view.

So you'd have a property on your model like

public class TestModel
{
    public bool ShowDialog { get; set; }
}

In your controller you would set that to be true based on your relevant conditions

public ActionResult Test()
{
    TestModel model = new TestModel();

    if (someCondition)
    {
        model.ShowDialog = true;
    }

    return View(model);
}

Then in your view you would show the modal if that variable is true, so here I am using jQuery UI's "dialog" plug-in

<script type="text/javascript">
    function show() {
        $("#dialog").dialog();
    }
</script>

@if (Model.ShowDialog)
{
    <div id="dialog" title="Test dialog">
        <p>Hello world</p>
    </div>

    <script>show();</script>
}