You cannot delete the button from form two. You can only do it in Form1's designer view.
However, you can do something like this in the Form2's constructor.
this.btnExit.Visible = false;
P.S.:
By the way, why do you need to do this? If your Form1 contains items not necessary for it's children, there must be a problem with design. Parent classes should only have things that are common to all its children.