Without seeing sample data for all tables In the query I would look at the select statement as I believe that it is returning more than one result.
You might be able to change your query to this also (but this has'nt been tested!)
UPDATE a SET a.Inst_Id = s.Institute_Id
FROM Applicants a
INNER JOIN Students s ON a.App_Id = s.Stud_Id