Returning a value or throwing an exception

The other day my colleague was probing around whether to return a value or to throw an exception in general scenarios and I initially told him that throwing exceptions almost always incur performance costs as compared to a returned value. We had then left the discussion there itself but...