Whenever we did schema changes against our tables in my previous employ, we ran scripts to drop all indexes first. You might need to check to see if you have any constraints on the table also. This link might be useful for that
sqlservercentral[
^] (just realised you have to register on the site - I already have - I've found it useful in the past)
Useful article
from msdn blogs[
^] on performance during schema changes.
As to the difference in timings based on column types ... this could just be a feature of the order you did them in (int first, varchar 2nd) in which case caching and paging, disk free space, network activity etc etc could be having an effect. Use monitoring tools to find out what is actually happening