The best solution is: don't use a varchar field and try and do any maths with it, it only leads to pain. Always use a numeric field for numeric data!
You can do it - SQL has a ISNUMERIC test - but what are you going to do with fields like 28:45? Ignore them? Treat them as matched? Unmatched?
And the ISNUMERIC returns 1 for 123.4 so you still have to play silly buggers even then:
SELECT * FROM tblSalaryRecordCompliance WHERE SalarySTDT ='1-Mar-2014' AND ISNUMERIC(OTHours) AND CAST(CAST(OTHours AS FLOAT) AS INT)>52
But you would be much, much better off using numeric fields!