Though I don't know whether or not it is the problem you're looking for, I see a problem in your code:
1) (no showstopper but unnecessary)
DataTable dt = new DataTable();
You're assigning an instance of DataTable here, but you don't use it afterwards. In the very next line, it gets overwritten by
dt = (DataTable)ViewState["Remarks"];
You can shorten that to
DataTable dt = (DataTable)ViewState["Remarks"];
2) (Maybe a real problem)
You're Iterating by row number from zero upwards. An you're eventually deleting a row based on ist number. And you're iterating further up from there.
Here you have to keep in mind that the row you deleted is in fact gone. Its row number is assigned to the next larger row. And so on. All following rows wander one row number down.
Since your iteration variable doesn't respond to that, you may be missing a row to delete.
So, if you know that you'll delete one row at most, break the Loop after deleting:
dt.Rows.RemoveAt(irows);
break;
If you don't know how many rows will be deleted, revert the loop's direction:
for (int irows = dt.Rows.Count - 1; irows >= 0; irows--)
I don't understand the rest, so no tipps for that.