Of course it will.
At start your loop you have i = 0
i mod 3 = 0 - you open <ul>
then add an element
and closed <ul> tag inside same iteration.
for next iteration you have i = 1
i mod 3 = 1 and it neither opens nor closes.
possible solution may look like this:
For i = 0 to dt.rows.count - 1
If i mod 3 = 2 then
str_val +="</ul>"
End If
If i mod 3 = 0 then
str_val +="<ul>"
End If
str_val +="<li>Item One</li>"
If i mod 3 0 then
str_val +="</ul>"
End If
Next
if dt.rows.count > 0 then
str_val +="</ul>"
end if
By the way why are you using 3, if you want to have 4 items in a group you should use 4 (and 3 in the first check which closes the tag).