The discussion in comment was distracting from your question. You could call some vote "down-vote" if it adds a negative value to reputation score, and "up-vote" if this is positive. The votes of 3 are pretty rare (most vote either 1 or 5 :-)).
But, as far as I remember, 1-3 are down-votes, in this respect ([EDIT] by ProgramFOX maintains that vote of 3 adds zero, so may by I'm wrong [END EDIT]).
If you really wants to check it up using present-day settings, we can make a little non-destructive
experiment. :-) Alternatively, ask here: http://www.codeproject.com/Forums/1645/Site-Bugs-Suggestions.aspx
We performed a little experiment with Maciej. The result tells us that the vote of 3 does not affect the member's reputation score at all.
Additionally, we could see that, if, say, I voted 5 (my vote added +40), and then re-vote to 3, previously added reputation points are not removed. In effect, 3 only affects the score on a separate post, not affecting the member's score in any way.
We tested it only on the answer score, not on a question.