You aren't using the parameter value you pass in so I suspect it is returning the average values for all TeamId's - it's possible you want this:
SELECT AVG(VoteValue) AS [AverageVoteValue]
FROM Vote
WHERE datediff(day,getdate(),LastModifiedDateTime) <= 7
AND TeamID = @TeamID
ORDER BY AverageVoteValue
But without your data and a better idea of exactly what you want to get, it's difficult to be sure.