Listening on multiple ports on a single server

This is almost certainly the wrong approach. Port numbers are a precious resource and if you have control over the protocol, it is much better to only listen on one, well known port and have the first message in the protocol be a switching message that chooses further details of future messages (in this case, 'I am type x/y/etc'). That message can be as simple as the client sending a byte (or short or int if you are thinking of more than 256 switchable types) as the first thing in the stream.

The type of connection should be a property of each connection model object, and once you've worked it out, it doesn't matter whether it was based on a port or a message (or other information, e.g. client IP, configuration settings, etc). So you shouldn't use something as scarce as port numbers to perform a menial switching task.

If (and only if) you do not control the protocol, you should do this simply by starting several server sockets and using the asynchronous socket I/O (e.g. call BeginAccept) on each one. But I do stress that if you have control over the architecture and the protocol, which it sounds like you do from your description, you should accept all communication through the same port and work everything else out from messages sent between client and server.