How to find a bottleneck

It's hard to answer based on this description.

If you've written a client that tries to behave like many clients, it really isn't a client. And if it's running on the same system as the server, it will obviously interfere with it.

With regard to an isolated server, what affects latency?

1. How long it takes the network layer (part of the platform and therefore outside your control) to react to an incoming packet.
2. How long the I/O thread that services sockets (I'm guessing TCP here) has to wait before it gets to run, given that it contends with other threads.
3. How many sockets have events. The I/O thread has to handle each one, so some will get handled earlier than others. The thread should turn each client's stream into a series of work items and queue them for a worker thread instead of processing them itself.
4. How long a work item sits on a work queue before it gets processed. The worker thread that handles it can send a response directly to the client.

That's about the best you can do. You can timestamp a work item when it gets created and queued, so that you can see how long it waited before a worker thread handled it, and you can gather statistics from this. You can also look at CPU usage to see if the server is getting overloaded (100% CPU usage). Heap usage is unlikely to be a factor except that the busier the server, the busier the garbage collector.

Maybe I've wasted time here, because I'm assuming you can modify the server. If it's some black-box code, then it should provide the kinds of statistics I'm talking about if it's something other than a toy demo.

Edit: Nothing here is really a "bottleneck" until the server is running at 100% CPU usage. At that point, it's a question of streamlining the code. Also, if work from clients can be classified as either "new" or "progress", then the server can implement an overload control policy that rejects (or just silently throws away) new work in order to maintain good service to clients that have already been accepted.