Hi,
I like what you have done however personally I feel the mix of custom thread management plus mangeing TPL tasks etc very problematic in this instance. TPL has many great features that do not require all the custom management.
Whilst I have not tried executing tasks in order of thread priority I would like to mention a couple of ideas that may help.
Using ParallelOptions you can manage the concurrency of your tasks. You can also manage the priority and other thread aspects purely from the TPL wrappers.
a good link is below:
http://aviadezra.blogspot.co.uk/2009/10/how-many-threads-tpl-concurrency.html[
^]
Secondly, the idea of executing something synchronously if the max threads has been reached will not help too much. If the all cores are busy on jobs, your task is going to wait anyway until something is free. I am not sure why you would want to do this but then I might be missing something.
I would also look into the System.Collections.Concurrency stuff. Using the ConcurrentBag will be better for thread safety as this is all done for you. Also take a look at the Concurrent Stack and queue. These could help.
System.Collections.Concurrency[
^]
Lastly, I must recommend the BlockingCollection. Basically what you can do here is add tasks / actions / stuff to the collection. You can then enumerate over it in parrallel. It will be thread safe and only give you the next item in the collection. This can then be parsed off for processing. The beauty of sticking with TPL is the ability to safely cancel actions that are rather problematic if relying on threads. You can also create continuations that can handle faults and clean up.
http://msdn.microsoft.com/en-us/library/dd267312.aspx[
^]
So, whilst not a direct solution, I hope some of this is useful. Best of luck!