best practice with long running task in IIS

Start by instantiating a ServiceHost class in a windows service. Then, on the service interface, define a method that would kick off your long running process. You can use the leaders-followers pattern to service requests using multiple threads. Alternatively, use a One-way method on your service interface so that the requestors don't wait for the windows service to spin up a thread or whatever it would do. Using a One-way method you must then queue requests on your service.

I suppose, the basic scenario is like following:
1) user initiates a the process, gives parameters - let' call it job
2) your application is starting to do the job, and creates the desired output
3) user is happy with he output :)
I also suppose, that your server does not have unlimited resources, thus a heavy load would inrease even more processing time measured during single request.

Now, in this situation I suggest the following approach:
1) Separate your user interface in two: job initiation and "inbox" for handing over finished jobs
2) Create a job model and a queue based on that model. This can be a simple database table, but something more complicated something for a complex job. Don't forget to link the job to the initiator user.
3) Create a windows service that looks for jobs in the queue, does the job and stores the result attached to the job in the queue.
4) The finished job is added to the inbox of the user. You can use ajax calls to refresh inbox on client side and/or send out email to notify user.
5) You have to choose the level of parallelism implemented in the windows service based on the job in general, the concrete job parameters and the load on the server.
6) You can even inform the user about the status of his jobs, by updating some sort of status or progress information on the job itself and the UI.

You could create and call a separate application from asp.net, by spawning a process - and thus escaping the limitations you know. But this way you could not schedule the jobs and the overall resource limits would impact performance. WCF can handle async calls, but you still have to handle asynchronicity on client side, and I think a windows service is more appropriate for such scenarios.