stdout and stderr Listener issues (causing frozen processes?)

Question

0.00/5 (No votes)

See more:

Hi,

I have inherited an application that calls other apps (robocopy, rsh, etc...) by invoking processed that my application then listens to and logs the outputs.

The method that was in use to perform the listening was floored (and quite comical) and caused processes to freeze. There was no way to free them unless I kill the process which causes an entire pipeline to fail which results in nasty email being sent to me from annoyed stake-holders.

I changed the way the the processed were listened to by using stdout and stderr events instead of stream readers. This works fine 99% of the time.

The 1% I am looking at seem odd: I had DebugViewer running which crashed (forgot to limit the depth) and I now have 3 robocopys (robocopies?) that can't continue. I killed DebugView and the robocopys continue, for a few seconds. I can only think that they ran until the output buffers filled up, and now they are frozen again.

I have restarted DebugView (with depth set this time) but the Robocopys are still frozen.

Is it the case that, because my listener events aren't themselves listened to, the output buffers are not actually being read from? My impression was that these events read from the buffers, whether they themselves have listeners attached or not.

My question is two part: How does this happen and how can I prevent it?

Please let me know if you would like any more detail

Thanks in advance ^_^

Here is the base Command.Execute code that listens to the outputs:

C#

public virtual int Execute()
{
    // execute the process
    ProcessStartInfo processInfo = new ProcessStartInfo(ExecutableName, Options);
    Log.WriteLineTrack("Command.Execute: " + ExecutableName + " " + Options);

    OnNewEvent("Command.Execute: " + ExecutableName + " " + Options, 0, "");

    // Notes:
    // 1. I think CreateNoWindow has to be true to prevent the launched cmd file truing to access a desktop (and failing)
    // 2. the credentials must have access to all drives required to fulfill the supplied command line

    processInfo.RedirectStandardOutput = true;
    processInfo.RedirectStandardError = true;
    processInfo.CreateNoWindow = true;
    processInfo.UseShellExecute = false;
    processInfo.WorkingDirectory = WorkingDirectory;

    Process rcpProcess = new Process { StartInfo = processInfo };

    //StringBuilder strOut = new StringBuilder();
    StringBuilder strErr = new StringBuilder();

    string exeName = Path.GetFileNameWithoutExtension(ExecutableName);

    _processId = 0;
    rcpProcess.OutputDataReceived += delegate(object sender, DataReceivedEventArgs args)
                                         {
                                             if (!string.IsNullOrEmpty(args.Data))
                                             {
                                                 //strOut.AppendLine(args.Data);
                                                 OnStdOut(args.Data, _processId, exeName);
                                             }
                                         };
    rcpProcess.ErrorDataReceived += delegate(object sender, DataReceivedEventArgs args)
                                        {
                                            if (!string.IsNullOrEmpty(args.Data))
                                            {
                                                if (args.Data.Substring(0, 2).ToUpper().Equals("@."))
                                                {
                                                    OnNewEvent(args.Data, _processId, exeName);
                                                }
                                                else
                                                {
                                                    strErr.AppendLine(args.Data);
                                                    OnStdErr(args.Data, _processId, exeName);
                                                }
                                            }
                                        };
    rcpProcess.Start();

    _processId = rcpProcess.Id;

    rcpProcess.BeginOutputReadLine();
    rcpProcess.BeginErrorReadLine();

    if (_timeout != 0)
    {
        rcpProcess.WaitForExit(_timeout);
    }
    else
    {
        rcpProcess.WaitForExit(); //can take a millisecs tiumeout arg
    }

    int rc = rcpProcess.ExitCode;

    StringBuilder strbOutput = new StringBuilder();
    if (_logAgainstRun)
    {
        if (strErr.Length > 0)
        {
            strbOutput.AppendLine(string.Format("Executable {0} returned errors", ExecutableName));
            strbOutput.Append("<stdout>");
            strbOutput.Append(strErr);
            strbOutput.AppendLine("</stdout>");
            strbOutput.AppendLine(string.Format("Return code: {0}", rc));
        }
        else
        {
            strbOutput.AppendLine(string.Format("Executable {0} completed", ExecutableName));
            strbOutput.AppendLine(string.Format("Return code: {0}", rc));
        }
        Log.LogForRunOnly(_runId, _logfileName, strbOutput.ToString());
    }

    rcpProcess.Dispose();
    return rc;
}

PS: what happens if the parent process crashes is also a concern. Can I release the redirection of the outputs before crash?

Posted 11-Jun-13 0:37am

Andy Lanng

Updated 11-Jun-13 0:44am

v2

Add a Solution

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Alan N · Answer 1 · 2013-06-11T03:55:00

I've seen something similar in a small program I created to run several instances of SysInternals sdelete.exe and collect their standard output.

I'll explain my situation so that you can determine if it's relevant to yours.

Each Process instance was created in a separate thread and the relevant part of the code was this fairly standard stuff:

C#

Process p = new Process();
p.StartInfo = psi;
p.Start();
String result = p.StandardOutput.ReadToEnd();
p.WaitForExit();

Occasionally TaskManager would show that a child process had ended but the corresponding ReadToEnd would block until one of the other longer running child processes ended. Something odd was going on as obviously that could not happen! I could show that the problem was nothing to do with sdelete.exe and came to the conclusion that the Process.Start method is not thread safe. Specifically if the execution of the Start method on different threads overlaps there will be a coupling between the processes' redirected output streams.

The solution was simply to prevent overlapping calls to Process.Start with lock object shared between all the threads. With this modification in place the processes no longer exhibited the strange coupling behaviour.

C#

Process p = new Process();
p.StartInfo = psi;
lock (startLock) {
  p.Start();
}
String result = p.StandardOutput.ReadToEnd();
p.WaitForExit();

Let me know if that's of any help.

Some other comments on your code, not that you asked, but it's useful to know stuff.

The output streams may deliver data after WaitForExit has returned making it unsafe to assume that you have all the data at that point. It also makes it very difficult to know when to call the Process.Dispose method. The solution is not to proceed until both output streams have returned null, the indicator that they have closed, i.e. DataReceivedEventArgs.Data == null.

Incidentally your DataReceived event handlers are discarding blank lines. This may be what you want but do be aware that args.Data == String.Empty is perfectly normal and means a blank line was written to the output stream. On the other hand args.Data == null occurs once only when the stream has closed.

Alan.