ReadLine on StreamReader with ftpWebRequest response stream throws ObjectDisposedException

Question

0.00/5 (No votes)

See more:

Hello!

I am trying to read file from FTP line by line:

C#

static void Main(string[] args)
{
var uri = new Uri( ....... );
var networkCredential = new NetworkCredential( ......... );
 
var ftpWebRequest = (FtpWebRequest)WebRequest.Create(uri);
ftpWebRequest.Method = WebRequestMethods.Ftp.DownloadFile;
ftpWebRequest.KeepAlive = true;
ftpWebRequest.Timeout = System.Threading.Timeout.Infinite;
 
ftpWebRequest.Credentials = networkCredential;
 
FtpWebResponse ftpWebResponse = null;
 
ftpWebResponse = (FtpWebResponse)ftpWebRequest.GetResponse();
 
using (var responseStream = ftpWebResponse.GetResponseStream())
using (var streamReader = new StreamReader(responseStream, Encoding.Default))
{
    var counter = 0;
    while (true)
    {
        if (streamReader.ReadLine() == null) break;
        counter++;
    }
}
}

Occasionally ReadLine thorws exception:

System.ObjectDisposedException was unhandled
  Message=Cannot access a disposed object.
Object name: 'System.Net.Sockets.NetworkStream'.
  Source=System
  ObjectName=System.Net.Sockets.NetworkStream
  StackTrace:
       at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
       at System.Net.FtpDataStream.Read(Byte[] buffer, Int32 offset, Int32 size)
       at System.IO.StreamReader.ReadBuffer()
       at System.IO.StreamReader.ReadLine()
       at Training.Program.Main(String[] args) .....
       at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
       at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
       at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
       at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading.ThreadHelper.ThreadStart()
  InnerException:

I don't understand that is wrong with my code. How to correctly read file from FTP line by line?

Posted 13-May-13 3:26am

Eugene Efimov

Add a Solution

Comments

Sergey Alexandrovich Kryukov 13-May-13 10:45am

Why doing so at all? And what if the file has no lines at all? Reading lines makes little to no sense, I would say...
—SA

Eugene Efimov 15-May-13 2:58am

I need to analyze text log located on a remote server accessible only via FTP. Text log structure requires analyzing it line by line. Is there a better way?

Sergey Alexandrovich Kryukov 15-May-13 10:45am

Why not transferring the whole file (log file, as I understand)? This is what FTP designed for...
—SA

Eugene Efimov 16-May-13 4:42am

I do transferring the whole file, do i not?

Sergey Alexandrovich Kryukov 16-May-13 9:50am

I'm not sure...
—SA

Eugene Efimov 17-May-13 4:36am

> while (true)
and
> if (streamReader.ReadLine() == null) break;
Will assure reading stream until the end (and until the end of underlying stream). In fact, that is what happens in my case - program reads file until the end, but then instead of returning null in ReadLine() it throws exception.

Dave Kreskowiak 17-May-13 9:57am

No, you're not. SAK is correct though, it would be better to copy the file locally, then anaylze it. At least then, you're getting a "snapshot" of the file and don't have to worry about it's contents changing as you're reading it for your analysis.

Eugene Efimov 21-May-13 7:34am

Could you be more specific about why not the whole file is read? Point out some code, cuz I expect ReadLine method in the loop to read all data.

And how can I get snapshot? The file can be modified during download process. I just can't get why should one download file content to memory, then flush it on disk and then again load to memory.

It may be a workaround for this particular task, but the true reason of the problem is still hidden.

Dave Kreskowiak 21-May-13 8:32am

You copy the file locally to a temp folder, not read the entire thing into memory. You know, File.Copy...

Then you can read the local copy of the file, line by line, not read the entire file into memory, and do your anaylsis.

Why not read the entire file into memory?? Because what if the file is a couple of GB is size?? It won't fit in memory...

Eugene Efimov 21-May-13 9:54am

I did not mean storing all data in memory at once. I get your point about temporary storing data in file.

But we already have stream to read from! I see no real reason to write that stream to file, and then create another stream (only now its from file not network). We can read and process data as it comes, on the fly! The problem is that I may be doing I am doing it wrong, but let me emphasize again - on the fly processing is the right way to do it. Using temporary storage must be justified with more than "This way it works and we have no idea about what is wrong with the previous code".

Dave Kreskowiak 21-May-13 10:48am

YES, you already have an UNRELIABLE stream to read from. That's why you download the file first locally then do the analysis.

Eugene Efimov 22-May-13 9:13am

If you claim the stream is "UNRELIABLE", how can you rely on it to download file? What's the difference between downloading file to local file or to memory for on-the-fly analysis?

Dave Kreskowiak 22-May-13 11:39am

DID YOU TRY IT??? If you're just going to shoot down everything everyone says, you're going to be stuck with this problem for a very long time.

It MIGHT just give you a clue as to what's going on! Ah, the wonders of troubleshooting.

Have a nice life...

2 solutions

Solution 1

So far, I could not explain what you describe, but… Why not using ftpWebRequest.DownloadData(/*...*/), as recommended in the second code sample on the MSDN help page: http://msdn.microsoft.com/en-us/library/system.net.ftpwebrequest.aspx[^]?

By the way, you did not really show the working code sample. The code you show won't compile: the curly brackets of first "using" statement are not balanced. Do you through out the non-null result of streamReader.ReadLine()?

—SA

Posted 17-May-13 3:42am

Sergey Alexandrovich Kryukov

Updated 21-May-13 3:51am

v2

Comments

Eugene Efimov 21-May-13 9:43am

Please, check again. All brackets are balanced and code complies fine. The first 'using' statement has no brackets at all, see nested using example here: http://stackoverflow.com/questions/1329739/nested-using-statements-in-c-sharp

Downloading all data at once to a single byte array could be painful for big files, besides question clearly states that line by line processing required, so no need to store whole data at once.

P.S. Current example does not contain part of processing streamReader.ReadLine() non-null result just to make example smaller.

Sergey Alexandrovich Kryukov 21-May-13 9:56am

I understand about the big files, but the downloading problem by copying the file to a local disk as you real. But this is done in universal way, be reading block-by-block of some constant size (except the last one). Doing it line-by-line is somewhat unpredictable, because you don't know the line size before you read, so it's generally problematic and bad for performance. I would not do it at all.

If your files are logically line-based (like messages), and the files are big, you should rather review your architecture.

—SA

Eugene Efimov 21-May-13 10:06am

>Doing it line-by-line is somewhat unpredictable, because you don't know the line size before you read, so it's generally problematic and bad for performance. I would not do it at all.
Why is this unpredictable? We need to receive a chunk of data, write it to buffer, scan buffer for newline separators, extract fully-received lines, yield them and remove them from buffer.

In fact, that is that StreamReader does as far I can see. The code above works fine in most cases, and it actually gets all content. Except for throwing exceptions some times in the end of file instead of returning null.

My current workaround is just to ignore ObjectDisposedException if it occurs in ReadLine(). And it works fine, processing date on the fly with unnecessary caching data on local drive. I'm just curious about the reasons of occasionally occuring ObjectDisposedException exceptions.

Sergey Alexandrovich Kryukov 21-May-13 10:31am

I already explained why, but generally what you describe is a working scenario, I just don't see a whole lot of sense... Bad workaround, anyway. You need to get to the root of the problem. But it's much better not to work with lines.
—SA

Eugene Efimov 22-May-13 9:17am

> but generally what you describe is a working scenario
Do you have experience working with sockets? Cuz if so, you know there is no way to know the size of incoming packages. And the scenario I described not just "generally working", but a valid and preferable approach (instead of fixing package size and read fix length pachages, what you suggest for in analogy to block-devices).

Sergey Alexandrovich Kryukov 22-May-13 12:03pm

Yes I do. If it's working, why do you have a problem in this very? Well, if I knew it right now, I would post a solution... sorry... I seriously advise you to review the design...
—SA

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Cyrus Neah · Accepted Answer · 2013-05-21T04:46:00

The problem is with FtpDataStream.Read(Byte[] buffer, Int32 offset, Int32 size) method. FtpDataStream closes when it is unable to read and gets disposed. You need to create your own custom FtpDataStream and override the Read method and catch ObjectDisposedException error (return 0) to get you out.The uncaught ObjectDisposedException is bubbling up.

Streamreader has a method called ReadBuffer which calls the FtpDataStream.Read(Byte[] buffer, Int32 offset, Int32 size) method. Since FtpDataStream closes and gets disposed , ReadBuffer never gets to be zero which it needs to be for Streamreader.readline to return null

Declare a custom stream class

C#

public class custStream : Stream
{
    internal Stream _stream;

    public custStream(Stream stream)
    {
        _stream = stream;
    }
    public override void WriteByte(byte value)
    {
        _stream.WriteByte(value);
    }

    public override int ReadByte()
    {

        try
        {
            return _stream.ReadByte();
        }
        catch (ObjectDisposedException ode)
        {
            return 0; //
        }
    }

    public override long Seek(long offset, SeekOrigin origin)
    {
        return _stream.Seek(offset, origin);
    }

    public override void Flush()
    {
        _stream.Flush();
    }
    public override long Position
    {
        get
        {
            return _stream.Position;
        }
        set
        {
            _stream.Position = value;
        }
    }

    public override void Write(byte[] buffer, int offset, int count)
    {
        _stream.Write(buffer, offset, count);
    }

    //this is the one we are really interested in
    public override int Read(byte[] buffer, int offset, int count)
    {
        try
        {
            return _stream.Read(buffer, offset, count);
        }
        catch (ObjectDisposedException ode)
        {
            return 0; //
        }
    }

    public override void SetLength(long value)
    {
        _stream.SetLength(value);
    }



    public override long Length { get { return _stream.Length; } }

    public override bool CanWrite { get { return _stream.CanWrite; } }

    public override bool CanSeek { get { return _stream.CanSeek; } }

    public override bool CanRead { get { return _stream.CanRead; } }

}

In your code change ftpWebResponse.GetResponseStream() to

VB

using (var responseStream = custStream(ftpWebResponse.GetResponseStream()))