403 forbidden error when loading url into xdocument

Question

0.00/5 (No votes)

See more:

The following code to download an RSS file was working perfectly for weeks in an application I'm putting together:

public class RssFeedReader
{
public IEnumerable<post>; ReadFeed(string url)
{
var rssFeed = XDocument.Load(url);

var posts = from item in rssFeed.Descendants(item)
select new Post
{
Title = item.Element(title).Value,
Description = item.Element(description).Value,
// PublishedDate = item.Element(.Value};

return posts;
}
}
url in the test case I'm using is
http://www.news.com.au/national/rss

The response I'm getting is:
"Exception thrown: System.Net.WebException in System.dll "
and

"Additional information: The remote server returned an error: (403) Forbidden."

System.Net.WebException occurred
HResult=-2146233079
Message=The remote server returned an error: (403) Forbidden.
Source=System
StackTrace: at System.Net.HttpWebRequest.GetResponse()
InnerException:

As mentioned this was all working so smoothly up until 3 days ago then stopped working for no apparent reason.. I've looked all over the place for an answer and tried various things like loading directly into an XDocument etc - but it still gives the same or similar error.

I took that code snippet, made no changes to it and wrote a small program b that calls it. I used exactly the same url - and it works fine!

So could it be something is configured incorrectly in the main program?

Any help very much appreciated!

What I have tried:

Amongst other things, I put this block in front of 'var rssFeed' and ran it:

WebClient webClient = new WebClient();
webClient.Headers.Add( user-agent , MyRSSReader/1.0 ;);

XmlReader readers = XmlReader.Create(webClient.OpenRead(url));
var xmlDocument = new XmlDocument();
xmlDocument.Load(readers);

and got exactly the same 403 error as it tried to run "xmlDocument.Load(readers); "

I read some suggestions on CodeProject about the 403 error and tried using https instead of just http: - it failed for a diffeet (but understandable) reason:

Exception thrown: System.Security.Authentication.AuthenticationException in System.dll

Additional information: The remote certificate is invalid according to the validation procedure.

Posted 20-Nov-16 16:21pm

Herilane

Updated 21-Nov-16 8:45am

Add a Solution

2 solutions

Solution 1

sometimes due to update on server end, permission changes for internal or new folder, try to give permission again to whole web folder to following users

=> network service
=> IIS_IUSRS

Plz try to give permissions to above users

Posted 21-Nov-16 6:30am

Mohtshm Zubair

Updated 21-Nov-16 22:36pm

v3

Comments

Herilane 21-Nov-16 18:53pm

Hi - I don't understand how the problem is at the Server end. It could be - I just don't see how at the moment. The biggest puzzle for me is that the code used to work fine for weeks and then suddenly the RSS feed stopped. I've changed the URL to several others - all with the same negative result. And works fine if I create a test project and call it inside there. By the way, this is a WPF application I'm writing, not asp.net.

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Richard Deeming · Accepted Answer · 2016-11-21T08:45:00

That site is filtering out requests which don't contain a user agent header. You'll need to replace the var rssFeed = XDocument.Load(url); call with a WebRequest call:

C#

static XDocument LoadRss(string url)
{
    var request = (HttpWebRequest)WebRequest.Create(url);
    request.UserAgent = "(Anything other than an empty string)";
    
    using (var response = request.GetResponse())
    using (var stream = response.GetResponseStream())
    {
        return XDocument.Load(stream);
    }
}

public IEnumerable<Post> ReadFeed(string url)
{
    var rssFeed = LoadRss(url);
    ...