Download a CSV file with unknown location from a URL

Question

0.00/5 (No votes)

See more:

Hi, would someone be able to explain me how I could download in VB or C# the CSV file from the following URL? As you can see, the URL doesn't contain the file name that I can download when I access the URL.

https://www.nasdaq.com/api/v1/historical/AMZN/stocks/2020-07-20/2020-07-22

When I access that URL I get the option to save the CSV in my PC but I need to do it in code, automatically, because I need to get several files with different dates.

Many thanks in advance!

What I have tried:

I have been researching the issue for sometime now and all I could find are solutions to do what I want when the file name is contained as part of the URL, but as you can see above, in my case the file name is not contained in the URL.

Posted 23-Jul-20 5:40am

migallo

Updated 24-Jul-20 6:37am

Add a Solution

3 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Sandeep Mewara · Answer 1 · 2020-07-23T06:03:00

Solution 1

If you use any network tool (easiest would be to use browser Developer Tools option), you can see that the request made for the url and the response received.

Response is of type: application/csv
Response Payload along with it.

This payload can be saved as the expected response type (CSV) as a simple file save (stream) on disk.

Posted 23-Jul-20 6:03am

Sandeep Mewara

Comments

migallo 24-Jul-20 10:07am

Thank you so much for this Sandeep, my coding skills are very limited. The issue I get is that when I try to get the response, it times out. The call below GetResponse() returns this exception: "System.Net.WebException: 'An error occurred while sending the request. Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond..' "

Could you please point me in the right direction?

Sub Test()
Dim req As HttpWebRequest = HttpWebRequest.Create("https://www.nasdaq.com/api/v1/historical/AMZN/stocks/2020-07-20/2020-07-22")
Dim res As HttpWebResponse = req.GetResponse()
'save req.GetResponse.GetResponseStream() to disk
res.Close()
End Sub

Thanks a lot!!

Sandeep Mewara 24-Jul-20 10:09am

using (Stream file = File.Create("quotes.csv"))
{
request = (HttpWebRequest)WebRequest.CreateDefault(new Uri(url));
request.Timeout = 30000;
var response = (HttpWebResponse)request.GetResponse();
using (Stream input = response.GetResponseStream())
{
CopyStream(input, file);
}
}

migallo 24-Jul-20 10:32am

Hi, still no luck... same issue. The GetResponse() call keeps returning the same Exception. I dont think it is a timeout issue... if you open the URL in a new page the response is just the file, no page at all is returned. I think it might have to do with it. The exact code that I am running is below:

static void Main(string[] args)
{

using (Stream file = File.Create("quotesAMZN.csv"))
{
var request = (HttpWebRequest)WebRequest.CreateDefault(new Uri("https://www.nasdaq.com/api/v1/historical/AMZN/stocks/2020-07-20/2020-07-22"));
request.Timeout = 30000;
var response = (HttpWebResponse)request.GetResponse();
using (Stream input = response.GetResponseStream())
{
input.CopyTo(file);
}
}
Console.WriteLine("Hello World!");
}

System.Net.WebException: 'An error occurred while sending the request. Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond..'

Sandeep Mewara 24-Jul-20 10:52am

Set this before the Webrequest call and see:

ServicePointManager.Expect100Continue = true;
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls
| SecurityProtocolType.Tls11
| SecurityProtocolType.Tls12
| SecurityProtocolType.Ssl3;

migallo 24-Jul-20 11:13am

Hi Sandeep, same exception as before... :( "Ssl3" is obsolete apparently and doesnt compile but I included "Tls13" instead which was available as an option.

The full code that I am running is below:

static void Main(string[] args)
{
ServicePointManager.Expect100Continue = true;
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12 | SecurityProtocolType.Tls13;
using (Stream file = File.Create("quotesAMZN.csv"))
{
var request = (HttpWebRequest)WebRequest.CreateDefault(new Uri("https://www.nasdaq.com/api/v1/historical/AMZN/stocks/2020-07-20/2020-07-22"));
request.Timeout = 30000;
var response = (HttpWebResponse)request.GetResponse();
using (Stream input = response.GetResponseStream())
{
input.CopyTo(file);
}
}
Console.WriteLine("Completed!");
}

migallo 24-Jul-20 12:35pm

Hi Sandeep, I managed to find the solution in another forum. The lines below made the trick! :)

request.AllowWriteStreamBuffering = false;
request.Accept = "*/*";

Sandeep Mewara 24-Jul-20 12:44pm

Great to know your query is solved!

migallo 24-Jul-20 13:04pm

Thanks a lot for your help!!

Richard MacCutchan · Answer 2 · 2020-07-23T06:04:00

Solution 2

The obvious place to look would be Nasdaq Data on Demand API Reference[^].

Posted 23-Jul-20 6:04am

Richard MacCutchan

v2

Comments

migallo 24-Jul-20 10:09am

Thank you for this Richard. I already spent some time reading the reference before asking the question but couldn't find anything useful to solve the problem I am facing.

migallo · Answer 3 · 2020-07-24T06:37:00

static void Main(string[] args)
{
using (Stream file = File.Create("C:\\Temp\\quotes.csv"))
{
var request = (HttpWebRequest)WebRequest.CreateDefault(new Uri("https://www.nasdaq.com/api/v1/historical/AMZN/stocks/2020-07-20/2020-07-22"));
request.Timeout = 30000;
request.AllowWriteStreamBuffering = false;
request.Accept = "*/*";

var response = (HttpWebResponse)request.GetResponse();
Stream input = response.GetResponseStream();
input.CopyTo(file);

}
Console.WriteLine("Completed!");
}