How to determine if the fetched URL has results

Question

1.00/5 (2 votes)

See more:

Hi everyone,

I'm building a project in which the user will fill in my website the search keyword and I will check in hundreds of websites if there is a result on the fetched websites.

my problem is that some websites that not have an API will not return 404 when there are no results and the response will look like 200 ok but there will not be any results.

I will be really glad to learn what is the best way to do it.

Thanks!

What I have tried:

Hi everyone,

I'm building a project in which the user will fill in my website the search keyword and I will check in hundreds of websites if there is a result on the fetched websites.

my problem is that some websites that not have an API will not return 404 when there are no results and the response will look like 200 ok but there will not be any results.

I will be really glad to learn what is the best way to do it.

Thanks!

Posted 4-Oct-20 22:57pm

Learning Learning

Updated 4-Oct-20 23:04pm

Add a Solution

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

F-ES Sitecore · Answer 1 · 2020-10-04T23:05:00

Solution 1

You'll have to analyse the response you do get back on a per-site basis to check for "No results found" or whatever that site does to indicate no results. Also a site shouldn't return 404 if there are no search results as the 404 indicates the URI you requested couldn't be found and a search result page that shows results still exists.

Posted 4-Oct-20 23:05pm

F-ES Sitecore

Comments

Learning Learning 5-Oct-20 5:07am

thanks! so I will need to scrape every website?

P.S - the 404 refers to an API's where there a 404 when no results.

Dave Kreskowiak 5-Oct-20 10:48am

That depends on each site. 404 means the requested URL couldn't be found, not that there are no search results.

There is a different between calling a Web API and a web page URL, like www.google.com. If it's a normal web page, yes, you'll have to write scrapper code for each and every one of your "hundreds" of sites. Hell, you'd have to do it for each and every site that's an API too! Good luck with that!