Click here to Skip to main content
15,912,021 members
Please Sign up or sign in to vote.
1.00/5 (1 vote)
See more:
Hi,
I would like to know if it is technically possible to develop an automated internet search algorithm.
Let me give you a random example of what I have in mind.

For example, let’s say I am the largest disaster relief organization in the world and I want to run an automated google search every single day to see if there is any crisis/disaster happening anywhere around the world. The reason I want an automated search (not manual) because I may have many different key word combinations that I want to use (let’s say 1000 different key word combinations) and it would be hectic to do it manually. Let me give few key word examples that this disaster relief organization might want to use every day:

- “Disaster” + “need help urgently”
- “Massive disaster” + “no help”
- “People are suffering” + “earthquake”
- “Where is the UN”
- “We are trapped”
- “Africa” + “hunger” + “no food”
- Etc.

There might be hundreds of key word combinations that the disaster relief organization want to type every single day and they want an automated search algorithm to run all these key word combinations daily and then extract the key links in a page or table. Of course in Google you can also restrict the date – for e.g. ask Google to only give new results for the last day … since the organization will be running the automated algorithm each day and it only wants to see the new results.

The above was a random example for a disaster relief organization, but it can be for any agency or company.

My two most important questions are:
- Is this technically feasible to do?
- If it is possible, then who is the type of person that can do it? What is his “job title” or expertise or qualifications?

What I have tried:

General Google search about the topic
Posted
Updated 13-Dec-16 0:01am
Comments
F-ES Sitecore 12-Dec-16 5:31am    
It is technically possible, your issue is if it is legally possible. You'd have to study the T&Cs of the search engines you are hoping to use to ensure what you are doing complies with their terms. Search engines don't make money by you scraping their hard work to pass off as your own.

1. Yes it is technically feasible.
2. The person required to do it is a software developer.
 
Share this answer
 
Comments
[no name] 13-Dec-16 14:55pm    
Countered
Richard MacCutchan 14-Dec-16 3:47am    
:thumbsup:
Thanks Richard for your reply.

What about alogrithm developers, do they do this type of job?
In general, are the experts who can do this rare? Or is it simple and widely available?
It would be great if you could provide the name of companies or individuals that provide this service, and customers that implemented such service for their business.
 
Share this answer
 
Comments
Richard MacCutchan 14-Dec-16 3:49am    
Sorry but that is not a question for a technical forum. This site is for helping people with programming issues. Consultancy and marketing are not part of it.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900