Java

A Web Crawler code...help

27-Nov-09 22:03

This does not appear to be a Java question. Try the Struts or Pentaho forums instead.

Write XML Attribute

toby3127-Nov-09 13:02

toby31

27-Nov-09 13:02

Does anyone know what am i supposed to do to write xml attribute in java?

example

<body id="Content">Text</body>

sangeeta200927-Nov-09 2:23

27-Nov-09 2:23

i have got a code for Web Crawler in java language....
Following is applet code...

import java.applet.Applet;
import java.text.*;
import java.awt.*;
import java.awt.List;
import java.awt.event.*;
import java.util.*;
import java.util.*;
import java.net.*;
import java.io.*;

/* <applet code=WebCrawler height=400 width=400>
</applet>
*/



public class WebCrawler extends Applet implements ActionListener, Runnable {
    public static final String SEARCH = "Search";
    public static final String STOP = "Stop";
    public static final String DISALLOW = "Disallow:";
    public static final int    SEARCH_LIMIT = 50;

    Panel   panelMain;
    List    listMatches;
    Label   labelStatus;

    // URLs to be searched
    Vector vectorToSearch;
    // URLs already searched
    Vector vectorSearched;
    // URLs which match
    Vector vectorMatches;

    Thread searchThread;

    TextField textURL;
    Choice    choiceType;

    public void init() {

	// set up the main UI panel
	panelMain = new Panel();
	panelMain.setLayout(new BorderLayout(5, 5));

	// text entry components
	Panel panelEntry = new Panel();
	panelEntry.setLayout(new BorderLayout(5, 5));

	Panel panelURL = new Panel();
	panelURL.setLayout(new FlowLayout(FlowLayout.LEFT, 5, 5));
	Label labelURL = new Label("Starting URL: ", Label.RIGHT);
	panelURL.add(labelURL);
	textURL = new TextField("", 40);
	panelURL.add(textURL);
	panelEntry.add("North", panelURL);

	Panel panelType = new Panel();
	panelType.setLayout(new FlowLayout(FlowLayout.LEFT, 5, 5));
	Label labelType = new Label("Content type: ", Label.RIGHT);
	panelType.add(labelType);
	choiceType = new Choice();
	choiceType.addItem("text/html");
	choiceType.addItem("audio/basic");
	choiceType.addItem("audio/au");
	choiceType.addItem("audio/aiff");
	choiceType.addItem("audio/wav");
	choiceType.addItem("video/mpeg");
	choiceType.addItem("video/x-avi");
	panelType.add(choiceType);
	panelEntry.add("South", panelType);

	panelMain.add("North", panelEntry);

	// list of result URLs
	Panel panelListButtons = new Panel();
	panelListButtons.setLayout(new BorderLayout(5, 5));

	Panel panelList = new Panel();
	panelList.setLayout(new BorderLayout(5, 5));
	Label labelResults = new Label("Search results");
	panelList.add("North", labelResults);
	Panel panelListCurrent = new Panel();
	panelListCurrent.setLayout(new BorderLayout(5, 5));
	listMatches = new List(10);
	panelListCurrent.add("North", listMatches);
	labelStatus = new Label("");
	panelListCurrent.add("South", labelStatus);
	panelList.add("South", panelListCurrent);

	panelListButtons.add("North", panelList);

	// control buttons
	Panel panelButtons = new Panel();
	Button buttonSearch = new Button(SEARCH);
	buttonSearch.addActionListener(this);
	panelButtons.add(buttonSearch);
	Button buttonStop = new Button(STOP);
	buttonStop.addActionListener(this);
	panelButtons.add(buttonStop);

	panelListButtons.add("South", panelButtons);

	panelMain.add("South", panelListButtons);

	add(panelMain);
	setVisible(true);

	repaint();

	// initialize search data structures
	vectorToSearch = new Vector();
	vectorSearched = new Vector();
	vectorMatches = new Vector();

	// set default for URL access
	URLConnection.setDefaultAllowUserInteraction(false);
    }

    public void start() {
    }

    public void stop() {
	if (searchThread != null) {
	    setStatus("stopping...");
	    searchThread = null;
	}
    }

    public void destroy() {
    }

    boolean robotSafe(URL url) {
	String strHost = url.getHost();

	// form URL of the robots.txt file
	String strRobot = "http://" + strHost + "/robots.txt";
	URL urlRobot;
	try {
	    urlRobot = new URL(strRobot);
	} catch (MalformedURLException e) {
	    // something weird is happening, so don't trust it
	    return false;
	}

	String strCommands;
	try {
	    InputStream urlRobotStream = urlRobot.openStream();

	    // read in entire file
	    byte b[] = new byte[1000];
	    int numRead = urlRobotStream.read(b);
	    strCommands = new String(b, 0, numRead);
	    while (numRead != -1) {
		if (Thread.currentThread() != searchThread)
		    break;
		numRead = urlRobotStream.read(b);
		if (numRead != -1) {
		    String newCommands = new String(b, 0, numRead);
		    strCommands += newCommands;
		}
	    }
	    urlRobotStream.close();
	} catch (IOException e) {
	    // if there is no robots.txt file, it is OK to search
	    return true;
	}

	// assume that this robots.txt refers to us and
	// search for "Disallow:" commands.
	String strURL = url.getFile();
	int index = 0;
	while ((index = strCommands.indexOf(DISALLOW, index)) != -1) {
	    index += DISALLOW.length();
	    String strPath = strCommands.substring(index);
	    StringTokenizer st = new StringTokenizer(strPath);

	    if (!st.hasMoreTokens())
		break;

	    String strBadPath = st.nextToken();

	    // if the URL starts with a disallowed path, it is not safe
	    if (strURL.indexOf(strBadPath) == 0)
		return false;
	}

	return true;
    }

    public void paint(Graphics g) {
      	//Draw a Rectangle around the applet's display area.
      	g.drawRect(0, 0, getSize().width - 1, getSize().height - 1);

	panelMain.paint(g);
	panelMain.paintComponents(g);
	// update(g);
	// panelMain.update(g);
    }

    public void run() {
	String strURL = textURL.getText();
	String strTargetType = choiceType.getSelectedItem();
	int numberSearched = 0;
	int numberFound = 0;

	if (strURL.length() == 0) {
	    setStatus("ERROR: must enter a starting URL");
	    return;
	}

	// initialize search data structures
	vectorToSearch.removeAllElements();
	vectorSearched.removeAllElements();
	vectorMatches.removeAllElements();
	listMatches.removeAll();

	vectorToSearch.addElement(strURL);

	while ((vectorToSearch.size() > 0)
	  && (Thread.currentThread() == searchThread)) {
	    // get the first element from the to be searched list
	    strURL = (String) vectorToSearch.elementAt(0);

	    setStatus("searching " + strURL);

	    URL url;
	    try {
		url = new URL(strURL);
	    } catch (MalformedURLException e) {
		setStatus("ERROR: invalid URL " + strURL);
		break;
	    }

	    // mark the URL as searched (we want this one way or the other)
	    vectorToSearch.removeElementAt(0);
	    vectorSearched.addElement(strURL);

	    // can only search http: protocol URLs
	    if (url.getProtocol().compareTo("http") != 0)
		break;

	    // test to make sure it is before searching
	    if (!robotSafe(url))
		break;

	    try {
		// try opening the URL
		URLConnection urlConnection = url.openConnection();

		urlConnection.setAllowUserInteraction(false);

		InputStream urlStream = url.openStream();
		String type
		  = urlConnection.guessContentTypeFromStream(urlStream);
		if (type == null)
		    break;
		if (type.compareTo("text/html") != 0)
		    break;

		// search the input stream for links
		// first, read in the entire URL
		byte b[] = new byte[1000];
		int numRead = urlStream.read(b);
		String content = new String(b, 0, numRead);
		while (numRead != -1) {
		    if (Thread.currentThread() != searchThread)
			break;
		    numRead = urlStream.read(b);
		    if (numRead != -1) {
			String newContent = new String(b, 0, numRead);
			content += newContent;
		    }
		}
		urlStream.close();

		if (Thread.currentThread() != searchThread)
		    break;

		String lowerCaseContent = content.toLowerCase();

		int index = 0;
		while ((index = lowerCaseContent.indexOf("<a", index)) != -1)
		{
		    if ((index = lowerCaseContent.indexOf("href", index)) == -1)
			break;
		    if ((index = lowerCaseContent.indexOf("=", index)) == -1)
			break;

		    if (Thread.currentThread() != searchThread)
			break;

		    index++;
		    String remaining = content.substring(index);

		    StringTokenizer st
		      = new StringTokenizer(remaining, "\t\n\r\">#");
		    String strLink = st.nextToken();

		    URL urlLink;
		    try {
			urlLink = new URL(url, strLink);
			strLink = urlLink.toString();
		    } catch (MalformedURLException e) {
			setStatus("ERROR: bad URL " + strLink);
			continue;
		    }

		    // only look at http links
		    if (urlLink.getProtocol().compareTo("http") != 0)
			break;

		    if (Thread.currentThread() != searchThread)
			break;

		    try {
			// try opening the URL
			URLConnection urlLinkConnection
			  = urlLink.openConnection();
			urlLinkConnection.setAllowUserInteraction(false);
			InputStream linkStream = urlLink.openStream();
			String strType
			  = urlLinkConnection.guessContentTypeFromStream(linkStream);
			linkStream.close();

			// if another page, add to the end of search list
			if (strType == null)
			    break;
			if (strType.compareTo("text/html") == 0) {
			    // check to see if this URL has already been
			    // searched or is going to be searched
			    if ((!vectorSearched.contains(strLink))
			      && (!vectorToSearch.contains(strLink))) {

				// test to make sure it is robot-safe!
				if (robotSafe(urlLink))
				    vectorToSearch.addElement(strLink);
			    }
			}

			// if the proper type, add it to the results list
			// unless we have already seen it
			if (strType.compareTo(strTargetType) == 0) {
			    if (vectorMatches.contains(strLink) == false) {
				listMatches.add(strLink);
				vectorMatches.addElement(strLink);
				numberFound++;
				if (numberFound >= SEARCH_LIMIT)
				    break;
			    }
			}
		    } catch (IOException e) {
			setStatus("ERROR: couldn't open URL " + strLink);
			continue;
		    }
		}
	    } catch (IOException e) {
		setStatus("ERROR: couldn't open URL " + strURL);
		break;
	    }

	    numberSearched++;
	    if (numberSearched >= SEARCH_LIMIT)
		break;
	}

	if (numberSearched >= SEARCH_LIMIT || numberFound >= SEARCH_LIMIT)
	    setStatus("reached search limit of " + SEARCH_LIMIT);
	else
	    setStatus("done");
	searchThread = null;
	// searchThread.stop();
    }

    void setStatus(String status) {
	labelStatus.setText(status);
    }

    public void actionPerformed(ActionEvent event) {
	String command = event.getActionCommand();

	if (command.compareTo(SEARCH) == 0) {
	    setStatus("searching...");

	    // launch a thread to do the search
	    if (searchThread == null) {
		searchThread = new Thread(this);
	    }
	    searchThread.start();
	}
	else if (command.compareTo(STOP) == 0) {
	    stop();
	}
    }
        public static void main (String argv[])
        {
                Frame f = new Frame("WebFrame");
                WebCrawler applet = new WebCrawler();
		f.add("Center", applet);

/*		Behind a firewall set your proxy and port here!
*/
                Properties props= new Properties(System.getProperties());
                props.put("http.proxySet", "true");
        	props.put("http.proxyHost", "webcache-cup");
        	props.put("http.proxyPort", "8080");

                Properties newprops = new Properties(props);
                System.setProperties(newprops);
/**/


                applet.init();
                applet.start();
                f.pack();
                f.show();
        }

}

But when i run it, and enter any URL it shows Following error in Cmd

Exception in thread "AWT-EventQueue-1" java.lang.IllegalThreadStateExcepti
        at java.lang.Thread.start(Thread.java:571)
        at WebCrawler.actionPerformed(WebCrawler.java:388)
        at java.awt.Button.processActionEvent(Button.java:388)
        at java.awt.Button.processEvent(Button.java:356)
        at java.awt.Component.dispatchEventImpl(Component.java:3955)
        at java.awt.Component.dispatchEvent(Component.java:3803)
        at java.awt.EventQueue.dispatchEvent(EventQueue.java:463)
        at java.awt.EventDispatchThread.pumpOneEventForHierarchy(EventDisp
        at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispat
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.jav
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.jav
        at java.awt.EventDispatchThread.run(EventDispatchThread.java:110)

Please look at it...please tell what i need to do in it....

and what i need to edit in this Code of webcrawler

Properties props= new Properties(System.getProperties());
                props.put("http.proxySet", "true");
        	props.put("http.proxyHost", "webcache-cup");
        	props.put("http.proxyPort", "8080");

Please help me out....
Please help.....

Richard MacCutchan27-Nov-09 2:32

27-Nov-09 2:32

This is far too much code for anyone to read through. Please identifiy the line of code where the error occurs and post only the small amount of code near that line. Have you looked at the documentation to see what an IllegalThreadStateException is, and what may be its causes?

sangeeta200927-Nov-09 2:54

27-Nov-09 2:54

   public static void main (String argv[])
        {
                Frame f = new Frame("WebFrame");
                WebCrawler applet = new WebCrawler();
		f.add("Center", applet);

/*		Behind a firewall set your proxy and port here!
*/
                Properties props= new Properties(System.getProperties());
                props.put("http.proxySet", "true");
        	props.put("http.proxyHost", "webcache-cup");
        	props.put("http.proxyPort", "8080");

                Properties newprops = new Properties(props);
                System.setProperties(newprops);
/**/


                applet.init();
                applet.start();
                f.pack();
                f.show();
        }

}

Ok ..just look at this MAIN
is there something needed to be done at
Properties prop=new Properties(System.getProperties());

Please compile it at your side ....and look at error..
there are some errors which i cannot explain...
please just once run it at your side...and put the inputs
and see what is that i am missing....
Please help me...

Richard MacCutchan27-Nov-09 3:55

27-Nov-09 3:55

sangeeta2009 wrote:
Please compile it at your side ....and look at error..

Well once I had commented out all the irrelevant code I could not see anything wrong; it compiles and runs. However, I don't see what this has to do with your original problem which as I recall, was some sort of ThreadException. My only suggestion is that you use whatever debug tools you have and try to step through your code. Assuming you wrote this code in the first place you should be able to extract small parts of it to test in isolation until you can at least narrow it down to a specific area of code.

Nagy Vilmos27-Nov-09 4:12

Nagy Vilmos

27-Nov-09 4:12

Richard MacCutchan wrote:
Assuming you wrote this code in the first place [...]

#wistles#

Panic, Chaos, Destruction.
My work here is done.

Richard MacCutchan27-Nov-09 5:13

27-Nov-09 5:13

Someone must have written it! Roll eyes | :rolleyes:

sangeeta200928-Nov-09 1:29

28-Nov-09 1:29

Ok if Possible please help me out...

I have run that code without any compilation error...
But when i input the url int the first Text box..
say: http://java.sun.com

in command prompt it give me errors..and doesn't display me any
of the Url's related to that site..

Hope you get it...Please help and see if that works on your side...

And please all other guys..stop commenting about the code.... Mad | :mad:

I haven't written that code, i have taken it form Sun.com
So if Somebody is interested in helping someone whose is in need...

Richard MacCutchan28-Nov-09 1:36

28-Nov-09 1:36

sangeeta2009 wrote:
I haven't written that code, i have taken it form Sun.com

Well I suggest you go back to Sun and ask them to fix it. I don't have the skills to work on a piece of code this complex, and as you haven't written it yourself I doubt that you have much understanding of it either.

sangeeta200928-Nov-09 1:54

28-Nov-09 1:54

Ohhh..!!!
That means you cannot help ..except fake posting...
If i will be knowing that...it will be better that
i will not be here..asking for help!!!

well thanks for time wasting... Frown | :(

Richard MacCutchan28-Nov-09 2:10

28-Nov-09 2:10

sangeeta2009 wrote:
That means you cannot help ..except fake posting.

On the contrary, I have tried to help, but you seem incapable of doing anything for yourself, even answering a simple question.

sangeeta2009 wrote:
well thanks for time wasting

Yes, I am sorry that I have wasted my time trying to assist you. I thought you were a serious programmer who wanted to learn something. Obviously I was mistaken.

routing table in java

vidzdas27-Nov-09 2:02

vidzdas

27-Nov-09 2:02

Please can somebody help me to find the source code for creating a routing table in java.. i googled but couldn't find anything useful there.. Frown | :(

Richard MacCutchan27-Nov-09 2:27

27-Nov-09 2:27

vidzdas wrote:
i googled but couldn't find anything useful there..

I googled and found lots of examples; why do think they are not useful?

vidzdas27-Nov-09 2:54

vidzdas

27-Nov-09 2:54

They do not describe how these tables are implemented. It only describes the nature of these tables.. Hmmm | :|

Richard MacCutchan27-Nov-09 3:01

27-Nov-09 3:01

This sample[^] contains Java code.

Serialize and Deserialize Excel File

vidzdas27-Nov-09 3:23

vidzdas

27-Nov-09 3:23

Thanks a lot.. Smile | :)

I think this should solve my problem.
I'l keep u updated..

2008.sobha26-Nov-09 14:34

2008.sobha

26-Nov-09 14:34

Hi Can anyone send a sample code to serialize & deserialize excel file?
Thanks in Advance.

Richard MacCutchan26-Nov-09 21:52

26-Nov-09 21:52

Try this[^].

Nagy Vilmos26-Nov-09 21:54

Nagy Vilmos

26-Nov-09 21:54

Danmed you me nemesis! You beat me by a click!

Do you start work at the same time as me then?

Poke tongue | ;-P

Panic, Chaos, Destruction.
My work here is done.

Richard MacCutchan26-Nov-09 23:25

26-Nov-09 23:25

Nagy Vilmos wrote:
You beat me by a click!

Two clicks, I believe. Laugh | :laugh:

Nagy Vilmos wrote:
Do you start work at the same time as me then?

I haven't 'worked' for about three years. That is to say I haven't received any remuneration for my efforts in that time; details here[^].