How I can scape data from two website and store information together.

Question

1.00/5 (1 vote)

See more:

I do not know it is correct or not. I try to web scraped data and store in sqlite databse but i cannot store them together because the information is from two website.

My problem is how I can scape data from different website and store the information together .

Now I try is to scape data from two website, but i cannot pass through this information together.thrown errors

Cannot resolve symbol 'name'
Cannot resolve symbol 'matric_no'

What I have tried:

Java

  1  package com;
  2  
  3  
  4  import org.jsoup.Jsoup;
  5  import org.jsoup.nodes.Document;
  6  import org.jsoup.nodes.Element;
  7  //import org.jsoup.select.Elements;
  8  
  9  import java.io.IOException;
 10  import java.sql.Connection;
 11  import java.sql.DriverManager;
 12  import java.sql.PreparedStatement;
 13  import java.sql.SQLException;
 14  
 15  public class studentlist {
 16  
 17      public static void main (String[] args)  {
 18  
 19          new studentlist().trr();
 20         // new studentlist().git();
 21      }
 22  
 23      public void trr() {
 24          final String url =
 25                  "https://github.com/STIW3054-A202/Main-Data/wiki/List_of_Student";
 26          final String urls =
 27                  "https://github.com/STIW3054-A202/Main-Data/issues/1";
 28          try {
 29              final Document document = Jsoup.connect(url).get();
 30  
 31              for (Element row : document.select(
 32                      "div.gollum-markdown-content tr")) {
 33                  if (row.select("td:nth-of-type(2)").text().equals("")) {
 34                      continue;
 35                  } else {
 36  
 37                      final String matric_no =
 38                              row.select("td:nth-of-type(2)").text();
 39  
 40                      final String name =
 41                              row.select("td:nth-of-type(3)").text();
 42                      studentlist t = new studentlist();
 43  
 44                  }
 45              }
 46             final Document doc = Jsoup.connect(urls).get();
 47             for (Element row : doc.select("div.edit-comment-hide tr")) {
 48              if (row.select("[href]").text().equals("")) {
 49                  continue;
 50              } else {
 51                  final String githubID = row.select("[href]").text();
 52  
 53                  studentlist t = new studentlist();
 54                  t.add(matric_no,name,githubID);
 55  
 56              }
 57          }
 58  
 59      }
 60          catch (IOException ex) {
 61              ex.printStackTrace();
 62          }
 63      }
 64  
 65      private Connection connect() {
 66          // SQLite connection string
 67          String url = "jdbc:sqlite:C://sqlite/db/test.db";
 68          Connection conn = null;
 69          try {
 70              conn = DriverManager.getConnection(url);
 71          } catch (SQLException e) {
 72              System.out.println(e.getMessage());
 73          }
 74          return conn;
 75      }
 76     public void add (String matric_no,String name ) {
 77          String sql = "INSERT INTO NameList(Matric_No,Name,GithubLink) VALUES(?,?,?)";
 78  
 79          try (Connection conn = this.connect();
 80               PreparedStatement pstmt = conn.prepareStatement(sql)) {
 81              pstmt.setString(1, matric_no);
 82              pstmt.setString(2, name);
 83              //pstmt.setString(3, githubID);
 84              pstmt.executeUpdate();
 85          } catch (SQLException e) {
 86              System.out.println(e.getMessage());
 87          }
 88      }
 89  
 90  }

Posted 5-Apr-21 20:59pm

Cow cow Lee

Updated 6-Apr-21 1:55am

Richard MacCutchan

v7

Add a Solution

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Richard MacCutchan · Accepted Answer · 2021-04-05T21:16:00

Solution 1

You are passing a null value in one of the parameters, for a field that does not allow null. Check the data that you are collecting and passing to the database. If necessary use the debugger to find out exactly what is happening.

Your code makes no sense. You have two separate methods (add and adds) to insert information into your database. You need only a single method that inserts all fields per row:

Java

public void add ( String matric_no, String name ) {
    String sql = "INSERT INTO NameList(Matric_No, Name) VALUES(?, ?)";
    
    try {
        Connection conn = this.connect();
        PreparedStatement pstmt = conn.prepareStatement(sql)) {
        pstmt.setString(1, matric_no);
        pstmt.setString(2, name);
        pstmt.executeUpdate();
    } catch (SQLException e) {
        System.out.println(e.getMessage());
    }
}

Posted 5-Apr-21 21:16pm

Richard MacCutchan

Updated 5-Apr-21 21:41pm

v2

Comments

Cow cow Lee 6-Apr-21 4:36am

oK,but i combine to one method also throwns new errors like

'add(java.lang.String, java.lang.String)' in 'com.studentlist' cannot be applied to '(java.lang.String)'
and my code t.add(matric.no) is red.

It is because i using two string?or how to correct it?

Richard MacCutchan 6-Apr-21 6:18am

You have declared the add method to accept three string values so you annot use something like:

t.add(name);

You must pass all required parameters at the same time.

I think maybe you need to go back to your Java reference guide and read up on methods.

Cow cow Lee 6-Apr-21 7:41am

ok i got it.But now i have problem to scrapping the data from website now.I think i updated question.

Richard MacCutchan 6-Apr-21 7:50am

There are so many variables involved in web scraping that you need to be precise about what data you are trying to extract. And saying, "I have a problem" does not give us any useful information about what that problem is.

Cow cow Lee 6-Apr-21 7:51am

ok i try to updated my question.Or you have some advice?

Richard MacCutchan 6-Apr-21 7:58am

You have created the variables matric_no and name inside the block between lines 35 and 44. You have then tried to refer to them in your call to t.add at line 54. But they have gone out of scope at that point so they no longer exist.

You need to study and understand the lifetime and scope of variables in Java.

Cow cow Lee 6-Apr-21 10:24am

ok thanks