Click here to Skip to main content
15,889,876 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
I want to read 50 CSV one by one from a folder path. In the CSV file, there are some links in all of them, I want to open the link one by one and fetch data into the text file for each link new text file should be created.
the code is running but doesn't give any output.
up to print(list_of_rows) it gives the output

What I have tried:

Python
from bs4 import BeautifulSoup
import requests
#import csv
import pandas
import os
os.environ["PYTHONIOENCODING"] = "utf-8"

df_list = []
path = "C:/Users/bubai/Desktop/try/scrapy/output"

for file in os.listdir(path):
    #print(file)
    df_list.append(path + '/' + file)
    
for i in df_list:  
    df = pandas.read_csv(i,delimiter=',',usecols = [1], index_col=None, header = None) 
    #print(df)
   
    list_of_rows = [list(row) for row in df.values]
    # Print list of lists i.e. rows
    print(list_of_rows)
    
    for link in list_of_rows:
        print(link)
        r = requests.get(link)

        soup = BeautifulSoup(r.text, 'lxml')
        print(soup.text)

        fname= "DAta"+ str(i)+".csv"
        f = open(fname,'w', newline='',encoding="utf-8")
        f = open(fname,'a', newline='',encoding="utf-8")

        classs = soup.find(id = 'ContentPlaceHolder1_lblContentText')

        f.write(classs.get_text())  
        
        f.close()
Posted
Updated 16-Jun-20 4:30am
v2
Comments
Richard MacCutchan 16-Jun-20 9:59am    
And did you have a question?
eshan sharma 16-Jun-20 10:21am    
the code is running but doesn't give any output
Richard MacCutchan 16-Jun-20 10:32am    
Then there is something wrong with it. However the only person who can find out why is you. Uncomment all the print statements in the code, and add some others so you can see what is happening as it runs. That should help to narrow down what is wrong.
eshan sharma 16-Jun-20 10:33am    
ok. thank you
Patrice T 16-Jun-20 10:06am    
And the question is ?

Compiling does not mean your code is right! :laugh:
Think of the development process as writing an email: compiling successfully means that you wrote the email in the right language - English, rather than German for example - not that the email contained the message you wanted to send.

So now you enter the second stage of development (in reality it's the fourth or fifth, but you'll come to the earlier stages later): Testing and Debugging.

Start by looking at what it does do, and how that differs from what you wanted. This is important, because it give you information as to why it's doing it. For example, if a program is intended to let the user enter a number and it doubles it and prints the answer, then if the input / output was like this:
Input   Expected output    Actual output
  1            2                 1
  2            4                 4
  3            6                 9
  4            8                16
Then it's fairly obvious that the problem is with the bit which doubles it - it's not adding itself to itself, or multiplying it by 2, it's multiplying it by itself and returning the square of the input.
So with that, you can look at the code and it's obvious that it's somewhere here:
C#
int Double(int value)
   {
   return value * value;
   }

Once you have an idea what might be going wrong, start using the debugger to find out why. Put a breakpoint on the first line of the method, and run your app. When it reaches the breakpoint, the debugger will stop, and hand control over to you. You can now run your code line-by-line (called "single stepping") and look at (or even change) variable contents as necessary (heck, you can even change the code and try again if you need to).
Think about what each line in the code should do before you execute it, and compare that to what it actually did when you use the "Step over" button to execute each line in turn. Did it do what you expect? If so, move on to the next line.
If not, why not? How does it differ?
Hopefully, that should help you locate which part of that code has a problem, and what the problem is.
This is a skill, and it's one which is well worth developing as it helps you in the real world as well as in development. And like all skills, it only improves by use!
 
Share this answer
 
Quote:
the code is running but doesn't give any output

Your code do not behave the way you expect, or you don't understand why !

There is an almost universal solution: Run your code on debugger step by step, inspect variables.
The debugger is here to show you what your code is doing and your task is to compare with what it should do.
There is no magic in the debugger, it don't know what your code is supposed to do, it don't find bugs, it just help you to by showing you what is going on. When the code don't do what is expected, you are close to a bug.
To see what your code is doing: Just set a breakpoint and see your code performing, the debugger allow you to execute lines 1 by 1 and to inspect variables as it execute.

Debugger - Wikipedia, the free encyclopedia[^]

Mastering Debugging in Visual Studio 2010 - A Beginner's Guide[^]
Basic Debugging with Visual Studio 2010 - YouTube[^]

27.3. pdb — The Python Debugger — Python 3.6.1 documentation[^]
Debugging in Python | Python Conquers The Universe[^]
pdb – Interactive Debugger - Python Module of the Week[^]

The debugger is here to only show you what your code is doing and your task is to compare with what it should do.
 
Share this answer
 

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900