Click here to Skip to main content
15,886,014 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
i just made a python script to scan for urls in a website and it keeps looping and not printing the urls it finds, i am working on a pen test tool but i just need this to work. bet is some really simple. if you know where i have gone wrong please tell me. Thanks in advace

Python
import urllib
import sqlite3
import re
import sys

links = []
colur_red = '\033[31m'
colur_end = '\033[0m'
color_green = '\033[32m'

def findurl(text):
    if str(text).find("http") == 0:
        return True
    elif str(text).find("ftp") == 0:
        return True
    elif str(text).find("https") == 0:
        return True
    else:
        return False

def openurl(url):
    try:
        html = urllib.urlopen(url).read()
        getlinks = re.findall(r"""<\s*a\s*href=["']([^=]+)["']""", html)
        for link in getlinks:
            a = findurl(link)
            if a == True:
                links.append(link)
    except:
        print(colur_red + "[-] Error Loading {}".format(str(url)) + colur_end)

def main(site):
    print(color_green + "[+] Starting Scan on {}".format(site) + colur_end)
    openurl(site)
    for url in links:
        print url
        if len(url) == 0:
            print(colur_red + "[-] No URLs Found" + colur_end)
        elif sys.stdin.read(1) == 'q':
            print(color_green + "[+] Stopping Scan" + colur_end)
        else:
            print(color_green + "[+] Found URl and adding it to database" + colur_end)
            #hostname = str(url).replace(':/', '').split('/')[1]
            #db = sqlite3.connect('database.db')
            #cursor = db.cursor()
            #cursor.execute('''INSERT INTO data_1(urls, hostname)VALUES(?,?)''', (url, hostname))
            #db.commit()
            #db.close()
            openurl(url) # i need to add away of exiting by pressing 'q'

command = sys.argv
length = len(command)
num = 0
if len(command) == 1:
    print('please use -u <url>')
else:
    for i in command:
        if i == "-u":
            num = num + 1
            starturl = command[num]
            main(starturl)
        else:
            num = num + 1



What I have tried:

I have tried doing some debuging but just to try and see whats wrong. i know that the it gets the urls fine and i think is to do with the getting urls from links.
Posted
Updated 12-Apr-19 6:49am

1 solution

Your openurl(url) method creates the links list, but does not return it to the caller. So in main after calling openurl main's links variable contains nothing. You should study Python Scopes and Namespaces[^] to understand why.
 
Share this answer
 
Comments
WOLF 2018 12-Apr-19 16:26pm    
I got it working it took some editing tho

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900