Python 命令提示符 - 自动提取链接

Question

我在 github 上找到了一个很好的工具，可以让你输入 URL 以从中提取链接：https://github.com/devharsh/Links-Extractor

但是，我想提取页面上的所有 URL，而不仅仅是可点击的链接，例如，在网站的 HTML 中：

<a href="www.example.com">test</a>
in plaintext HTML: www.example.com
and <img src="www.example.com/picture.png">

会打印出来：

www.example.com
www.example.com
www.example.com/picture.png

我是 python 新手，我还没有找到任何在线工具可以让你从多个页面中提取 URL（我想要它，所以你输入多个 URL，运行它会从每个 URL 中提取所有 URL输入），它们只允许输入一个 URL 并从该页面提取链接（一次一个）。

它只打印出 HTML 标签 url，而不是全部。

这是 python 代码（经过编辑以处理 UTF-8 和百分比编码）：

#!/usr/bin/python

__author__ = "Devharsh Trivedi"
__copyright__ = "Copyright 2018, Devharsh Trivedi"
__license__ = "GPL"
__version__ = "1.4"
__maintainer__ = "Devharsh Trivedi"
__email__ = "devharsh@live.in"
__status__ = "Production"

import sys
import requests
from bs4 import BeautifulSoup
from urllib.parse import urlparse

try:

    for link in sys.argv[1:]:
        page = requests.get(link)
        soup = BeautifulSoup(page.text, "lxml")
        extlist = set()
        intlist = set()

        for a in soup.findAll("a", attrs={"href":True}):
            if len(a['href'].strip()) > 1 and a['href'][0] != '#' and 'javascript:' not in a['href'].strip() and 'mailto:' not in a['href'].strip() and 'tel:' not in a['href'].strip():
                if 'http' in a['href'].strip() or 'https' in a['href'].strip():
                    if urlparse(link).netloc.lower() in urlparse(a['href'].strip()).netloc.lower():
                        intlist.add(a['href'])
                    else:
                        extlist.add(a['href'])
                else:
                    intlist.add(a['href'])

        print('\n')
        print(link)
        print('---------------------')
        print('\n')
        print(str(len(intlist)) + ' internal links found:')
        print('\n')
        for il in intlist:
            print(il.encode("utf-8"))
        print('\n')
        print(str(len(extlist)) + ' external links found:')
        print('\n')
        for el in extlist:
            print(el.encode("utf-8"))
        print('\n')

except Exception as e:
    print(e)

编辑：解决方案

#!/usr/bin/python

__author__ = "Devharsh Trivedi"
__copyright__ = "Copyright 2018, Devharsh Trivedi"
__license__ = "GPL"
__version__ = "1.4"
__maintainer__ = "Devharsh Trivedi"
__email__ = "devharsh@live.in"
__status__ = "Production"

import re
import requests
import sys

def find_urls(links):
  url_list = []
  for link in links:
    page = requests.get(link).text
    parts = re.findall('(http|ftp|https)://([\w_-]+(?:(?:\.[\w_-]+)+))([\w.,@?^=%&:/~+#-]*[\w@?^=%&/~+#-])?', page)
    true_url = [p + '://' + d + sd for p, d, sd in parts]
    url_list.extend(true_url)
  return url_list

for url in find_urls(sys.argv[1:]): print(url);

感谢 maninthecomputer (https://stackoverflow.com/users/3650306/maninthecomputer)。

Answer 1

这里有一个快速的正则表达式来识别一个 URL：

(http|ftp|https)://([\w_-]+(?:(?:\.[\w_-]+)+))([\w.,@?^=%&:/~+#-]*[\w@?^=%&/~+#-])?

实际上，这看起来像：

import re
import requests
import sys

def find_urls(links):
  url_list = []
  for link in links:
    page = requests.get(link).text
    parts = re.findall('(http|ftp|https)://([\w_-]+(?:(?:\.[\w_-]+)+))([\w.,@?^=%&:/~+#-]*[\w@?^=%&/~+#-])?', page)
    true_url = [p + '://' + d + sd for p, d, sd in parts]
    url_list.extend(true_url)
  return url_list

print(find_urls(sys.argv[1:]))

输出：

print(find_urls(['https://www.google.com']))

是：

['http://schema.org/WebPage', 'https://www.google.com/imghp?hl=en&tab=wi', 'https://maps.google.com/maps?hl=en&tab=wl', 'https://play.google.com/?hl=en&tab=w8', 'https://www.youtube.com/?gl=US&tab=w1', 'https://news.google.com/nwshp?hl=en&tab=wn', 'https://mail.google.com/mail/?tab=wm', 'https://drive.google.com/?tab=wo', 'https://www.google.com/intl/en/about/products?tab=wh', 'http://www.google.com/history/optout?hl=en', 'https://accounts.google.com/ServiceLogin?hl=en&passive=true&continue=https://www.google.com/']

感谢 Rajeev here 提供正则表达式

编辑：鉴于作者更新的用例，我做了一些试验和错误，发现了这个新的正则表达式：

((https?:\/\/.+)?(\/.*)+)

这是在实践中：

def find_urls(links):
  url_list = []
  for link in links:
    page = requests.get(link).text
    parts = re.findall('((https?:\/\/.+)?(\/.*)+)', page)
    url_list.extend(parts)
  return url_list

我不保证这适用于所有用例（我不是正则表达式专家），但它应该适用于您在网页中找到的大多数 URL/文件路径