使用 python 正则表达式从文本中提取某些 URL答案

【问题标题】：using python regex to extract certain URLs from text使用 python 正则表达式从文本中提取某些 URL
【发布时间】：2015-01-16 17:17:17
【问题描述】：

所以我有来自 NPR 页面的 HTML，我想使用正则表达式为我提取某些 URL（这些 URL 调用嵌套在页面中的特定故事的 URL）。实际链接在文本中（手动检索）显示为：

<a href="http://www.npr.org/blogs/parallels/2014/11/11/363018388/how-the-islamic-state-wages-its-propaganda-war">
<a href="http://www.npr.org/blogs/thetwo-way/2014/11/11/363309020/asked-to-stop-praying-alaska-school-won-t-host-state-tournament">
<a href="http://www.npr.org/2014/11/11/362817642/a-marines-parents-story-their-memories-that-you-should-hear">
<a href="http://www.npr.org/blogs/thetwo-way/2014/11/11/363288744/comets-rugged-landscape-makes-landing-a-roll-of-the-dice">
<a href="http://www.npr.org/blogs/thetwo-way/2014/11/11/363293514/for-dyslexics-a-font-and-a-dictionary-that-are-meant-to-help">

显然，如果我希望能够始终如一地使用它，我就不能继续使用手动检索。到目前为止，我有这个代码：

import nltk
import re

f = open("/Users/shannonmcgregor/Desktop/npr.txt")
npr_lines = f.readlines()
f.close()

我有这段代码可以抓取 (

for line in npr_lines:
re.findall('<a href="?\'?([^"\'>]*)', line)

但这会抓取所有网址。我尝试添加类似：

(parallels|thetwo-way|a-marines)

但这什么也没返回。那么我做错了什么？如何将较大的 URL 剥离器与这些针对给定 URL 的特定词结合起来？

谢谢你:)

【问题讨论】：

您的输入和预期输出是什么？
使用 HTML 解析器，crummy.com/software/BeautifulSoup
你能把/Users/shannonmcgregor/Desktop/npr.txt文件的内容连同预期的输出一起发布吗？

标签： python regex url

【解决方案1】：

通过专门用于解析html和xml文件[BeautifulSoup]的工具，

>>> from bs4 import BeautifulSoup
>>> s = """<a href="http://www.npr.org/blogs/parallels/2014/11/11/363018388/how-the-islamic-state-wages-its-propaganda-war">
<a href="http://www.npr.org/blogs/thetwo-way/2014/11/11/363309020/asked-to-stop-praying-alaska-school-won-t-host-state-tournament">
<a href="http://www.npr.org/2014/11/11/362817642/a-marines-parents-story-their-memories-that-you-should-hear">
<a href="http://www.npr.org/blogs/thetwo-way/2014/11/11/363288744/comets-rugged-landscape-makes-landing-a-roll-of-the-dice">
<a href="http://www.npr.org/blogs/thetwo-way/2014/11/11/363293514/for-dyslexics-a-font-and-a-dictionary-that-are-meant-to-help">"""
>>> soup = BeautifulSoup(s) # or pass the file directly into BS like >>> soup = BeautifulSoup(open('/Users/shannonmcgregor/Desktop/npr.txt'))
>>> atag = soup.find_all('a')
>>> links = [i['href'] for i in atag]
>>> import re
>>> for i in links:
        if re.match(r'.*(parallels|thetwo-way|a-marines).*', i):
            print(i)


http://www.npr.org/blogs/parallels/2014/11/11/363018388/how-the-islamic-state-wages-its-propaganda-war
http://www.npr.org/blogs/thetwo-way/2014/11/11/363309020/asked-to-stop-praying-alaska-school-won-t-host-state-tournament
http://www.npr.org/2014/11/11/362817642/a-marines-parents-story-their-memories-that-you-should-hear
http://www.npr.org/blogs/thetwo-way/2014/11/11/363288744/comets-rugged-landscape-makes-landing-a-roll-of-the-dice
http://www.npr.org/blogs/thetwo-way/2014/11/11/363293514/for-dyslexics-a-font-and-a-dictionary-that-are-meant-to-help

【讨论】：

re.match(r'.*(parallels|thetwo-way|a-marines).*', i) 在这种情况下最好拼写为re.search(r'parallels|thetwo-way|a-marines', i)

【解决方案2】：

您可以使用re.search 函数来匹配行中的正则表达式，如果匹配则打印该行

>>> file  = open('/Users/shannonmcgregor/Desktop/npr.txt', 'r')
>>> for line in file:
...     if re.search('<a href=[^>]*(parallels|thetwo-way|a-marines)', line):
...             print line

将输出为

<a href="http://www.npr.org/blogs/parallels/2014/11/11/363018388/how-the-islamic-state-wages-its-propaganda-war">
<a href="http://www.npr.org/blogs/thetwo-way/2014/11/11/363309020/asked-to-stop-praying-alaska-school-won-t-host-state-tournament">
<a href="http://www.npr.org/2014/11/11/362817642/a-marines-parents-story-their-memories-that-you-should-hear">
<a href="http://www.npr.org/blogs/thetwo-way/2014/11/11/363288744/comets-rugged-landscape-makes-landing-a-roll-of-the-dice">
<a href="http://www.npr.org/blogs/thetwo-way/2014/11/11/363293514/for-dyslexics-a-font-and-a-dictionary-that-are-meant-to-help">

【讨论】：

【解决方案3】：

您可以使用 lookahead：

<a href="?\'?((?=[^"\'>]*(?:thetwo\-way|parallels|a\-marines))[^"\'>]+)

Debuggex Demo

【讨论】：