如何从pandas中解析的html页面中提取文本

Question

考虑这个简单的例子

df = pd.DataFrame({'link' : ['https://en.wikipedia.org/wiki/World%27s_funniest_joke',
                             'https://en.wikipedia.org/wiki/The_Funniest_Joke_in_the_World']})

df
Out[169]: 
                                                           link
0         https://en.wikipedia.org/wiki/World%27s_funniest_joke
1  https://en.wikipedia.org/wiki/The_Funniest_Joke_in_the_World

我想使用beautiful soup 解析每个链接，并将解析后的内容存储到我的数据框的另一列中。以下似乎运作良好：

def puller(mylink):
    doc = requests.get(mylink)
    return BeautifulSoup(doc.content, 'html5lib')

df['parsed'] = df.apply(lambda x: puller(x))
df['mytag'] = df.parsed.apply(lambda x: x.find_all('p'))

问题是我正在获取列表，我需要处理其中的文本。特别是，我试图仅将提及joke 的段落保留在文本中的某处，但我无法这样做。

def extractor(mylist):
    return list(filter(lambda x: re.search('joke', x), mylist))

df.mytag.apply(lambda x: extractor(x))
TypeError: expected string or bytes-like object

在这里进行的最佳方式是什么？

谢谢！

Answer 1

df[mytag] 的每个条目都是 beautifulsoup '<p>' 元素的列表。您可以编写一个函数来获取此列表并返回包含您的单词的文本。然后使用 .apply 而不是 df[mytag] 让它适用于所有行。

def myfunc(list_of_ps, word='joke'):
    '''
    This will return a list of string text paragraphs 
    containing the word.
    '''
    result_ps = []
    for p in list of ps:
        if word in p.text:
            result_ps.append(p.text) # p if p itself is needed

    return result_ps if result_ps else None

df['mytag'].apply(myfunc)

编辑：
您问题中的错误反映了上面斜体中提到的事实。 re.search 期望字符串作为参数。换句话说，该函数调用中的x 必须是字符串或类似字节的对象。在这种情况下，它是一个 BeautifulSoup 对象，作为一个单独的 <p> 元素。该错误可以通过获取元素的字符串文本为x.text来解决。

Answer 2

pandas api 旨在用于更原始的数据类型；你最好写一个函数来转换你想要的链接 -> 文本然后调用apply。这是一种解决方案：

import pandas as pd
from bs4 import BeautifulSoup

df = pd.DataFrame({'link' : [
        'https://en.wikipedia.org/wiki/World%27s_funniest_joke',
        'https://en.wikipedia.org/wiki/The_Funniest_Joke_in_the_World'
    ]
})

def parse_link(mylink):
    doc = requests.get(mylink)
    return BeautifulSoup(doc.content, 'html5lib')

def matching_paragraphs(soup, text):
    res = [p.get_text() for p in soup.find_all("p") if text in p.get_text()]
    return res
   
def apply_func(link, text):
    soup = parse_link(link)
    res = matching_paragraphs(soup, text=text)
    return res
    

df['text'] = df.link.apply(apply_func, args=("joke",))

输出：

                                                link                                               text
0  https://en.wikipedia.org/wiki/World%27s_funnie...  [The "world's funniest joke" is a term used by...
1  https://en.wikipedia.org/wiki/The_Funniest_Jok...  ["The Funniest Joke in the World" (also "Joke ...

更明智地使用数据框，您可以将字符串列表转换为行：

df.explode(column="text", ignore_index=True)

结果：

                                                 link                                               text
0   https://en.wikipedia.org/wiki/World%27s_funnie...  The "world's funniest joke" is a term used by ...
1   https://en.wikipedia.org/wiki/World%27s_funnie...  The winning joke, which was later found to be ...
2   https://en.wikipedia.org/wiki/World%27s_funnie...  Researchers also included five computer-genera...
3   https://en.wikipedia.org/wiki/The_Funniest_Jok...  "The Funniest Joke in the World" (also "Joke W...
4   https://en.wikipedia.org/wiki/The_Funniest_Jok...  The sketch appeared in the first episode of th...
5   https://en.wikipedia.org/wiki/The_Funniest_Jok...  The sketch is framed in a documentary style an...
6   https://en.wikipedia.org/wiki/The_Funniest_Jok...  The British Army are soon eager to determine "...
7   https://en.wikipedia.org/wiki/The_Funniest_Jok...  The German version is described as being "over...
8   https://en.wikipedia.org/wiki/The_Funniest_Jok...  The Germans attempt counter-jokes, but each at...
9   https://en.wikipedia.org/wiki/The_Funniest_Jok...  The British joke is said to have been laid to ...
10  https://en.wikipedia.org/wiki/The_Funniest_Jok...  The footage of Adolf Hitler is taken from Leni...
11  https://en.wikipedia.org/wiki/The_Funniest_Jok...  If the German version of the joke is entered i...