在 python 中使用 bs4 从多个链接中获取律师详细信息

Question

我是使用 Python 进行 Web Scraping 的绝对初学者，并且对 Python 编程知之甚少。我只是想提取田纳西州律师的信息。在网页中，有多个链接，其中还有更多关于律师类别的链接，其中还有律师的详细信息。

我已经将各个城市的链接提取到一个列表中，并且还提取了每个城市链接中可用的各种律师类别。现在我正在尝试获取每个城市的每个类别下的每个律师的个人资料链接，我将从那里检索律师的详细信息。但是正在返回空列表。可以做什么。如果可能的话建议。

import requests
from bs4 import BeautifulSoup as bs
import pandas as pd

res = requests.get('https://attorneys.superlawyers.com/tennessee/', headers = {'User-agent': 'Super Bot 9000'})
soup = bs(res.content, 'lxml')

cities = [item['href'] for item in soup.select('#browse_view a')]
for c in cities:
    r=requests.get(c)
    s1=bs(r.content,'lxml')
    categories = [item['href'] for item in s1.select('.three_browse_columns:nth-of-type(2) a')]
    #print(categories)
    for c1 in categories:
        r1=requests.get(c1)
        s2=bs(r1.content,'lxml')
        lawyers = [item['href'] for item in s2.select('.directory_profile a')]
        print(lawyers)

“我希望输出是每个类别的律师的每个个人资料的链接，但它返回的是空列表。”

“[][][][][][][]”

Answer 1

来自另一个post：

发生这种情况是因为您不能将 nth-of-type() 与分类标签一起使用，它只能用于这样的标签：table:nth-of-type(4)。

因此，您的 categories 变量返回一个空列表。

解决方法在同一篇文章中给出：

categories = [item['href'] for item in s1.select('.three_browse_columns a')][1]

Answer 2

当您使用第一个问题的类选择器时，您已经处于a 标签级别。

我在下面使用了一个不同的选择器并测试了掩盖它们是为同一位律师的事实的 url。我分成结束 url，所以我可以使用 set 删除重复项。

我使用 Session 来提高重用连接的效率。我将律师资料添加到列表中，并通过集合理解展平列表以删除任何重复项。

import requests
from bs4 import BeautifulSoup as bs

final = []
with requests.Session() as s:
    res = s.get('https://attorneys.superlawyers.com/tennessee/', headers = {'User-agent': 'Super Bot 9000'})
    soup = bs(res.content, 'lxml')
    cities = [item['href'] for item in soup.select('#browse_view a')]
    for c in cities:
        r = s.get(c)
        s1 = bs(r.content,'lxml')
        categories = [item['href'] for item in s1.select('.three_browse_columns:nth-of-type(2) a')]
        for c1 in categories:
            r1 = s.get(c1)
            s2 = bs(r1.content,'lxml')
            lawyers = [item['href'].split('*')[1] if '*' in item['href'] else item['href'] for item in s2.select('.indigo_text .directory_profile')]
            final.append(lawyers)
final_list = {item for sublist in final for item in sublist}

Answer 3

我尝试了以下方法：

import requests
from bs4 import BeautifulSoup as bs
import pandas as pd

res = requests.get('https://attorneys.superlawyers.com/tennessee/', headers = {'User-agent': 'Super Bot 9000'})
soup = bs(res.content, 'lxml')

cities = [item['href'] for item in soup.select('#browse_view a')]
for c in cities:
    r=requests.get(c)
    s1=bs(r.content,'lxml')
    categories = [item['href'] for item in s1.select('.three_browse_columns:nth-of-type(2) a')]
    #print(categories)
    for c1 in categories:
        r1=requests.get(c1)
        s2=bs(r1.content,'lxml')
        lawyers = [item['href'] for item in s2.select('#lawyer_0_main a')]
        print(lawyers)

“它不仅打印个人资料链接，还打印关于和其他不需要的相关链接。我只想要律师的个人资料链接。”

"输出显示为"

"`['https://profiles.superlawyers.com/tennessee/alamo/lawyer/jim-emison/c99a7c4f-3a42-4953-9260-3750f46ed4bd.html', 'https://www.superlawyers.com/about/selection_process.html']
['https://profiles.superlawyers.com/tennessee/alamo/lawyer/jim-emison/c99a7c4f-3a42-4953-9260-3750f46ed4bd.html', 'https://www.superlawyers.com/about/selection_process.html']
['https://profiles.superlawyers.com/tennessee/alamo/lawyer/jim-emison/c99a7c4f-3a42-4953-9260-3750f46ed4bd.html', 'https://www.superlawyers.com/about/selection_process.html']`"