使用 python 和 beautifulsoup 获取联系信息答案

【问题标题】：Grabbing contact information with python and beautifulsoup使用 python 和 beautifulsoup 获取联系信息
【发布时间】：2019-07-06 23:36:51
【问题描述】：

我正在尝试从页面中获取联系信息。我需要姓名、职位、电话和电子邮件地址。

我正在学习 Python，并尝试根据我知道的数据编写代码。我能够提取包含各个联系人的 div 块，但我不确定一旦拥有它们后如何爬过它们。

tags = soup.find_all('div', attrs={'class':'tshowcase-inner-box'})

但后来我想爬过儿童 div，但没有运气。

    fullname = soup.find('div', attrs={'class':'tshowcase-box-title'})
    title = soup('div', attrs={'class':'tshowcase-single-position'})
    phone = soup('div', attrs={'class':'tshowcase-single-telephone'})
    email = soup('div', attrs={'class':'tshowcase-box-social'})

我不确定接下来会发生什么，并感谢任何指点。

这里是示例 HTML：

<div class="tshowcase-inner-box ts-float-left ">
    <div class="tshowcase-box-info ts-align-left  ">
        <div class="tshowcase-box-title">FULL NAME</div>
        <div class="tshowcase-box-details">
            <div class="tshowcase-single-position"><i class="fa fa-chevron-circle-right"></i>JOB TITLE</div>
            <div class="tshowcase-single-telephone"><i class="fa fa-phone-square"></i><a href="tel:PHONE">PHONE</a></div>
        </div>
        <div class="tshowcase-box-social"><a href="mailto:EMAIL" rel="nofollow" target="_blank"><i class="fa fa-envelope-o fa-lg"></i></a></div>
    </div>
</div>

【问题讨论】：

是否有与此相关的公共网址？您只想要全名等的第一个匹配项（每页只有一个匹配项吗？）？

标签： python html beautifulsoup

【解决方案1】：

如果您对每个列表进行循环，您可以测试是否存在并采取相应措施

from bs4 import BeautifulSoup as bs
import requests

html = '''
<div class="tshowcase-inner-box ts-float-left ">
    <div class="tshowcase-box-info ts-align-left  ">
        <div class="tshowcase-box-title">FULL NAME</div>
        <div class="tshowcase-box-details">
            <div class="tshowcase-single-position"><i class="fa fa-chevron-circle-right"></i>JOB TITLE</div>
            <div class="tshowcase-single-telephone"><i class="fa fa-phone-square"></i><a href="tel:PHONE">PHONE</a></div>
        </div>
        <div class="tshowcase-box-social"><a href="mailto:EMAIL" rel="nofollow" target="_blank"><i class="fa fa-envelope-o fa-lg"></i></a></div>
    </div>
</div>
<div class="tshowcase-inner-box ts-float-left ">
    <div class="tshowcase-box-info ts-align-left  ">
        <div class="tshowcase-box-title">FULL NAME2</div>
        <div class="tshowcase-box-details">
            <div class="tshowcase-single-position"><i class="fa fa-chevron-circle-right"></i>JOB TITLE2</div>
            <div class="tshowcase-single-telephone"><i class="fa fa-phone-square"></i><a href="tel:PHONE">PHONE2</a></div>
        </div>
        <div class="tshowcase-box-social"><a href="mailto:EMAIL2" rel="nofollow" target="_blank"><i class="fa fa-envelope-o fa-lg"></i></a></div>
    </div>
</div>
'''
soup = bs(html, 'lxml')
results = []

for listing in soup.select('.tshowcase-inner-box'):
    name = listing.select_one('.tshowcase-box-title')
    job = listing.select_one('.tshowcase-single-position')
    tel = listing.select_one('.tshowcase-single-telephone')
    email = listing.select_one('[href^=mailto]')
    if name is None:
        name = 'Not present'
    else:
        name = name.text
    if job is None:
        job = 'Not present'
    else:
        job = job.text
    if tel is None:
        tel = 'Not present'
    else:
        tel = tel.text
    if email is None:
        email = 'Not present'
    else:
        email = email['href'].replace('mailto:','')
    results.append({ 'name' : name, 'job' : job, 'tel': tel, 'email': email })
print(results)

【讨论】：

【解决方案2】：

您可以使用soup.find_all 定位元素，然后访问text 和href 值：

from bs4 import BeautifulSoup as soup
import re
d = soup(html, 'html.parser')
s = [i.text for i in d.find_all('div', {'class':re.compile('title$|position$|telephone$')})]
result = [*s, d.find('div', {'class':'tshowcase-box-social'}).a['href'][7:]]

输出：

['FULL NAME', 'JOB TITLE', 'PHONE', 'EMAIL']

如果您尝试抓取页面上的多个联系人块，您可以将上面的代码转换为接受bs4 对象的函数以抓取单个列表并遍历所有块divs：

def get_contact(d):
   s = [i.text for i in d.find_all('div', {'class':re.compile('title$|position$|telephone$')})]
   return [*s, d.find('div', {'class':'tshowcase-box-social'}).a['href'][7:]]

results = [get_contact(i) for i in soup(html, 'html.parser').find_all('div', {'class':'tshowcase-inner-box'})]

输出：

[['FULL NAME', 'JOB TITLE', 'PHONE', 'EMAIL']]

【讨论】：

我想我按照你在做的，但我收到一个错误：results = [get_contact(i) for i in soup(html, 'html.parser').find_all('div', {'class':'tshowcase-inner-box'})] File "python3.7/site-packages/beautifulsoup4-4.7.1-py3.7.egg/bs4/element.py", line 1620, in __getattr__ "ResultSet object has no attribute '%s'. You're probably treating a list of items like a single item. Did you call find_all() when you meant to call find()?" % key
@NicoleC.Baratta 我建议升级BeautifulSoup: pip install beautifulsoup4 --upgrade。

【解决方案3】：

我获取联系方式的版本：

data = '''<div class="tshowcase-inner-box ts-float-left ">
    <div class="tshowcase-box-info ts-align-left  ">
        <div class="tshowcase-box-title">FULL NAME</div>
        <div class="tshowcase-box-details">
            <div class="tshowcase-single-position"><i class="fa fa-chevron-circle-right"></i>JOB TITLE</div>
            <div class="tshowcase-single-telephone"><i class="fa fa-phone-square"></i><a href="tel:PHONE">PHONE</a></div>
        </div>
        <div class="tshowcase-box-social"><a href="mailto:EMAIL" rel="nofollow" target="_blank"><i class="fa fa-envelope-o fa-lg"></i></a></div>
    </div>
</div>'''

from bs4 import BeautifulSoup

soup = BeautifulSoup(data, 'lxml')

data = []
for div in soup.select('.tshowcase-inner-box'):
    data.append([])
    data[-1].extend(txt.strip() for txt in div.get_text(separator='|').split('|') if txt.strip())
    data[-1].extend(a['href'].replace('mailto:', '') for a in div.select('a[href*="mailto:"]'))

print(data)

打印：

[['FULL NAME', 'JOB TITLE', 'PHONE', 'EMAIL']]

【讨论】：