想要从给定站点抓取所有成员的个人资料链接

Question

from bs4 import BeautifulSoup
import requests

r = requests.get('http://iacts.org/members/')
soup = BeautifulSoup(r.text,'lxml')

for link in soup.find('div',id_='members-dir-list'):
  for link1 in link.find_all('div',class_='item'):
     for link2 in link1.find_all('a'):
        for link3 in link2.find('href'):
            print(link3.text)

没有获得会员链接，想从给定网站上抓取会员资料链接。

Answer 1

试试下面的代码。您可以打开 chrome 开发人员工具 (F12) 并查看单击搜索按钮时发送到 post 请求的表单数据。然后通过在请求中传递数据来模拟请求。

import requests
import lxml
from lxml import html

data = {
    "iacts_search": "Search",
    "bp_profile_search": "true"
}

r = requests.post('http://iacts.org/members/', data=data)

tree = html.fromstring(r.content)

member_list = tree.xpath('//*[@id="members-list"]//div[@class="item-title"]//a')

# print(len(member_list))

for member in member_list:
    print(member.get('href'))

结果：

http://iacts.org/members/Jc
http://iacts.org/members/Jc2
http://iacts.org/members/Ramesh-test
http://iacts.org/members/Raj
http://iacts.org/members/Arvind
http://iacts.org/members/Nhf
http://iacts.org/members/Man
http://iacts.org/members/Srujan
http://iacts.org/members/Vikas
http://iacts.org/members/Manohar
http://iacts.org/members/Sanjeeva Rao
http://iacts.org/members/Atul Kumar Gupta
http://iacts.org/members/Charlesmundwp
http://iacts.org/members/Amjad M
http://iacts.org/members/Dr Mrithyunjaya V
http://iacts.org/members/Siddharth