使用 BeautifulSoup 从 javascript 函数中收集返回值

Question

我想使用 BeautifulSoup 从以下网址收集代理商电话号码：https://www.cv-library.co.uk/companies/agencies/0-9。

但问题是，我必须先单击一个链接，该链接绑定到一个名为 "contactDetails()" 的 JavaScript 函数才能显示一个数字。我设法使用 Selenium 单击所有链接。但是我现在如何收集这些数字？

那么，我现在应该怎么做才能克服这个问题？

提前致谢。

注意：我是网络抓取的新手。

import requests,bs4
from selenium import webdriver

site_url = "https://www.cv-library.co.uk/companies/agencies/0-9"

#---------------------------------- Opening Firefox with Selenium Webdrivre ---------------
#browser = webdriver.Firefox() 
#I need my Firefox browser's current profile for a reason.
profile = webdriver.FirefoxProfile(r"C:\Users\USER\AppData\Roaming\Mozilla\Firefox\Profiles\i27jf7iw.default")
browser = webdriver.Firefox(firefox_profile=profile)
browser.get(site_url)

#---------------------------------- Clicking Phone Buttons ---------------------
phone_btn = browser.find_elements_by_link_text("Phone - Click to View")
for i in range(0,20):
    phone_btn[i].click()

Answer 1

它点击所有按钮，然后得到数字。

但经过几次测试后，我得到“已达到联系人详细信息视图限制”:) 所以点击次数是有限制的。

from selenium import webdriver

site_url = "https://www.cv-library.co.uk/companies/agencies/0-9"

# --- Opening Firefox with Selenium Webdrivre ---

browser = webdriver.Firefox() 

#I need my Firefox browser's current profile for a reason.
#profile = webdriver.FirefoxProfile(r"C:\Users\USER\AppData\Roaming\Mozilla\Firefox\Profiles\i27jf7iw.default")
#browser = webdriver.Firefox(firefox_profile=profile)

browser.get(site_url)

# --- Clicking Phone Buttons ---

phone_btn = browser.find_elements_by_link_text("Phone - Click to View")
for btn in phone_btn:
    btn.click()

numbers = browser.find_elements_by_class_name('company-profile-phone')
for num in numbers:
    print('number:', num.text)

---

没有Selenium的版本

每个链接"Phone - click to View" 在属性onclick（即contactDetails( this, 154513 )）中都有编号，JavaScript 使用该编号从服务器获取编号，始终使用与该编号相同的URL - 即。 https://www.cv-library.co.uk/account-contact-details?id=1545‌​13。

它工作了一段时间 - 可能我达到了点击的限制:)

import requests
import bs4

site_url = "https://www.cv-library.co.uk/companies/agencies/0-9"
phone_url = "https://www.cv-library.co.uk/account-contact-details?id="

session = requests.Session()
session.headers.update({
    #"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", 
#    "Accept-Encoding": "gzip, deflate", 
    #"Accept-Language": "pl,en-US;q=0.7,en;q=0.3", 
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0"
})
print(session.headers)

r = session.get(site_url)
print(r.status_code)
soup = bs4.BeautifulSoup(r.text, 'html.parser')

#print(r.text)
all_p = soup.find_all('p', class_='company-profile-phone')

for p in all_p:
    number = p.a['onclick'][22:-2]
    print('Phone ID:', number)

    session.headers.update({
        'X-Requested-With': 'XMLHttpRequest',
        'Accept': 'application/json, text/javascript, */*; q=0.01',
    })

    r = session.get(phone_url + number)

    if r.status_code != 200:
        print("Contact details view limit reached")
    else:
        data = r.json()

        if "email" in data:
            print('email:', data['email'])
        if "phone" in data:
            print('phone:', data['telephone'])
    print('---')