在没有硒和 API 的情况下抓取linkedin 配置文件

Question

我想通过 URL 抓取 LinkedIn 的个人资料

喜欢; https://www.linkedin.com/in/andrew-marson-90a74015/

我想从中获取一些数据

我以前使用过 selenum，但我想让它更快

所以我想使用 request.get(url, auth=**** ) 但是要获取您需要登录的数据，我可以通过 auth = ('user', 'pass')

获取页面

import requests
from requests.auth import HTTPBasicAuth
test = requests.get('https://www.linkedin.com/in/andrew-marson-90a74015/', auth=HTTPBasicAuth('user', 'pass'))
print(test.text)

Answer 1

import requests
from bs4 import BeautifulSoup

email = ""
password = ""

client = requests.Session()

HOMEPAGE_URL = 'https://www.linkedin.com'
LOGIN_URL = 'https://www.linkedin.com/uas/login'

html = client.get(HOMEPAGE_URL).content
soup = BeautifulSoup(html, "html.parser")
csrf = soup.find('input', {'name': 'loginCsrfParam'}).get('value')

login_information = {
    'session_key': email,
    'session_password': password,
    'loginCsrfParam': csrf,
    'trk': 'guest_homepage-basic_sign-in-submit'
}

client.post(LOGIN_URL, data=login_information)

response = client.get('https://www.linkedin.com/in/andrew-marson-90a74015/')
print(response.text)

此代码无效

Answer 2

使用beautifulsoup 或scrapy，两者都适用于不需要硒的任务。

from requests import Session
from bs4 import BeautifulSoup as bs

with Session() as s:
    site = s.get("https://www.linkedin.com/")
    bs_content = bs(site.content, "html.parser")
    token = bs_content.find("input", {"name": "loginCsrfParam"})["value"]
    login_data = {"username": "admin",
                  "password": "12345", "loginCsrfParam": token}
    s.post("https://www.linkedin.com/login", login_data)
    home_page = s.get("https://www.linkedin.com/")
    print(home_page.content)