Python BeautifulSoup findAll 不返回所有元素？

Question

我正在尝试从该 URL https://99airdrops.com/page/1/ 中提取数据。

我写的代码如下。

import requests
from bs4 import BeautifulSoup

url_str = 'https://99airdrops.com/page/1/'

page = requests.get(url_str, headers={'User-Agent': 'Mozilla Firefox'})

# soup = BeautifulSoup(page.text, 'lxml')
soup = BeautifulSoup(page.text, 'html.parser')

# print(soup.prettify())

print(len(soup.findAll('div')))

print(soup.find('div', class_='title'))

我的问题是print(len(soup.findAll('div'))) 行仅返回 23，而print(soup.find('div', class_='title')) 行打印None。即使有多个实例，find 命令也找不到带有class_='title' 的 div 元素，并且 div 元素深深嵌套在 html 页面中，但这从未给我带来过问题。

我尝试过使用lxml 和html.parser，但都没有返回所有的div 元素。我还尝试将 html 写入文件，将其读入，然后使用它运行 BeautifulSoup，但我得到了相同的结果。谁能告诉我这是什么问题？

我也尝试了这里的建议Beautiful Soup - `findAll` not capturing all tags in SVG (`ElementTree` does) 来更新我的 lxml 包，但我仍然遇到了同样的问题。

我也尝试了BeautifulSoup doesn't find correctly parsed elements这里的解决方案，但没有运气。

Answer 1

您似乎可以通过一个请求获得您正在寻找的所有数据。

>>> import requests
>>> r = requests.get('https://cdn.99airdrops.com/static/airdrops.json')
>>> data = r.json()
>>> len(data)
133

例如：

>>> import json; print(json.dumps(data.popitem(), indent=2))
[
  "pointium",
  {
    "unique": "pointium",
    "name": "Pointium",
    "currency": "PNT",
    "description": "Global Decentralized Platform for Point Management & Loyalty Program",
    "instructions": "<ol><li>Join Telegram <a href=\"https://t.me/pointium\" target=\"_blank\">@Pointium</a> and click \"Join Airdrop\" (+500 PNT) </li><li>Enter your e-mail (+200 PNT) </li><li><a href=\"https://twitter.com/POINTIUM_ICO\" target=\"_blank\">Follow Twitter</a> and submit your username (+500 PNT) </li><li>Confirm your details</li></ol>",
    "rating": "7.30",
    "addDate": "2018-04-20 06:23:03",
    "expirationDate": "2018-05-07",
    "startDate": "2018-04-07",
    "image": "https://cdn.99airdrops.com/static/pointium.jpeg",
    "joinLink": "https://www.pointium.org/airdrop",
    "sponsored": "0",
    "status": "0",
    "startDateFormatted": "7th of April",
    "expirationDateFormatted": "7th of May",
    "attributes": {
      "bitcointalk": "0",
      "category": "airdrop",
      "email": "1",
      "facebook": "0",
      "kyc": "0",
      "news": "https://twitter.com/POINTIUM_ICO",
      "opinion": "O parere personala este ca merge acest sistem foarte bine. Doar ca mai avem de lucrat la el sa fie bomba!",
      "other": "0",
      "phone": "0",
      "ratingConcept": "7",
      "ratingTeam": "5.5",
      "ratingWebsite": "7",
      "ratingWhitepaper": "8",
      "reddit": "0",
      "telegram": "1",
      "tokenGiven": "1200",
      "tokenPrice": "0.007",
      "tokenSupply": "1,600,000,000",
      "tokenType": "ERC20",
      "twitter": "1",
      "website": "www.pointium.org"
    }
  }
]