有没有办法使用 bs4 获取具有特定属性的标签中的文本？

Question

假设我在 HTML 文件中有这个：

<a rel="nofollow" class="result__a" href="some_link">Foo-baz</a>

如何使用 bs4 仅提取 Foo-baz？

现在，我可以使用 href 属性获得

headers = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:84.0) Gecko/20100101 Firefox/84.0",
}

page = requests.get('the_url', headers=headers).text
soup = BeautifulSoup(page, 'html.parser').find_all("a", class_="result__a")

for link in soup:
    print(link['href'])

但是，我无法提取具有这些特定属性的特定标签中的单词。
我已经尝试了文档和 StackOverflow 中的不同解决方案，但它们似乎不起作用。或者我可能无法执行它，因为我是 bs4 的新手。

链接是：https://html.duckduckgo.com/html/?q=test

感谢任何帮助。

Answer 1

用link.text代替link['href']：

import requests
from bs4 import BeautifulSoup

url = "https://html.duckduckgo.com/html/?q=test"

headers = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:84.0) Gecko/20100101 Firefox/84.0",
}

page = requests.get(url, headers=headers).text
soup = BeautifulSoup(page, "html.parser").find_all("a", class_="result__a")

for link in soup:
    print(link.text)

打印：

Test | Definition of Test by Merriam-Webster
Speedtest by Ookla - The Global Broadband Speed Test
Internet Speed Test | Fast.com
Tests.com Practice Tests
Speed test
Testing for COVID-19 | CDC
Xfinity Speed Test - Check Your Internet Speed
Internet Speed Test - AT&T Official Site
Speed Test by Speedcheck - Test your internet speed
Mic Test
Test - definition of test by The Free Dictionary
CPS Test - Check Clicks per Second
Free Personality Test | 16Personalities
CEA Test: MedlinePlus Medical Test
Test Innovators | Prep for Success
Join a Test Meeting - Zoom
TEST Synonyms: 83 Synonyms & Antonyms for TEST | Thesaurus.com
Speakeasy Internet Speed Test - Check Your Broadband Speed ...
Speed Test - Telstra
IQTest.com--The Original Free Online IQ Test
Speedtest by Ookla - Teste de Velocidade de Conexão da ...
Speedtest - Google Search
ADHD Test - Psych Central
Cisco Webex | Test online meeting
Login | Salesforce
Practice Tests, Tutoring & Prep Courses | Kaplan Test Prep
test | Origin and meaning of test by Online Etymology ...
Test English - Prepare for your English exam
Internet Speed Test - Check Your Internet Speed | Cox
A1C test - Mayo Clinic

Answer 2

查看SelectorGadget Chrome 扩展以获取CSS 选择器。

如果你想抓取 javascript 版本的 DuckDuckGo，你可以使用 request-html 或 selenium。 注意：这不是最快的解决方案。

您可以通过解析来自<script> 标记的数据来实现，但这需要更多的工作。

代码（非javascript）：

import requests, lxml
from bs4 import BeautifulSoup

headers = {
    'User-agent':
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
}

html = requests.get('https://html.duckduckgo.com/html/?q=test&kl=us-en', headers=headers)
soup = BeautifulSoup(html.text, 'lxml')

for result in soup.select('.result__body'):
  title = result.select_one('.result__a').text
  url = result.select_one('.result__snippet')['href'].replace('//', '')
  snippet = result.select_one('.result__snippet').text
  print(f"{title}\n{url}\n{snippet}\n")

-----
'''
Test | Definition of Test by Merriam-Webster
duckduckgo.com/l/?uddg=https%3A%2F%2Fwww.merriam%2Dwebster.com%2Fdictionary%2Ftest&rut=4749db61adf540ccd15b6f5aa6a68d4e6604b2c86e5bffe0a5380e06513eff93
Test definition is - a means of testing: such as. How to use test in a sentence.
...
'''

---

或者，您可以使用来自 SerpApi 的 DuckDuckGo Organic Results API。这是一个带有免费计划的付费 API。

不同之处在于，您在此处解析的是非 JavaScript DuckDuckGo 页面（请参阅此 URL，您将看到“最新消息”、“内联图像”结果），而 SerpApi 是解析页面的javascript版本。而唯一真正需要做的就是遍历 JSON 字符串并获取您想要的数据。

要集成的代码：

import json, os
from serpapi import GoogleSearch

params = {
  "api_key": os.getenv("API_KEY"),
  "engine": "duckduckgo",
  "q": "fus ro dah",
  "kl": "us-en"
}

search = GoogleSearch(params)
results = search.get_dict()

for result in results['organic_results']:
    print(json.dumps(result, indent=2, ensure_ascii=False))

------------
'''
{
  "position": 1,
  "title": "Test | Definition of Test by Merriam-Webster",
  "link": "https://www.merriam-webster.com/dictionary/test",
  "snippet": "Test definition is - a means of testing: such as. How to use test in a sentence.",
  "favicon": "https://external-content.duckduckgo.com/ip3/www.merriam-webster.com.ico"
}
...
'''

免责声明，我为 SerpApi 工作。