如何使用漂亮的汤从存储在列表中的链接中抓取内容？

Question

我想抓取主网站上所有帖子的标题。 main 是一个包含 6 或 7 个 url 的列表：

import requests 
from bs4 import Beautifulsoup 

r=requests.get("https://forums.oneplus.com/")
s=BeautifulSoup(r.content)
links=s.find_all("a",{"class" : "focus-content"})
url2=[] 
for link in links:
    url2.append(link.get("href"))

url1="https://forums.oneplus.com/"
for u in url2:
    main=url1+u
    print(main)

for m in main:
    r1=requests.get(m)
    s1=BeautifulSoup(r1)
    title=s1.find("span", {"class" : "title"})
    print(title)

Answer 1

您需要将变量main 声明为列表。在您的代码中，您在循环的每次迭代中更新变量main。最后，main 将是包含来自url2 列表的最后一个 url 的字符串。如果您随后将 main 提供给下一个循环，它将遍历单个字符。

经过一些外观上的改变，这应该会得到你的标题：

import requests
from bs4 import BeautifulSoup

r=requests.get("https://forums.oneplus.com/")
s=BeautifulSoup(r.content, 'lxml')
links=s.find_all("a",{"class" : "focus-content"})
url2=[]
for link in links:
    url2.append(link.get("href"))

url1 = "https://forums.oneplus.com/"
main = []
for u in url2:
    main.append(url1 + u)

for m in main:
    r1 = requests.get(m)
    s1 = BeautifulSoup(r1.text, 'lxml')
    title=s1.find("span", {"class" : "title"})
    print(title.text.strip())

打印：

Weekly 240: We release the updates and get reading
Shot on OnePlus: Part 6 – Best Slo-mo Video / Animal Photos
Android P Beta Developer Preview 3 for OnePlus 6
[Let's Talk] To whom are you gonna give your appreciation in this Community?
[Let's Talk] What Does Your OnePlus Device Replace?
[Let's Talk]  Loyalty to tech companies