如何在 Beautifulsoup 中实际使用循环？

Question

我正在尝试制作一个网络爬虫。我在里面使用了一些循环。该循环在第一个循环中运行良好，但在第二个循环中运行良好。我总是收到这条消息：“在处理上述异常期间，发生了另一个异常”

import requests
from bs4 import BeautifulSoup


result = 
requests.get("http://desaku.bandungkab.go.id/desaonline/")
#This url is the main web, inside this web there are 270 links of 
#other website. I get into that 270 webs and open every article in 
#each 
web
src = result.content
soup = BeautifulSoup(src, 'lxml')
links = soup.find_all('a')
urls = []
for link in links:
    if "www" in link.text:
        url = link.attrs['href']
        urls.append(url)


num1=len(urls)
b=0
while b<num1:
    result2 = requests.get(urls[b])
    src2 = result2.content
    soup = BeautifulSoup(src2, 'lxml')
    links2 = soup.find_all('a')
    urls2 = []
    for link in links2:
        if "selengkapnya" in link.text:
            url2 = link.attrs['href']
            urls2.append(url2)
    b+=1
#the code run well until this part. If i print this, it will result 
#url that take me directly to specific article

num=len(urls2)
i=0
while i<num:
    result2 = requests.get(urls2[i])
    src2 = result2.content
    soup = BeautifulSoup(src2, 'lxml')
    links2 = soup.find_all('a')
    artikel=[]
    isi = link.attrs['href']
    artikel.append(isi)
    print(artikel)
    i+=1

我希望从网站上获取文章的所有链接并将它们放入名为 artikel=[] 的列表中

Answer 1

问题是您在每次循环迭代时都分配给urls2 = [] 和artikel=[]，从而有效地替换了列表。在迭代结束时，您有空数组。您可以使用此代码作为开始：

import requests
from bs4 import BeautifulSoup

result = requests.get("http://desaku.bandungkab.go.id/desaonline/")
#This url is the main web, inside this web there are 270 links of
#other website. I get into that 270 webs and open every article in
#each web

src = result.content
soup = BeautifulSoup(src, 'lxml')

urls = []
urls2 = []

for link in soup.select('a[href]:contains(www)'):
    urls.append(link['href'])

print('Urls:')
for url in urls:
    print('Downloading {}'.format(url))
    result2 = requests.get(url)
    src2 = result2.content
    soup = BeautifulSoup(src2, 'lxml')
    for link in soup.select('a[href]:contains(selengkapnya)'):
        print('\tFound link {}'.format(link['href']))
        urls2.append(link['href'])

print('Articles:')
articles = []
for url2 in urls2:
    print('Downloading {}'.format(url2))
    result2 = requests.get(url2)
    src2 = result2.content
    soup = BeautifulSoup(src2, 'lxml')
    for link in soup.find_all('a[href]'):
        articles.append(link['href'])

print(articles)

打印：

Urls:
Downloading http://www.ancolmekar.desa.id
    Found link http://www.ancolmekar.desa.id/first/artikel/423
    Found link http://www.ancolmekar.desa.id/first/artikel/421
    Found link http://www.ancolmekar.desa.id/first/artikel/420
    Found link http://www.ancolmekar.desa.id/first/artikel/419
    Found link http://www.ancolmekar.desa.id/first/artikel/414
    Found link http://www.ancolmekar.desa.id/first/artikel/413
    Found link http://www.ancolmekar.desa.id/first/artikel/412
    Found link http://www.ancolmekar.desa.id/first/artikel/410
    Found link http://www.ancolmekar.desa.id/first/artikel/410
    Found link http://www.ancolmekar.desa.id/first/artikel/100
Downloading http://www.arjasari.desa.id
    Found link http://www.arjasari.desa.id/first/artikel/180
    Found link http://www.arjasari.desa.id/first/artikel/190

...and so on.