NoneType 不可迭代，使用 Python 3.8 进行网络抓取时导致错误 [重复]

Question

我目前的任务是制作一个用于拉取链接的网络爬虫。我可以成功提取这些数据：

/
/users/sign_up
/topics
/smarties
/posts
/users/sign_in
/users/sign_up
/posts/installing-anaconda-python-data-science-platform
/topics/python
/topics/anaconda-python
/topics/machine-learning
/jordan
/posts/python-libraries-to-import-for-data-science-programs
/topics/python
/topics/data-science
/topics/machine-learning
/jordan
/posts/shortcut-for-opening-the-object-inspector-in-python-spyder
/topics/python
/topics/anaconda-python
/topics/spyder-python
/topics/machine-learning
/jordan
/posts/python-script-for-replacing-missing-data-in-a-machine-learning-algorithm
/topics/machine-learning
/topics/python
/jordan
/posts/python-script-for-pulling-in-the-same-column-from-an-array-of-arrays
/topics/python
/jordan
/posts/how-to-implement-fizzbuzz-in-python
/topics/fizzbuzz
/topics/python
/jordan
/posts/how-to-think-like-a-computer-scientist
/topics/computer-science
/topics/python
/topics/programming
/jordan
/posts/base-case-example-for-how-to-test-a-python-class
/topics/python
/topics/tdd
/jordan
/posts/installing-and-working-with-pipenv
/topics/pipenv
/topics/python
/jordan
/posts/steps-for-building-a-flask-api-application-with-python-3
/topics/flask
/topics/tutorial
/topics/python
/jordan
None
/topics/python?page=2
/topics/python?page=3
/topics/python?page=4
/topics/python?page=2
/topics/python?page=4

在我运行这段代码之后

import requests
from bs4 import BeautifulSoup as bs

r = requests.get('http://www.dailysmarty.com/topics/python')

soup = bs(r.text, 'html.parser')

for link in soup.find_all('a'):
    print(link.get('href'))

但是当我运行我正在开发的这个生成器时：

def generator(web):
    titles = []
    for link in web:
        if 'posts' in link.get('href'):
            print(link.get('href'))
        else:
            pass


data = soup.find_all('a')
#generator(data)

我得到了这些数据和这些回调错误：

/posts
/posts/installing-anaconda-python-data-science-platform
/posts/python-libraries-to-import-for-data-science-programs
/posts/shortcut-for-opening-the-object-inspector-in-python-spyder
/posts/python-script-for-replacing-missing-data-in-a-machine-learning-algorithm
/posts/python-script-for-pulling-in-the-same-column-from-an-array-of-arrays
/posts/how-to-implement-fizzbuzz-in-python
/posts/how-to-think-like-a-computer-scientist
/posts/base-case-example-for-how-to-test-a-python-class
/posts/installing-and-working-with-pipenv
/posts/steps-for-building-a-flask-api-application-with-python-3
Traceback (most recent call last):
  File "C:\Users\joshu\AppData\Local\Programs\Python\Python38\classes.py", line 18, in <module>
    generator(data)
  File "C:\Users\joshu\AppData\Local\Programs\Python\Python38\classes.py", line 13, in generator
    if 'posts' in link.get('href'):
TypeError: argument of type 'NoneType' is not iterable

我怎样才能做到这一点，以便当我运行我的生成器时，我可以在我的 for 循环中传递 None 而不会导致它在代码中给我错误？

Answer 1

您必须检查链接是否确实具有"href" 属性：

if link.has_attr('href') and 'posts' in link.get('href'):