从 url 获取数据并将其放入 DataFrame

Question

大家好，我目前正在尝试从 url 获取一些数据，然后尝试预测该文章应该属于哪个类别。 到目前为止，我已经这样做了，但它有一个错误：

    info = pd.read_csv('labeled_urls.tsv',sep='\t',header=None)
    html, category = [], []
    for i in info.index:
        response = requests.get(info.iloc[i,0])
        soup = BeautifulSoup(response.text, 'html.parser')
        html.append([re.sub(r'<.*?>','', 
                      str(soup.findAll(['p','h1','\href="/avtorji/'])))])
        category.append(info.iloc[0,i])

    data = pd.DataFrame()
    data['html'] = html
    data['category'] = category

错误是这样的：

IndexError: 单个位置索引器超出范围。

有人可以帮帮我吗？

Answer 1

您可以避免 iloc 调用并改用iterrows，我认为您将不得不使用loc 而不是iloc，因为您正在对索引进行操作，但在使用iloc 和loc循环通常效率不高。您可以尝试以下代码（插入等待时间）：

import time

info = pd.read_csv('labeled_urls.tsv',sep='\t',header=None)
html, category = [], []
for i, row in info.iterrows():
    url= row.iloc[0]
    time.sleep(2.5)  # wait 2.5 seconds
    response = requests.get(url)  # you can use row[columnname] instead here as well (i only use iloc, because I don't know the column names)
    soup = BeautifulSoup(response.text, 'html.parser')
    html.append([re.sub(r'<.*?>','', 
                  str(soup.findAll(['p','h1','\href="/avtorji/'])))])
    # the following iloc was probably raising the error, because you access the ith column in the first row of your df
    # category.append(info.iloc[0,i])
    category.append(row.iloc[0])  # not sure which field you wanted to access here, you should also replace it by row['name']

data = pd.DataFrame()
data['html'] = html
data['category'] = category

如果您真的只需要循环中的 url，请替换：

for i, row in info.iterrows():
    url= row.iloc[0]

通过类似的方式：

for url in info[put_the_name_of_the_url_column_here]: # or info.iloc[:,0] as proposed by serge

Answer 2

错误可能是由于将索引传递给iloc 引起的：loc 期望索引值和列名，而iloc 期望行和列的数字位置。此外，您已将category 的行和列位置与category.append(info.iloc[0,i]) 互换。所以你至少应该这样做：

for i in range(len(info)):
    response = requests.get(info.iloc[i,0])
    ...
    category.append(info.iloc[i,0])

但是当您尝试迭代数据框的第一列时，上面的代码不是 Pythonic。最好直接使用列：

for url in info.loc[:, 0]:
    response = requests.get(url)
    ...
    category.append(url)