sec.gov 使用嵌套的 for 循环进行抓取，遇到 Unicode 错误

Question

我正在尝试获取 sec 以获取通过输入选择的公司的所有 10-K 归档链接。 该程序从 1993 年至今每年的每个季度（QTR1-4）循环。 我从https://codingandfun.com/scraping-sec-edgar-python/ 得到了代码 执行时遇到：UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 13013584: invalid continuation byte 如果没有我多年来的 for 循环和固定的年份/季度，它可以工作 - 那么这里的问题是什么？

import bs4 as bs
import requests
import pandas as pd
import re
from datetime import datetime

def get_base():
    company = input('Which company?: ')
    filing = '10-K'
    year = [*range(1993,datetime.now().year + 1)]
    quarter = ['QTR1','QTR2','QTR3','QTR4']
    #get all filings for each quarter(QTR1-4) in each year(beginning 1993 until actual year)
    for x in year:
        for y in quarter:
            download = requests.get(f'https://www.sec.gov/Archives/edgar/full-index/{x}/{y}/master.idx').content
            download = download.decode("utf-8").split('\n')
            for item in download:
                #company name and report type
                if (company in item) and (filing in item): 
                   
                    company = item
                    company = company.strip()
                    splitted_company = company.split('|')
                    url = splitted_company[-1]
                    
                    #build second part of the url
                    url2 = url.split('-') 
                    url2 = url2[0] + url2[1] + url2[2]
                    url2 = url2.split('.txt')[0] 

                    # build third part of the url
                    to_get_html_site = 'https://www.sec.gov/Archives/' + url
                    data = requests.get(to_get_html_site).content
                    data = data.decode("utf-8") 
                    data = data.split('FILENAME>')
                    data = data[1].split('\n')[0]

                    #combine
                    url_to_use = 'https://www.sec.gov/Archives/'+ url2 + '/'+data
                    print(url_to_use)
                    

get_base()

Answer 1

您需要指定标题：

headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"}

download = requests.get(f'https://www.sec.gov/Archives/edgar/full-index/{x}/{y}/master.idx', headers=headers).text

在行尾将 .content 更改为 .text。 对于下载文件的过滤器标题，将过滤器视为：

data = [row.split('|') for row in download.split("\n") if '|' in row]
data_10k = [row for row in data if row[2] == "10-K"]
data_10Q = [row for row in data if row[2] == "10-Q"]

我希望这能够让您在工作中畅通无阻，但我认为这不是一种有效的方式来做您想做的事情。 （可能很长，你需要知道公司的确切名称......）