我在抓取多个 URL 时遇到问题

Question

我在抓取多个网址时遇到问题。基本上我只能为一种类型运行它，但第二种我包含它停止工作的其他链接。

目标是获取数据并将其放入包含电影标题、网址和流派的 csv 文件中。任何帮助将不胜感激！

from urllib.request import urlopen as uReq
from bs4 import BeautifulSoup as soup

my_url = ['https://www.netflix.com/browse/genre/1365', 'https://www.netflix.com/browse/genre/7424']

uClient = uReq(my_url)
page_html = uClient.read()
uClient.close()

page_soup = soup(page_html,"html.parser")

containers = page_soup.findAll("li",{"class":"nm-content-horizontal-row-item"})


# name the output file to write to local disk
out_filename = "netflixaction2.csv"
# header of csv file to be written
headers = "Movie_Name, Movie_ID \n"

# opens file, and writes headers
f = open(out_filename, "w")
f.write(headers)



for container in containers:
    
    title_container = container.findAll("a",{"class":"nm-collections-title nm-collections-link"})
    title_container = title_container[0].text

    movieid = container.findAll("a",{"class":"nm-collections-title nm-collections-link"})
    movieid = movieid[0].attrs['href']

    print("Movie Name: " + title_container, "\n")
    print("Movie ID: " , movieid, "\n")

    f.write(title_container + ", " + movieid + "\n")
f.close()  # Close the file

Answer 1

您收到错误的原因是您尝试对列表执行 GET 请求。

my_url = ['https://www.netflix.com/browse/genre/1365', 'https://www.netflix.com/browse/genre/7424']

uClient = uReq(my_url)

我建议在这里做的是遍历每个链接等：

my_url = ['https://www.netflix.com/browse/genre/1365', 'https://www.netflix.com/browse/genre/7424']

for link in my_url:
    uClient = uReq(link)
    page_html = uClient.read()
    ....

要提一下，如果您只是为循环应用代码，它将覆盖您的 f.write 函数。您需要做的是：

新编辑：

import csv

import requests
from bs4 import BeautifulSoup as soup

# All given URLS
my_url = ['https://www.netflix.com/browse/genre/1365', 'https://www.netflix.com/browse/genre/7424']

# Create and open CSV file
with open("netflixaction2.csv", 'w', encoding='utf-8') as csv_file:
    # Headers for CSV
    headers_for_csv = ['Movie Name', 'Movie Link']

    # Small function for csv DictWriter
    csv_writer = csv.DictWriter(csv_file, delimiter=',', lineterminator='\n', fieldnames=headers_for_csv)
    csv_writer.writeheader()

    # We need to loop through each URL from the list
    for link in my_url:

        # Do a simple GET requests with the URL
        response = requests.get(link)

        page_soup = soup(response.text, "html.parser")

        # Find all nm-content-horizontal-row-item
        containers = page_soup.findAll("li", {"class": "nm-content-horizontal-row-item"})

        # Loop through each found "li"
        for container in containers:
            movie_name = container.text.strip()
            movie_link = container.find("a")['href']

            print(f"Movie Name: {movie_name} | Movie link: {movie_link}")

            # Write to CSV
            csv_writer.writerow({
                'Movie Name': movie_name,
                'Movie Link': movie_link,
            })

# Close the file
csv_file.close()

这应该是您的解决方案 :) 如果我遗漏了什么，请随时发表评论！