无法将数据放入 Pandas DataFrame 中。每当我尝试将这两个数据放入 Pandas 中时。它没有完成

Question

我无法创建 data_list 和 data_1_list 的数据框。每当我这样做时，我都不会收到错误，但只显示了整个数据的一行。请在您自己的代码中尝试此操作。无法解释用文字。你尝试制作一个数据框并打印它，你会发现它并没有全部打印出来

import pandas as pd 
import lxml
from bs4 import BeautifulSoup
import html5lib
from selenium import webdriver    

for des in soup.find_all('div',class_='pt-2'):
    data = des.find('p',class_='mt-1').text
    data_list = []
    data_list.append(data)
    print(data_list)
for title in soup.find_all('a',class_='stretched-link'):
    data_1 = title.h2.text
    data_1_list = []
    data_1_list.append(data_1)
    print(data_1_list)
    #Unable to put data in Pandas DataFrame.Whenever i try to put these two data in Pandas.It is not done `

Answer 1

这是因为您在循环内声明了empty list。请将其放在循环外。

data_list = []
for des in soup.find_all('div', class_='pt-2'):
    data = des.find('p', class_='mt-1').text    
    data_list.append(data)
print(data_list)

data_1_list = []
for title in soup.find_all('a', class_='stretched-link'):
    data_1 = title.h2.text    
    data_1_list.append(data_1)
print(data_1_list)

现在你可以使用它了

df = pd.DataFrame({"Data_1":data_list,"Data_2":data_1_list})
print(df)

---

更新

由于javascripts渲染的数据是先用selenium加载数据，再用bs4和pandas加载到DataFrame中。

from selenium import webdriver
from bs4 import BeautifulSoup
import pandas as pd

driver=webdriver.Chrome(executable_path="path/to/chromedriver")
driver.get("https://realpython.com/search?q=web%20scraping")
time.sleep(3)
soup=BeautifulSoup(driver.page_source,"html.parser")
title=[]
title_content=[]
for item in soup.find_all('div', class_='row')[1:]:
    title.append(item.find('a',class_='stretched-link').find_next('h2').text)
    title_content.append(item.find('p',class_='text-muted my-0 mt-1 small').text)


df = pd.DataFrame({"Data_1":title,"Data_2":title_content})
print(df)

输出：

                                            Data_1                                             Data_2
0                  Learning Path: Python Web Scraping  Web scraping is about downloading structured d...
1                Topic: Python Web Scraping Tutorials  Web scraping is about downloading structured d...
2    Practical Introduction to Web Scraping in Python  Learn the basics of web scraping with Python u...
3                Web Scraping with Scrapy and MongoDB  This tutorial covers how to write a Python web...
4   Web Scraping and Crawling with Scrapy and MongoDB  Part 2 in this tutorial series covers how to e...
5   The Real Python Podcast – Episode #12: Web Scr...  Do you want to get started with web scraping u...
6     Beautiful Soup: Build a Web Scraper With Python  In this tutorial, you'll walk through the main...
7      Modern Web Automation With Python and Selenium  Your guide to learning advanced Python web aut...
8                                    Course Overviews  An overview of the contents of the Real Python...
9                       The Real Python Course Bundle  Learn Python from the ground up and gain real-...
10                            What are people saying?  What the Python community says about the Real ...
11                                 Python Basics Book  Master fundamental concepts for Python beginne...
12       The Ultimate List of Python YouTube Channels  We couldn't find a good and updated list of Py...
13  Flask by Example – Text Processing with Reques...  In part three of this series, we're going scra...
14                              The Best Python Books  Find the right books to help you get started w...
15                             Python Books & Courses  At Real Python we offer a number of Python tra...
16  Switching to Python From Another Programming L...  Here you'll find the best resources and tips f...
17              How can I learn the basics of Python?  How to learn Python programming as a complete ...
18                                          Resources  Free tutorials and resources, developed by the...
19                    Learn Python Programming Online  Learn Python online: Web development tutorials...