将字典或元组转换为数据框

Question

我正在做一个网络抓取练习，其中抓取的单词按出现次数计算。我想将计数的单词和频率转换为数据框并保存为 excel 格式。

我已经尝试了每个示例，但没有任何效果。 我想转换这个列表（顶部），看起来像这样

 Print (top)
 [('the', 1)]
 [('one', 1)]
 [('of', 1)]
 [('the', 1)]
 [('most', 1)]
 ...........

进入这样的数据框：

 index Word count
  ..    the   1
  ..    one   1
  ..    of    1
  ..    the   1
  ..    most  1
  ..    ...   ..

下面是代码

 for word in clean_list: 
       if word in word_count: 
           word_count[word] += 1
       else: 
           word_count[word] = 1

  #To get count of each word in 
      #the crawled page --> 

  c = Counter(word_count)

  # returns the most occuring elements 
  top = c.most_common(100)

这是我不工作的代码：

df=pd.DataFrame.from_records(top, columns=["word","count"])
df.to_excel("mine" + ".xls")
print(top)

它只保存最后一行而不是整个列表。 如果有人可以提供帮助，我会很高兴。谢谢！

完整代码为： `

# Python3 program for a word frequency 
# counter after crawling a web-page 
import requests 
from bs4 import BeautifulSoup 
import operator 
from collections import Counter 
import pandas as pd
from datetime import datetime
import time
import pandas as pd
import numpy as np
from itertools import chain

'''Function defining the web-crawler/core 
spider, which will fetch information from 
a given website, and push the contents to 
the second function clean_wordlist()'''
def start(url): 

# empty list to store the contents of 
# the website fetched from our web-crawler 
    wordlist = [] 
    source_code = requests.get(url).text 

# BeautifulSoup object which will 
# ping the requested url for data 
    soup = BeautifulSoup(source_code, 'html.parser') 

# Text in given web-page is stored under 
# the <div> tags with class <entry-content> 
    for each_text in soup.findAll('div', {'class':'entry-content'}): 
        content = each_text.text 

# use split() to break the sentence into 
# words and convert them into lowercase 
    words = content.lower().split() 

    for each_word in words: 
        wordlist.append(each_word) 
        clean_wordlist(wordlist)



# Function removes any unwanted symbols 
def clean_wordlist(wordlist): 

    clean_list =[] 
    for word in wordlist: 
        symbols = '!@#$%^&*()_-+={[}]|\;:"<>?/., '

    for i in range (0, len(symbols)): 
        word = word.replace(symbols[i], '') 

    if len(word) > 0: 
        clean_list.append(word) 
    create_dictionary(clean_list)


# Creates a dictionary conatining each word's 
# count and top_20 ocuuring words 
def create_dictionary(clean_list): 
    word_count = {} 
    dateObj =time.strftime("%d.%m.%Y")
    df=[]
    other={}

    for word in clean_list: 
        if word in word_count: 
            word_count[word] += 1
        else: 
            word_count[word] = 1

    c = Counter(word_count)

    # returns the most occuring elements 
    top = c.most_common(100)

    #df=pd.DataFrame(chain.from_iterable(top), columns=['Word', 'Count'])
    df=pd.DataFrame.from_records([i[0] for i in top])
    df.to_excel("mine" + ".xls")
    #print(top)
    print(top)


# Driver code 
if __name__ == '__main__': 
    start("https://www.geeksforgeeks.org/programming-language-choose/")`

Answer 1

您的代码中有几个错误。最常见的错误是在循环期间覆盖数据。

像这里一样，你一遍又一遍地替换内容！

for each_text in soup.findAll('div', {'class':'entry-content'}): 
        content = each_text.text

在这里，你一遍又一遍地覆盖你的输出文件。

for each_word in words: 
        wordlist.append(each_word) 
        clean_wordlist(wordlist)

我已将其重写为单个函数，如果您愿意，可以将其拆分为更多函数。

#Add imports...


def process(url): 

    # the website fetched from our web-crawler 
    source_code = requests.get(url).text 

    # BeautifulSoup object which will 
    # ping the requested url for data 
    soup = BeautifulSoup(source_code, 'html.parser') 

    # Text in given web-page is stored under 
    # the <div> tags with class <entry-content>
    entry_content = ''
    for content in soup.findAll('div', {'class':'entry-content'}): 
        entry_content += content.text.strip()

    #clean
    entry_content = entry_content.lower()
    symbols = list('!@#$%^&*()_-+={[}]|\;:"<>?/., ')
    symbols.append('\n') #new line char

    for symbol in symbols:
        entry_content = entry_content.replace(symbol,' ')
    #print(entry_content)

    #split in words
    words = entry_content.split()

    #conut words
    c = Counter(words)

    #remove white spaces
    if ' ' in c:
        del c[' ']

    #write to excell the most common words
    most_common_words = c.most_common(100)
    df = pd.DataFrame.from_records(most_common_words, columns=['Word', 'Count'])
    df.to_excel("mine.xls")

# Driver code 
if __name__ == '__main__': 
    process("https://www.geeksforgeeks.org/programming-language-choose/")