python中的文本数据清理问题

Question

我正在开发一个使用网络爬取方法来爬取互联网文章的程序。
该程序是通过输入网站的起始页和结束页来启动的。

此程序按以下顺序运行。

1. 文章信息的网络爬取（标题、排序、时间、内容）
2. 删除特殊字符
3. 只提取名词。

问题可能出在对文章内容进行清洗的过程中提取名词。它一直有效到名词提取之前的阶段。

报错信息如下
ValueError：传递值的长度是 4，索引意味着 5
为了解决这个问题，我使用了添加DataFrame append的方法进行编码。 但这并不能解决问题。

使用konlypy方法（韩语词素分析器）

import urllib.request
import urllib.parse
from bs4 import BeautifulSoup
import pandas as pd
import requests
import re
from konlpy.tag import Okt
from pandas import Series

i = input('Start page? : ')
k = input('End page? : ')

startpage = int(i)
lastpage = int(k)
count = int(i)

# Definition of text cleaning function
def text_cleaning(text):
    hangul = re.compile('[^ㄱ-ㅣ가-힣]+')
    result = hangul.sub(' ', text)
    return result

# Definition of nouns extraction function
def get_nouns(x):
    nouns_tagger = Okt()
    nouns = nouns_tagger.nouns(x)
    nouns = [noun for noun in nouns if len(noun)>1]
    nouns = [noun for noun in nouns if noun not in stopwords]
    return nouns

# dataframe formation
columns = ['Title', 'Sort', 'Datetime', 'Article']
news_info = pd.DataFrame(columns=columns)
idx = 0

网站页面循环

while startpage<lastpage + 1:
  url = f'http://www.koscaj.com/news/articleList.html?page={startpage}&total=72698&box_idxno=&sc_section_code=S1N2&view_type=sm'
  html = urllib.request.urlopen(url).read()
  soup = BeautifulSoup(html, 'html.parser')
  links = soup.find_all(class_='list-titles')

  print(f'-----{count}page result-----')
# Articles loop in the web-site page
  for link in links:
    news_url = "http://www.koscaj.com"+link.find('a')['href']
    news_link = urllib.request.urlopen(news_url).read()
    soup2 = BeautifulSoup(news_link, 'html.parser')

    # an article's title
    title = soup2.find('div', {'class':'article-head-title'})

    if title:
        title = soup2.find('div', {'class':'article-head-title'}).text
    else:
        title = ''
           
    # an article's sort
    sorts = soup2.find('nav', {'class':'article-head-nav auto-marbtm-10'})
    try:
        sorts2 = sorts.find_all('a')
        sort = sorts2[2].text
    except:
        sort =''
    
    # an article's time
    date = soup2.find('div',{'class':'info-text'})
    try:
        datetime = date.find('i', {'class':'fa fa-clock-o fa-fw'}).parent.text.strip()
        datetime = datetime.replace("승인", "")
    except:
        datetime = ''

    # an article's content
    article = soup2.find('div', {'id':'article-view-content-div'})
    if article:
        article = soup2.find('div', {'id':'article-view-content-div'}).text
        article = article.replace("\n", "")
        article = article.replace("\r", "")
        article = article.replace("\t", "")
        article = article.replace("[전문건설신문] koscaj@kosca.or.kr", "")
        article = article.replace("저작권자 © 대한전문건설신문 무단전재 및 재배포 금지", "")
        article = article.replace("전문건설신문", "")
        article = article.replace("다른기사 보기", "")

    else:
        article = ''

    # Remove special characters
    news_info['Title'] = news_info['Title'].apply(lambda x: text_cleaning(x))
    news_info['Sort'] = news_info['Sort'].apply(lambda x: text_cleaning(x))
    news_info['Article'] = news_info['Article'].apply(lambda x: text_cleaning(x))

到目前为止，程序运行没有任何问题。但是如果你看到程序报错信息，说明由于输入值和索引不同，操作不成功。

提取名词的文本数据清理

    # Dataframe for storing after crawling individual articles
    row = [title, sort, datetime, article]
    series = pd.Series(row, index=news_info.columns)
    news_info = news_info.append(series, ignore_index=True)
    
    
    
    # Load Korean stopword dictionary file    
    path = "C:/Users/이바울/Desktop/이바울/코딩파일/stopwords-ko.txt"
    with open(path, encoding = 'utf-8') as f:
        stopwords = f.readlines()
    
    stopwords = [x.strip() for x in stopwords]

    news_info['Nouns'] = news_info['Article'].apply(lambda x: get_nouns(x))    


  startpage += 1
  count += 1

news_info.to_excel(f'processing{lastpage-int(1)}-{startpage-int(1)}.xlsx')

print('Complete')

设置Pandas DataFrame中已有的4列后，使用append将提取为名词的列添加为第5列。我知道无论索引名称如何，此方法都会添加一列。如果您查看底部的图像链接，结果是第一篇文章被爬取并显示结果。从下一篇文章开始，它不起作用并发生错误。

enter image description here(程序错误结果)
enter link description here(韩语停用词词典)

Answer 1

我解决了这个问题。 这取决于代码在 for 循环语句中的位置。 由于继续重新定位有问题的区域，除了之前工作的代码，我已经能够解决问题。 我通过在下面的代码中只应用两次退格来解决了这个问题。

news_info['Nouns'] = news_info['Article'].apply(lambda x: get_nouns(x))