作为一个列表元素读入段落

Question

我有一个包含数千条这样的商品评论的文本文件：

+1  This book is such a life saver.  It has been so helpful to be able to go back to track trends, answer pediatrician questions, or communicate with each other when you are up at different times of the night with a newborn.  I think it is one of those things that everyone should be required to have before they leave the hospital.  We went through all the pages of the newborn version, then moved to the infant version, and will finish up the second infant book (third total) right as our baby turns 1.  See other things that are must haves for baby at [...]
+1  I bought this a few times for my older son and have bought it again for my newborn. This is super easy to use and helps me keep track of his daily routine. When he started going to the sitter when I went back to work, it helped me know how his day went to better prepare me for how the evening would most likely go. When he was sick, it help me keep track of how many diapers a day he was producing to make sure he was getting dehydrated. The note sections to the side and bottom are useful too because his sitter writes in small notes about whether or not he liked his lunch or if the playtime included going for a walk, etc.Excellent for moms who are wanting to keep track of their kids daily routine even though they are at work. Excellent for dads to keep track as my husband can quickly forget what time he fed our son. LOL
+1  This is great for basics, but I wish the space to write things in was bigger. A lot times I need struggle trying to read what the caretaker wrote in because the spaces go together.
+1  This book is perfect!  I'm a first time new mom, and this book made it so easy to keep track of feedings, diaper changes, sleep.  Definitely would recommend this for new moms.  Plus it's small enough that I throw in the diaper back for doctor visits.

每条评论由一个新行分隔，评论情绪与评论之间由一个标签分隔。

这是我的代码，可以正确地将每个情绪和评论放入各自的数组中：

# read in training data, 18506 reviews
trainingFile = open(r"D:\Desktop\\1565964985_2925534_train_file.data", "r")

# arrays for the sentiments and reviews
sentiment = []
review = []

# for loop that reads each line
for line in trainingFile:
    # data field array separated by tab
    dataFields = line.split("\t")

    # sentiment holds the positive or negative sentiment of the review
    sentiment.append(dataFields[0])
    # review holds the text from the review
    review.append(dataFields[1])

# test print statement
for x in range(len(sentiment)):
    print(sentiment[x])

for x in range(len(review)):
    print(review[x])

问题出在这里：我正在对这些评论段落和当前代码进行词袋分析

print(review[0])
print(type(review[0]))
count = CountVectorizer()
docs = numpy.array(review[0])
bag = count.fit_transform(docs)
print(bag.toarray())

错误：

TypeError: iteration over a 0-d array    

所以我尝试以不同的方式阅读评论：

review.append(dataFields[1].split())

现在这是我的结果：

['This', 'book', 'is', 'such', 'a', 'life', 'saver.', 'It', 'has', 'been', 'so', 'helpful', 'to', 'be', 'able', 'to', 'go', 'back', 'to', 'track', 'trends,', 'answer', 'pediatrician', 'questions,', 'or', 'communicate', 'with', 'each', 'other', 'when', 'you', 'are', 'up', 'at', 'different', 'times', 'of', 'the', 'night', 'with', 'a', 'newborn.', 'I', 'think', 'it', 'is', 'one', 'of', 'those', 'things', 'that', 'everyone', 'should', 'be', 'required', 'to', 'have', 'before', 'they', 'leave', 'the', 'hospital.', 'We', 'went', 'through', 'all', 'the', 'pages', 'of', 'the', 'newborn', 'version,', 'then', 'moved', 'to', 'the', 'infant', 'version,', 'and', 'will', 'finish', 'up', 'the', 'second', 'infant', 'book', '(third', 'total)', 'right', 'as', 'our', 'baby', 'turns', '1.', 'See', 'other', 'things', 'that', 'are', 'must', 'haves', 'for', 'baby', 'at', '[...]']
<class 'list'>
[[0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 ...
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]]

这就是我想要的：

review[0] = ["This book is such a life saver.  It has been so helpful to be able to go back to track trends, answer"
             " pediatrician questions, or communicate with each other when you are up at different times of the night"
             " with a newborn.  I think it is one of those things that everyone should be required to have before they"
             " leave the hospital.  We went through all the pages of the newborn version, then moved to the infant"
             " version, and will finish up the second infant book (third total) right as our baby turns 1."
             "  See other things that are must haves for baby at [...]"]
print(type(review[0]))
count = CountVectorizer()
docs = numpy.array(review[0])
bag = count.fit_transform(docs)
print(count.get_feature_names())
print(bag.toarray())

输出：

<class 'list'>
['able', 'all', 'and', 'answer', 'are', 'as', 'at', 'baby', 'back', 'be', 'been', 'before', 'book', 'communicate', 'different', 'each', 'everyone', 'finish', 'for', 'go', 'has', 'have', 'haves', 'helpful', 'hospital', 'infant', 'is', 'it', 'leave', 'life', 'moved', 'must', 'newborn', 'night', 'of', 'one', 'or', 'other', 'our', 'pages', 'pediatrician', 'questions', 'required', 'right', 'saver', 'second', 'see', 'should', 'so', 'such', 'that', 'the', 'then', 'they', 'things', 'think', 'third', 'this', 'those', 'through', 'times', 'to', 'total', 'track', 'trends', 'turns', 'up', 'version', 'we', 'went', 'when', 'will', 'with', 'you']
[[1 1 1 1 2 1 2 2 1 2 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 1 1 1 1 2 1 3 1
  1 2 1 1 1 1 1 1 1 1 1 1 1 1 2 6 1 1 2 1 1 1 1 1 1 5 1 1 1 1 2 2 1 1 1 1
  2 1]]

所以我最大的问题是如何将我的文本阅读成我想要的格式？

Full code + outputs:

import re
from sklearn.feature_extraction.text import CountVectorizer
import numpy
import math

# def euclideanDistance


# read in training data, 18506 reviews
trainingFile = open(r"D:\Desktop\\1565964985_2925534_train_file.data", "r")

# arrays for the sentiments and reviews
sentiment = []
review = []

# for loop that reads each line
for line in trainingFile:
    # data field array separated by tab
    dataFields = line.split("\t")

    # sentiment holds the positive or negative sentiment of the review
    sentiment.append(dataFields[0])
    # review holds the text from the review
    review.append(dataFields[1].split())

# test print statement
#for x in range(len(sentiment)):
   # print(sentiment[x])

#for x in range(len(review)):
   # print(review[x])

print(review[0])
print(type(review[0]))
count = CountVectorizer()
docs = numpy.array(review[0])
bag = count.fit_transform(docs)
print(bag.toarray())
print("\n\n\n")

review[0] = ["This book is such a life saver.  It has been so helpful to be able to go back to track trends, answer"
             " pediatrician questions, or communicate with each other when you are up at different times of the night"
             " with a newborn.  I think it is one of those things that everyone should be required to have before they"
             " leave the hospital.  We went through all the pages of the newborn version, then moved to the infant"
             " version, and will finish up the second infant book (third total) right as our baby turns 1."
             "  See other things that are must haves for baby at [...]"]
print(type(review[0]))
count = CountVectorizer()
docs = numpy.array(review[0])
bag = count.fit_transform(docs)
print(count.get_feature_names())
print(bag.toarray())

trainingFile.close()

['This', 'book', 'is', 'such', 'a', 'life', 'saver.', 'It', 'has', 'been', 'so', 'helpful', 'to', 'be', 'able', 'to', 'go', 'back', 'to', 'track', 'trends,', 'answer', 'pediatrician', 'questions,', 'or', 'communicate', 'with', 'each', 'other', 'when', 'you', 'are', 'up', 'at', 'different', 'times', 'of', 'the', 'night', 'with', 'a', 'newborn.', 'I', 'think', 'it', 'is', 'one', 'of', 'those', 'things', 'that', 'everyone', 'should', 'be', 'required', 'to', 'have', 'before', 'they', 'leave', 'the', 'hospital.', 'We', 'went', 'through', 'all', 'the', 'pages', 'of', 'the', 'newborn', 'version,', 'then', 'moved', 'to', 'the', 'infant', 'version,', 'and', 'will', 'finish', 'up', 'the', 'second', 'infant', 'book', '(third', 'total)', 'right', 'as', 'our', 'baby', 'turns', '1.', 'See', 'other', 'things', 'that', 'are', 'must', 'haves', 'for', 'baby', 'at', '[...]']
<class 'list'>
[[0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 ...
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]]




<class 'list'>
['able', 'all', 'and', 'answer', 'are', 'as', 'at', 'baby', 'back', 'be', 'been', 'before', 'book', 'communicate', 'different', 'each', 'everyone', 'finish', 'for', 'go', 'has', 'have', 'haves', 'helpful', 'hospital', 'infant', 'is', 'it', 'leave', 'life', 'moved', 'must', 'newborn', 'night', 'of', 'one', 'or', 'other', 'our', 'pages', 'pediatrician', 'questions', 'required', 'right', 'saver', 'second', 'see', 'should', 'so', 'such', 'that', 'the', 'then', 'they', 'things', 'think', 'third', 'this', 'those', 'through', 'times', 'to', 'total', 'track', 'trends', 'turns', 'up', 'version', 'we', 'went', 'when', 'will', 'with', 'you']
[[1 1 1 1 2 1 2 2 1 2 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 1 1 1 1 2 1 3 1
  1 2 1 1 1 1 1 1 1 1 1 1 1 1 2 6 1 1 2 1 1 1 1 1 1 5 1 1 1 1 2 2 1 1 1 1
  2 1]]

Answer 1

假设您的情绪值可能是[+1, 0, -1] 之一，下面的代码应该可以解决问题。

数据

我修改了您的示例数据，以确保其中包含所有类型的情绪值。

data_string = """
-1  This book is such a life saver.  It has been so helpful to be able to go back to track trends, answer pediatrician questions, or communicate with each other when you are up at different times of the night with a newborn.  I think it is one of those things that everyone should be required to have before they leave the hospital.  We went through all the pages of the newborn version, then moved to the infant version, and will finish up the second infant book (third total) right as our baby turns 1.  See other things that are must haves for baby at [...]
+1  I bought this a few times for my older son and have bought it again for my newborn. This is super easy to use and helps me keep track of his daily routine. When he started going to the sitter when I went back to work, it helped me know how his day went to better prepare me for how the evening would most likely go. When he was sick, it help me keep track of how many diapers a day he was producing to make sure he was getting dehydrated. The note sections to the side and bottom are useful too because his sitter writes in small notes about whether or not he liked his lunch or if the playtime included going for a walk, etc.Excellent for moms who are wanting to keep track of their kids daily routine even though they are at work. Excellent for dads to keep track as my husband can quickly forget what time he fed our son. LOL
0  This is great for basics, but I wish the space to write things in was bigger. A lot times I need struggle trying to read what the caretaker wrote in because the spaces go together.
+1  This book is perfect!  I'm a first time new mom, and this book made it so easy to keep track of feedings, diaper changes, sleep.  Definitely would recommend this for new moms.  Plus it's small enough that I throw in the diaper back for doctor visits.
"""

# convert to a list of lines (strings)
data_lines = [line.strip() for line in data_string.split('\n') if line.strip()!='']
# print data_lines
data_lines

您可以这样做以从文件中以行列表的形式读取数据。

with open("filename.csv", "r") as f:
    data_lines = f.readlines()

输出：

['-1  This book is such a life saver.  It has been so helpful to be able to go back to track trends, answer pediatrician questions, or communicate with each other when you are up at different times of the night with a newborn.  I think it is one of those things that everyone should be required to have before they leave the hospital.  We went through all the pages of the newborn version, then moved to the infant version, and will finish up the second infant book (third total) right as our baby turns 1.  See other things that are must haves for baby at [...]',
 '+1  I bought this a few times for my older son and have bought it again for my newborn. This is super easy to use and helps me keep track of his daily routine. When he started going to the sitter when I went back to work, it helped me know how his day went to better prepare me for how the evening would most likely go. When he was sick, it help me keep track of how many diapers a day he was producing to make sure he was getting dehydrated. The note sections to the side and bottom are useful too because his sitter writes in small notes about whether or not he liked his lunch or if the playtime included going for a walk, etc.Excellent for moms who are wanting to keep track of their kids daily routine even though they are at work. Excellent for dads to keep track as my husband can quickly forget what time he fed our son. LOL',
 '0  This is great for basics, but I wish the space to write things in was bigger. A lot times I need struggle trying to read what the caretaker wrote in because the spaces go together.',
 "+1  This book is perfect!  I'm a first time new mom, and this book made it so easy to keep track of feedings, diaper changes, sleep.  Definitely would recommend this for new moms.  Plus it's small enough that I throw in the diaper back for doctor visits."]

解决方案：

sentiments = list()
reviews = list()
for i, line in enumerate(data_lines):
    s = ''.join(re.findall("^[+1]*[-1]*[0]*", line))
    r = line.replace(s,'').strip()
    print('line:{} \n\t sentiment: {} \n\t review: {}'.format(i, s, r))
    sentiments.append(s)
    reviews.append(r)

输出

line:0 
     sentiment: -1 
     review: This book is such a life saver.  It has been so helpful to be able to go back to track trends, answer pediatrician questions, or communicate with each other when you are up at different times of the night with a newborn.  I think it is one of those things that everyone should be required to have before they leave the hospital.  We went through all the pages of the newborn version, then moved to the infant version, and will finish up the second infant book (third total) right as our baby turns 1.  See other things that are must haves for baby at [...]
line:1 
     sentiment: +1 
     review: I bought this a few times for my older son and have bought it again for my newborn. This is super easy to use and helps me keep track of his daily routine. When he started going to the sitter when I went back to work, it helped me know how his day went to better prepare me for how the evening would most likely go. When he was sick, it help me keep track of how many diapers a day he was producing to make sure he was getting dehydrated. The note sections to the side and bottom are useful too because his sitter writes in small notes about whether or not he liked his lunch or if the playtime included going for a walk, etc.Excellent for moms who are wanting to keep track of their kids daily routine even though they are at work. Excellent for dads to keep track as my husband can quickly forget what time he fed our son. LOL
line:2 
     sentiment: 0 
     review: This is great for basics, but I wish the space to write things in was bigger. A lot times I need struggle trying to read what the caretaker wrote in because the spaces go together.
line:3 
     sentiment: +1 
     review: This book is perfect!  I'm a first time new mom, and this book made it so easy to keep track of feedings, diaper changes, sleep.  Definitely would recommend this for new moms.  Plus it's small enough that I throw in the diaper back for doctor visits.

我希望这会有所帮助。