将两列文本文档转换为单行以进行文本挖掘答案

【问题标题】：convert two columns text document to single line for text mining将两列文本文档转换为单行以进行文本挖掘
【发布时间】：2017-06-01 03:21:14
【问题描述】：

我使用pdftools从pdf中提取文本并将结果保存为txt。

有没有一种有效的方法可以将 2 列的 txt 转换为 1 列的文件。

这是我所拥有的一个例子：

Alice was beginning to get very      into the book her sister was reading,
tired of sitting by her sister       but it had no pictures or conversations
on the bank, and of having nothing   in it, `and what is the use of a book,' 
to do: once or twice she had peeped  thought Alice `without pictures or conversation?`

而不是

    Alice was beginning to get very tired of sitting by her sister on the bank, and 
of having nothing to do: once or twice she had peeped into the book her sister was 
reading, but it had no pictures or conversations in it, `and what is the use of a 
book,' thought Alice `without pictures or conversation?'

基于Extract Text from Two-Column PDF with R我稍微修改了函数得到：

library(readr)    
trim = function (x) gsub("(?<=[\\s])\\s*|^\\s+|\\s+$", "", x,  perl=TRUE)

QTD_COLUMNS = 2

read_text = function(text) {
  result = ''
  #Get all index of " " from page.
  lstops = gregexpr(pattern =" ",text)
  #Puts the index of the most frequents ' ' in a vector.
  stops = as.integer(names(sort(table(unlist(lstops)),decreasing=TRUE)[1:2]))
  #Slice based in the specified number of colums (this can be improved)
  for(i in seq(1, QTD_COLUMNS, by=1))
  {
    temp_result = sapply(text, function(x){
      start = 1
      stop =stops[i] 
      if(i > 1)            
        start = stops[i-1] + 1
      if(i == QTD_COLUMNS)#last column, read until end.
        stop = nchar(x)+1
      substr(x, start=start, stop=stop)
    }, USE.NAMES=FALSE)
    temp_result = trim(temp_result)
    result = append(result, temp_result)
  }
  result
}

txt = read_lines("alice_in_wonderland.txt")

result = ''

for (i in 1:length(txt)) { 
  page = txt[i]
  t1 = unlist(strsplit(page, "\n"))      
  maxSize = max(nchar(t1))
  t1 = paste0(t1,strrep(" ", maxSize-nchar(t1)))
  result = append(result,read_text(t1))
}

result

但有些文件没有运气。我想知道是否有更通用/更好的正则表达式来实现结果。

提前非常感谢！

【问题讨论】：

我很想找到非 PDF 替代品。如果你想使用那个特定的故事，这里有一个纯文本版本：gutenberg.org/files/11/11-0.txt。如果做不到这一点，请寻找另一个 PDF 到文本的转换工具，它将转换为 1 列输出。
看起来像一个固定宽度的文件 - 如果两列的宽度始终保持不变，dat <- read.fwf(file, widths=c(37,48), stringsAsFactors=FALSE) 会给你一个很好的开始。
saved my sanity 意识到pdftohtml 有一个非常有用的 XML 输出模式。

标签： r perl

【解决方案1】：

使用固定宽度的左列，我们可以将每行拆分为前 37 个字符和其余字符，将它们添加到左列和右列的字符串中。例如，使用正则表达式

use warnings;
use strict;

my $file = 'two_column.txt'
open my $fh, '<', $file or die "Can't open $file: $!";

my ($left_col, $right_col);

while (<$fh>) 
{
    my ($left, $right) = /(.{37})(.*)/;

    $left =~ s/\s*$/ /;

    $left_col  .= $left;
    $right_col .= $right;
}
close $fh;

print $left_col, $right_col, "\n";

这将打印整个文本。或加入专栏，my $text = $left_col . $right_col;

正则表达式模式(.{37}) 匹配任何字符 (.) 并匹配 37 次 ({37})，使用 () 捕获； (.*) 捕获所有剩余的。这些由正则表达式返回并分配。 $left 中的尾随空格被压缩为一个。然后将两者都附加 (.=)。

或者从命令行

perl -wne'
    ($l, $r) = /(.{37})(.*)/; $l =~ s/\s*$/ /; $cL .= $l; $cR .= $r; 
     }{ print $cL,$cR,"\n"
' two_column.txt

}{ 启动 END 块的位置，该块在退出之前运行（在处理完所有行之后）。

【讨论】：

@pachamaltese 为了清楚起见，我进行了一些编辑，并添加了一些语句。

【解决方案2】：

如果两列的宽度总是恒定的，则看起来像一个固定宽度的文件：

dat <- read.fwf(textConnection(txt), widths=c(37,48), stringsAsFactors=FALSE)
gsub("\\s+", " ", paste(unlist(dat), collapse=" "))

这会将所有内容放在一个大长字符串中：

[1] "Alice was beginning to get very tired of sitting by her sister on the bank, and of having nothing to do: once or twice she had peeped into the book her sister was reading, but it had no pictures or conversations in it, `and what is the use of a book,' thought Alice `without pictures or conversation?"

【讨论】：