文件中重复单词的迭代器

Question

假设，我正在编写一个函数来查找文本文件中的“重复单词”。例如，在aaa aaa bb cc cc bb dd 中重复的单词是aaa 和cc，但不是bb，因为两个bb 实例不会并排出现。

该函数接收一个迭代器并像这样返回迭代器：

def foo(in: Iterator[String]): Iterator[String] = ???

foo(Iterator("aaa", "aaa", "bb", "cc", "cc", "bb")) // Iterator("aaa", "cc")
foo(Iterator("a", "a", "a", "b", "c", "b"))         // Iterator("a")

你会怎么写foo？请注意，输入量很大，所有单词都无法放入内存（但重复单词的数量相对较少）。

附：我还想稍后增强 foo 以返回重复单词的位置、重复次数等。

Answer 1

更新：

那好吧。让我们指定你想要的东西：

 input       | expected    
             |             
 a           |             
 aa          | a           
 abc         |             
 aabc        | a           
 aaabbbbbbc  | ab          
 aabaa       | aa          
 aabbaa      | aba         
 aabaa       | aa    

这是真的吗？如果是这样，这是可行的解决方案。不确定性能，但至少它很懒（不要将所有内容都加载到内存中）。

//assume we have no nulls in iterator.
def foo[T >: Null](it:Iterator[T]) = {
  (Iterator(null) ++ it).sliding(3,1).collect {
    case x @ Seq(a,b,c) if b == c && a != b => c
  }
}

我们需要这个丑陋的Iterator(null) ++，因为我们正在寻找 3 个元素，并且我们需要一种方法来查看前两个是否相同。

这是纯粹的实现，它比命令式具有一些优势（例如，在其他答案中）。最重要的是它很懒：

//infinite iterator!!!
val it = Iterator.iterate('a')(s => (s + (if(Random.nextBoolean) 1 else 0)).toChar)
//it'll take only as much as needs to take this 10 items.
//should not blow up
foo(it).take(10)
//imperative implementation will blow up in such situation.
fooImp(it).take(10)

以下是本主题中看到的这篇文章和其他文章的所有实现： https://scalafiddle.io/sf/w5yozTA/15

有索引和位置

您在评论中询问是否可以轻松添加重复单词的数量及其索引。我想了一会儿，我做了这样的东西。不确定它是否具有出色的性能，但它应该是惰性的（例如，应该适用于大文件）。

/** returns Iterator that replace consecutive items with (item, index, count). 
It contains all items from orginal iterator.  */
def pack[T >: Null](it:Iterator[T]) = {
  //Two nulls, each for one sliding(...) 
  (Iterator(null:T) ++ it ++ Iterator(null:T))
  .sliding(2,1).zipWithIndex
  //skip same items
  .filter { case (x, _) => x(0) != x(1) }
  //calculate how many items was skipped
  .sliding(2,1).collect {
    case Seq((a, idx1), (b, idx2)) => (a(1), idx1 ,idx2-idx1)  
  }
}

def foo[T >: Null](it:Iterator[T]) = pack(it).filter(_._3 > 1)

旧答案（更新问题之前）

另一个（更简单的）解决方案可能是这样的：

import scala.collection.immutable._

//Create new iterator each time we'll print it.
def it = Iterator("aaa", "aaa", "bb", "cc", "cc", "bb", "dd", "dd", "ee",  "ee", "ee", "ee", "ee", "aaa", "aaa", "ff", "ff", "zz", "gg", "aaa", "aaa")

//yep... this is whole implementation :)
def foo(it:Iterator[String]) = it.sliding(2,1).collect { case Seq(a,b) if a == b => a } 


println(foo(it).toList) //dont care about duplication
//List(aaa, cc, dd, ee, ee, ee, ff)

println(foo(it).toSet) //throw away duplicats but don't keeps order
//Set(cc, aaa, ee, ff, dd)

println(foo(it).to[ListSet]) //throw away duplicats and keeps order
//ListSet(aaa, cc, dd, ee, ff)

//oh... and keep result longer than 5 items while testing. 
//Scala collections (eg: Sets) behaves bit diffrently up to this limit (they keeps order)
//just test with bit bigger Sequences :)

https://scalafiddle.io/sf/w5yozTA/1

（如果回答有帮助，请投票）

Answer 2

这是一个带有累加器的解决方案：

  case class Acc(word: String = "", count: Int = 0, index: Int = 0)

  def foo(in: Iterator[String]) =
    in.zipWithIndex
      .foldLeft(List(Acc())) { case (Acc(w, c, i) :: xs, (word: String, index)) =>
        if (word == w) // keep counting
          Acc(w, c + 1, i) :: xs
        else
          Acc(word, 1, index) :: Acc(w, c, i) :: xs
      }.filter(_.count > 1)
      .reverse

  val it = Iterator("aaa", "aaa", "bb", "cc", "cc", "bb", "dd", "aaa", "aaa", "aaa", "aaa")

这将返回List(Acc(aaa,2,0), Acc(cc,2,3), Acc(aaa,4,7))

如果同一个单词有另一组重复单词，它也会处理。

你有出现次数的索引和计数。

如果您需要更多解释，请告诉我。

Answer 3

这是一个仅使用原始迭代器的解决方案。没有中间集合。所以一切都保持完全惰性，适用于非常大的输入数据。

def foo(in: Iterator[String]): Iterator[String] =
  Iterator.unfold(in.buffered){ itr =>   // <--- Scala 2.13
    def loop :Option[String] =
      if (!itr.hasNext) None
      else {
        val str = itr.next()
        if (!itr.hasNext) None
        else if (itr.head == str) {
          while (itr.hasNext && itr.head == str) itr.next() //remove repeats
          Some(str)
        }
        else loop
      }
    loop.map(_ -> itr)
  }

测试：

val it = Iterator("aaa", "aaa", "aaa", "bb", "cc", "cc", "bb", "dd")
foo(it) // Iterator("aaa", "cc")

//pseudo-infinite iterator
val piIt = Iterator.iterate(8)(_+1).map(_/3)  //2,3,3,3,4,4,4,5,5,5, etc.
foo(piIt.map(_.toString))                     //3,4,5,6, etc.

Answer 4

与其他答案相比，它有些复杂，但它使用的额外内存相对较小。而且可能更快。

def repeatedWordsIndex(in: Iterator[String]): java.util.Iterator[String] = {
  val initialCapacity = 4096
  val res = new java.util.ArrayList[String](initialCapacity) // or mutable.Buffer or mutable.Set, if you want Scala
  var prev: String = null
  var next: String = null
  var prevEquals = false
  while (in.hasNext) {
    next = in.next()
    if (next == prev) {
      if (!prevEquals) res.add(prev)
      prevEquals = true
    } else {
      prevEquals = false
    }
    prev = next
  }
  res.iterator // may be need to call distinct
}

Answer 5

您可以使用 foldLeft 遍历集合，其累加器是 Map 和 String 的元组，以跟踪前一个单词的条件单词计数，然后是 collect，如下所示：

def foo(in: Iterator[String]): Iterator[String] =
  in.foldLeft((Map.empty[String, Int], "")){ case ((m, prev), word) =>
      val count = if (word == prev) m.getOrElse(word, 0) + 1 else 1
      (m + (word -> count), word)
    }._1.
    collect{ case (word, count) if count > 1 => word }.
    iterator

foo(Iterator("aaa", "aaa", "bb", "cc", "cc", "bb", "dd")).toList
// res1: List[String] =  List("aaa", "cc")

要捕获重复的字数和索引，只需索引集合并对条件字数应用类似的策略：

def bar(in: Iterator[String]): Map[(String, Int), Int] =
  in.zipWithIndex.foldLeft((Map.empty[(String, Int), Int], "", 0)){
      case ((m, pWord, pIdx), (word, idx)) =>
        val idx1 = if (word == pWord) idx min pIdx else idx
        val count = if (word == pWord) m.getOrElse((word, idx1), 0) + 1 else 1
        (m + ((word, idx1) -> count), word, idx1)
    }._1.
    filter{ case ((_, _), count) => count > 1 }

bar(Iterator("aaa", "aaa", "bb", "cc", "cc", "bb", "dd", "cc", "cc", "cc"))
// res2: Map[(String, Int), Int] = Map(("cc", 7) -> 3, ("cc", 3) -> 2, ("aaa", 0) -> 2)

---

更新：

根据修订后的要求，为了最大限度地减少内存使用，一种方法是通过删除计数 1 的元素（如果重复的单词很少，这将是大多数）来保持 Map 的最小尺寸 -在foldLeft 遍历期间飞行。下面方法baz是bar的修改版：

def baz(in: Iterator[String]): Map[(String, Int), Int] =
  (in ++ Iterator("")).zipWithIndex.
    foldLeft((Map.empty[(String, Int), Int], (("", 0), 0), 0)){
      case ((m, pElem, pIdx), (word, idx)) =>
        val sameWord = word == pElem._1._1
        val idx1 = if (sameWord) idx min pIdx else idx
        val count = if (sameWord) m.getOrElse((word, idx1), 0) + 1 else 1
        val elem = ((word, idx1), count)
        val newMap = m + ((word, idx1) -> count)
        if (sameWord) {
          (newMap, elem, idx1)
        } else
          if (pElem._2 == 1)
            (newMap - pElem._1, elem, idx1)
          else
            (newMap, elem, idx1)
    }._1.
    filter{ case ((word, _), _) => word != "" }

baz(Iterator("aaa", "aaa", "bb", "cc", "cc", "bb", "dd", "cc", "cc", "cc"))
// res3: Map[(String, Int), Int] = Map(("aaa", 0) -> 2, ("cc", 3) -> 2, ("cc", 7) -> 3)

请注意，附加到输入集合的虚拟空字符串是为了确保最后一个单词也得到正确处理。