PHP - 文本挖掘缓慢过程的文本预处理

Question

我对大型数据库的文本挖掘进行了文本预处理，我想将数据库上所有文章中的 camus 数据制作成数组，但这需要很长时间。

$multiMem   = memory_get_usage();
$xstart = microtime(TRUE);
$word = "";
$sql = mysql_query("SELECT * FROM tbl_content");
while($data = mysql_fetch_assoc($sql)){
  $word = $word."".$data['article'];
}

$preprocess = new preprocess($word);
$word= $preprocess->preprocess($word);
print_r($kata);

$xfinish = microtime(TRUE);

这是我的班级预处理

class preprocess {

  var $teks;

  function preprocess($teks){
  /*start process segmentation*/
  $teks = trim($teks);

  //menghapus tanda baca
  $teks = str_replace("'", "", $teks);
  $teks = str_replace("-", "", $teks);
  $teks = str_replace(")", "", $teks);
  $teks = str_replace("(", "", $teks);
  $teks = str_replace("=", "", $teks);
  $teks = str_replace(".", "", $teks);
  $teks = str_replace(",", "", $teks);
  $teks = str_replace(":", "", $teks);
  $teks = str_replace(";", "", $teks);
  $teks = str_replace("!", "", $teks);
  $teks = str_replace("?", "", $teks);

  //remove HTML tags
  $teks = strip_tags($teks);
  $teks = preg_replace('@<(\w+)\b.*?>.*?</\1>@si', '', $teks);
  /*end proses segmentation*/

  /*start case folding*/
  $teks = strtolower($teks);

  $teks = preg_replace('/[0-9]+/', '', $teks);
  /*end case folding*/

  /*start of tokenizing*/
  $teks = explode(" ", $teks);

  /*end of tokenizing*/

  /*start of filtering*/
  //stopword
  $file = file_get_contents('stopword.txt', FILE_USE_INCLUDE_PATH);
  $stopword = explode("\n", $file);

  //remove stopword
  $teks = preg_replace('/\b('.implode('|',$stopword).')\b/','',$teks);

  /*end of filtering*/

  /*start of stemming*/
  require_once('stemming.php');
  foreach($teks as $t => $value){
    $teks[$t] = stemming($value);
  }
  /*end of stemming*/

  $teks = array_filter($teks);
  $teks = array_values($teks);

  return $teks;
 }
}

有人知道我的程序可以快速处理吗？请帮助
感谢提前

Answer 1

还有一些地方可以改进...

1. 建立$word 后，您可以释放查询结果$sql 和data

   $word = '';
   $sql = mysql_query("SELECT * FROM tbl_content");
   while($data = mysql_fetch_assoc($sql)){
     $word = $word . $data['article'];
   }
   mysql_free_result($sql);
   unset($sql, $data);
   

2. 这个块：

   $teks = str_replace("'", "", $teks);
   $teks = str_replace("-", "", $teks);
   $teks = str_replace(")", "", $teks);
   $teks = str_replace("(", "", $teks);
   $teks = str_replace("=", "", $teks);
   $teks = str_replace(".", "", $teks);
   $teks = str_replace(",", "", $teks);
   $teks = str_replace(":", "", $teks);
   $teks = str_replace(";", "", $teks);
   $teks = str_replace("!", "", $teks);
   $teks = str_replace("?", "", $teks);
   

可以这样写：

    $teks = str_replace(array('(','-',')',',','.','=',';','!','?'), '', $teks);

1. 由于您稍后在代码中将数字替换为正则表达式，因此您可以在上面的 str_replace 调用中添加数字，或者将上面的字符添加到 preg_replace

   $teks = str_replace(array('0','1','2','3','4','5','6','7','8','9','(','-',')',',','.','=',';','!','?'), '', $teks);
   

   或

   $teks = preg_replace('/[0-9,\(\)\-\=\.\,\;\!\?]+/', '', $teks);
   

2. $teks = strip_tags($teks); 应该足够了。如果不是，则仅使用以下preg_replace，因为它正在做同样的事情。

3. 使用file insted file_get_contentsfollowed by theexplodesince thefilereturns an array directly. Also there is no need to explode the $teks

      $stopword = file('stopword.txt');
      array_walk($stopword, function(&$item1){
        $item1 = '/\b' . $item1 . '\b/';
      });
      $teks = preg_replace($stopword, '', $teks);
   

4. 一般不要使用""，因为处理器会尝试评估内容并且需要更长时间。

5. 如果stopword.txt 列表没有改变，最好将其作为数组直接放在代码中，然后访问文件系统来读取它。