使用 ICUTokenizer JAVA 标记泰语句子答案

【问题标题】：Tokenize Thai sentence with ICUTokenizer JAVA使用 ICUTokenizer JAVA 标记泰语句子
【发布时间】：2017-09-08 16:39:51
【问题描述】：

我正在尝试以下代码来获取泰语句子中的所有标记。它抛出异常。谁能指点我在 JAVA 中标记泰语？

    import org.apache.lucene.analysis.Analyzer.TokenStreamComponents;
import org.apache.lucene.analysis.TokenFilter;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.icu.ICUNormalizer2Filter;
import org.apache.lucene.analysis.icu.segmentation.ICUTokenizer;
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;

public class Tokenizer{

    public static void main(String[] args) throws IOException {
        ICUTokenizer tokenizer = new ICUTokenizer(new StringReader("การที่ได้ต้องแสดงว่างานดี"));
        TokenFilter filter = new ICUNormalizer2Filter(tokenizer);
        TokenStreamComponents tt = new TokenStreamComponents(tokenizer, filter);
        TokenStream ts = tt.getTokenStream();
        CharTermAttribute cattr  = ts.addAttribute(CharTermAttribute.class);
        ts.reset();
        while(ts.incrementToken()){
            System.out.println(cattr.toString()+"-----");
        }
    }
}

异常如下

Exception in thread "main" java.lang.ExceptionInInitializerError
    at org.apache.lucene.analysis.icu.segmentation.ICUTokenizer.<init>(ICUTokenizer.java:72)
    at com.tokenizer.tt.main(tt.java:22)
Caused by: java.lang.RuntimeException: java.io.IOException: ICU data file error: Not an ICU data file
    at org.apache.lucene.analysis.icu.segmentation.DefaultICUTokenizerConfig.readBreakIterator(DefaultICUTokenizerConfig.java:128)
    at org.apache.lucene.analysis.icu.segmentation.DefaultICUTokenizerConfig.<clinit>(DefaultICUTokenizerConfig.java:66)
    ... 2 more
Caused by: java.io.IOException: ICU data file error: Not an ICU data file
    at com.ibm.icu.impl.ICUBinary.readHeader(ICUBinary.java:577)
    at com.ibm.icu.text.RBBIDataWrapper.get(RBBIDataWrapper.java:173)
    at com.ibm.icu.text.RuleBasedBreakIterator.getInstanceFromCompiledRules(RuleBasedBreakIterator.java:71)
    at org.apache.lucene.analysis.icu.segmentation.DefaultICUTokenizerConfig.readBreakIterator(DefaultICUTokenizerConfig.java:123)
    ... 3 more

【问题讨论】：

异常信息是什么意思？
添加了异常堆栈跟踪

标签： java lucene tokenize icu4j

【解决方案1】：

终于弄明白了如何在java程序中使用ICU4J

import java.io.IOException;
import java.io.Reader;
import java.io.StringReader;
import org.apache.lucene.analysis.icu.segmentation.ICUTokenizer;
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;

public class icuEstes {

public static void main(String[] args) throws IOException {
    Reader reader = new StringReader("การที่ได้ต้องแสดงว่างานดี  This is a test ກວ່າດອກ");
    ICUTokenizer icut = new ICUTokenizer();
    icut.setReader(reader);
    icut.addAttribute(CharTermAttribute.class);
    icut.reset();
    while (icut.incrementToken()) {
        System.out.println(icut.toString());
        System.out.println(icut.getAttribute(CharTermAttribute.class));
    }
    icut.close();
}}

【讨论】：