Unknown Word Detection for Chinese by a Corpus-based Learning Method Export

@article{citeulike:578081, abstract = {One of the most prominent problems in computer processing of the Chinese language is identification of the words in a sentence. Since there are no blanks to mark word boundaries, identifying words is difficult because of segmentation ambiguities and occurrences of out-of-vocabulary words (i.e., unknown words). In this paper, a corpus-based learning method is proposed which derives sets of syntactic rules that are applied to distinguish monosyllabic words from monosyllabic morphemes which may be parts of unknown words or typographical errors. The corpus-based learning approach has the advantages of: 1. automatic rule learning, 2. automatic evaluation of the performance of each rule, and 3. balancing of recall and precision rates through dynamic rule set selection. The experimental results show that the rule set derived using the proposed method outperformed hand-crafted rules produced by human experts in detecting unknown words.}, author = {Chen, Keh-Jiann and Bai, Ming-Hong}, citeulike-article-id = {578081}, journal = {International Journal of Computational linguistics and Chinese Language Processing}, keywords = {chinese, corpus, detection, learning, sinica, word}, number = {1}, pages = {27--44}, posted-at = {2006-04-06 06:33:03}, priority = {5}, title = {Unknown Word Detection for Chinese by a Corpus-based Learning Method}, volume = {3}, year = {1998} }

TY - JOUR ID - citeulike:578081 L3 - citeulike-article-id:578081 N2 - One of the most prominent problems in computer processing of the Chinese language is identification of the words in a sentence. Since there are no blanks to mark word boundaries, identifying words is difficult because of segmentation ambiguities and occurrences of out-of-vocabulary words (i.e., unknown words). In this paper, a corpus-based learning method is proposed which derives sets of syntactic rules that are applied to distinguish monosyllabic words from monosyllabic morphemes which may be parts of unknown words or typographical errors. The corpus-based learning approach has the advantages of: 1. automatic rule learning, 2. automatic evaluation of the performance of each rule, and 3. balancing of recall and precision rates through dynamic rule set selection. The experimental results show that the rule set derived using the proposed method outperformed hand-crafted rules produced by human experts in detecting unknown words. IS - 1 JF - International Journal of Computational linguistics and Chinese Language Processing EP - 44 TI - Unknown Word Detection for Chinese by a Corpus-based Learning Method VL - 3 SP - 27 KW - chinese KW - corpus KW - detection KW - learning KW - sinica KW - word AU - Chen, Keh-Jiann AU - Bai, Ming-Hong PY - 1998/// ER -