Unknown Word Extraction for Chinese Documents Export

@inproceedings{citeulike:578084, abstract = {There is no blank to mark word boundaries in Chinese text. As a result, identifying words is difficult, because of segmentation ambiguities and occurrences of unknow words. Most previous works focus their attention only on the resolution of ambiguous segmentation. The problem of unknown word identification is considered more difficult and needs further investigation. Convertionally unknown words were extracted by statistical methods for statistical methods are simple and efficient. Howevere the statistical methods without using linguistic knowledge suffer the drawbacks of low precision and low recall. Because character strings with statistical significance might be phrases or partical phrases instead of words and low frequency new words are hardly identifiable by statistic methods. In addition to statistical information, we try to use as much information as possible, such as morphology, syntax, semantics, and world knowledge. The identification system fully utilizes the context and content information of unknown word in the steps of detection process, extraction process, and verification process. A practical unknown word extraction system was implemented wich online identifies new words, including low frequency new words, with high precision and high recall rates.}, author = {Chen, Keh-Jiann and Ma, Wei-Yun}, citeulike-article-id = {578084}, journal = {Proceedings of Coling 2002}, keywords = {chinese, detection, extraction, sinica, word}, pages = {169--175}, posted-at = {2006-04-06 06:41:49}, priority = {5}, title = {Unknown Word Extraction for Chinese Documents}, year = {2002} }

TY - CONF ID - citeulike:578084 L3 - citeulike-article-id:578084 N2 - There is no blank to mark word boundaries in Chinese text. As a result, identifying words is difficult, because of segmentation ambiguities and occurrences of unknow words. Most previous works focus their attention only on the resolution of ambiguous segmentation. The problem of unknown word identification is considered more difficult and needs further investigation. Convertionally unknown words were extracted by statistical methods for statistical methods are simple and efficient. Howevere the statistical methods without using linguistic knowledge suffer the drawbacks of low precision and low recall. Because character strings with statistical significance might be phrases or partical phrases instead of words and low frequency new words are hardly identifiable by statistic methods. In addition to statistical information, we try to use as much information as possible, such as morphology, syntax, semantics, and world knowledge. The identification system fully utilizes the context and content information of unknown word in the steps of detection process, extraction process, and verification process. A practical unknown word extraction system was implemented wich online identifies new words, including low frequency new words, with high precision and high recall rates. JF - Proceedings of Coling 2002 EP - 175 TI - Unknown Word Extraction for Chinese Documents SP - 169 KW - chinese KW - detection KW - extraction KW - sinica KW - word AU - Chen, Keh-Jiann AU - Ma, Wei-Yun PY - 2002/// ER -