Entropy-based Pruning of Backoff Language Models Export

@inproceedings{citeulike:5932703, abstract = {A criterion for pruning parameters from N-gram backoff language models is developed, based on the relative entropy between the original and the pruned model. It is shown that the relative entropy resulting from pruning a single N-gram can be computed exactly and efficiently for backoff models. The relative entropy measure can be expressed as a relative change in training set perplexity. This leads to a simple pruning criterion whereby all N-grams that change perplexity by less than a threshold are removed from the model. Experiments show that a production-quality Hub4 LM can be reduced to 26\% its original size without increasing recognition error. We also compare the approach to a heuristic pruning criterion by Seymore and Rosenfeld [9], and show that their approach can be interpreted as an approximation to the relative entropy criterion. Experimentally, both approaches select similar sets of N-grams (about 85\% overlap), with the exact relative entropy criterion giving marginally bette...}, author = {Stolcke, Andreas}, booktitle = {In Proc. DARPA Broadcast News Transcription and Understanding Workshop}, citeulike-article-id = {5932703}, citeulike-linkout-0 = {http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.56.8924}, keywords = {lm, pruning}, pages = {270--274}, posted-at = {2009-10-13 06:15:33}, priority = {0}, title = {Entropy-based Pruning of Backoff Language Models}, url = {http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.56.8924}, year = {1998} }

TY - CONF ID - citeulike:5932703 L3 - citeulike-article-id:5932703 N2 - A criterion for pruning parameters from N-gram backoff language models is developed, based on the relative entropy between the original and the pruned model. It is shown that the relative entropy resulting from pruning a single N-gram can be computed exactly and efficiently for backoff models. The relative entropy measure can be expressed as a relative change in training set perplexity. This leads to a simple pruning criterion whereby all N-grams that change perplexity by less than a threshold are removed from the model. Experiments show that a production-quality Hub4 LM can be reduced to 26% its original size without increasing recognition error. We also compare the approach to a heuristic pruning criterion by Seymore and Rosenfeld [9], and show that their approach can be interpreted as an approximation to the relative entropy criterion. Experimentally, both approaches select similar sets of N-grams (about 85% overlap), with the exact relative entropy criterion giving marginally bette... EP - 274 TI - Entropy-based Pruning of Backoff Language Models T2 - In Proc. DARPA Broadcast News Transcription and Understanding Workshop SP - 270 KW - lm KW - pruning AU - Stolcke, Andreas PY - 1998/// UR - http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.56.8924 ER -