An Extensive Empirical Study of Feature Selection Metrics for Text Classification Export

@article{citeulike:1915886, abstract = {Machine learning for text classification is the cornerstone of document categorization, news filtering, document routing, and personalization. In text domains, effective feature selection is essential to make the learning task efficient and more accurate. This paper presents an empirical comparison of twelve feature selection methods (e.g. Information Gain) evaluated on a benchmark of 229 text classification problem instances that were gathered from Reuters, TREC, OHSUMED, etc. The results are analyzed from multiple goal perspectives-accuracy, F-measure, precision, and recall-since each is appropriate in different situations. The results reveal that a new feature selection metric we call 'Bi-Normal Separation' (BNS), outperformed the others by a substantial margin in most situations. This margin widened in tasks with high class skew, which is rampant in text classification problems and is particularly challenging for induction algorithms. A new evaluation methodology is offered that focuses on the needs of the data mining practitioner faced with a single dataset who seeks to choose one (or a pair of) metrics that are most likely to yield the best performance. From this perspective, BNS was the top single choice for all goals except precision, for which Information Gain yielded the best result most often. This analysis also revealed, for example, that Information Gain and Chi-Squared have correlated failures, and so they work poorly together. When choosing optimal pairs of metrics for each of the four performance goals, BNS is consistently a member of the pair---e.g., for greatest recall, the pair BNS + F1-measure yielded the best performance on the greatest number of tasks by a considerable margin.}, author = {Forman, George}, citeulike-article-id = {1915886}, citeulike-linkout-0 = {http://www.jmlr.org/papers/volume3/forman03a/forman03a.pdf}, journal = {Journal of Machine Learning Research}, keywords = {bns, classification, empirical, feature, selection, text}, month = {March}, pages = {1289--1305}, posted-at = {2008-01-29 10:58:06}, priority = {0}, title = {An Extensive Empirical Study of Feature Selection Metrics for Text Classification}, url = {http://www.jmlr.org/papers/volume3/forman03a/forman03a.pdf}, volume = {3}, year = {2003} }

TY - JOUR ID - citeulike:1915886 L3 - citeulike-article-id:1915886 N2 - Machine learning for text classification is the cornerstone of document categorization, news filtering, document routing, and personalization. In text domains, effective feature selection is essential to make the learning task efficient and more accurate. This paper presents an empirical comparison of twelve feature selection methods (e.g. Information Gain) evaluated on a benchmark of 229 text classification problem instances that were gathered from Reuters, TREC, OHSUMED, etc. The results are analyzed from multiple goal perspectives-accuracy, F-measure, precision, and recall-since each is appropriate in different situations. The results reveal that a new feature selection metric we call 'Bi-Normal Separation' (BNS), outperformed the others by a substantial margin in most situations. This margin widened in tasks with high class skew, which is rampant in text classification problems and is particularly challenging for induction algorithms. A new evaluation methodology is offered that focuses on the needs of the data mining practitioner faced with a single dataset who seeks to choose one (or a pair of) metrics that are most likely to yield the best performance. From this perspective, BNS was the top single choice for all goals except precision, for which Information Gain yielded the best result most often. This analysis also revealed, for example, that Information Gain and Chi-Squared have correlated failures, and so they work poorly together. When choosing optimal pairs of metrics for each of the four performance goals, BNS is consistently a member of the pair---e.g., for greatest recall, the pair BNS + F1-measure yielded the best performance on the greatest number of tasks by a considerable margin. JF - Journal of Machine Learning Research EP - 1305 TI - An Extensive Empirical Study of Feature Selection Metrics for Text Classification VL - 3 SP - 1289 KW - bns KW - classification KW - empirical KW - feature KW - selection KW - text AU - Forman, George PY - 2003/03// UR - http://www.jmlr.org/papers/volume3/forman03a/forman03a.pdf ER -