Testing heuristics: We have it all wrong Export

@article{hooker95testing, abstract = {The competitive nature of most algorithmic experimentation is a source of problems that are all too familiar to the research community. It is hard to make fair comparisons between algorithms and to assemble realistic test problems. Competitive testing tells us which algorithm is faster but not why. Because it requires polished code, it consumes time and energy that could be spent doing more experiments. This paper argues that a more scientific approach of controlled experimentation, similar to that used in other empirical sciences, avoids or alleviates these problems. We have confused research and development; competitive testing is suited only for the latter. Most experimental studies of heuristic algorithms resemble track meets more than scientific endeavors. Typically an investigator has a bright idea for a new algorithm and wants to show that it works better, in some sense, than known algorithms.}, author = {Hooker, J. N.}, citeulike-article-id = {3170710}, citeulike-linkout-0 = {http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.34.4103}, journal = {Journal of Heuristics}, keywords = {experimental-method}, pages = {33--42}, posted-at = {2008-08-28 21:53:33}, priority = {2}, title = {Testing heuristics: We have it all wrong}, url = {http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.34.4103}, volume = {1}, year = {1995} }

TY - JOUR ID - hooker95testing L3 - citeulike-article-id:3170710 N2 - The competitive nature of most algorithmic experimentation is a source of problems that are all too familiar to the research community. It is hard to make fair comparisons between algorithms and to assemble realistic test problems. Competitive testing tells us which algorithm is faster but not why. Because it requires polished code, it consumes time and energy that could be spent doing more experiments. This paper argues that a more scientific approach of controlled experimentation, similar to that used in other empirical sciences, avoids or alleviates these problems. We have confused research and development; competitive testing is suited only for the latter. Most experimental studies of heuristic algorithms resemble track meets more than scientific endeavors. Typically an investigator has a bright idea for a new algorithm and wants to show that it works better, in some sense, than known algorithms. JF - Journal of Heuristics EP - 42 TI - Testing heuristics: We have it all wrong VL - 1 SP - 33 KW - experimental-method AU - Hooker, JN PY - 1995/// UR - http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.34.4103 ER -