Fallacies in evaluating decentralized systems Export

@inproceedings{citeulike:5766778, abstract = {Research on decentralized systems such as peer-to-peer overlays and ad hoc networks has been hampered by the fact that few systems of this type are in production use, and the space of possible applications is still poorly understood. As a consequence, new ideas have mostly been evaluated using common synthetic workloads, traces from a few existing systems, testbeds like PlanetLab, and simulators like ns-2. Some of these methods have, in fact, become the ” gold standard ” for evaluating new systems, and are often a prerequisite for getting papers accepted at top conferences in the field. In this paper, we examine the current practice of evaluating decentralized systems under these specific sets of conditions and point out pitfalls associated with this practice. In particular, we argue that (i) despite authors ' best intentions, results from such evaluations often end up being inappropriately generalized; (ii) there is an incentive not to deviate from the accepted standard of evaluation, even if that is technically appropriate; (iii) research may gravitate towards systems that are feasible and perform well when evaluated in the accepted environments; and, (iv) in the worst-case, research may become ossified as a result. We close with a call to action for the community to develop tools, data, and best practices that allow systems to be evaluated across a space of workloads and environments. 1.}, author = {Haeberlen, Andreas and Mislove, Alan and Post, Ansley and Druschel, Peter}, booktitle = {In Proceedings of IPTPS}, citeulike-article-id = {5766778}, citeulike-linkout-0 = {http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.138.7754}, comment = {This is a very well-written paper discussing the pitfalls that the research community can fall into when evaluating systems -- here the paper is focused on P2P, but the general guidelines apply to all research. One of the papers that you have to reread every once in a while.}, keywords = {evaluation, fallacies, i-love-it, p2p}, posted-at = {2009-09-10 10:53:38}, priority = {0}, title = {Fallacies in evaluating decentralized systems}, url = {http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.138.7754}, year = {2006} }

TY - CONF ID - citeulike:5766778 L3 - citeulike-article-id:5766778 N2 - Research on decentralized systems such as peer-to-peer overlays and ad hoc networks has been hampered by the fact that few systems of this type are in production use, and the space of possible applications is still poorly understood. As a consequence, new ideas have mostly been evaluated using common synthetic workloads, traces from a few existing systems, testbeds like PlanetLab, and simulators like ns-2. Some of these methods have, in fact, become the “gold standard ” for evaluating new systems, and are often a prerequisite for getting papers accepted at top conferences in the field. In this paper, we examine the current practice of evaluating decentralized systems under these specific sets of conditions and point out pitfalls associated with this practice. In particular, we argue that (i) despite authors ’ best intentions, results from such evaluations often end up being inappropriately generalized; (ii) there is an incentive not to deviate from the accepted standard of evaluation, even if that is technically appropriate; (iii) research may gravitate towards systems that are feasible and perform well when evaluated in the accepted environments; and, (iv) in the worst-case, research may become ossified as a result. We close with a call to action for the community to develop tools, data, and best practices that allow systems to be evaluated across a space of workloads and environments. 1. TI - Fallacies in evaluating decentralized systems T2 - In Proceedings of IPTPS KW - evaluation KW - fallacies KW - i-love-it KW - p2p N1 - This is a very well-written paper discussing the pitfalls that the research community can fall into when evaluating systems -- here the paper is focused on P2P, but the general guidelines apply to all research. One of the papers that you have to reread every once in a while. AU - Haeberlen, Andreas AU - Mislove, Alan AU - Post, Ansley AU - Druschel, Peter PY - 2006/// UR - http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.138.7754 ER -