Efficient access to many small files in a filesystem for grid computing Export

@inproceedings{citeulike:3109554, abstract = {Many potential users of grid computing systems have a need to manage large numbers of small files. However, computing and storage grids are generally optimized for the management of large files. As a result, users with small files achieve performance several orders of magnitude worse than possible. Archival tools and custom storage structures can be used to improve small-file performance, but this requires the end user to change the behavior of the application, which is not always practical. To address this problem, we augment the protocol of the Chirp filesystem for grid computing to improve small file performance. We describe in detail how this protocol compares to FTP and NFS, which are widely used in similar situations. In addition, we observe that changes to the system call interface are necessary to invoke the protocol properly. We demonstrate an order-of-magnitude performance improvement over existing protocols for copying files and manipulating large directory trees.}, author = {Thain, D. and Moretti, C.}, booktitle = {Grid Computing, 2007 8th IEEE/ACM International Conference on}, citeulike-article-id = {3109554}, citeulike-linkout-0 = {http://dx.doi.org/10.1109/GRID.2007.4354139}, citeulike-linkout-1 = {http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4354139}, doi = {10.1109/GRID.2007.4354139}, journal = {Grid Computing, 2007 8th IEEE/ACM International Conference on}, keywords = {computing, data, grid}, pages = {243--250}, posted-at = {2008-08-11 20:58:52}, priority = {2}, title = {Efficient access to many small files in a filesystem for grid computing}, url = {http://dx.doi.org/10.1109/GRID.2007.4354139}, year = {2007} }

TY - CONF ID - citeulike:3109554 L3 - citeulike-article-id:3109554 N2 - Many potential users of grid computing systems have a need to manage large numbers of small files. However, computing and storage grids are generally optimized for the management of large files. As a result, users with small files achieve performance several orders of magnitude worse than possible. Archival tools and custom storage structures can be used to improve small-file performance, but this requires the end user to change the behavior of the application, which is not always practical. To address this problem, we augment the protocol of the Chirp filesystem for grid computing to improve small file performance. We describe in detail how this protocol compares to FTP and NFS, which are widely used in similar situations. In addition, we observe that changes to the system call interface are necessary to invoke the protocol properly. We demonstrate an order-of-magnitude performance improvement over existing protocols for copying files and manipulating large directory trees. JF - Grid Computing, 2007 8th IEEE/ACM International Conference on EP - 250 TI - Efficient access to many small files in a filesystem for grid computing T2 - Grid Computing, 2007 8th IEEE/ACM International Conference on SP - 243 KW - computing KW - data KW - grid AU - Thain, D AU - Moretti, C PY - 2007/// UR - http://dx.doi.org/10.1109/GRID.2007.4354139 ER -