Dynamic tables: An architecture for managing evolving, heterogeneous biomedical data in relational database management systems

Data sparsity and schema evolution issues affecting bioinformatics and medical informatics communities have forced the adoption of vertical or object-attribute-value based database schemas to overcome limitations posed when using conventional relational database technology. Through our collaboration with the Yale Center for Medial Informatics (YCMI), we explore the reasons for this and show why their data is difficult to model using conventional relational techniques. We propose a solution to these obstacles based on a relational database engine using a sparse, column-store architecture. We provide benchmarks comparing the performance of queries and schema-modification operations using three different strategies: (1) the standard conventional relational design, (2) past approaches used by clinical and neuroinformatics researchers, and (3) our sparse, column-store architecture. Our performance results show that our architecture is a promising technique for storing and processing many types of data that were not handled well by conventional nor other derived semantic data models. 1