Charles Explorer logo
🇬🇧

Column-oriented Data Store for Astrophysical Data

Publication

Abstract

A hybrid scientific database that combines the benefits of two approaches. The metadata, which are often listed and searched by traditional (SQL) means are stored in classical RDBMS.

The scientific data (stelar spectra in our case) are stored in a custom binary format which was designed as a column-oriented data store. The binary files may be read by various system APIs, which have different benefits and drawbacks.

We have tested them all and present a performance comparison. The best selected methods achieved speedup of two orders of magnitude with respect to the original data representation (FITS format) used by astrophysicists.