Faced with a long-term endeavor to gather and process vast amounts of visual information from the universe, the Vera C. Rubin Observatory in the mountains of Chile turned into an open source, time series database, InfluxDB, developed by InfluxData.
The observatory and its 8.4-meter optical telescope are being constructed to survey the area of space viewable in the southern hemisphere for 10 years, capture about 1,000 pictures of the sky to a nightly basis. The project, known as the Legacy Survey of Space and Time, is expected to create 500 petabytes of visual data astronomers should be able to utilize to better understand the cosmos.
The Rubin Observatory, funded by the National Science Foundation and the Department of Energy, will aim to assemble data on several 37 billion stars and galaxies, and gain further insight on leading phenomena like dark matter, dark energy, and asteroid movement.
Operating complex astronomical telescopes requires a sound understanding of the instrumentation, states Frossie Economou, project manager for the Rubin Observatory Science Platform. Although the observatory is currently in Chile, scientists around the globe have interest in the information, she says.
As the project progressed, Economou claims that the team recognized they needed to concentrate on that intricate work rather than be tied up dealing with storing and processing a flood of instrument readings. When the 3-gigapixel camera, telescope, and other gear are fully assembled, the observatory is expected to create significant data in a high frequency, she says. “The telemetry is high volume. Even without the telescope in full construction we are already collecting about a terabyte of telemetry a day.”
The team now uses the open source edition of InfluxDB, says Angelo Fausti, software engineer with the observatory, though they are updating to another tier. “We are currently planning the migration to InfluxDB 2.0,” he states. This migration will include a brand new user interface with new visualization capabilities for different scatter plots, heat maps, and histograms, Fausti states. “It’s a tool made for developers and we, as scientists and engineers, are also developers.”
The observatory made a prior attempt to build a traditional MySQL, relational database, Economou states, to store and examine telemetry, but it had been challenge. The team was already using Apache Kafka and InfluxData for another use case at the time, she states, and recognized those resources could be utilised to collect data at a higher frequency, quantity, and throughput. “We realized that our telemetry was a very good fit for this,” she states. The observatory team then built their engineering fac