Borealis Data Files¶
The Borealis software module data_write.py is responsible for writing all data files. Different flags can be given to the module to write various types of files. See the documentation for Borealis Processes
The following data file types can be generated by Borealis in HDF5 format. The standard Borealis release mode run by the scheduler generates HDF5 files for rawacf, antennas_iq and bfiq.
These are the Borealis filetypes produced by the radar software, from most processed to least processed.
The correlated data from the main and interferometer arrays. Produced by Borealis in release mode.
The beamformed iq data from both arrays. Produced by Borealis in release mode.
The iq data from every antenna. Produced by Borealis in release mode.
The unfiltered, full receive bandwidth data from every antenna. Only produced by Borealis in debug modes.
Post-processed dmap files can be created from the hdf5 rawacf or bfiq files using the pyDARNio package.
For more information on the data files and the fields stored within them, check the data file information for the correct Borealis software version.
Borealis current version¶
The Borealis software version can affect the data fields in the file format so be sure to check if your data is of the most up to date version. The current Borealis software version is v0.6.
- rawacf v0.6
- bfiq v0.6
- antennas_iq v0.6
- rawrf v0.6
v0.2, v0.3, and v0.4 follow the v0.4 format.
v0.5 follows the v0.5 format.
- rawacf v0.4
- bfiq v0.4
- antennas_iq v0.4
- rawrf v0.4
- rawacf v0.5
- bfiq v0.5
- antennas_iq v0.5
- rawrf v0.5
To read the files in python, we recommend using h5py, PyTables, or deepdish packages. If you are looking to generate SuperDARN standard plots, we recommend using the the pyDARN package, which can read Borealis files specifically. After converting to dmap, standard SuperDARN plots including RTI plots and fan plot can be produced.
Data Storage and Deletion¶
Borealis file sizes can add up quickly to fill all available hard drive space, especially if antennas_iq and/or bfiq data types are being generated. However, it is convenient and recommended to keep a backlog of lower level data products such as antennas_iq for a period of time. These files are useful for debugging hardware issues and reproducing RAWACF files.
In order to prevent system failure due to hard drives filling up, a method for deleting the oldest data files is employed for SuperDARN Canada radars. This is referred to as rotating the files.
A utility script is scheduled via cron to check the filesystem that Borealis files are written to. If the filesystem usage is too high, it searches for and deletes the oldest files in a loop until the filesystem usage goes below the threshold. See the SuperDARN Canada data flow repository for more information.