Borealis Data Files

Data Generation

The Borealis software module data_write.py is responsible for writing all data files. Different flags can be given to the module to write various types of files. See the documentation for Borealis Processes

Borealis writes files into HDF5 format. Examples of how to use HDF5 files can be found here for multiple languages: HDF5 Examples

The following data file types can be generated by Borealis in HDF5 format. The standard Borealis release mode run by the scheduler generates HDF5 files for rawacf, antennas_iq and bfiq.

Borealis filetypes

These are the Borealis filetypes produced by the radar software, from most processed to least processed.

• rawacf

  The correlated data from the main and interferometer arrays. Produced by Borealis in release mode.

• bfiq

  The beamformed iq data from both arrays. Produced by Borealis in release mode.

• antennas_iq

  The iq data from every antenna. Produced by Borealis in release mode.

• rawrf

  The unfiltered, full receive bandwidth data from every antenna. Only produced by Borealis in debug modes.

Post-processed dmap files can be created from the hdf5 rawacf or bfiq files using the pyDARN package.

For more information on the data files and the fields stored within them, check the data file information for the correct Borealis software version.

Borealis current version

The Borealis software version can affect the data fields in the file format so be sure to check if your data is of the most up to date version. The current Borealis software version is v0.5.

• rawacf v0.5
  • rawacf array files
  • rawacf site files
  • Site/Array Restructuring
  • rawacf to rawacf SDARN (DMap) Conversion
• bfiq v0.5
  • bfiq array files
  • bfiq site files
  • Site/Array Restructuring
  • bfiq to iqdat SDARN (DMap) Conversion
• antennas_iq v0.5
  • antennas_iq array files
  • antennas_iq site files
  • Site/Array Restructuring
• rawrf v0.5
  • rawrf site files
  • Site/Array Restructuring

Previous versions

• v0.2, v0.3, and v0.4 follow the v0.4 format.

Reading Data

To read the files in python, we recommend using PyTables or deepdish packages. If you are looking to generate SuperDARN standard plots, we recommend using the the pyDARN package, which can read Borealis files specifically. After converting to dmap, standard SuperDARN plots including RTI plots and fan plot can be produced.

Data Storage and Deletion

Borealis file sizes can add up quickly to fill all available hard drive space, especially if antennas_iq and/or bfiq data types are being generated. However, it is convenient and recommended to keep a backlog of lower level data products such as antennas_iq for a period of time. These files are useful for debugging hardware issues and reproducing RAWACF files.

A utility script is scheduled via cron to check the filesystem that Borealis files are written to. If the filesystem usage is too high, it searches for and deletes the oldest files in a loop until the filesystem usage goes below the threshold. See the SuperDARN Canada data flow repository for more information.

In order to prevent system failure due to hard drives filling up, a method for deleting the oldest data files is employed for SuperDARN Canada radars. This is referred to as rotating the files.