=================== Borealis Data Files =================== Data Generation --------------- The Borealis software module ``data_write.py`` is responsible for writing all data files. Different flags can be given to the module to write various types of files. See the documentation for :ref:`Data Write`. Borealis writes files into `HDF5 format `_. Examples of how to use HDF5 files can be found here for multiple languages: `HDF5 Examples `_ The following data file types can be generated by Borealis in HDF5 format. The standard Borealis ``release`` mode run by the scheduler generates HDF5 files for rawacf and antennas_iq types. Borealis filetypes ------------------ These are the Borealis filetypes produced by the radar software, from most processed to least processed. * **rawacf:** The correlated data from the main and interferometer arrays. Produced by Borealis in release mode. * **bfiq:** The beamformed IQ data from both arrays. * **antennas_iq:** The IQ data from every antenna. Produced by Borealis in release mode. * **rawrf:** The unfiltered, full receive bandwidth data from every antenna. Only produced by Borealis in debug modes. Post-processed DMAP files can be created from the hdf5 rawacf or bfiq files using the `pyDARNio package `_. For more information on the data files and the fields stored within them, check the data file information for the correct Borealis software version. Reading Data ------------ To read the files in python, we recommend using `pyDARNio `_. If you are looking to generate SuperDARN standard plots, we recommend using `pyDARN `_, which can read Borealis files specifically. After converting to DMAP, standard SuperDARN plots including RTI plots and fan plots can be produced. Data Storage and Deletion ------------------------- Borealis file sizes can add up quickly to fill all available hard drive space, especially if antennas_iq and/or bfiq data types are being generated. However, it is convenient and recommended to keep a backlog of lower level data products such as antennas_iq for a period of time. These files are useful for debugging hardware issues and reproducing RAWACF files. .. _current_file_types: Current version --------------- .. tabs:: .. tab:: info The Borealis software version can affect the data fields in the file format so be sure to check if your data is of the most up to date version. .. tab:: rawacf .. include:: file_types/rawacf.txt .. tab:: bfiq .. include:: file_types/bfiq.txt .. tab:: antennas_iq .. include:: file_types/antennas_iq.txt .. tab:: rawrf .. include:: file_types/rawrf.txt .. _dmap-equivalents: DMAP-equivalent files --------------------- .. tabs:: .. tab:: info Mappings between ``rawacf`` and ``bfiq`` HDF5 files and their DMAP equivalents ``rawacf`` and ``iqdat`` are detailed in the corresponding tabs. .. tab:: bfiq .. include:: file_types/iqdat_mapping.txt .. tab:: rawacf .. include:: file_types/rawacf_mapping.txt Previous Versions ----------------- In prior Borealis versions, data files could be in one of two "structures": ``site`` or ``array``. ``site`` structured files were written directly by the radar, consisting of one HDF5 group per averaging period of the experiment. Each group, or "record", contained the same metadata and data fields, even for static metadata (i.e. not dependent on the specific averaging period, for example, the experiment name). The naming convention of site-structured files is:: [YYYYMMDD].[HHMM].[SS].[station_id].[slice_id].[file type].hdf5.site ``array`` structured files could be created using pyDARNio, where the HDF5 groups were removed and all matching fields from different records combined into one dataset. This structure is more ergonomic, and reduces the size of the file. However, converting between the structures can take a significant amount of time and memory. The naming convention of array-structured files is:: [YYYYMMDD].[HHMM].[SS].[station_id].[slice_id].[file type].hdf5 .. tabs:: .. tab:: v0.2 and v0.3 These versions follow the v0.4 format. .. tab:: v0.4 .. tabs:: .. tab:: rawacf .. tabs:: .. tab:: array .. include:: archive/rawacf-v04-array.txt .. tab:: site .. include:: archive/rawacf-v04-site.txt .. tab:: bfiq .. tabs:: .. tab:: array .. include:: archive/bfiq-v04-array.txt .. tab:: site .. include:: archive/bfiq-v04-site.txt .. tab:: antennas_iq .. tabs:: .. tab:: array .. include:: archive/antiq-v04-array.txt .. tab:: site .. include:: archive/antiq-v04-site.txt .. tab:: rawrf .. include:: archive/rawrf-v04.txt .. tab:: v0.5 .. tabs:: .. tab:: rawacf .. tabs:: .. tab:: array .. include:: archive/rawacf-v05-array.txt .. tab:: site .. include:: archive/rawacf-v05-site.txt .. tab:: bfiq .. tabs:: .. tab:: array .. include:: archive/bfiq-v05-array.txt .. tab:: site .. include:: archive/bfiq-v05-site.txt .. tab:: antennas_iq .. tabs:: .. tab:: array .. include:: archive/antiq-v05-array.txt .. tab:: site .. include:: archive/antiq-v05-site.txt .. tab:: rawrf .. include:: archive/rawrf-v05.txt .. tab:: v0.6 .. tabs:: .. tab:: rawacf .. tabs:: .. tab:: array .. include:: archive/rawacf-v06-array.txt .. tab:: site .. include:: archive/rawacf-v06-site.txt .. tab:: bfiq .. tabs:: .. tab:: array .. include:: archive/bfiq-v06-array.txt .. tab:: site .. include:: archive/bfiq-v06-site.txt .. tab:: antennas_iq .. tabs:: .. tab:: array .. include:: archive/antiq-v06-array.txt .. tab:: site .. include:: archive/antiq-v06-site.txt .. tab:: rawrf .. include:: archive/rawrf-v06.txt .. tab:: v0.7 .. tabs:: .. tab:: rawacf .. tabs:: .. tab:: array .. include:: archive/rawacf-v07-array.txt .. tab:: site .. include:: archive/rawacf-v07-site.txt .. tab:: bfiq .. tabs:: .. tab:: array .. include:: archive/bfiq-v07-array.txt .. tab:: site .. include:: archive/bfiq-v07-site.txt .. tab:: antennas_iq .. tabs:: .. tab:: array .. include:: archive/antiq-v07-array.txt .. tab:: site .. include:: archive/antiq-v07-site.txt .. tab:: rawrf .. include:: archive/rawrf-v07.txt