Borealis Data Files¶
Data Generation¶
The Borealis software module data_write.py is responsible for writing all data files. Different
flags can be given to the module to write various types of files. See the documentation for
Data Write.
Borealis writes files into HDF5 format. Examples of how to use HDF5 files can be found here for multiple languages: HDF5 Examples
The following data file types can be generated by Borealis in HDF5 format. The standard Borealis
release mode run by the scheduler generates HDF5 files for rawacf and antennas_iq types.
Borealis filetypes¶
These are the Borealis filetypes produced by the radar software, from most processed to least processed.
rawacf: The correlated data from the main and interferometer arrays. Produced by Borealis in release mode.
bfiq: The beamformed IQ data from both arrays.
antennas_iq: The IQ data from every antenna. Produced by Borealis in release mode.
rawrf: The unfiltered, full receive bandwidth data from every antenna. Only produced by Borealis in debug modes.
Post-processed DMAP files can be created from the hdf5 rawacf or bfiq files using the pyDARNio package.
For more information on the data files and the fields stored within them, check the data file information for the correct Borealis software version.
Reading Data¶
To read the files in python, we recommend using pyDARNio. If you are looking to generate SuperDARN standard plots, we recommend using pyDARN, which can read Borealis files specifically. After converting to DMAP, standard SuperDARN plots including RTI plots and fan plots can be produced.
Data Storage and Deletion¶
Borealis file sizes can add up quickly to fill all available hard drive space, especially if antennas_iq and/or bfiq data types are being generated. However, it is convenient and recommended to keep a backlog of lower level data products such as antennas_iq for a period of time. These files are useful for debugging hardware issues and reproducing RAWACF files.
Current version¶
The Borealis software version can affect the data fields in the file format so be sure to check if your data is of the most up to date version.
Each field contains metadata that determines how the field is written to file.
descriptionA description of the field.
dim_labelsIf applicable, a brief descriptor for each dimension of the dataset. This could be different for different
groupvalues. If so, this metadata will be a dict, with the keys being thegroupname and the values the associated list of dimension labels.dim_scalesIf applicable, dimension scales will be associated to the field. These are datasets that match one of the dimensions of the data, such as timestamps to go along with an array of collected data. Note that some dimensions may be associated with multiple fields. If a dimension has no associated dataset, the list will have a
Noneentry.groupsThe types of data file that need this field to be written.
levelThe level within the file that the data will be stored at. Either
fileorrecord, indicating that the field is either written once per file, or once per record.nicknameA nickname for the field, used for making Dimension Scale names.
unitsUnits for the data.
- agc_status_word:
32 bits, a 1 in bit position corresponds to an AGC fault on that transmitter
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- antenna_locations:
Relative antenna locations
dim_labels:
['antenna', 'local_coord']dim_scales:
['antennas', 'local_coord']level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']units:
m
- antennas:
Labels for each antenna of the radar
dim_labels:
['antenna']level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- averaging_method:
Averaging method, e.g. mean, median
level:
filerequired_for:
['rawacf']
- beam_azms:
Beams azimuths for each beam in degrees CW of boresight
dim_labels:
['beam']level:
recordnickname:
beam directionrequired_for:
['antennas_iq', 'bfiq', 'rawacf']units:
degrees
- beam_nums:
Beam numbers used in this slice
dim_labels:
['beam']level:
recordnickname:
beam numberrequired_for:
['antennas_iq', 'bfiq', 'rawacf']
- blanked_samples:
Samples blanked during transmission of a pulse
dim_labels:
['time']level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- borealis_git_hash:
Version and commit hash of Borealis at runtime
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- cfs_freqs:
Frequencies measured during clear frequency search
dim_labels:
['freq']level:
filenickname:
freqrequired_for:
[]units:
Hz
- cfs_masks:
Mask for cfs_freqs restricting freqs available for setting cfs slice freq
dim_labels:
['freq']dim_scales:
['cfs_freqs']level:
recordrequired_for:
[]
- cfs_noise:
Power measured during clear frequency search
dim_labels:
['freq']dim_scales:
['cfs_freqs']level:
recordrequired_for:
[]units:
a.u. ~ dBW
- cfs_range:
Lower and upper bound of frequencies examined by clear frequency search
level:
filerequired_for:
[]units:
Hz
- data_normalization_factor:
Cumulative scale of all of the filters for a total scaling factor to normalize by
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf']
- experiment_comment:
Comment about the whole experiment
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- experiment_id:
Number used to identify experiment
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- experiment_name:
Name of the experiment class
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- first_range:
Distance to first range in km
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf']units:
km
- first_range_rtt:
Round trip time of flight to first range in microseconds
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf']units:
μs
- freq:
Frequency used for this experiment slice, in kHz
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']units:
kHz
- global_coord:
Descriptors for global coordinates
dim_labels:
['global_coord']level:
filenickname:
global coordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- gps_locked:
True if the GPS was locked during the entire averaging period
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- gps_to_system_time_diff:
Max time diff in seconds between GPS and system/NTP time during the averaging period
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']units:
s
- int_time:
Integration time in seconds
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']units:
s
- intf_acfs:
Interferometer array autocorrelations
dim_labels:
['beam', 'range', 'lag']dim_scales:
[['beam_azms', 'beam_nums'], 'range_gates', 'lag_numbers']level:
recordrequired_for:
[]units:
a.u. ~ W
- lags:
Lag indices
level:
filenickname:
lagrequired_for:
['rawacf']
- lag_numbers:
Difference in units of tau_spacing of unique pairs of pulse in the pulse array
level:
filenickname:
lagrequired_for:
['rawacf']units:
tau_spacing
- lag_pulses:
Unique pairs of pulses in pulse array, in units of tau_spacing
dim_labels:
['lag', 'pulse']dim_scales:
['lags', 'lag_pulse_descriptors']level:
filerequired_for:
['rawacf']units:
tau_spacing
- lag_pulse_descriptors:
Descriptor of the pulse pairs used in a lag
level:
filerequired_for:
['rawacf']
- local_coord:
Descriptors for local coordinates
dim_labels:
['local_coord']level:
filenickname:
local coordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- lp_status_word:
32 bits, a 1 in bit position corresponds to a low power condition on that transmitter
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- main_acfs:
Main array autocorrelations
dim_labels:
['beam', 'range', 'lag']dim_scales:
[['beam_azms', 'beam_nums'], 'range_gates', 'lag_numbers']level:
recordrequired_for:
['rawacf']units:
a.u. ~ W
- num_sequences:
Number of sampling periods in the averaging period
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- num_slices:
Number of slices in the experiment for this averaging period
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- pulses:
Pulse sequence in units of tau_spacing
dim_labels:
['pulse']level:
filenickname:
pulserequired_for:
['antennas_iq', 'bfiq', 'rawacf']
- range_gates:
Range gates of interest for the experiment, beginning at
first_rangeand spaced byrange_seplevel:
filenickname:
range gaterequired_for:
['antennas_iq', 'bfiq', 'rawacf']
- range_sep:
Range gate separation (equivalent distance between samples) in km
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf']units:
km
- rx_intf_antennas:
Indices into
antenna_locationsof the interferometer array antennas used in this experimentlevel:
filerequired_for:
[]
- rx_main_antennas:
Indices into
antenna_locationsof the main array antennas used in this experimentlevel:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- rx_sample_rate:
Sampling rate of the samples being written to file in Hz
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']units:
Hz
- rx_main_excitations:
Complex excitations of main array receive antennas for each antenna. Magnitude between 0 (off) and 1 (full power)
dim_labels:
['beam', 'antenna']dim_scales:
[['beam_azms', 'beam_nums'], 'rx_main_antennas']level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- rx_intf_excitations:
Complex excitations of interferometer array receive antennas for each antenna. Magnitude between 0 (off) and 1 (full power)
dim_labels:
['beam', 'antenna']dim_scales:
[['beam_azms', 'beam_nums'], 'rx_intf_antennas']level:
recordrequired_for:
[]
- samples_data_type:
C data type of the samples
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- scan_start_marker:
Designates if the record is the first in a scan
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- scheduling_mode:
Type of scheduling time at the time of this dataset
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- slice_comment:
Comment that describes the slice
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf']
- slice_id:
Slice ID of the file and dataset
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf']
- slice_interfacing:
Interfacing of this slice to other slices
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf']
- sqn_timestamps:
GPS timestamps of start of first pulse for each sampling period in the averaging period
dim_labels:
['sequence']level:
recordnickname:
timestamprequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']units:
seconds since 1970-01-01 00:00:00 UTC
- station:
Three letter radar identifier
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- station_location:
Location of the radar
dim_labels:
['global_coord']dim_scales:
['global_coord']level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- tau_spacing:
Unit of spacing between pulses in microseconds
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf']units:
μs
- tx_antennas:
Indices into
antenna_locationsof the antennas used for transmitting in this experimentlevel:
filenickname:
tx antennarequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- tx_excitations:
Complex excitations of transmit signal for each antenna. Magnitude between 0 (off) and 1 (full power)
dim_labels:
['antenna']dim_scales:
['tx_antennas']level:
recordrequired_for:
[]units:
a.u.
- tx_pulse_len:
Length of the pulse in microseconds
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf']units:
μs
- xcfs:
Cross-correlations between main and interferometer arrays
dim_labels:
['beam', 'range', 'lag']dim_scales:
[['beam_azms', 'beam_nums'], 'range_gates', 'lag_numbers']level:
recordrequired_for:
[]units:
a.u. ~ W
Each field contains metadata that determines how the field is written to file.
descriptionA description of the field.
dim_labelsIf applicable, a brief descriptor for each dimension of the dataset. This could be different for different
groupvalues. If so, this metadata will be a dict, with the keys being thegroupname and the values the associated list of dimension labels.dim_scalesIf applicable, dimension scales will be associated to the field. These are datasets that match one of the dimensions of the data, such as timestamps to go along with an array of collected data. Note that some dimensions may be associated with multiple fields. If a dimension has no associated dataset, the list will have a
Noneentry.groupsThe types of data file that need this field to be written.
levelThe level within the file that the data will be stored at. Either
fileorrecord, indicating that the field is either written once per file, or once per record.nicknameA nickname for the field, used for making Dimension Scale names.
unitsUnits for the data.
- agc_status_word:
32 bits, a 1 in bit position corresponds to an AGC fault on that transmitter
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- antenna_arrays:
Descriptor of each antenna array contained in the data
level:
filenickname:
arrayrequired_for:
['bfiq']
- antenna_locations:
Relative antenna locations
dim_labels:
['antenna', 'local_coord']dim_scales:
['antennas', 'local_coord']level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']units:
m
- antennas:
Labels for each antenna of the radar
dim_labels:
['antenna']level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- beam_azms:
Beams azimuths for each beam in degrees CW of boresight
dim_labels:
['beam']level:
recordnickname:
beam directionrequired_for:
['antennas_iq', 'bfiq', 'rawacf']units:
degrees
- beam_nums:
Beam numbers used in this slice
dim_labels:
['beam']level:
recordnickname:
beam numberrequired_for:
['antennas_iq', 'bfiq', 'rawacf']
- bfiq_data:
Beamformed I&Q complex voltage samples for each antenna array
dim_labels:
['array', 'sequence', 'beam', 'time']dim_scales:
['antenna_arrays', 'sqn_timestamps', ['beam_nums', 'beam_azms'], 'sample_time']level:
recordrequired_for:
['bfiq']units:
a.u. ~ V
- blanked_samples:
Samples blanked during transmission of a pulse
dim_labels:
['time']level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- borealis_git_hash:
Version and commit hash of Borealis at runtime
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- cfs_freqs:
Frequencies measured during clear frequency search
dim_labels:
['freq']level:
filenickname:
freqrequired_for:
[]units:
Hz
- cfs_masks:
Mask for cfs_freqs restricting freqs available for setting cfs slice freq
dim_labels:
['freq']dim_scales:
['cfs_freqs']level:
recordrequired_for:
[]
- cfs_noise:
Power measured during clear frequency search
dim_labels:
['freq']dim_scales:
['cfs_freqs']level:
recordrequired_for:
[]units:
a.u. ~ dBW
- cfs_range:
Lower and upper bound of frequencies examined by clear frequency search
level:
filerequired_for:
[]units:
Hz
- data_normalization_factor:
Cumulative scale of all of the filters for a total scaling factor to normalize by
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf']
- experiment_comment:
Comment about the whole experiment
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- experiment_id:
Number used to identify experiment
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- experiment_name:
Name of the experiment class
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- first_range:
Distance to first range in km
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf']units:
km
- first_range_rtt:
Round trip time of flight to first range in microseconds
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf']units:
μs
- freq:
Frequency used for this experiment slice, in kHz
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']units:
kHz
- global_coord:
Descriptors for global coordinates
dim_labels:
['global_coord']level:
filenickname:
global coordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- gps_locked:
True if the GPS was locked during the entire averaging period
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- gps_to_system_time_diff:
Max time diff in seconds between GPS and system/NTP time during the averaging period
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']units:
s
- int_time:
Integration time in seconds
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']units:
s
- lags:
Lag indices
level:
filenickname:
lagrequired_for:
['rawacf']
- lag_numbers:
Difference in units of tau_spacing of unique pairs of pulse in the pulse array
level:
filenickname:
lagrequired_for:
['rawacf']units:
tau_spacing
- lag_pulses:
Unique pairs of pulses in pulse array, in units of tau_spacing
dim_labels:
['lag', 'pulse']dim_scales:
['lags', 'lag_pulse_descriptors']level:
filerequired_for:
['rawacf']units:
tau_spacing
- lag_pulse_descriptors:
Descriptor of the pulse pairs used in a lag
level:
filerequired_for:
['rawacf']
- local_coord:
Descriptors for local coordinates
dim_labels:
['local_coord']level:
filenickname:
local coordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- lp_status_word:
32 bits, a 1 in bit position corresponds to a low power condition on that transmitter
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- num_sequences:
Number of sampling periods in the averaging period
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- num_slices:
Number of slices in the experiment for this averaging period
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- pulse_phase_offset:
Phase offset in degrees for each pulse in pulses
dim_labels:
['sequence', 'pulse']dim_scales:
['sqn_timestamps', 'pulses']level:
recordrequired_for:
[]units:
degrees
- pulses:
Pulse sequence in units of tau_spacing
dim_labels:
['pulse']level:
filenickname:
pulserequired_for:
['antennas_iq', 'bfiq', 'rawacf']
- range_gates:
Range gates of interest for the experiment, beginning at
first_rangeand spaced byrange_seplevel:
filenickname:
range gaterequired_for:
['antennas_iq', 'bfiq', 'rawacf']
- range_sep:
Range gate separation (equivalent distance between samples) in km
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf']units:
km
- rx_intf_antennas:
Indices into
antenna_locationsof the interferometer array antennas used in this experimentlevel:
filerequired_for:
[]
- rx_main_antennas:
Indices into
antenna_locationsof the main array antennas used in this experimentlevel:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- rx_sample_rate:
Sampling rate of the samples being written to file in Hz
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']units:
Hz
- rx_main_excitations:
Complex excitations of main array receive antennas for each antenna. Magnitude between 0 (off) and 1 (full power)
dim_labels:
['beam', 'antenna']dim_scales:
[['beam_azms', 'beam_nums'], 'rx_main_antennas']level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- rx_intf_excitations:
Complex excitations of interferometer array receive antennas for each antenna. Magnitude between 0 (off) and 1 (full power)
dim_labels:
['beam', 'antenna']dim_scales:
[['beam_azms', 'beam_nums'], 'rx_intf_antennas']level:
recordrequired_for:
[]
- samples_data_type:
C data type of the samples
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- sample_time:
Time of measurement relative to the first pulse in the sequence
dim_labels:
['time']level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawrf']units:
μs
- scan_start_marker:
Designates if the record is the first in a scan
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- scheduling_mode:
Type of scheduling time at the time of this dataset
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- slice_comment:
Comment that describes the slice
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf']
- slice_id:
Slice ID of the file and dataset
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf']
- slice_interfacing:
Interfacing of this slice to other slices
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf']
- sqn_timestamps:
GPS timestamps of start of first pulse for each sampling period in the averaging period
dim_labels:
['sequence']level:
recordnickname:
timestamprequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']units:
seconds since 1970-01-01 00:00:00 UTC
- station:
Three letter radar identifier
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- station_location:
Location of the radar
dim_labels:
['global_coord']dim_scales:
['global_coord']level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- tau_spacing:
Unit of spacing between pulses in microseconds
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf']units:
μs
- tx_antennas:
Indices into
antenna_locationsof the antennas used for transmitting in this experimentlevel:
filenickname:
tx antennarequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- tx_excitations:
Complex excitations of transmit signal for each antenna. Magnitude between 0 (off) and 1 (full power)
dim_labels:
['antenna']dim_scales:
['tx_antennas']level:
recordrequired_for:
[]units:
a.u.
- tx_pulse_len:
Length of the pulse in microseconds
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf']units:
μs
Each field contains metadata that determines how the field is written to file.
descriptionA description of the field.
dim_labelsIf applicable, a brief descriptor for each dimension of the dataset. This could be different for different
groupvalues. If so, this metadata will be a dict, with the keys being thegroupname and the values the associated list of dimension labels.dim_scalesIf applicable, dimension scales will be associated to the field. These are datasets that match one of the dimensions of the data, such as timestamps to go along with an array of collected data. Note that some dimensions may be associated with multiple fields. If a dimension has no associated dataset, the list will have a
Noneentry.groupsThe types of data file that need this field to be written.
levelThe level within the file that the data will be stored at. Either
fileorrecord, indicating that the field is either written once per file, or once per record.nicknameA nickname for the field, used for making Dimension Scale names.
unitsUnits for the data.
- agc_status_word:
32 bits, a 1 in bit position corresponds to an AGC fault on that transmitter
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- antenna_locations:
Relative antenna locations
dim_labels:
['antenna', 'local_coord']dim_scales:
['antennas', 'local_coord']level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']units:
m
- antennas:
Labels for each antenna of the radar
dim_labels:
['antenna']level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- antennas_iq_data:
Filtered and downsampled I&Q complex voltage samples for each antenna
dim_labels:
['antenna', 'sequence', 'time']dim_scales:
['rx_antennas', 'sqn_timestamps', 'sample_time']level:
recordrequired_for:
['antennas_iq']units:
a.u. ~ V
- beam_azms:
Beams azimuths for each beam in degrees CW of boresight
dim_labels:
['beam']level:
recordnickname:
beam directionrequired_for:
['antennas_iq', 'bfiq', 'rawacf']units:
degrees
- beam_nums:
Beam numbers used in this slice
dim_labels:
['beam']level:
recordnickname:
beam numberrequired_for:
['antennas_iq', 'bfiq', 'rawacf']
- blanked_samples:
Samples blanked during transmission of a pulse
dim_labels:
['time']level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- borealis_git_hash:
Version and commit hash of Borealis at runtime
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- cfs_freqs:
Frequencies measured during clear frequency search
dim_labels:
['freq']level:
filenickname:
freqrequired_for:
[]units:
Hz
- cfs_masks:
Mask for cfs_freqs restricting freqs available for setting cfs slice freq
dim_labels:
['freq']dim_scales:
['cfs_freqs']level:
recordrequired_for:
[]
- cfs_noise:
Power measured during clear frequency search
dim_labels:
['freq']dim_scales:
['cfs_freqs']level:
recordrequired_for:
[]units:
a.u. ~ dBW
- cfs_range:
Lower and upper bound of frequencies examined by clear frequency search
level:
filerequired_for:
[]units:
Hz
- data_normalization_factor:
Cumulative scale of all of the filters for a total scaling factor to normalize by
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf']
- experiment_comment:
Comment about the whole experiment
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- experiment_id:
Number used to identify experiment
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- experiment_name:
Name of the experiment class
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- first_range:
Distance to first range in km
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf']units:
km
- first_range_rtt:
Round trip time of flight to first range in microseconds
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf']units:
μs
- freq:
Frequency used for this experiment slice, in kHz
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']units:
kHz
- global_coord:
Descriptors for global coordinates
dim_labels:
['global_coord']level:
filenickname:
global coordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- gps_locked:
True if the GPS was locked during the entire averaging period
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- gps_to_system_time_diff:
Max time diff in seconds between GPS and system/NTP time during the averaging period
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']units:
s
- int_time:
Integration time in seconds
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']units:
s
- lags:
Lag indices
level:
filenickname:
lagrequired_for:
['rawacf']
- lag_numbers:
Difference in units of tau_spacing of unique pairs of pulse in the pulse array
level:
filenickname:
lagrequired_for:
['rawacf']units:
tau_spacing
- lag_pulses:
Unique pairs of pulses in pulse array, in units of tau_spacing
dim_labels:
['lag', 'pulse']dim_scales:
['lags', 'lag_pulse_descriptors']level:
filerequired_for:
['rawacf']units:
tau_spacing
- lag_pulse_descriptors:
Descriptor of the pulse pairs used in a lag
level:
filerequired_for:
['rawacf']
- local_coord:
Descriptors for local coordinates
dim_labels:
['local_coord']level:
filenickname:
local coordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- lp_status_word:
32 bits, a 1 in bit position corresponds to a low power condition on that transmitter
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- num_sequences:
Number of sampling periods in the averaging period
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- num_slices:
Number of slices in the experiment for this averaging period
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- pulse_phase_offset:
Phase offset in degrees for each pulse in pulses
dim_labels:
['sequence', 'pulse']dim_scales:
['sqn_timestamps', 'pulses']level:
recordrequired_for:
[]units:
degrees
- pulses:
Pulse sequence in units of tau_spacing
dim_labels:
['pulse']level:
filenickname:
pulserequired_for:
['antennas_iq', 'bfiq', 'rawacf']
- range_gates:
Range gates of interest for the experiment, beginning at
first_rangeand spaced byrange_seplevel:
filenickname:
range gaterequired_for:
['antennas_iq', 'bfiq', 'rawacf']
- range_sep:
Range gate separation (equivalent distance between samples) in km
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf']units:
km
- rx_antennas:
Indices into
antenna_locationsof the antennas with recorded datalevel:
filenickname:
rx antennarequired_for:
['antennas_iq', 'rawrf']
- rx_intf_antennas:
Indices into
antenna_locationsof the interferometer array antennas used in this experimentlevel:
filerequired_for:
[]
- rx_main_antennas:
Indices into
antenna_locationsof the main array antennas used in this experimentlevel:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- rx_sample_rate:
Sampling rate of the samples being written to file in Hz
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']units:
Hz
- rx_main_excitations:
Complex excitations of main array receive antennas for each antenna. Magnitude between 0 (off) and 1 (full power)
dim_labels:
['beam', 'antenna']dim_scales:
[['beam_azms', 'beam_nums'], 'rx_main_antennas']level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- rx_intf_excitations:
Complex excitations of interferometer array receive antennas for each antenna. Magnitude between 0 (off) and 1 (full power)
dim_labels:
['beam', 'antenna']dim_scales:
[['beam_azms', 'beam_nums'], 'rx_intf_antennas']level:
recordrequired_for:
[]
- samples_data_type:
C data type of the samples
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- sample_time:
Time of measurement relative to the first pulse in the sequence
dim_labels:
['time']level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawrf']units:
μs
- scan_start_marker:
Designates if the record is the first in a scan
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- scheduling_mode:
Type of scheduling time at the time of this dataset
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- slice_comment:
Comment that describes the slice
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf']
- slice_id:
Slice ID of the file and dataset
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf']
- slice_interfacing:
Interfacing of this slice to other slices
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf']
- sqn_timestamps:
GPS timestamps of start of first pulse for each sampling period in the averaging period
dim_labels:
['sequence']level:
recordnickname:
timestamprequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']units:
seconds since 1970-01-01 00:00:00 UTC
- station:
Three letter radar identifier
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- station_location:
Location of the radar
dim_labels:
['global_coord']dim_scales:
['global_coord']level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- tau_spacing:
Unit of spacing between pulses in microseconds
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf']units:
μs
- tx_antennas:
Indices into
antenna_locationsof the antennas used for transmitting in this experimentlevel:
filenickname:
tx antennarequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- tx_excitations:
Complex excitations of transmit signal for each antenna. Magnitude between 0 (off) and 1 (full power)
dim_labels:
['antenna']dim_scales:
['tx_antennas']level:
recordrequired_for:
[]units:
a.u.
- tx_pulse_len:
Length of the pulse in microseconds
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf']units:
μs
Each field contains metadata that determines how the field is written to file.
descriptionA description of the field.
dim_labelsIf applicable, a brief descriptor for each dimension of the dataset. This could be different for different
groupvalues. If so, this metadata will be a dict, with the keys being thegroupname and the values the associated list of dimension labels.dim_scalesIf applicable, dimension scales will be associated to the field. These are datasets that match one of the dimensions of the data, such as timestamps to go along with an array of collected data. Note that some dimensions may be associated with multiple fields. If a dimension has no associated dataset, the list will have a
Noneentry.groupsThe types of data file that need this field to be written.
levelThe level within the file that the data will be stored at. Either
fileorrecord, indicating that the field is either written once per file, or once per record.nicknameA nickname for the field, used for making Dimension Scale names.
unitsUnits for the data.
- agc_status_word:
32 bits, a 1 in bit position corresponds to an AGC fault on that transmitter
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- antenna_locations:
Relative antenna locations
dim_labels:
['antenna', 'local_coord']dim_scales:
['antennas', 'local_coord']level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']units:
m
- antennas:
Labels for each antenna of the radar
dim_labels:
['antenna']level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- blanked_samples:
Samples blanked during transmission of a pulse
dim_labels:
['time']level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- borealis_git_hash:
Version and commit hash of Borealis at runtime
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- experiment_comment:
Comment about the whole experiment
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- experiment_id:
Number used to identify experiment
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- experiment_name:
Name of the experiment class
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- freq:
Frequency used for this experiment slice, in kHz
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']units:
kHz
- global_coord:
Descriptors for global coordinates
dim_labels:
['global_coord']level:
filenickname:
global coordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- gps_locked:
True if the GPS was locked during the entire averaging period
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- gps_to_system_time_diff:
Max time diff in seconds between GPS and system/NTP time during the averaging period
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']units:
s
- int_time:
Integration time in seconds
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']units:
s
- local_coord:
Descriptors for local coordinates
dim_labels:
['local_coord']level:
filenickname:
local coordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- lp_status_word:
32 bits, a 1 in bit position corresponds to a low power condition on that transmitter
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- num_sequences:
Number of sampling periods in the averaging period
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- num_slices:
Number of slices in the experiment for this averaging period
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- rawrf_data:
I&Q complex voltage samples for each antenna
dim_labels:
['sequence', 'antenna', 'time']dim_scales:
['sqn_timestamps', 'rx_antennas', 'sample_time']level:
recordrequired_for:
['rawrf']units:
a.u. ~ V
- rx_antennas:
Indices into
antenna_locationsof the antennas with recorded datalevel:
filenickname:
rx antennarequired_for:
['antennas_iq', 'rawrf']
- rx_intf_antennas:
Indices into
antenna_locationsof the interferometer array antennas used in this experimentlevel:
filerequired_for:
[]
- rx_main_antennas:
Indices into
antenna_locationsof the main array antennas used in this experimentlevel:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- rx_center_freq:
Center frequency of the data in kHz
level:
filerequired_for:
['rawrf']units:
kHz
- rx_sample_rate:
Sampling rate of the samples being written to file in Hz
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']units:
Hz
- rx_main_excitations:
Complex excitations of main array receive antennas for each antenna. Magnitude between 0 (off) and 1 (full power)
dim_labels:
['beam', 'antenna']dim_scales:
[['beam_azms', 'beam_nums'], 'rx_main_antennas']level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- rx_intf_excitations:
Complex excitations of interferometer array receive antennas for each antenna. Magnitude between 0 (off) and 1 (full power)
dim_labels:
['beam', 'antenna']dim_scales:
[['beam_azms', 'beam_nums'], 'rx_intf_antennas']level:
recordrequired_for:
[]
- samples_data_type:
C data type of the samples
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- sample_time:
Time of measurement relative to the first pulse in the sequence
dim_labels:
['time']level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawrf']units:
μs
- scan_start_marker:
Designates if the record is the first in a scan
level:
recordrequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- scheduling_mode:
Type of scheduling time at the time of this dataset
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- sqn_timestamps:
GPS timestamps of start of first pulse for each sampling period in the averaging period
dim_labels:
['sequence']level:
recordnickname:
timestamprequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']units:
seconds since 1970-01-01 00:00:00 UTC
- station:
Three letter radar identifier
level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- station_location:
Location of the radar
dim_labels:
['global_coord']dim_scales:
['global_coord']level:
filerequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- tx_antennas:
Indices into
antenna_locationsof the antennas used for transmitting in this experimentlevel:
filenickname:
tx antennarequired_for:
['antennas_iq', 'bfiq', 'rawacf', 'rawrf']
- tx_excitations:
Complex excitations of transmit signal for each antenna. Magnitude between 0 (off) and 1 (full power)
dim_labels:
['antenna']dim_scales:
['tx_antennas']level:
recordrequired_for:
[]units:
a.u.
DMAP-equivalent files¶
Mappings between rawacf and bfiq HDF5 files and their DMAP equivalents rawacf and iqdat are detailed in the corresponding tabs.
This conversion is done in pyDARNio here in the __convert_bfiq_record method: Link to Source
SDARN DMAP FIELD NAME
type
SDARN description
|
Borealis Conversion |
|---|---|
radar.revision.major
char
Major version number
|
borealis_git_hash major version number
or 255 if not a commit with a version tag
|
radar.revision.minor
char
Minor version number
|
borealis_git_hash minor version number
or 255 if not a commit with a version tag
|
origin.code
char
Code indicating origin of data
|
= 100, this can be used as a flag that the
origin code was Borealis
|
origin.time
string
ASCII representation of when
the data was generated
|
timestamp_of_write conversion
|
origin.command
string
The command line or control
program used to generate the
data
|
Borealis vXXX + borealis_git_hash +
experiment_name
|
cp
short
Control program identifier
|
experiment_id, truncated to short
|
stid
short
Station identifier
|
station conversion
|
time.yr
short
Year
|
sqn_timestamps [0] conversion
|
time.mo
short
Month
|
sqn_timestamps [0] conversion
|
time.dy
short
Day
|
sqn_timestamps [0] conversion
|
time.hr
short
Hour
|
sqn_timestamps [0] conversion
|
time.mt
short
Minute
|
sqn_timestamps [0] conversion
|
time.sc
short
Second
|
sqn_timestamps [0] conversion
|
time.us
short
Microsecond
|
sqn_timestamps [0] conversion
|
txpow
short
Transmitted power (kW)
|
= -1 (filler)
|
nave
short
Number of pulse sequences
transmitted
|
num_sequences
|
atten
short
Attenuation level
|
= 0 (filler)
|
lagfr
short
Lag to first range
(microseconds)
|
first_range_rtt
|
smsep
short
Sample separation
(microseconds)
|
(rx_sample_rate)^ -1
|
ercod
short
Error code
|
= 0 (filler)
|
stat.agc
short
AGC status word
|
= agc_status_word
|
stat.lopwr
short
LOPWR status word
|
= lp_status_word
|
noise.search
float
Calculated noise from clear
frequency search
|
noise_at_freq [0] conversion
|
noise.mean
float
Average noise across frequency
band
|
= 0 (filler)
|
channel
short
Channel number for a stereo
radar (zero for all others)
|
slice_id
|
bmnum
short
Beam number
|
beam_nums [i]
|
bmazm
float
Beam azimuth
|
beam_azms [i]
|
scan
short
Scan flag
|
scan_start_marker (0 or 1)
|
offset
short
Offset between channels for a
stereo radar (zero for all
others)
|
= 0 (filler)
|
rxrise
short
Receiver rise time
(microseconds)
|
= 0.0
|
intt.sc
short
Whole number of seconds of
integration time.
|
int_time conversion
|
intt.us
short
Fractional number of
microseconds of integration
time
|
int_time conversion
|
txpl
short
Transmit pulse length
(microseconds)
|
tx_pulse_len
|
mpinc
short
Multi-pulse increment
(microseconds)
|
tau_spacing
|
mppul
short
Number of pulses in sequence
|
len(pulses)
|
mplgs
short
Number of lags in sequence
|
lags.shape[0]
|
nrang
short
Number of ranges
|
num_ranges
|
frang
short
Distance to first range
(kilometers)
|
first_range
|
rsep
short
Range separation (kilometers)
|
range_sep
|
xcf
short
XCF flag
|
If xcfs exist, then =1
|
tfreq
short
Transmitted frequency
|
freq
|
mxpwr
int
Maximum power (kHz)
|
= -1 (filler)
|
lvmax
int
Maximum noise level allowed
|
= 20000 (filler)
|
iqdata.revision.major
int
Major version number of the
iqdata library
|
= 1 (meaning Borealis conversion)
|
iqdata.revision.minor
int
Minor version number of the
iqdata library
|
= 0 (Borealis conversion)
|
combf
string
Comment buffer
|
Original Borealis filename, ‘converted
from Borealis file ’ , number of beams in
this original record (len(beam_nums)),
experiment_comment and slice_comment
from the file
|
seqnum
int
Number of pulse sequences
transmitted
|
num_sequences
|
chnnum
int
Number of channels sampled
(both I and Q quadrature
samples)
|
len(antenna_arrays_order)
|
smpnum
int
Number of samples taken per
sequence
|
num_samps
|
skpnum
int
Number of samples to skip
before the first valid sample
|
math.ceil(first_range/range_sep). In
theory this should =0 due to Borealis
functionality(no rise time).
However make_raw in RST requires this to
be indicative of the first range so we
provide this.
|
ptab[mppul]
short
Pulse table
|
pulses
|
ltab[2][mplgs]
short
Lag table
|
np.transpose(lags)
|
tsc[seqnum]
int
Seconds component of time past
epoch of pulse sequence
|
sqn_timestamps conversion
|
tus[seqnum]
int
Microsecond component of time
past epoch of pulse sequence
|
sqn_timestamps conversion
|
tatten[seqnum]
short
Attenuator setting for each
pulse sequence
|
= [0,0…] (fillers)
|
tnoise[seqnum]
float
Noise value for each pulse
sequence
|
noise_at_freq conversion
|
toff[seqnum]
int
Offset into the sample buffer
for each pulse sequence
|
Offset = 2 * num_samps *
len(antenna_arrays_order), toff = [i *
offset for i in range(v[‘num_sequences’])]
|
tsze[seqnum]
int
Number of words stored for this
pulse sequence
|
= [offset, offset, offset….]
|
data[totnum]
int
Array of raw I and Q samples,
arranged: [[[smpnum(i),
smpnum(q)] * chnnum] * seqnum],
so totnum =
2*seqnum*chnnum*smpnum
|
Data conversion for correct dimensions
and scaled to max int (-32768 to 32767)
|
If blanked_samples != ptab, or pulse_phase_offset contains non-zeroes, no conversion to iqdat is possible.
This conversion is done in pyDARNio here in the __convert_rawacf_record method: Link to Source
SDARN DMAP FIELD NAME
type
SDARN description
|
Borealis Conversion |
|---|---|
radar.revision.major
char
Major version number
|
borealis_git_hash major version number
or 255 if not a commit with a version tag
|
radar.revision.minor
char
Minor version number
|
borealis_git_hash minor version number
or 255 if not a commit with a version tag
|
origin.code
char
Code indicating origin of data
|
= 100, this can be used as a flag that the
origin code was Borealis
|
origin.time
string
ASCII representation of when
the data was generated
|
timestamp_of_write conversion
|
origin.command
string
The command line or control
program used to generate the
data
|
Borealis vXXX + borealis_git_hash +
experiment_name
|
cp
short
Control program identifier
|
experiment_id, truncated to short
|
stid
short
Station identifier
|
station conversion
|
time.yr
short
Year
|
sqn_timestamps [0] conversion
|
time.mo
short
Month
|
sqn_timestamps [0] conversion
|
time.dy
short
Day
|
sqn_timestamps [0] conversion
|
time.hr
short
Hour
|
sqn_timestamps [0] conversion
|
time.mt
short
Minute
|
sqn_timestamps [0] conversion
|
time.sc
short
Second
|
sqn_timestamps [0] conversion
|
time.us
short
Microsecond
|
sqn_timestamps [0] conversion
|
txpow
short
Transmitted power (kW)
|
= -1 (filler)
|
nave
short
Number of pulse sequences
transmitted
|
num_sequences
|
atten
short
Attenuation level
|
= 0 (filler)
|
lagfr
short
Lag to first range
(microseconds)
|
first_range_rtt
|
smsep
short
Sample separation
(microseconds)
|
(rx_sample_rate)^ -1
|
ercod
short
Error code
|
= 0 (filler)
|
stat.agc
short
AGC status word
|
= agc_status_word
|
stat.lopwr
short
LOPWR status word
|
= lp_status_word
|
noise.search
float
Calculated noise from clear
frequency search
|
noise_at_freq [0] conversion
|
noise.mean
float
Average noise across frequency
band
|
= 0 (filler)
|
channel
short
Channel number for a stereo
radar (zero for all others)
|
slice_id
|
bmnum
short
Beam number
|
beam_nums [i]
|
bmazm
float
Beam azimuth
|
beam_azms [i]
|
scan
short
Scan flag
|
scan_start_marker (0 or 1)
|
offset
short
Offset between channels for a
stereo radar (zero for all
others)
|
= 0 (filler)
|
rxrise
short
Receiver rise time
(microseconds)
|
= 0.0
|
intt.sc
short
Whole number of seconds of
integration time.
|
int_time conversion
|
intt.us
short
Fractional number of
microseconds of integration
time
|
int_time conversion
|
txpl
short
Transmit pulse length
(microseconds)
|
tx_pulse_len
|
mpinc
short
Multi-pulse increment
(microseconds)
|
tau_spacing
|
mppul
short
Number of pulses in sequence
|
len(pulses)
|
mplgs
short
Number of lags in sequence
|
lags.shape[0]
|
nrang
short
Number of ranges
|
correlation_dimensions [1]
|
frang
short
Distance to first range
(kilometers)
|
first_range
|
rsep
short
Range separation (kilometers)
|
range_sep
|
xcf
short
XCF flag
|
If xcfs exist, then =1
|
tfreq
short
Transmitted frequency
|
freq
|
mxpwr
int
Maximum power (kHz)
|
= -1 (filler)
|
lvmax
int
Maximum noise level allowed
|
= 20000 (filler)
|
rawacf.revision.major
int
Major version number of the
rawacf format
|
= 255
|
rawacf.revision.minor
int
Minor version number of the
rawacf format
|
= 255
|
combf
string
Comment buffer
Comment buffer
|
Original Borealis filename, ‘converted
from Borealis file beam number ’ X,
number of beams in this original record
(len(beam_nums)), experiment_comment and
slice_comment from the file
|
thr
float
Thresholding factor
|
= 0.0 (filler)
|
ptab[mppul]
short
Pulse table
|
pulses
|
ltab[2][mplgs]
short
Lag table
|
np.transpose(lags)
|
pwr0[nrang]
[float]
Lag zero power for main
|
Calculated from main_acfs
|
slist[0-nrang]
[short]
List of stored ranges, length
dependent on SNR. Lists the
range gate of each stored ACF
|
range(0,*correlation_dimensions*.size[1])
|
acfd[2][mplgs][0-nrang]
[short]
Calculated ACFs
|
main_acfs conversion, real and imag
|
xcfd[2][mplgs][0-nrang]
[short]
Calculated XCFs
|
xcfs conversion, real and imag
|
If blanked_samples != ptab, or pulse_phase_offset contains non-zeroes, no conversion to dmap rawacf is possible.
Previous Versions¶
In prior Borealis versions, data files could be in one of two “structures”: site or array.
site structured files were written directly by the radar, consisting of one HDF5 group per averaging period
of the experiment. Each group, or “record”, contained the same metadata and data fields, even for static metadata
(i.e. not dependent on the specific averaging period, for example, the experiment name). The naming convention of site-structured files is:
[YYYYMMDD].[HHMM].[SS].[station_id].[slice_id].[file type].hdf5.site
array structured files could be created using pyDARNio, where the HDF5 groups were removed and all matching fields
from different records combined into one dataset. This structure is more ergonomic, and reduces the size of the file.
However, converting between the structures can take a significant amount of time and memory. The naming convention of array-structured files is:
[YYYYMMDD].[HHMM].[SS].[station_id].[slice_id].[file type].hdf5
These versions follow the v0.4 format.
FIELD NAME
type
[dimensions]
|
description |
|---|---|
beam_azms
float64
[num_records x
max_num_beams]
|
A list of the beam azimuths for each beam
in degrees off boresite. Note that this
is padded with zeroes for any record
which has num_beams less than the
max_num_beams. The num_beams field should
be used to read the correct number of
beams for each record.
|
beam_nums
uint32
[num_records x
max_num_beams]
|
A list of beam numbers used in this slice
in this record. Note that this is padded
with zeroes for any record which has
num_beams less than the max_num_beams.
The num_beams field should be used to
read the correct number of beams for each
record.
|
blanked_samples
uint32
[number of blanked
samples]
|
Samples that should be blanked because
they occurred during transmission times,
given by sample number (index into
decimated data). Can differ from the
pulses array due to multiple slices in a
single sequence. Assumed shared between
records which was a bug fixed in v0.5.
|
borealis_git_hash
unicode
|
Identifies the version of Borealis that
made this data. Contains git commit hash
characters. Typically begins with the
latest git tag of the software.
|
correlation_descriptors
unicode
[4]
|
Denotes what each correlation dimension
(in main_acfs, intf_acfs, xcfs)
represents. = ‘num_records’,
‘max_num_beams’, ‘num_ranges’, ‘num_lags’
|
data_normalization_factor
float32
|
Scale of all the filters used,
multiplied, for a total scale to
normalize the data by.
|
experiment_comment
unicode
|
Comment provided in experiment about the
experiment as a whole.
|
experiment_id
int64
|
Number used to identify the experiment.
|
experiment_name
unicode
|
Name of the experiment file.
|
first_range
float32
|
Distance to use for first range in km.
|
first_range_rtt
float32
|
Round trip time of flight to first range
in microseconds.
|
freq
uint32
|
The frequency used for this experiment,
in kHz. This is the frequency the data
has been filtered to.
|
int_time
float32
[num_records]
|
Integration time in seconds.
|
intf_acfs
complex64
[num_records x
max_num_beams x
num_ranges x
num_lags]
|
Interferometer array correlations. Note
that records that do not have num_beams =
max_num_beams will have padded zeros. The
num_beams array should be used to
determine the correct number of beams to
read for the record.
|
intf_antenna_count
uint32
|
Number of interferometer array antennas
|
lags
uint32
[number of lags, 2]
|
The lags created from two pulses in the
pulses array. Values have to be from
pulses array. The lag number is lag[1] -
lag[0] for each lag pair.
|
main_acfs
complex64
[num_records x
max_num_beams x
num_ranges x
num_lags]
|
Main array correlations. Note
that records that do not have num_beams =
max_num_beams will have padded zeros. The
num_beams array should be used to
determine the correct number of beams to
read for the record.
|
main_antenna_count
uint32
|
Number of main array antennas
|
noise_at_freq
float64
[num_records x
max_num_sequences]
|
Noise at the receive frequency, with
dimension = number of sequences.
20191114: not currently implemented and
filled with zeros. Still a TODO. Note
that records that do not have
num_sequences = max_num_sequences will
have padded zeros. The num_sequences
array should be used to determine the
correct number of sequences to read for
the record.
|
num_beams
uint32
[num_records]
|
The number of beams calculated for each
record. Allows the user to correctly read
the data up to the correct number and
remove the padded zeros in the data
array.
|
num_sequences
int64
[num_records]
|
Number of sampling periods (equivalent to
number sequences transmitted) in the
integration time for each record. Allows
the user to correctly read the data up to
the correct number and remove the padded
zeros in the data array.
|
num_slices
int64
[num_records]
|
Number of slices used simultaneously in
the record by the experiment. If more
than 1, data should exist in another file
for the same time period as that record
for the other slice.
|
pulses
uint32
[number of pulses]
|
The pulse sequence in units of the
tau_spacing.
|
range_sep
float32
|
Range gate separation (conversion from
time (1/rx_sample_rate) to equivalent
distance between samples), in km.
|
rx_sample_rate
float64
|
Sampling rate of the samples in this
file’s data in Hz.
|
samples_data_type
unicode
|
C data type of the samples, provided for
user friendliness. = ‘complex float’
|
scan_start_marker
bool
[num_records]
|
Designates if the record is the first in
a scan (scan is defined by the
experiment).
|
slice_comment
unicode
|
Additional text comment that describes
the slice written in this file.
|
sqn_timestamps
float64
[num_records x
max_num_sequences]
|
A list of GPS timestamps corresponding to
the beginning of transmission for each
sampling period in the integration time.
These timestamps come back from the USRP
driver and the USRPs are GPS disciplined
and synchronized using the Octoclock.
Provided in seconds since epoch.
Note that records that do not have
num_sequences = max_num_sequences will
have padded zeros. The num_sequences
array should be used to determine the
correct number of sequences to read for
the record.
|
station
unicode
|
Three-letter radar identifier.
|
tau_spacing
uint32
|
The minimum spacing between pulses in
microseconds. Spacing between pulses is
always a multiple of this.
|
tx_pulse_len
uint32
|
Length of the transmit pulse in
microseconds.
|
xcfs
complex64
[num_records x
max_num_beams x
num_ranges x
num_lags]
|
Cross correlations of interferometer to
main array. Note
that records that do not have num_beams =
max_num_beams will have padded zeros. The
num_beams array should be used to
determine the correct number of beams to
read for the record.
|
Field name
type
|
description |
|---|---|
beam_azms
[float64, ]
|
A list of the beam azimuths for each
beam in degrees off boresite.
|
beam_nums
[uint32, ]
|
A list of beam numbers used in this slice
in this record.
|
blanked_samples
[uint32, ]
|
Samples that should be blanked because
they occurred during transmission times,
given by sample number (index into
decimated data). Can differ from the
pulses array due to multiple slices in a
single sequence.
|
borealis_git_hash
unicode
|
Identifies the version of Borealis that
made this data. Contains git commit hash
characters. Typically begins with the
latest git tag of the software.
|
correlation_descriptors
[unicode, ]
|
Denotes what each correlation dimension
(in main_acfs, intf_acfs, xcfs)
represents. (‘num_beams, ‘num_ranges’,
‘num_lags’)
|
correlation_dimensions
[uint32, ]
|
The dimensions in which to reshape the
acf or xcf datasets. Dimensions
correspond to correlation_descriptors.
|
data_normalization_factor
float32
|
Scale of all the filters used, multiplied
for a total scale to normalize the data
by.
|
experiment_comment
unicode
|
Comment provided in experiment about the
experiment as a whole.
|
experiment_id
int64
|
Number used to identify the experiment.
|
experiment_name
unicode
|
Name of the experiment file.
|
first_range
float32
|
Distance to use for first range in km.
|
first_range_rtt
float32
|
Round trip time of flight to first range
in microseconds.
|
freq
uint32
|
The frequency used for this experiment,
in kHz. This is the frequency the data
has been filtered to.
|
int_time
float32
|
Integration time in seconds.
|
intf_acfs
[complex64, ]
|
Interferometer array correlations.
|
intf_antenna_count
uint32
|
Number of interferometer array antennas
|
lags
[[uint32, ], ]
|
The lags created from two pulses in the
pulses array. Dimensions are number of
lags x 2. Values have to be from pulses
array. The lag number is lag[1] - lag[0]
for each lag pair.
|
main_acfs
[complex64, ]
|
Main array correlations.
|
main_antenna_count
uint32
|
Number of main array antennas
|
noise_at_freq
[float64, ]
|
Noise at the receive frequency, with
dimension = number of sequences.
20191114: not currently implemented and
filled with zeros. Still a TODO.
|
num_sequences
int64
|
Number of sampling periods (equivalent to
number sequences transmitted) in the
integration time.
|
num_slices
int64
|
Number of slices used simultaneously in
this record by the experiment. If more
than 1, data should exist in another file
for this time period for the other slice.
|
pulses
[uint32, ]
|
The pulse sequence in units of the
tau_spacing.
|
range_sep
float32
|
Range gate separation (conversion from
time (1/rx_sample_rate) to equivalent
distance between samples), in km.
|
rx_sample_rate
float64
|
Sampling rate of the samples in this
file’s data in Hz.
|
samples_data_type
unicode
|
C data type of the samples, provided for
user friendliness. = ‘complex float’
|
scan_start_marker
bool
|
Designates if the record is the first in
a scan (scan is defined by the
experiment).
|
slice_comment
unicode
|
Additional text comment that describes
the slice written in this file.
|
sqn_timestamps
[float64, ]
|
A list of GPS timestamps corresponding to
the beginning of transmission for each
sampling period in the integration time.
These timestamps come from the USRP
driver and the USRPs are GPS disciplined
and synchronized using the Octoclock.
Provided in seconds since epoch.
|
station
unicode
|
Three-letter radar identifier.
|
tau_spacing
uint32
|
The minimum spacing between pulses in
microseconds. Spacing between pulses is
always a multiple of this.
|
tx_pulse_len
uint32
|
Length of the transmit pulse in
microseconds.
|
xcfs
[complex64, ]
|
Cross correlations of interferometer to
main array.
|
FIELD NAME
type
[dimensions]
|
description |
|---|---|
antenna_arrays_order
unicode
[num_antenna_arrays]
|
States what order the data is in and
describes the data layout for the
num_antenna_arrays data dimension
|
beam_azms
float64
[num_records x
max_num_beams]
|
A list of the beam azimuths for each beam
in degrees off boresite. Note that this
is padded with zeroes for any record
which has num_beams less than the
max_num_beams. The num_beams field should
be used to read the correct number of
beams for each record.
|
beam_nums
uint32
[num_records x
max_num_beams]
|
A list of beam numbers used in this slice
in this record. Note that this is padded
with zeroes for any record which has
num_beams less than the max_num_beams.
The num_beams field should be used to
read the correct number of beams for each
record.
|
blanked_samples
uint32
[number of blanked
samples]
|
Samples that should be blanked because
they occurred during transmission times,
given by sample number (index into
decimated data). Can differ from the
pulses array due to multiple slices in a
single sequence. Assumed shared between
records which was a bug fixed in v0.5.
|
borealis_git_hash
unicode
|
Identifies the version of Borealis that
made this data. Contains git commit hash
characters. Typically begins with the
latest git tag of the software.
|
data
complex64
[num_records x
num_antenna_arrays x
max_num_sequences x
max_num_beams x
num_samps]
|
A set of samples (complex float) at given
sample rate. Note that records that do not
have num_sequences = max_num_sequences or
num_beams = max_num_beams will have
padded zeros. The num_sequences and
num_beams arrays should be used to
determine the correct number of sequences
and beams to read for the record.
|
data_descriptors
unicode
[5]
|
Denotes what each data dimension
represents. = ‘num_records’,
‘num_antenna_arrays’,
‘max_num_sequences’, ‘max_num_beams’,
‘num_samps’
|
data_normalization_factor
float32
|
Scale of all the filters used,
multiplied, for a total scale to
normalize the data by.
|
experiment_comment
unicode
|
Comment provided in experiment about the
experiment as a whole.
|
experiment_id
int64
|
Number used to identify the experiment.
|
experiment_name
unicode
|
Name of the experiment file.
|
first_range
float32
|
Distance to use for first range in km.
|
first_range_rtt
float32
|
Round trip time of flight to first range
in microseconds.
|
freq
uint32
|
The frequency used for this experiment,
in kHz. This is the frequency the data
has been filtered to.
|
int_time
float32
[num_records]
|
Integration time in seconds.
|
intf_antenna_count
uint32
|
Number of interferometer array antennas
|
lags
uint32
[number of lags, 2]
|
The lags created from two pulses in the
pulses array. Values have to be from
pulses array. The lag number is lag[1] -
lag[0] for each lag pair.
|
main_antenna_count
uint32
|
Number of main array antennas
|
noise_at_freq
float64
[num_records x
max_num_sequences]
|
Noise at the receive frequency, with
dimension = number of sequences.
20191114: not currently implemented and
filled with zeros. Still a TODO. Note
that records that do not have
num_sequences = max_num_sequences will
have padded zeros. The num_sequences
array should be used to determine the
correct number of sequences to read for
the record.
|
num_beams
uint32
[num_records]
|
The number of beams calculated for each
record. Allows the user to correctly read
the data up to the correct number and
remove the padded zeros in the data
array.
|
num_ranges
uint32
|
Number of ranges to calculate
correlations for.
|
num_samps
uint32
|
Number of samples in the sampling
period. Each sequence has its own
sampling period. Will also be provided
as the last data_dimension value.
|
num_sequences
int64
[num_records]
|
Number of sampling periods (equivalent to
number sequences transmitted) in the
integration time for each record. Allows
the user to correctly read the data up to
the correct number and remove the padded
zeros in the data array.
|
num_slices
int64
[num_records]
|
Number of slices used simultaneously in
the record by the experiment. If more
than 1, data should exist in another file
for the same time period as that record
for the other slice.
|
pulse_phase_offset
float32
[number of pulses]
|
For pulse encoding phase, in degrees
offset. Contains one phase offset per
pulse in pulses.
|
pulses
uint32
[number of pulses]
|
The pulse sequence in units of the
tau_spacing.
|
range_sep
float32
|
Range gate separation (conversion from
time (1/rx_sample_rate) to equivalent
distance between samples), in km.
|
rx_sample_rate
float64
|
Sampling rate of the samples in this
file’s data in Hz.
|
samples_data_type
unicode
|
C data type of the samples, provided for
user friendliness. = ‘complex float’
|
scan_start_marker
bool
[num_records]
|
Designates if the record is the first in
a scan (scan is defined by the
experiment).
|
slice_comment
unicode
|
Additional text comment that describes
the slice written in this file. The slice
number of this file is provided in the
filename.
|
sqn_timestamps
float64
[num_records x
max_num_sequences]
|
A list of GPS timestamps corresponding to
the beginning of transmission for each
sampling period in the integration time.
These timestamps come back from the USRP
driver and the USRPs are GPS disciplined
and synchronized using the Octoclock.
Provided in seconds since epoch.
Note that records that do not have
num_sequences = max_num_sequences will
have padded zeros. The num_sequences
array should be used to determine the
correct number of sequences to read for
the record.
|
station
unicode
|
Three-letter radar identifier.
|
tau_spacing
uint32
|
The minimum spacing between pulses in
microseconds. Spacing between pulses is
always a multiple of this.
|
tx_pulse_len
uint32
|
Length of the transmit pulse in
microseconds.
|
Field name
type
|
description |
|---|---|
antenna_arrays_order
[unicode, ]
|
States what order the data is in and
describes the data layout for the
num_antenna_arrays data dimension
|
beam_azms
[float64, ]
|
A list of the beam azimuths for each
beam in degrees off boresite.
|
beam_nums
[uint32, ]
|
A list of beam numbers used in this slice
in this record.
|
blanked_samples
[uint32, ]
|
Samples that should be blanked because
they occurred during transmission times,
given by sample number (index into
decimated data). Can differ from the
pulses array due to multiple slices in a
single sequence.
|
borealis_git_hash
unicode
|
Identifies the version of Borealis that
made this data. Contains git commit hash
characters. Typically begins with the
latest git tag of the software.
|
data
[complex64, ]
|
A contiguous set of samples (complex
float) at given sample rate. Needs to be
reshaped by data_dimensions to be
correctly read.
|
data_descriptors
[unicode, ]
|
Denotes what each data dimension
represents. = ‘num_antenna_arrays’,
‘num_sequences’, ‘num_beams’, ‘num_samps’
for bfiq
|
data_dimensions
[uint32, ]
|
The dimensions in which to reshape the
data. Dimensions correspond to
data_descriptors.
|
data_normalization_factor
float32
|
Scale of all the filters used, multiplied
for a total scale to normalize the data
by.
|
experiment_comment
unicode
|
Comment provided in experiment about the
experiment as a whole.
|
experiment_id
int64
|
Number used to identify the experiment.
|
experiment_name
unicode
|
Name of the experiment file.
|
first_range
float32
|
Distance to use for first range in km.
|
first_range_rtt
float32
|
Round trip time of flight to first range
in microseconds.
|
freq
uint32
|
The frequency used for this experiment,
in kHz. This is the frequency the data
has been filtered to.
|
int_time
float32
|
Integration time in seconds.
|
intf_antenna_count
uint32
|
Number of interferometer array antennas
|
lags
[[uint32, ], ]
|
The lags created from two pulses in the
pulses array. Dimensions are number of
lags x 2. Values have to be from pulses
array. The lag number is lag[1] - lag[0]
for each lag pair.
|
main_antenna_count
uint32
|
Number of main array antennas
|
noise_at_freq
[float64, ]
|
Noise at the receive frequency, with
dimension = number of sequences.
20191114: not currently implemented and
filled with zeros. Still a TODO.
|
num_ranges
uint32
|
Number of ranges to calculate
correlations for.
|
num_samps
uint32
|
Number of samples in the sampling
period. Each sequence has its own
sampling period. Will also be provided
as the last data_dimension value.
|
num_sequences
int64
|
Number of sampling periods (equivalent to
number sequences transmitted) in the
integration time.
|
num_slices
int64
|
Number of slices used simultaneously in
this record by the experiment. If more
than 1, data should exist in another file
for this time period for the other slice.
|
pulse_phase_offset
[float32, ]
|
For pulse encoding phase, in degrees
offset. Contains one phase offset per
pulse in pulses.
|
pulses
[uint32, ]
|
The pulse sequence in units of the
tau_spacing.
|
range_sep
float32
|
Range gate separation (conversion from
time (1/rx_sample_rate) to equivalent
distance between samples), in km.
|
rx_sample_rate
float64
|
Sampling rate of the samples in this
file’s data in Hz.
|
samples_data_type
unicode
|
C data type of the samples, provided for
user friendliness. = ‘complex float’
|
scan_start_marker
bool
|
Designates if the record is the first in
a scan (scan is defined by the
experiment).
|
slice_comment
unicode
|
Additional text comment that describes
the slice written in this file.
|
sqn_timestamps
[float64, ]
|
A list of GPS timestamps corresponding to
the beginning of transmission for each
sampling period in the integration time.
These timestamps come from the USRP
driver and the USRPs are GPS disciplined
and synchronized using the Octoclock.
Provided in seconds since epoch.
|
station
unicode
|
Three-letter radar identifier.
|
tau_spacing
uint32
|
The minimum spacing between pulses in
microseconds. Spacing between pulses is
always a multiple of this.
|
tx_pulse_len
uint32
|
Length of the transmit pulse in
microseconds.
|
FIELD NAME
type
[dimensions]
|
description |
|---|---|
antenna_arrays_order
unicode
[num_antennas]
|
States what order the data is in and
describes the data layout for the
num_antennas data dimension
Antennas are recorded main array
ascending and then interferometer array
ascending
|
beam_azms
float64
[num_records x
max_num_beams]
|
A list of the beam azimuths for each beam
in degrees off boresite. Note that this
is padded with zeroes for any record
which has num_beams less than the
max_num_beams. The num_beams field should
be used to read the correct number of
beams for each record.
|
beam_nums
uint32
[num_records x
max_num_beams]
|
A list of beam numbers used in this slice
in this record. Note that this is padded
with zeroes for any record which has
num_beams less than the max_num_beams.
The num_beams field should be used to
read the correct number of beams for each
record.
|
borealis_git_hash
unicode
|
Identifies the version of Borealis that
made this data. Contains git commit hash
characters. Typically begins with the
latest git tag of the software.
|
data
complex64
[num_records x
num_antennas x
max_num_sequences x
num_samps]
|
A set of samples (complex float) at given
sample rate. Note that records that do not
have num_sequences = max_num_sequences
will have padded zeros. The num_sequences
array should be used to determine the
correct number of sequences to read for
the record.
|
data_descriptors
unicode
[4]
|
Denotes what each data dimension
represents. = ‘num_records’,
‘num_antennas’, ‘max_num_sequences’,
‘num_samps’
|
data_normalization_factor
float32
|
Scale of all the filters used,
multiplied, for a total scale to
normalize the data by.
|
experiment_comment
unicode
|
Comment provided in experiment about the
experiment as a whole.
|
experiment_id
int64
|
Number used to identify the experiment.
|
experiment_name
unicode
|
Name of the experiment file.
|
freq
uint32
|
The frequency used for this experiment,
in kHz. This is the frequency the data
has been filtered to.
|
int_time
float32
[num_records]
|
Integration time in seconds.
|
intf_antenna_count
uint32
|
Number of interferometer array antennas
|
main_antenna_count
uint32
|
Number of main array antennas
|
noise_at_freq
float64
[num_records x
max_num_sequences]
|
Noise at the receive frequency, with
dimension = number of sequences.
20191114: not currently implemented and
filled with zeros. Still a TODO. Note
that records that do not have
num_sequences = max_num_sequences will
have padded zeros. The num_sequences
array should be used to determine the
correct number of sequences to read for
the record.
|
num_beams
uint32
[num_records]
|
The number of beams to calculate for each
record.
|
num_samps
uint32
|
Number of samples in the sampling
period. Each sequence has its own
sampling period. Will also be provided
as the last data_dimension value.
|
num_sequences
int64
[num_records]
|
Number of sampling periods (equivalent to
number sequences transmitted) in the
integration time for each record. Allows
the user to correctly read the data up to
the correct number and remove the padded
zeros in the data array.
|
num_slices
int64
[num_records]
|
Number of slices used simultaneously in
the record by the experiment. If more
than 1, data should exist in another file
for the same time period as that record
for the other slice.
|
pulse_phase_offset
float32
[number of pulses]
|
For pulse encoding phase, in degrees
offset. Contains one phase offset per
pulse in pulses.
|
pulses
uint32
[number of pulses]
|
The pulse sequence in units of the
tau_spacing.
|
rx_sample_rate
float64
|
Sampling rate of the samples in this
file’s data in Hz.
|
samples_data_type
unicode
|
C data type of the samples, provided for
user friendliness. = ‘complex float’
|
scan_start_marker
bool
[num_records]
|
Designates if the record is the first in
a scan (scan is defined by the
experiment).
|
slice_comment
unicode
|
Additional text comment that describes
the slice written in this file. The slice
number of this file is provided in the
filename.
|
sqn_timestamps
float64
[num_records x
max_num_sequences]
|
A list of GPS timestamps corresponding to
the beginning of transmission for each
sampling period in the integration time.
These timestamps come back from the USRP
driver and the USRPs are GPS disciplined
and synchronized using the Octoclock.
Provided in seconds since epoch.
Note that records that do not have
num_sequences = max_num_sequences will
have padded zeros. The num_sequences
array should be used to determine the
correct number of sequences to read for
the record.
|
station
unicode
|
Three-letter radar identifier.
|
tau_spacing
uint32
|
The minimum spacing between pulses in
microseconds. Spacing between pulses is
always a multiple of this.
|
tx_pulse_len
uint32
|
Length of the transmit pulse in
microseconds.
|
Field name
type
|
description |
|---|---|
antenna_arrays_order
[unicode, ]
|
States what order the data is in and
describes the data layout for the
num_antennas data dimension. Antennas are
recorded main array ascending and then
interferometer array ascending.
|
beam_azms
[float64, ]
|
A list of the beam azimuths for each
beam in degrees off boresite.
|
beam_nums
[uint32, ]
|
A list of beam numbers used in this slice
in this record.
|
borealis_git_hash
unicode
|
Identifies the version of Borealis that
made this data. Contains git commit hash
characters. Typically begins with the
latest git tag of the software.
|
data
[complex64, ]
|
A contiguous set of samples (complex
float) at given sample rate. Needs to be
reshaped by data_dimensions to be
correctly read.
|
data_descriptors
[unicode, ]
|
Denotes what each data dimension
represents. = ‘num_antennas’,
‘num_sequences’, ‘num_samps’ for
antennas_iq
|
data_dimensions
[uint32, ]
|
The dimensions in which to reshape the
data. Dimensions correspond to
data_descriptors.
|
data_normalization_factor
float32
|
Scale of all the filters used, multiplied
for a total scale to normalize the data
by.
|
experiment_comment
unicode
|
Comment provided in experiment about the
experiment as a whole.
|
experiment_id
int64
|
Number used to identify the experiment.
|
experiment_name
unicode
|
Name of the experiment file.
|
freq
uint32
|
The frequency used for this experiment,
in kHz. This is the frequency the data
has been filtered to.
|
int_time
float32
|
Integration time in seconds.
|
intf_antenna_count
uint32
|
Number of interferometer array antennas
|
main_antenna_count
uint32
|
Number of main array antennas
|
noise_at_freq
[float64, ]
|
Noise at the receive frequency, with
dimension = number of sequences.
20191114: not currently implemented and
filled with zeros. Still a TODO.
|
num_samps
uint32
|
Number of samples in the sampling
period. Each sequence has its own
sampling period. Will also be provided
as the last data_dimension value.
|
num_sequences
int64
|
Number of sampling periods (equivalent to
number sequences transmitted) in the
integration time.
|
num_slices
int64
|
Number of slices used simultaneously in
this record by the experiment. If more
than 1, data should exist in another file
for this time period for the other slice.
|
pulse_phase_offset
[float32, ]
|
For pulse encoding phase, in degrees
offset. Contains one phase offset per
pulse in pulses.
|
pulses
[uint32, ]
|
The pulse sequence in units of the
tau_spacing.
|
rx_sample_rate
float64
|
Sampling rate of the samples in this
file’s data in Hz.
|
samples_data_type
unicode
|
C data type of the samples, provided for
user friendliness. = ‘complex float’
|
scan_start_marker
bool
|
Designates if the record is the first in
a scan (scan is defined by the
experiment).
|
slice_comment
unicode
|
Additional text comment that describes
the slice written in this file.
|
sqn_timestamps
[float64, ]
|
A list of GPS timestamps corresponding to
the beginning of transmission for each
sampling period in the integration time.
These timestamps come from the USRP
driver and the USRPs are GPS disciplined
and synchronized using the Octoclock.
Provided in seconds since epoch.
|
station
unicode
|
Three-letter radar identifier.
|
tau_spacing
uint32
|
The minimum spacing between pulses in
microseconds. Spacing between pulses is
always a multiple of this.
|
tx_pulse_len
uint32
|
Length of the transmit pulse in
microseconds.
|
The pyDARNio format class for this format is BorealisRawrfv0_4 found in the borealis_formats.
This format is intended to hold high bandwidth, non-filtered raw data from every antenna.
This format is only produced in a site-style, record by record format and is only available to be produced on request. Please note that this format can cause radar operating delays and may reduce number of averages in an integration, for example.
Site files are produced by the Borealis code package and have the data in a record by record style format. In site files, the hdf5 group names (ie record names) are given as the timestamp in ms past epoch of the first sequence or sampling period recorded in the record.
The naming convention of the rawrf site-structured files are:
[YYYYmmDD].[HHMM].[SS].[station_id].rawrf.hdf5.site
For example:
20191105.1400.02.sas.rawrf.hdf5.site
This is the file that began writing at 14:00:02 UT on November 5 2019 at the Saskatoon site, and it provides data the experiment that ran at that time. Since rawrf is pre-filtered, this data does not need a slice identifier because it contains all the samples being taken at that time. Some familiarity with the experiment may be necessary to understand the data. This is primarily a debug format for engineering purposes.
These files are often bzipped after they are produced.
The file fields under the record name in rawrf site files are:
Field name
type
|
description |
|---|---|
borealis_git_hash
unicode
|
Identifies the version of Borealis that
made this data. Contains git commit hash
characters. Typically begins with the
latest git tag of the software.
|
data
[complex64, ]
|
A contiguous set of samples (complex
float) at given sample rate. Needs to be
reshaped by data_dimensions to be
correctly read.
|
data_descriptors
[unicode, ]
|
Denotes what each data dimension
represents. = ‘num_sequences’,
‘num_antennas’, ‘num_samps’ for
rawrf
|
data_dimensions
[uint32, ]
|
The dimensions in which to reshape the
data. Dimensions correspond to
data_descriptors.
|
experiment_comment
unicode
|
Comment provided in experiment about the
experiment as a whole.
|
experiment_id
int64
|
Number used to identify the experiment.
|
experiment_name
unicode
|
Name of the experiment file.
|
int_time
float32
|
Integration time in seconds.
|
intf_antenna_count
uint32
|
Number of interferometer array antennas
|
main_antenna_count
uint32
|
Number of main array antennas
|
num_samps
uint32
|
Number of samples in the sampling
period. Each sequence has its own
sampling period. Will also be provided
as the last data_dimension value.
|
num_sequences
int64
|
Number of sampling periods (equivalent to
number sequences transmitted) in the
integration time.
|
num_slices
int64
|
Number of slices used simultaneously in
this record by the experiment. If more
than 1, data should exist in another file
for this time period for the other slice.
|
rx_center_freq
float64
|
Center frequency of the sampled data
in kHz.
|
rx_sample_rate
float64
|
Sampling rate of the samples in this
file’s data in Hz.
|
samples_data_type
unicode
|
C data type of the samples, provided for
user friendliness. = ‘complex float’
|
scan_start_marker
bool
|
Designates if the record is the first in
a scan (scan is defined by the
experiment).
|
sqn_timestamps
[float64, ]
|
A list of GPS timestamps corresponding to
the beginning of transmission for each
sampling period in the integration time.
These timestamps come from the USRP
driver and the USRPs are GPS disciplined
and synchronized using the Octoclock.
Provided in seconds since epoch.
|
station
unicode
|
Three-letter radar identifier.
|
FIELD NAME
type
[dimensions]
|
description |
|---|---|
averaging_method
unicode
|
A string describing the averaging method.
Default is ‘mean’ but an experiment can
set this to ‘median’ to get the median of
all sequences in an integration period,
and other methods to combine all
sequences in an integration period could
be added in the future.
|
beam_azms
float64
[num_records x
max_num_beams]
|
A list of the beam azimuths for each beam
in degrees off boresite. Note that this
is padded with zeroes for any record
which has num_beams less than the
max_num_beams. The num_beams field should
be used to read the correct number of
beams for each record.
|
beam_nums
uint32
[num_records x
max_num_beams]
|
A list of beam numbers used in this slice
in this record. Note that this is padded
with zeroes for any record which has
num_beams less than the max_num_beams.
The num_beams field should be used to
read the correct number of beams for each
record.
|
blanked_samples
uint32
[num_records x
max_num_blanked_samples ]
|
Samples that should be blanked because
they occurred during transmission times,
given by sample number (index into
decimated data). Can differ from the
pulses array due to multiple slices in a
single sequence and can differ from
record to record if a new slice is added.
|
borealis_git_hash
unicode
|
Identifies the version of Borealis that
made this data. Contains git commit hash
characters. Typically begins with the
latest git tag of the software.
|
correlation_descriptors
unicode
[4]
|
Denotes what each correlation dimension
(in main_acfs, intf_acfs, xcfs)
represents. = ‘num_records’,
‘max_num_beams’, ‘num_ranges’, ‘num_lags’
|
data_normalization_factor
float32
|
Scale of all the filters used,
multiplied, for a total scale to
normalize the data by.
|
experiment_comment
unicode
|
Comment provided in experiment about the
experiment as a whole.
|
experiment_id
int64
|
Number used to identify the experiment.
|
experiment_name
unicode
|
Name of the experiment file.
|
first_range
float32
|
Distance to use for first range in km.
|
first_range_rtt
float32
|
Round trip time of flight to first range
in microseconds.
|
freq
uint32
|
The frequency used for this experiment,
in kHz. This is the frequency the data
has been filtered to.
|
int_time
float32
[num_records]
|
Integration time in seconds.
|
intf_acfs
complex64
[num_records x
max_num_beams x
num_ranges x
num_lags]
|
Interferometer array correlations. Note
that records that do not have num_beams =
max_num_beams will have padded zeros. The
num_beams array should be used to
determine the correct number of beams to
read for the record.
|
intf_antenna_count
uint32
|
Number of interferometer array antennas
|
lags
uint32
[number of lags, 2]
|
The lags created from two pulses in the
pulses array. Values have to be from
pulses array. The lag number is lag[1] -
lag[0] for each lag pair.
|
main_acfs
complex64
[num_records x
max_num_beams x
num_ranges x
num_lags]
|
Main array correlations. Note
that records that do not have num_beams =
max_num_beams will have padded zeros. The
num_beams array should be used to
determine the correct number of beams to
read for the record.
|
main_antenna_count
uint32
|
Number of main array antennas
|
noise_at_freq
float64
[num_records x
max_num_sequences]
|
Noise at the receive frequency, with
dimension = number of sequences.
20191114: not currently implemented and
filled with zeros. Still a TODO. Note
that records that do not have
num_sequences = max_num_sequences will
have padded zeros. The num_sequences
array should be used to determine the
correct number of sequences to read for
the record.
|
num_beams
uint32
[num_records]
|
The number of beams calculated for each
record. Allows the user to correctly read
the data up to the correct number and
remove the padded zeros in the data
array.
|
num_blanked_samples
uint32
[num_records]
|
The number of blanked samples for each
record.
|
num_sequences
int64
[num_records]
|
Number of sampling periods (equivalent to
number sequences transmitted) in the
integration time for each record. Allows
the user to correctly read the data up to
the correct number and remove the padded
zeros in the data array.
|
num_slices
int64
[num_records]
|
Number of slices used simultaneously in
the record by the experiment. If more
than 1, data should exist in another file
for the same time period as that record
for the other slice.
|
pulses
uint32
[number of pulses]
|
The pulse sequence in units of the
tau_spacing.
|
range_sep
float32
|
Range gate separation (conversion from
time (1/rx_sample_rate) to equivalent
distance between samples), in km.
|
rx_sample_rate
float64
|
Sampling rate of the samples in this
file’s data in Hz.
|
samples_data_type
unicode
|
C data type of the samples, provided for
user friendliness. = ‘complex float’
|
scan_start_marker
bool
[num_records]
|
Designates if the record is the first in
a scan (scan is defined by the
experiment).
|
scheduling_mode
unicode
|
The mode being run during this time
period (ex. ‘common’, ‘special’,
‘discretionary’).
|
slice_comment
unicode
|
Additional text comment that describes
the slice written in this file.
|
slice_id
uint32
|
The slice id of this file.
|
slice_interfacing
unicode
[num_records]
|
The interfacing of this slice to
other slices for each record. String
representation of the python dictionary
of {slice : interface_type, … }. Can
differ between records if slices updated.
|
sqn_timestamps
float64
[num_records x
max_num_sequences]
|
A list of GPS timestamps corresponding to
the beginning of transmission for each
sampling period in the integration time.
These timestamps come back from the USRP
driver and the USRPs are GPS disciplined
and synchronized using the Octoclock.
Provided in seconds since epoch.
Note that records that do not have
num_sequences = max_num_sequences will
have padded zeros. The num_sequences
array should be used to determine the
correct number of sequences to read for
the record.
|
station
unicode
|
Three-letter radar identifier.
|
tau_spacing
uint32
|
The minimum spacing between pulses in
microseconds. Spacing between pulses is
always a multiple of this.
|
tx_pulse_len
uint32
|
Length of the transmit pulse in
microseconds.
|
xcfs
complex64
[num_records x
max_num_beams x
num_ranges x
num_lags]
|
Cross correlations of interferometer to
main array. Note
that records that do not have num_beams =
max_num_beams will have padded zeros. The
num_beams array should be used to
determine the correct number of beams to
read for the record.
|
Field name
type
|
description |
|---|---|
averaging_method
unicode
|
A string describing the averaging method.
Default is ‘mean’ but an experiment can
set this to ‘median’ to get the median of
all sequences in an integration period,
and other methods to combine all
sequences in an integration period could
be added in the future.
|
beam_azms
[float64, ]
|
A list of the beam azimuths for each
beam in degrees off boresite.
|
beam_nums
[uint32, ]
|
A list of beam numbers used in this slice
in this record.
|
blanked_samples
[uint32, ]
|
Samples that should be blanked because
they occurred during transmission times,
given by sample number (index into
decimated data). Can differ from the
pulses array due to multiple slices in a
single sequence.
|
borealis_git_hash
unicode
|
Identifies the version of Borealis that
made this data. Contains git commit hash
characters. Typically begins with the
latest git tag of the software.
|
correlation_descriptors
[unicode, ]
|
Denotes what each correlation dimension
(in main_acfs, intf_acfs, xcfs)
represents. (‘num_beams, ‘num_ranges’,
‘num_lags’)
|
correlation_dimensions
[uint32, ]
|
The dimensions in which to reshape the
acf or xcf datasets. Dimensions
correspond to correlation_descriptors.
|
data_normalization_factor
float32
|
Scale of all the filters used, multiplied
for a total scale to normalize the data
by.
|
experiment_comment
unicode
|
Comment provided in experiment about the
experiment as a whole.
|
experiment_id
int64
|
Number used to identify the experiment.
|
experiment_name
unicode
|
Name of the experiment file.
|
first_range
float32
|
Distance to use for first range in km.
|
first_range_rtt
float32
|
Round trip time of flight to first range
in microseconds.
|
freq
uint32
|
The frequency used for this experiment,
in kHz. This is the frequency the data
has been filtered to.
|
int_time
float32
|
Integration time in seconds.
|
intf_acfs
[complex64, ]
|
Interferometer array correlations.
|
intf_antenna_count
uint32
|
Number of interferometer array antennas
|
lags
[[uint32, ], ]
|
The lags created from two pulses in the
pulses array. Dimensions are number of
lags x 2. Values have to be from pulses
array. The lag number is lag[1] - lag[0]
for each lag pair.
|
lp_status_word
uint32
|
Low power status word. Bit position
corresponds to the USRP motherboard/
transmitter. A ‘1’ indicates low power
occurred at least once during integration
|
main_acfs
[complex64, ]
|
Main array correlations.
|
main_antenna_count
uint32
|
Number of main array antennas
|
noise_at_freq
[float64, ]
|
Noise at the receive frequency, with
dimension = number of sequences.
20191114: not currently implemented and
filled with zeros. Still a TODO.
|
num_sequences
int64
|
Number of sampling periods (equivalent to
number sequences transmitted) in the
integration time.
|
num_slices
int64
|
Number of slices used simultaneously in
this record by the experiment. If more
than 1, data should exist in another file
for this time period for the other slice.
|
pulses
[uint32, ]
|
The pulse sequence in units of the
tau_spacing.
|
range_sep
float32
|
Range gate separation (conversion from
time (1/rx_sample_rate) to equivalent
distance between samples), in km.
|
rx_sample_rate
float64
|
Sampling rate of the samples in this
file’s data in Hz.
|
samples_data_type
unicode
|
C data type of the samples, provided for
user friendliness. = ‘complex float’
|
scan_start_marker
bool
|
Designates if the record is the first in
a scan (scan is defined by the
experiment).
|
scheduling_mode
unicode
|
The mode being run during this time
period (ex. ‘common’, ‘special’,
‘discretionary’).
|
slice_comment
unicode
|
Additional text comment that describes
the slice written in this file.
|
slice_id
uint32
|
The slice id of this file.
|
slice_interfacing
unicode
|
The interfacing of this slice to
other slices. String representation of
the python dictionary of
{slice : interface_type, … }
|
sqn_timestamps
[float64, ]
|
A list of GPS timestamps corresponding to
the beginning of transmission for each
sampling period in the integration time.
These timestamps come from the USRP
driver and the USRPs are GPS disciplined
and synchronized using the Octoclock.
Provided in seconds since epoch.
|
station
unicode
|
Three-letter radar identifier.
|
tau_spacing
uint32
|
The minimum spacing between pulses in
microseconds. Spacing between pulses is
always a multiple of this.
|
tx_pulse_len
uint32
|
Length of the transmit pulse in
microseconds.
|
xcfs
[complex64, ]
|
Cross correlations of interferometer to
main array.
|
FIELD NAME
type
[dimensions]
|
description |
|---|---|
antenna_arrays_order
unicode
[num_antenna_arrays]
|
States what order the data is in and
describes the data layout for the
num_antenna_arrays data dimension
|
beam_azms
float64
[num_records x
max_num_beams]
|
A list of the beam azimuths for each beam
in degrees off boresite. Note that this
is padded with zeroes for any record
which has num_beams less than the
max_num_beams. The num_beams field should
be used to read the correct number of
beams for each record.
|
beam_nums
uint32
[num_records x
max_num_beams]
|
A list of beam numbers used in this slice
in this record. Note that this is padded
with zeroes for any record which has
num_beams less than the max_num_beams.
The num_beams field should be used to
read the correct number of beams for each
record.
|
blanked_samples
uint32
[num_records x
max_num_blanked_samples ]
|
Samples that should be blanked because
they occurred during transmission times,
given by sample number (index into
decimated data). Can differ from the
pulses array due to multiple slices in a
single sequence and can differ from
record to record if a new slice is added.
|
borealis_git_hash
unicode
|
Identifies the version of Borealis that
made this data. Contains git commit hash
characters. Typically begins with the
latest git tag of the software.
|
data
complex64
[num_records x
num_antenna_arrays x
max_num_sequences x
max_num_beams x
num_samps]
|
A set of samples (complex float) at given
sample rate. Note that records that do not
have num_sequences = max_num_sequences or
num_beams = max_num_beams will have
padded zeros. The num_sequences and
num_beams arrays should be used to
determine the correct number of sequences
and beams to read for the record.
|
data_descriptors
unicode
[5]
|
Denotes what each data dimension
represents. = ‘num_records’,
‘num_antenna_arrays’,
‘max_num_sequences’, ‘max_num_beams’,
‘num_samps’
|
data_normalization_factor
float32
|
Scale of all the filters used,
multiplied, for a total scale to
normalize the data by.
|
experiment_comment
unicode
|
Comment provided in experiment about the
experiment as a whole.
|
experiment_id
int64
|
Number used to identify the experiment.
|
experiment_name
unicode
|
Name of the experiment file.
|
first_range
float32
|
Distance to use for first range in km.
|
first_range_rtt
float32
|
Round trip time of flight to first range
in microseconds.
|
freq
uint32
|
The frequency used for this experiment,
in kHz. This is the frequency the data
has been filtered to.
|
int_time
float32
[num_records]
|
Integration time in seconds.
|
intf_antenna_count
uint32
|
Number of interferometer array antennas
|
lags
uint32
[number of lags, 2]
|
The lags created from two pulses in the
pulses array. Values have to be from
pulses array. The lag number is lag[1] -
lag[0] for each lag pair.
|
main_antenna_count
uint32
|
Number of main array antennas
|
noise_at_freq
float64
[num_records x
max_num_sequences]
|
Noise at the receive frequency, with
dimension = number of sequences.
20191114: not currently implemented and
filled with zeros. Still a TODO. Note
that records that do not have
num_sequences = max_num_sequences will
have padded zeros. The num_sequences
array should be used to determine the
correct number of sequences to read for
the record.
|
num_beams
uint32
[num_records]
|
The number of beams calculated for each
record. Allows the user to correctly read
the data up to the correct number and
remove the padded zeros in the data
array.
|
num_blanked_samples
uint32
[num_records]
|
The number of blanked samples for each
record.
|
num_ranges
uint32
|
Number of ranges to calculate
correlations for.
|
num_samps
uint32
|
Number of samples in the sampling
period. Each sequence has its own
sampling period. Will also be provided
as the last data_dimension value.
|
num_sequences
int64
[num_records]
|
Number of sampling periods (equivalent to
number sequences transmitted) in the
integration time for each record. Allows
the user to correctly read the data up to
the correct number and remove the padded
zeros in the data array.
|
num_slices
int64
[num_records]
|
Number of slices used simultaneously in
the record by the experiment. If more
than 1, data should exist in another file
for the same time period as that record
for the other slice.
|
pulse_phase_offset
float32
[number of pulses]
|
For pulse encoding phase, in degrees
offset. Contains one phase offset per
pulse in pulses.
|
pulses
uint32
[number of pulses]
|
The pulse sequence in units of the
tau_spacing.
|
range_sep
float32
|
Range gate separation (conversion from
time (1/rx_sample_rate) to equivalent
distance between samples), in km.
|
rx_sample_rate
float64
|
Sampling rate of the samples in this
file’s data in Hz.
|
samples_data_type
unicode
|
C data type of the samples, provided for
user friendliness. = ‘complex float’
|
scan_start_marker
bool
[num_records]
|
Designates if the record is the first in
a scan (scan is defined by the
experiment).
|
scheduling_mode
unicode
|
The mode being run during this time
period (ex. ‘common’, ‘special’,
‘discretionary’).
|
slice_comment
unicode
|
Additional text comment that describes
the slice written in this file. The slice
number of this file is provided in the
filename.
|
slice_id
uint32
|
The slice id of this file.
|
slice_interfacing
unicode
[num_records]
|
The interfacing of this slice to
other slices for each record. String
representation of the python dictionary
of {slice : interface_type, … }. Can
differ between records if slices updated.
|
sqn_timestamps
float64
[num_records x
max_num_sequences]
|
A list of GPS timestamps corresponding to
the beginning of transmission for each
sampling period in the integration time.
These timestamps come back from the USRP
driver and the USRPs are GPS disciplined
and synchronized using the Octoclock.
Provided in seconds since epoch.
Note that records that do not have
num_sequences = max_num_sequences will
have padded zeros. The num_sequences
array should be used to determine the
correct number of sequences to read for
the record.
|
station
unicode
|
Three-letter radar identifier.
|
tau_spacing
uint32
|
The minimum spacing between pulses in
microseconds. Spacing between pulses is
always a multiple of this.
|
tx_pulse_len
uint32
|
Length of the transmit pulse in
microseconds.
|
Field name
type
|
description |
|---|---|
antenna_arrays_order
[unicode, ]
|
States what order the data is in and
describes the data layout for the
num_antenna_arrays data dimension
|
beam_azms
[float64, ]
|
A list of the beam azimuths for each
beam in degrees off boresite.
|
beam_nums
[uint32, ]
|
A list of beam numbers used in this slice
in this record.
|
blanked_samples
[uint32, ]
|
Samples that should be blanked because
they occurred during transmission times,
given by sample number (index into
decimated data). Can differ from the
pulses array due to multiple slices in a
single sequence.
|
borealis_git_hash
unicode
|
Identifies the version of Borealis that
made this data. Contains git commit hash
characters. Typically begins with the
latest git tag of the software.
|
data
[complex64, ]
|
A contiguous set of samples (complex
float) at given sample rate. Needs to be
reshaped by data_dimensions to be
correctly read.
|
data_descriptors
[unicode, ]
|
Denotes what each data dimension
represents. = ‘num_antenna_arrays’,
‘num_sequences’, ‘num_beams’, ‘num_samps’
for bfiq
|
data_dimensions
[uint32, ]
|
The dimensions in which to reshape the
data. Dimensions correspond to
data_descriptors.
|
data_normalization_factor
float32
|
Scale of all the filters used, multiplied
for a total scale to normalize the data
by.
|
experiment_comment
unicode
|
Comment provided in experiment about the
experiment as a whole.
|
experiment_id
int64
|
Number used to identify the experiment.
|
experiment_name
unicode
|
Name of the experiment file.
|
first_range
float32
|
Distance to use for first range in km.
|
first_range_rtt
float32
|
Round trip time of flight to first range
in microseconds.
|
freq
uint32
|
The frequency used for this experiment,
in kHz. This is the frequency the data
has been filtered to.
|
int_time
float32
|
Integration time in seconds.
|
intf_antenna_count
uint32
|
Number of interferometer array antennas
|
lags
[[uint32, ], ]
|
The lags created from two pulses in the
pulses array. Dimensions are number of
lags x 2. Values have to be from pulses
array. The lag number is lag[1] - lag[0]
for each lag pair.
|
main_antenna_count
uint32
|
Number of main array antennas
|
noise_at_freq
[float64, ]
|
Noise at the receive frequency, with
dimension = number of sequences.
20191114: not currently implemented and
filled with zeros. Still a TODO.
|
num_ranges
uint32
|
Number of ranges to calculate
correlations for.
|
num_samps
uint32
|
Number of samples in the sampling
period. Each sequence has its own
sampling period. Will also be provided
as the last data_dimension value.
|
num_sequences
int64
|
Number of sampling periods (equivalent to
number sequences transmitted) in the
integration time.
|
num_slices
int64
|
Number of slices used simultaneously in
this record by the experiment. If more
than 1, data should exist in another file
for this time period for the other slice.
|
pulse_phase_offset
[float32, ]
|
For pulse encoding phase, in degrees
offset. Contains one phase offset per
pulse in pulses.
|
pulses
[uint32, ]
|
The pulse sequence in units of the
tau_spacing.
|
range_sep
float32
|
Range gate separation (conversion from
time (1/rx_sample_rate) to equivalent
distance between samples), in km.
|
rx_sample_rate
float64
|
Sampling rate of the samples in this
file’s data in Hz.
|
samples_data_type
unicode
|
C data type of the samples, provided for
user friendliness. = ‘complex float’
|
scan_start_marker
bool
|
Designates if the record is the first in
a scan (scan is defined by the
experiment).
|
scheduling_mode
unicode
|
The mode being run during this time
period (ex. ‘common’, ‘special’,
‘discretionary’).
|
slice_comment
unicode
|
Additional text comment that describes
the slice written in this file.
|
slice_id
uint32
|
The slice id of this file.
|
slice_interfacing
unicode
|
The interfacing of this slice to
other slices. String representation of
the python dictionary of
{slice : interface_type, … }
|
sqn_timestamps
[float64, ]
|
A list of GPS timestamps corresponding to
the beginning of transmission for each
sampling period in the integration time.
These timestamps come from the USRP
driver and the USRPs are GPS disciplined
and synchronized using the Octoclock.
Provided in seconds since epoch.
|
station
unicode
|
Three-letter radar identifier.
|
tau_spacing
uint32
|
The minimum spacing between pulses in
microseconds. Spacing between pulses is
always a multiple of this.
|
tx_pulse_len
uint32
|
Length of the transmit pulse in
microseconds.
|
FIELD NAME
type
[dimensions]
|
description |
|---|---|
antenna_arrays_order
unicode
[num_antennas]
|
States what order the data is in and
describes the data layout for the
num_antennas data dimension
Antennas are recorded main array
ascending and then interferometer array
ascending
|
beam_azms
float64
[num_records x
max_num_beams]
|
A list of the beam azimuths for each beam
in degrees off boresite. Note that this
is padded with zeroes for any record
which has num_beams less than the
max_num_beams. The num_beams field should
be used to read the correct number of
beams for each record.
|
beam_nums
uint32
[num_records x
max_num_beams]
|
A list of beam numbers used in this slice
in this record. Note that this is padded
with zeroes for any record which has
num_beams less than the max_num_beams.
The num_beams field should be used to
read the correct number of beams for each
record.
|
blanked_samples
uint32
[num_records x
max_num_blanked_samples ]
|
Samples that should be blanked because
they occurred during transmission times,
given by sample number (index into
decimated data). Can differ from the
pulses array due to multiple slices in a
single sequence and can differ from
record to record if a new slice is added.
|
borealis_git_hash
unicode
|
Identifies the version of Borealis that
made this data. Contains git commit hash
characters. Typically begins with the
latest git tag of the software.
|
data
complex64
[num_records x
num_antennas x
max_num_sequences x
num_samps]
|
A set of samples (complex float) at given
sample rate. Note that records that do not
have num_sequences = max_num_sequences
will have padded zeros. The num_sequences
array should be used to determine the
correct number of sequences to read for
the record.
|
data_descriptors
unicode
[4]
|
Denotes what each data dimension
represents. = ‘num_records’,
‘num_antennas’, ‘max_num_sequences’,
‘num_samps’
|
data_normalization_factor
float32
|
Scale of all the filters used,
multiplied, for a total scale to
normalize the data by.
|
experiment_comment
unicode
|
Comment provided in experiment about the
experiment as a whole.
|
experiment_id
int64
|
Number used to identify the experiment.
|
experiment_name
unicode
|
Name of the experiment file.
|
freq
uint32
|
The frequency used for this experiment,
in kHz. This is the frequency the data
has been filtered to.
|
int_time
float32
[num_records]
|
Integration time in seconds.
|
intf_antenna_count
uint32
|
Number of interferometer array antennas
|
main_antenna_count
uint32
|
Number of main array antennas
|
noise_at_freq
float64
[num_records x
max_num_sequences]
|
Noise at the receive frequency, with
dimension = number of sequences.
20191114: not currently implemented and
filled with zeros. Still a TODO. Note
that records that do not have
num_sequences = max_num_sequences will
have padded zeros. The num_sequences
array should be used to determine the
correct number of sequences to read for
the record.
|
num_beams
uint32
[num_records]
|
The number of beams to calculate for each
record.
|
num_blanked_samples
uint32
[num_records]
|
The number of blanked samples for each
record.
|
num_samps
uint32
|
Number of samples in the sampling
period. Each sequence has its own
sampling period. Will also be provided
as the last data_dimension value.
|
num_sequences
int64
[num_records]
|
Number of sampling periods (equivalent to
number sequences transmitted) in the
integration time for each record. Allows
the user to correctly read the data up to
the correct number and remove the padded
zeros in the data array.
|
num_slices
int64
[num_records]
|
Number of slices used simultaneously in
the record by the experiment. If more
than 1, data should exist in another file
for the same time period as that record
for the other slice.
|
pulse_phase_offset
float32
[number of pulses]
|
For pulse encoding phase, in degrees
offset. Contains one phase offset per
pulse in pulses.
|
pulses
uint32
[number of pulses]
|
The pulse sequence in units of the
tau_spacing.
|
rx_sample_rate
float64
|
Sampling rate of the samples in this
file’s data in Hz.
|
samples_data_type
unicode
|
C data type of the samples, provided for
user friendliness. = ‘complex float’
|
scan_start_marker
bool
[num_records]
|
Designates if the record is the first in
a scan (scan is defined by the
experiment).
|
scheduling_mode
unicode
|
The mode being run during this time
period (ex. ‘common’, ‘special’,
‘discretionary’).
|
slice_comment
unicode
|
Additional text comment that describes
the slice written in this file. The slice
number of this file is provided in the
filename.
|
slice_id
uint32
|
The slice id of this file.
|
slice_interfacing
unicode
[num_records]
|
The interfacing of this slice to
other slices for each record. String
representation of the python dictionary
of {slice : interface_type, … }. Can
differ between records if slices updated.
|
sqn_timestamps
float64
[num_records x
max_num_sequences]
|
A list of GPS timestamps corresponding to
the beginning of transmission for each
sampling period in the integration time.
These timestamps come back from the USRP
driver and the USRPs are GPS disciplined
and synchronized using the Octoclock.
Provided in seconds since epoch.
Note that records that do not have
num_sequences = max_num_sequences will
have padded zeros. The num_sequences
array should be used to determine the
correct number of sequences to read for
the record.
|
station
unicode
|
Three-letter radar identifier.
|
tau_spacing
uint32
|
The minimum spacing between pulses in
microseconds. Spacing between pulses is
always a multiple of this.
|
tx_pulse_len
uint32
|
Length of the transmit pulse in
microseconds.
|
Field name
type
|
description |
|---|---|
antenna_arrays_order
[unicode, ]
|
States what order the data is in and
describes the data layout for the
num_antennas data dimension. Antennas are
recorded main array ascending and then
interferometer array ascending.
|
beam_azms
[float64, ]
|
A list of the beam azimuths for each
beam in degrees off boresite.
|
beam_nums
[uint32, ]
|
A list of beam numbers used in this slice
in this record.
|
blanked_samples
[uint32, ]
|
Samples that should be blanked because
they occurred during transmission times,
given by sample number (index into
decimated data). Can differ from the
pulses array due to multiple slices in a
single sequence.
|
borealis_git_hash
unicode
|
Identifies the version of Borealis that
made this data. Contains git commit hash
characters. Typically begins with the
latest git tag of the software.
|
data
[complex64, ]
|
A contiguous set of samples (complex
float) at given sample rate. Needs to be
reshaped by data_dimensions to be
correctly read.
|
data_descriptors
[unicode, ]
|
Denotes what each data dimension
represents. = ‘num_antennas’,
‘num_sequences’, ‘num_samps’ for
antennas_iq
|
data_dimensions
[uint32, ]
|
The dimensions in which to reshape the
data. Dimensions correspond to
data_descriptors.
|
data_normalization_factor
float32
|
Scale of all the filters used, multiplied
for a total scale to normalize the data
by.
|
experiment_comment
unicode
|
Comment provided in experiment about the
experiment as a whole.
|
experiment_id
int64
|
Number used to identify the experiment.
|
experiment_name
unicode
|
Name of the experiment file.
|
freq
uint32
|
The frequency used for this experiment,
in kHz. This is the frequency the data
has been filtered to.
|
int_time
float32
|
Integration time in seconds.
|
intf_antenna_count
uint32
|
Number of interferometer array antennas
|
main_antenna_count
uint32
|
Number of main array antennas
|
noise_at_freq
[float64, ]
|
Noise at the receive frequency, with
dimension = number of sequences.
20191114: not currently implemented and
filled with zeros. Still a TODO.
|
num_samps
uint32
|
Number of samples in the sampling
period. Each sequence has its own
sampling period. Will also be provided
as the last data_dimension value.
|
num_sequences
int64
|
Number of sampling periods (equivalent to
number sequences transmitted) in the
integration time.
|
num_slices
int64
|
Number of slices used simultaneously in
this record by the experiment. If more
than 1, data should exist in another file
for this time period for the other slice.
|
pulse_phase_offset
[float32, ]
|
For pulse encoding phase, in degrees
offset. Contains one phase offset per
pulse in pulses.
|
pulses
[uint32, ]
|
The pulse sequence in units of the
tau_spacing.
|
rx_sample_rate
float64
|
Sampling rate of the samples in this
file’s data in Hz.
|
samples_data_type
unicode
|
C data type of the samples, provided for
user friendliness. = ‘complex float’
|
scan_start_marker
bool
|
Designates if the record is the first in
a scan (scan is defined by the
experiment).
|
scheduling_mode
unicode
|
The mode being run during this time
period (ex. ‘common’, ‘special’,
‘discretionary’).
|
slice_comment
unicode
|
Additional text comment that describes
the slice written in this file.
|
slice_id
uint32
|
The slice id of this file.
|
slice_interfacing
unicode
|
The interfacing of this slice to
other slices. String representation of
the python dictionary of
{slice : interface_type, … }
|
sqn_timestamps
[float64, ]
|
A list of GPS timestamps corresponding to
the beginning of transmission for each
sampling period in the integration time.
These timestamps come from the USRP
driver and the USRPs are GPS disciplined
and synchronized using the Octoclock.
Provided in seconds since epoch.
|
station
unicode
|
Three-letter radar identifier.
|
tau_spacing
uint32
|
The minimum spacing between pulses in
microseconds. Spacing between pulses is
always a multiple of this.
|
tx_pulse_len
uint32
|
Length of the transmit pulse in
microseconds.
|
The pyDARNio format class for this format is BorealisRawrfv0_5 found in the borealis_formats.
This format is intended to hold high bandwidth, non-filtered raw data from every antenna.
This format is only produced in a site-style, record by record format and is only available to be produced on request. Please note that this format can cause radar operating delays and may reduce number of averages in an integration, for example.
Site files are produced by the Borealis code package and have the data in a record by record style format. In site files, the hdf5 group names (ie record names) are given as the timestamp in ms past epoch of the first sequence or sampling period recorded in the record.
The naming convention of the rawrf site-structured files are:
[YYYYmmDD].[HHMM].[SS].[station_id].rawrf.hdf5.site
For example:
20191105.1400.02.sas.rawrf.hdf5.site
This is the file that began writing at 14:00:02 UT on November 5 2019 at the Saskatoon site, and it provides data the experiment that ran at that time. Since rawrf is not filtered, this data does not need a slice identifier because it contains all the samples being taken at that time. Some familiarity with the experiment may be necessary to understand the data, or some access to the other file types produced concurrently. This is primarily a debug format for engineering purposes and should only be produced for special cases.
These files are often bzipped after they are produced.
The file fields under the record name in rawrf site files are:
Field name
type
|
description |
|---|---|
blanked_samples
uint32
[number of blanked
samples]
|
Samples that should be blanked because
they occurred during transmission times,
given by sample number (index into
decimated data). Can differ from the
pulses array due to multiple slices in a
single sequence.
|
borealis_git_hash
unicode
|
Identifies the version of Borealis that
made this data. Contains git commit hash
characters. Typically begins with the
latest git tag of the software.
|
data
[complex64, ]
|
A contiguous set of samples (complex
float) at given sample rate. Needs to be
reshaped by data_dimensions to be
correctly read.
|
data_descriptors
[unicode, ]
|
Denotes what each data dimension
represents. = ‘num_sequences’,
‘num_antennas’, ‘num_samps’ for
rawrf
|
data_dimensions
[uint32, ]
|
The dimensions in which to reshape the
data. Dimensions correspond to
data_descriptors.
|
experiment_comment
unicode
|
Comment provided in experiment about the
experiment as a whole.
|
experiment_id
int64
|
Number used to identify the experiment.
|
experiment_name
unicode
|
Name of the experiment file.
|
int_time
float32
|
Integration time in seconds.
|
intf_antenna_count
uint32
|
Number of interferometer array antennas
|
main_antenna_count
uint32
|
Number of main array antennas
|
num_samps
uint32
|
Number of samples in the sampling
period. Each sequence has its own
sampling period. Will also be provided
as the last data_dimension value.
|
num_sequences
int64
|
Number of sampling periods (equivalent to
number sequences transmitted) in the
integration time.
|
num_slices
int64
|
Number of slices used simultaneously in
this record by the experiment. If more
than 1, data should exist in another file
for this time period for the other slice.
|
rx_center_freq
float64
|
Center frequency of the sampled data
in kHz.
|
rx_sample_rate
float64
|
Sampling rate of the samples in this
file’s data in Hz.
|
samples_data_type
unicode
|
C data type of the samples, provided for
user friendliness. = ‘complex float’
|
scan_start_marker
bool
|
Designates if the record is the first in
a scan (scan is defined by the
experiment).
|
scheduling_mode
unicode
|
The mode being run during this time
period (ex. ‘common’, ‘special’,
‘discretionary’).
|
sqn_timestamps
[float64, ]
|
A list of GPS timestamps corresponding to
the beginning of transmission for each
sampling period in the integration time.
These timestamps come from the USRP
driver and the USRPs are GPS disciplined
and synchronized using the Octoclock.
Provided in seconds since epoch.
|
station
unicode
|
Three-letter radar identifier.
|
FIELD NAME
type
[dimensions]
|
description |
|---|---|
agc_status_word
uint32
[num_records]
|
AGC status word. Bit position
corresponds to the USRP motherboard/
transmitter. A ‘1’ indicates an agc fault
occurred at least once during integration
|
averaging_method
unicode
|
A string describing the averaging method.
Default is ‘mean’ but an experiment can
set this to ‘median’ to get the median of
all sequences in an integration period,
and other methods to combine all
sequences in an integration period could
be added in the future.
|
beam_azms
float64
[num_records x
max_num_beams]
|
A list of the beam azimuths for each beam
in degrees off boresite. Note that this
is padded with zeroes for any record
which has num_beams less than the
max_num_beams. The num_beams field should
be used to read the correct number of
beams for each record.
|
beam_nums
uint32
[num_records x
max_num_beams]
|
A list of beam numbers used in this slice
in this record. Note that this is padded
with zeroes for any record which has
num_beams less than the max_num_beams.
The num_beams field should be used to
read the correct number of beams for each
record.
|
blanked_samples
uint32
[num_records x
max_num_blanked_samples ]
|
Samples that should be blanked because
they occurred during transmission times,
given by sample number (index into
decimated data). Can differ from the
pulses array due to multiple slices in a
single sequence and can differ from
record to record if a new slice is added.
|
borealis_git_hash
unicode
|
Identifies the version of Borealis that
made this data. Contains git commit hash
characters. Typically begins with the
latest git tag of the software.
|
correlation_descriptors
unicode
[4]
|
Denotes what each correlation dimension
(in main_acfs, intf_acfs, xcfs)
represents. = ‘num_records’,
‘max_num_beams’, ‘num_ranges’, ‘num_lags’
|
data_normalization_factor
float32
|
Scale of all the filters used,
multiplied, for a total scale to
normalize the data by.
|
experiment_comment
unicode
|
Comment provided in experiment about the
experiment as a whole.
|
experiment_id
int16
|
Number used to identify the experiment.
|
experiment_name
unicode
|
Name of the experiment file.
|
first_range
float32
|
Distance to use for first range in km.
|
first_range_rtt
float32
|
Round trip time of flight to first range
in microseconds.
|
freq
uint32
|
The frequency used for this experiment,
in kHz. This is the frequency the data
has been filtered to.
|
gps_locked
bool
[num_records]
|
Designates if the local GPS had a lock
during the entire integration period.
False if it unlocked at least once.
|
gps_to_system_time_diff
float32
[num_records]
|
The max time difference between box_time
GPS time) and system time (NTP) during the
integration. Negative when GPS time is
ahead of system time.
|
int_time
float32
[num_records]
|
Integration time in seconds.
|
intf_acfs
complex64
[num_records x
max_num_beams x
num_ranges x
num_lags]
|
Interferometer array correlations. Note
that records that do not have num_beams =
max_num_beams will have padded zeros. The
num_beams array should be used to
determine the correct number of beams to
read for the record.
|
intf_antenna_count
uint32
|
Number of interferometer array antennas
|
lags
uint32
[number of lags, 2]
|
The lags created from two pulses in the
pulses array. Values have to be from
pulses array. The lag number is lag[1] -
lag[0] for each lag pair.
|
lp_status_word
uint32
[num_records]
|
Low power status word. Bit position
corresponds to the USRP motherboard/
transmitter. A ‘1’ indicates low power
occurred at least once during integration
|
main_acfs
complex64
[num_records x
max_num_beams x
num_ranges x
num_lags]
|
Main array correlations. Note
that records that do not have num_beams =
max_num_beams will have padded zeros. The
num_beams array should be used to
determine the correct number of beams to
read for the record.
|
main_antenna_count
uint32
|
Number of main array antennas
|
noise_at_freq
float64
[num_records x
max_num_sequences]
|
Noise at the receive frequency, with
dimension = number of sequences.
20191114: not currently implemented and
filled with zeros. Still a TODO. Note
that records that do not have
num_sequences = max_num_sequences will
have padded zeros. The num_sequences
array should be used to determine the
correct number of sequences to read for
the record.
|
num_beams
uint32
[num_records]
|
The number of beams calculated for each
record. Allows the user to correctly read
the data up to the correct number and
remove the padded zeros in the data
array.
|
num_blanked_samples
uint32
[num_records]
|
The number of blanked samples for each
record.
|
num_sequences
int64
[num_records]
|
Number of sampling periods (equivalent to
number sequences transmitted) in the
integration time for each record. Allows
the user to correctly read the data up to
the correct number and remove the padded
zeros in the data array.
|
num_slices
int64
[num_records]
|
Number of slices used simultaneously in
the record by the experiment. If more
than 1, data should exist in another file
for the same time period as that record
for the other slice.
|
pulses
uint32
[number of pulses]
|
The pulse sequence in units of the
tau_spacing.
|
range_sep
float32
|
Range gate separation (conversion from
time (1/rx_sample_rate) to equivalent
distance between samples), in km.
|
rx_sample_rate
float64
|
Sampling rate of the samples in this
file’s data in Hz.
|
samples_data_type
unicode
|
C data type of the samples, provided for
user friendliness. = ‘complex float’
|
scan_start_marker
bool
[num_records]
|
Designates if the record is the first in
a scan (scan is defined by the
experiment).
|
scheduling_mode
unicode
|
The mode being run during this time
period (ex. ‘common’, ‘special’,
‘discretionary’).
|
slice_comment
unicode
|
Additional text comment that describes
the slice written in this file.
|
slice_id
uint32
|
The slice id of this file.
|
slice_interfacing
unicode
[num_records]
|
The interfacing of this slice to
other slices for each record. String
representation of the python dictionary
of {slice : interface_type, … }. Can
differ between records if slices updated.
|
sqn_timestamps
float64
[num_records x
max_num_sequences]
|
A list of GPS timestamps corresponding to
the beginning of transmission for each
sampling period in the integration time.
These timestamps come back from the USRP
driver and the USRPs are GPS disciplined
and synchronized using the Octoclock.
Provided in seconds since epoch.
Note that records that do not have
num_sequences = max_num_sequences will
have padded zeros. The num_sequences
array should be used to determine the
correct number of sequences to read for
the record.
|
station
unicode
|
Three-letter radar identifier.
|
tau_spacing
uint32
|
The minimum spacing between pulses in
microseconds. Spacing between pulses is
always a multiple of this.
|
tx_pulse_len
uint32
|
Length of the transmit pulse in
microseconds.
|
xcfs
complex64
[num_records x
max_num_beams x
num_ranges x
num_lags]
|
Cross correlations of interferometer to
main array. Note
that records that do not have num_beams =
max_num_beams will have padded zeros. The
num_beams array should be used to
determine the correct number of beams to
read for the record.
|
Field name
type
|
description |
|---|---|
agc_status_word
uint32
|
AGC status word. Bit position
corresponds to the USRP motherboard/
transmitter. A ‘1’ indicates an agc fault
occurred at least once during integration
|
averaging_method
unicode
|
A string describing the averaging method.
Default is ‘mean’ but an experiment can
set this to ‘median’ to get the median of
all sequences in an integration period,
and other methods to combine all
sequences in an integration period could
be added in the future.
|
beam_azms
[float64, ]
|
A list of the beam azimuths for each
beam in degrees off boresite.
|
beam_nums
[uint32, ]
|
A list of beam numbers used in this slice
in this record.
|
blanked_samples
[uint32, ]
|
Samples that should be blanked because
they occurred during transmission times,
given by sample number (index into
decimated data). Can differ from the
pulses array due to multiple slices in a
single sequence.
|
borealis_git_hash
unicode
|
Identifies the version of Borealis that
made this data. Contains git commit hash
characters. Typically begins with the
latest git tag of the software.
|
correlation_descriptors
[unicode, ]
|
Denotes what each correlation dimension
(in main_acfs, intf_acfs, xcfs)
represents. (‘num_beams, ‘num_ranges’,
‘num_lags’)
|
correlation_dimensions
[uint32, ]
|
The dimensions in which to reshape the
acf or xcf datasets. Dimensions
correspond to correlation_descriptors.
|
data_normalization_factor
float32
|
Scale of all the filters used, multiplied
for a total scale to normalize the data
by.
|
experiment_comment
unicode
|
Comment provided in experiment about the
experiment as a whole.
|
experiment_id
int16
|
Number used to identify the experiment.
|
experiment_name
unicode
|
Name of the experiment file.
|
first_range
float32
|
Distance to use for first range in km.
|
first_range_rtt
float32
|
Round trip time of flight to first range
in microseconds.
|
freq
uint32
|
The frequency used for this experiment,
in kHz. This is the frequency the data
has been filtered to.
|
gps_locked
bool
|
Designates if the local GPS had a lock
during the entire integration period.
|
gps_to_system_time_diff
float32
|
The max time difference between box_time
GPS time) and system time (NTP) during the
integration. Negative when GPS time is
ahead of system time.
|
int_time
float32
|
Integration time in seconds.
|
intf_acfs
[complex64, ]
|
Interferometer array correlations.
|
intf_antenna_count
uint32
|
Number of interferometer array antennas
|
lags
[[uint32, ], ]
|
The lags created from two pulses in the
pulses array. Dimensions are number of
lags x 2. Values have to be from pulses
array. The lag number is lag[1] - lag[0]
for each lag pair.
|
lp_status_word
uint32
|
Low power status word. Bit position
corresponds to the USRP motherboard/
transmitter. A ‘1’ indicates low power
occurred at least once during integration
|
main_acfs
[complex64, ]
|
Main array correlations.
|
main_antenna_count
uint32
|
Number of main array antennas
|
noise_at_freq
[float64, ]
|
Noise at the receive frequency, with
dimension = number of sequences.
20191114: not currently implemented and
filled with zeros. Still a TODO.
|
num_sequences
int64
|
Number of sampling periods (equivalent to
number sequences transmitted) in the
integration time.
|
num_slices
int64
|
Number of slices used simultaneously in
this record by the experiment. If more
than 1, data should exist in another file
for this time period for the other slice.
|
pulses
[uint32, ]
|
The pulse sequence in units of the
tau_spacing.
|
range_sep
float32
|
Range gate separation (conversion from
time (1/rx_sample_rate) to equivalent
distance between samples), in km.
|
rx_sample_rate
float64
|
Sampling rate of the samples in this
file’s data in Hz.
|
samples_data_type
unicode
|
C data type of the samples, provided for
user friendliness. = ‘complex float’
|
scan_start_marker
bool
|
Designates if the record is the first in
a scan (scan is defined by the
experiment).
|
scheduling_mode
unicode
|
The mode being run during this time
period (ex. ‘common’, ‘special’,
‘discretionary’).
|
slice_comment
unicode
|
Additional text comment that describes
the slice written in this file.
|
slice_id
uint32
|
The slice id of this file.
|
slice_interfacing
unicode
|
The interfacing of this slice to
other slices. String representation of
the python dictionary of
{slice : interface_type, … }
|
sqn_timestamps
[float64, ]
|
A list of GPS timestamps corresponding to
the beginning of transmission for each
sampling period in the integration time.
These timestamps come from the USRP
driver and the USRPs are GPS disciplined
and synchronized using the Octoclock.
Provided in seconds since epoch.
|
station
unicode
|
Three-letter radar identifier.
|
tau_spacing
uint32
|
The minimum spacing between pulses in
microseconds. Spacing between pulses is
always a multiple of this.
|
tx_pulse_len
uint32
|
Length of the transmit pulse in
microseconds.
|
xcfs
[complex64, ]
|
Cross correlations of interferometer to
main array.
|
FIELD NAME
type
[dimensions]
|
description |
|---|---|
agc_status_word
uint32
[num_records]
|
AGC status word. Bit position
corresponds to the USRP motherboard/
transmitter. A ‘1’ indicates an agc fault
occurred at least once during integration
|
antenna_arrays_order
unicode
[num_antenna_arrays]
|
States what order the data is in and
describes the data layout for the
num_antenna_arrays data dimension
|
beam_azms
float64
[num_records x
max_num_beams]
|
A list of the beam azimuths for each beam
in degrees off boresite. Note that this
is padded with zeroes for any record
which has num_beams less than the
max_num_beams. The num_beams field should
be used to read the correct number of
beams for each record.
|
beam_nums
uint32
[num_records x
max_num_beams]
|
A list of beam numbers used in this slice
in this record. Note that this is padded
with zeroes for any record which has
num_beams less than the max_num_beams.
The num_beams field should be used to
read the correct number of beams for each
record.
|
blanked_samples
uint32
[num_records x
max_num_blanked_samples ]
|
Samples that should be blanked because
they occurred during transmission times,
given by sample number (index into
decimated data). Can differ from the
pulses array due to multiple slices in a
single sequence and can differ from
record to record if a new slice is added.
|
borealis_git_hash
unicode
|
Identifies the version of Borealis that
made this data. Contains git commit hash
characters. Typically begins with the
latest git tag of the software.
|
data
complex64
[num_records x
num_antenna_arrays x
max_num_sequences x
max_num_beams x
num_samps]
|
A set of samples (complex float) at given
sample rate. Note that records that do not
have num_sequences = max_num_sequences or
num_beams = max_num_beams will have
padded zeros. The num_sequences and
num_beams arrays should be used to
determine the correct number of sequences
and beams to read for the record.
|
data_descriptors
unicode
[5]
|
Denotes what each data dimension
represents. = ‘num_records’,
‘num_antenna_arrays’,
‘max_num_sequences’, ‘max_num_beams’,
‘num_samps’
|
data_normalization_factor
float32
|
Scale of all the filters used,
multiplied, for a total scale to
normalize the data by.
|
experiment_comment
unicode
|
Comment provided in experiment about the
experiment as a whole.
|
experiment_id
int16
|
Number used to identify the experiment.
|
experiment_name
unicode
|
Name of the experiment file.
|
first_range
float32
|
Distance to use for first range in km.
|
first_range_rtt
float32
|
Round trip time of flight to first range
in microseconds.
|
freq
uint32
|
The frequency used for this experiment,
in kHz. This is the frequency the data
has been filtered to.
|
gps_locked
bool
[num_records]
|
Designates if the local GPS had a lock
during the entire integration period.
False if it unlocked at least once.
|
gps_to_system_time_diff
float32
[num_records]
|
The max time difference between box_time
GPS time) and system time (NTP) during the
integration. Negative when GPS time is
ahead of system time.
|
int_time
float32
[num_records]
|
Integration time in seconds.
|
intf_antenna_count
uint32
|
Number of interferometer array antennas
|
lags
uint32
[number of lags, 2]
|
The lags created from two pulses in the
pulses array. Values have to be from
pulses array. The lag number is lag[1] -
lag[0] for each lag pair.
|
lp_status_word
uint32
[num_records]
|
Low power status word. Bit position
corresponds to the USRP motherboard/
transmitter. A ‘1’ indicates low power
occurred at least once during integration
|
main_antenna_count
uint32
|
Number of main array antennas
|
noise_at_freq
float64
[num_records x
max_num_sequences]
|
Noise at the receive frequency, with
dimension = number of sequences.
20191114: not currently implemented and
filled with zeros. Still a TODO. Note
that records that do not have
num_sequences = max_num_sequences will
have padded zeros. The num_sequences
array should be used to determine the
correct number of sequences to read for
the record.
|
num_beams
uint32
[num_records]
|
The number of beams calculated for each
record. Allows the user to correctly read
the data up to the correct number and
remove the padded zeros in the data
array.
|
num_blanked_samples
uint32
[num_records]
|
The number of blanked samples for each
record.
|
num_ranges
uint32
|
Number of ranges to calculate
correlations for.
|
num_samps
uint32
|
Number of samples in the sampling
period. Each sequence has its own
sampling period. Will also be provided
as the last data_dimension value.
|
num_sequences
int64
[num_records]
|
Number of sampling periods (equivalent to
number sequences transmitted) in the
integration time for each record. Allows
the user to correctly read the data up to
the correct number and remove the padded
zeros in the data array.
|
num_slices
int64
[num_records]
|
Number of slices used simultaneously in
the record by the experiment. If more
than 1, data should exist in another file
for the same time period as that record
for the other slice.
|
pulse_phase_offset
float32
[] or [num pulses x 1]
|
For pulse encoding phase, in degrees
offset. Contains one phase offset per
pulse in pulses
|
pulses
uint32
[number of pulses]
|
The pulse sequence in units of the
tau_spacing.
|
range_sep
float32
|
Range gate separation (conversion from
time (1/rx_sample_rate) to equivalent
distance between samples), in km.
|
rx_sample_rate
float64
|
Sampling rate of the samples in this
file’s data in Hz.
|
samples_data_type
unicode
|
C data type of the samples, provided for
user friendliness. = ‘complex float’
|
scan_start_marker
bool
[num_records]
|
Designates if the record is the first in
a scan (scan is defined by the
experiment).
|
scheduling_mode
unicode
|
The mode being run during this time
period (ex. ‘common’, ‘special’,
‘discretionary’).
|
slice_comment
unicode
|
Additional text comment that describes
the slice written in this file. The slice
number of this file is provided in the
filename.
|
slice_id
uint32
|
The slice id of this file.
|
slice_interfacing
unicode
[num_records]
|
The interfacing of this slice to
other slices for each record. String
representation of the python dictionary
of {slice : interface_type, … }. Can
differ between records if slices updated.
|
sqn_timestamps
float64
[num_records x
max_num_sequences]
|
A list of GPS timestamps corresponding to
the beginning of transmission for each
sampling period in the integration time.
These timestamps come back from the USRP
driver and the USRPs are GPS disciplined
and synchronized using the Octoclock.
Provided in seconds since epoch.
Note that records do not have
num_sequences = max_num_sequences will
have padded zeros. The num_sequences
array should be used to determine the
correct number of sequences to read for
the record.
|
station
unicode
|
Three-letter radar identifier.
|
tau_spacing
uint32
|
The minimum spacing between pulses in
microseconds. Spacing between pulses is
always a multiple of this.
|
tx_pulse_len
uint32
|
Length of the transmit pulse in
microseconds.
|
Field name
type
|
description |
|---|---|
agc_status_word
uint32
|
AGC status word. Bit position
corresponds to the USRP motherboard/
transmitter. A ‘1’ indicates an agc fault
occurred at least once during integration
|
antenna_arrays_order
[unicode, ]
|
States what order the data is in and
describes the data layout for the
num_antenna_arrays data dimension
|
beam_azms
[float64, ]
|
A list of the beam azimuths for each
beam in degrees off boresite.
|
beam_nums
[uint32, ]
|
A list of beam numbers used in this slice
in this record.
|
blanked_samples
[uint32, ]
|
Samples that should be blanked because
they occurred during transmission times,
given by sample number (index into
decimated data). Can differ from the
pulses array due to multiple slices in a
single sequence.
|
borealis_git_hash
unicode
|
Identifies the version of Borealis that
made this data. Contains git commit hash
characters. Typically begins with the
latest git tag of the software.
|
data
[complex64, ]
|
A contiguous set of samples (complex
float) at given sample rate. Needs to be
reshaped by data_dimensions to be
correctly read.
|
data_descriptors
[unicode, ]
|
Denotes what each data dimension
represents. = ‘num_antenna_arrays’,
‘num_sequences’, ‘num_beams’, ‘num_samps’
for bfiq
|
data_dimensions
[uint32, ]
|
The dimensions in which to reshape the
data. Dimensions correspond to
data_descriptors.
|
data_normalization_factor
float32
|
Scale of all the filters used, multiplied
for a total scale to normalize the data
by.
|
experiment_comment
unicode
|
Comment provided in experiment about the
experiment as a whole.
|
experiment_id
int16
|
Number used to identify the experiment.
|
experiment_name
unicode
|
Name of the experiment file.
|
first_range
float32
|
Distance to use for first range in km.
|
first_range_rtt
float32
|
Round trip time of flight to first range
in microseconds.
|
freq
uint32
|
The frequency used for this experiment,
in kHz. This is the frequency the data
has been filtered to.
|
gps_locked
bool
|
Designates if the local GPS had a lock
during the entire integration period.
|
gps_to_system_time_diff
float32
|
The max time difference between box_time
GPS time) and system time (NTP) during the
integration. Negative when GPS time is
ahead of system time.
|
int_time
float32
|
Integration time in seconds.
|
intf_antenna_count
uint32
|
Number of interferometer array antennas
|
lags
[[uint32, ], ]
|
The lags created from two pulses in the
pulses array. Dimensions are number of
lags x 2. Values have to be from pulses
array. The lag number is lag[1] - lag[0]
for each lag pair.
|
lp_status_word
uint32
|
Low power status word. Bit position
corresponds to the USRP motherboard/
transmitter. A ‘1’ indicates low power
occurred at least once during integration
|
main_antenna_count
uint32
|
Number of main array antennas
|
noise_at_freq
[float64, ]
|
Noise at the receive frequency, with
dimension = number of sequences.
20191114: not currently implemented and
filled with zeros. Still a TODO.
|
num_ranges
uint32
|
Number of ranges to calculate
correlations for.
|
num_samps
uint32
|
Number of samples in the sampling
period. Each sequence has its own
sampling period. Will also be provided
as the last data_dimension value.
|
num_sequences
int64
|
Number of sampling periods (equivalent to
number sequences transmitted) in the
integration time.
|
num_slices
int64
|
Number of slices used simultaneously in
this record by the experiment. If more
than 1, data should exist in another file
for this time period for the other slice.
|
pulse_phase_offset
[float32, ]
|
For pulse encoding phase, in degrees
offset. Contains one phase offset per
pulse in pulses, or none.
|
pulses
[uint32, ]
|
The pulse sequence in units of the
tau_spacing.
|
range_sep
float32
|
Range gate separation (conversion from
time (1/rx_sample_rate) to equivalent
distance between samples), in km.
|
rx_sample_rate
float64
|
Sampling rate of the samples in this
file’s data in Hz.
|
samples_data_type
unicode
|
C data type of the samples, provided for
user friendliness. = ‘complex float’
|
scan_start_marker
bool
|
Designates if the record is the first in
a scan (scan is defined by the
experiment).
|
scheduling_mode
unicode
|
The mode being run during this time
period (ex. ‘common’, ‘special’,
‘discretionary’).
|
slice_comment
unicode
|
Additional text comment that describes
the slice written in this file.
|
slice_id
uint32
|
The slice id of this file.
|
slice_interfacing
unicode
|
The interfacing of this slice to
other slices. String representation of
the python dictionary of
{slice : interface_type, … }
|
sqn_timestamps
[float64, ]
|
A list of GPS timestamps corresponding to
the beginning of transmission for each
sampling period in the integration time.
These timestamps come from the USRP
driver and the USRPs are GPS disciplined
and synchronized using the Octoclock.
Provided in seconds since epoch.
|
station
unicode
|
Three-letter radar identifier.
|
tau_spacing
uint32
|
The minimum spacing between pulses in
microseconds. Spacing between pulses is
always a multiple of this.
|
tx_pulse_len
uint32
|
Length of the transmit pulse in
microseconds.
|
FIELD NAME
type
[dimensions]
|
description |
|---|---|
agc_status_word
uint32
[num_records]
|
AGC status word. Bit position
corresponds to the USRP motherboard/
transmitter. A ‘1’ indicates an agc fault
occurred at least once during integration
|
antenna_arrays_order
unicode
[num_antennas]
|
States what order the data is in and
describes the data layout for the
num_antennas data dimension
Antennas are recorded main array
ascending and then interferometer array
ascending
|
beam_azms
float64
[num_records x
max_num_beams]
|
A list of the beam azimuths for each beam
in degrees off boresite. Note that this
is padded with zeroes for any record
which has num_beams less than the
max_num_beams. The num_beams field should
be used to read the correct number of
beams for each record.
|
beam_nums
uint32
[num_records x
max_num_beams]
|
A list of beam numbers used in this slice
in this record. Note that this is padded
with zeroes for any record which has
num_beams less than the max_num_beams.
The num_beams field should be used to
read the correct number of beams for each
record.
|
blanked_samples
uint32
[num_records x
max_num_blanked_samples ]
|
Samples that should be blanked because
they occurred during transmission times,
given by sample number (index into
decimated data). Can differ from the
pulses array due to multiple slices in a
single sequence and can differ from
record to record if a new slice is added.
|
borealis_git_hash
unicode
|
Identifies the version of Borealis that
made this data. Contains git commit hash
characters. Typically begins with the
latest git tag of the software.
|
data
complex64
[num_records x
num_antennas x
max_num_sequences x
num_samps]
|
A set of samples (complex float) at given
sample rate. Note that records that do not
have num_sequences = max_num_sequences
will have padded zeros. The num_sequences
array should be used to determine the
correct number of sequences to read for
the record.
|
data_descriptors
unicode
[4]
|
Denotes what each data dimension
represents. = ‘num_records’,
‘num_antennas’, ‘max_num_sequences’,
‘num_samps’
|
data_normalization_factor
float32
|
Scale of all the filters used,
multiplied, for a total scale to
normalize the data by.
|
experiment_comment
unicode
|
Comment provided in experiment about the
experiment as a whole.
|
experiment_id
int16
|
Number used to identify the experiment.
|
experiment_name
unicode
|
Name of the experiment file.
|
freq
uint32
|
The frequency used for this experiment,
in kHz. This is the frequency the data
has been filtered to.
|
gps_locked
bool
[num_records]
|
Designates if the local GPS had a lock
during the entire integration period.
False if it unlocked at least once.
|
gps_to_system_time_diff
float32
[num_records]
|
The max time difference between box_time
(GPS time) and system time (NTP) during
the integration. Negative when GPS time
is ahead of system time.
|
int_time
float32
[num_records]
|
Integration time in seconds.
|
intf_antenna_count
uint32
|
Number of interferometer array antennas
|
lp_status_word
uint32
[num_records]
|
Low power status word. Bit position
corresponds to the USRP motherboard/
transmitter. A ‘1’ indicates low power
occurred at least once during integration
|
main_antenna_count
uint32
|
Number of main array antennas
|
noise_at_freq
float64
[num_records x
max_num_sequences]
|
Noise at the receive frequency, with
dimension = number of sequences.
20191114: not currently implemented and
filled with zeros. Still a TODO. Note
that records that do not have
num_sequences = max_num_sequences will
have padded zeros. The num_sequences
array should be used to determine the
correct number of sequences to read for
the record.
|
num_beams
uint32
[num_records]
|
The number of beams to calculate for each
record.
|
num_blanked_samples
uint32
[num_records]
|
The number of blanked samples for each
record.
|
num_samps
uint32
|
Number of samples in the sampling
period. Each sequence has its own
sampling period. Will also be provided
as the last data_dimension value.
|
num_sequences
int64
[num_records]
|
Number of sampling periods (equivalent to
number sequences transmitted) in the
integration time for each record. Allows
the user to correctly read the data up to
the correct number and remove the padded
zeros in the data array.
|
num_slices
int64
[num_records]
|
Number of slices used simultaneously in
the record by the experiment. If more
than 1, data should exist in another file
for the same time period as that record
for the other slice.
|
pulse_phase_offset
float32
[] or [num pulses x 1]
|
For pulse encoding phase, in degrees
offset. Contains one phase offset per
pulse in pulses
|
pulses
uint32
[number of pulses]
|
The pulse sequence in units of the
tau_spacing.
|
rx_sample_rate
float64
|
Sampling rate of the samples in this
file’s data in Hz.
|
samples_data_type
unicode
|
C data type of the samples, provided for
user friendliness. = ‘complex float’
|
scan_start_marker
bool
[num_records]
|
Designates if the record is the first in
a scan (scan is defined by the
experiment).
|
scheduling_mode
unicode
|
The mode being run during this time
period (ex. ‘common’, ‘special’,
‘discretionary’).
|
slice_comment
unicode
|
Additional text comment that describes
the slice written in this file. The slice
number of this file is provided in the
filename.
|
slice_id
uint32
|
The slice id of this file.
|
slice_interfacing
unicode
[num_records]
|
The interfacing of this slice to
other slices for each record. String
representation of the python dictionary
of {slice : interface_type, … }. Can
differ between records if slices updated.
|
sqn_timestamps
float64
[num_records x
max_num_sequences]
|
A list of GPS timestamps corresponding to
the beginning of transmission for each
sampling period in the integration time.
These timestamps come back from the USRP
driver and the USRPs are GPS disciplined
and synchronized using the Octoclock.
Provided in seconds since epoch.
Note that records that do not have
num_sequences = max_num_sequences will
have padded zeros. The num_sequences
array should be used to determine the
correct number of sequences to read for
the record.
|
station
unicode
|
Three-letter radar identifier.
|
tau_spacing
uint32
|
The minimum spacing between pulses in
microseconds. Spacing between pulses is
always a multiple of this.
|
tx_pulse_len
uint32
|
Length of the transmit pulse in
microseconds.
|
Field name
type
|
description |
|---|---|
agc_status_word
uint32
|
AGC status word. Bit position
corresponds to the USRP motherboard/
transmitter. A ‘1’ indicates an agc fault
occurred at least once during integration
|
antenna_arrays_order
[unicode, ]
|
States what order the data is in and
describes the data layout for the
num_antennas data dimension. Antennas are
recorded main array ascending and then
interferometer array ascending.
|
beam_azms
[float64, ]
|
A list of the beam azimuths for each
beam in degrees off boresite.
|
beam_nums
[uint32, ]
|
A list of beam numbers used in this slice
in this record.
|
blanked_samples
[uint32, ]
|
Samples that should be blanked because
they occurred during transmission times,
given by sample number (index into
decimated data). Can differ from the
pulses array due to multiple slices in a
single sequence.
|
borealis_git_hash
unicode
|
Identifies the version of Borealis that
made this data. Contains git commit hash
characters. Typically begins with the
latest git tag of the software.
|
data
[complex64, ]
|
A contiguous set of samples (complex
float) at given sample rate. Needs to be
reshaped by data_dimensions to be
correctly read.
|
data_descriptors
[unicode, ]
|
Denotes what each data dimension
represents. = ‘num_antennas’,
‘num_sequences’, ‘num_samps’ for
antennas_iq
|
data_dimensions
[uint32, ]
|
The dimensions in which to reshape the
data. Dimensions correspond to
data_descriptors.
|
data_normalization_factor
float32
|
Scale of all the filters used, multiplied
for a total scale to normalize the data
by.
|
experiment_comment
unicode
|
Comment provided in experiment about the
experiment as a whole.
|
experiment_id
int16
|
Number used to identify the experiment.
|
experiment_name
unicode
|
Name of the experiment file.
|
freq
uint32
|
The frequency used for this experiment,
in kHz. This is the frequency the data
has been filtered to.
|
gps_locked
bool
|
Designates if the local GPS had a lock
during the entire integration period.
|
gps_to_system_time_diff
float32
|
The max time difference between box_time
GPS time) and system time (NTP) during the
integration. Negative when GPS time is
ahead of system time.
|
int_time
float32
|
Integration time in seconds.
|
intf_antenna_count
uint32
|
Number of interferometer array antennas
|
lp_status_word
uint32
|
Low power status word. Bit position
corresponds to the USRP motherboard/
transmitter. A ‘1’ indicates low power
occurred at least once during integration
|
main_antenna_count
uint32
|
Number of main array antennas
|
noise_at_freq
[float64, ]
|
Noise at the receive frequency, with
dimension = number of sequences.
20191114: not currently implemented and
filled with zeros. Still a TODO.
|
num_samps
uint32
|
Number of samples in the sampling
period. Each sequence has its own
sampling period. Will also be provided
as the last data_dimension value.
|
num_sequences
int64
|
Number of sampling periods (equivalent to
number sequences transmitted) in the
integration time.
|
num_slices
int64
|
Number of slices used simultaneously in
this record by the experiment. If more
than 1, data should exist in another file
for this time period for the other slice.
|
pulse_phase_offset
[float32, ]
|
For pulse encoding phase, in degrees
offset. Contains one phase offset per
pulse in pulses, or none.
|
pulses
[uint32, ]
|
The pulse sequence in units of the
tau_spacing.
|
rx_sample_rate
float64
|
Sampling rate of the samples in this
file’s data in Hz.
|
samples_data_type
unicode
|
C data type of the samples, provided for
user friendliness. = ‘complex float’
|
scan_start_marker
bool
|
Designates if the record is the first in
a scan (scan is defined by the
experiment).
|
scheduling_mode
unicode
|
The mode being run during this time
period (ex. ‘common’, ‘special’,
‘discretionary’).
|
slice_comment
unicode
|
Additional text comment that describes
the slice written in this file.
|
slice_id
uint32
|
The slice id of this file.
|
slice_interfacing
unicode
|
The interfacing of this slice to
other slices. String representation of
the python dictionary of
{slice : interface_type, … }
|
sqn_timestamps
[float64, ]
|
A list of GPS timestamps corresponding to
the beginning of transmission for each
sampling period in the integration time.
These timestamps come from the USRP
driver and the USRPs are GPS disciplined
and synchronized using the Octoclock.
Provided in seconds since epoch.
|
station
unicode
|
Three-letter radar identifier.
|
tau_spacing
uint32
|
The minimum spacing between pulses in
microseconds. Spacing between pulses is
always a multiple of this.
|
tx_pulse_len
uint32
|
Length of the transmit pulse in
microseconds.
|
The pyDARNio format class for this format is BorealisRawrf found in the borealis_formats.
The rawrf format is intended to hold high bandwidth, non-filtered raw data from every antenna.
This format is only produced in a site-style, record by record format and is only available to be produced on request. Please note that this format can cause radar operating delays and may reduce number of averages in an integration, for example.
Site files are produced by the Borealis code package and have the data in a record by record style format. In site files, the hdf5 group names (ie record names) are given as the timestamp in ms past epoch of the first sequence or sampling period recorded in the record.
The naming convention of the rawrf site-structured files are:
[YYYYmmDD].[HHMM].[SS].[station_id].rawrf.hdf5.site
For example:
20191105.1400.02.sas.rawrf.hdf5.site
This is the file that began writing at 14:00:02 UT on November 5 2019 at the Saskatoon site, and it provides data the experiment that ran at that time. Since rawrf is not filtered, this data does not need a slice identifier because it contains all the samples being taken at that time. Some familiarity with the experiment may be necessary to understand the data, or some access to the other file types produced concurrently. This is primarily a debug format for engineering purposes and should only be produced for special cases.
These files are often bzipped after they are produced.
The file fields under the record name in rawrf site files are:
Field name
type
|
description |
|---|---|
agc_status_word
uint32
|
AGC status word. Bit position
corresponds to the USRP motherboard/
transmitter. A ‘1’ indicates an agc fault
occurred at least once during integration
|
blanked_samples
uint32
[number of blanked
samples]
|
Samples that should be blanked because
they occurred during transmission times,
given by sample number (index into
decimated data). Can differ from the
pulses array due to multiple slices in a
single sequence.
|
borealis_git_hash
unicode
|
Identifies the version of Borealis that
made this data. Contains git commit hash
characters. Typically begins with the
latest git tag of the software.
|
data
[complex64, ]
|
A contiguous set of samples (complex
float) at given sample rate. Needs to be
reshaped by data_dimensions to be
correctly read.
|
data_descriptors
[unicode, ]
|
Denotes what each data dimension
represents. = ‘num_sequences’,
‘num_antennas’, ‘num_samps’ for
rawrf
|
data_dimensions
[uint32, ]
|
The dimensions in which to reshape the
data. Dimensions correspond to
data_descriptors.
|
experiment_comment
unicode
|
Comment provided in experiment about the
experiment as a whole.
|
experiment_id
int16
|
Number used to identify the experiment.
|
experiment_name
unicode
|
Name of the experiment file.
|
gps_locked
bool
|
Designates if the local GPS had a lock
during the entire integration period.
|
gps_to_system_time_diff
float32
|
The max time difference between box_time
GPS time) and system time (NTP) during the
integration. Negative when GPS time is
ahead of system time.
|
int_time
float32
|
Integration time in seconds.
|
intf_antenna_count
uint32
|
Number of interferometer array antennas
|
lp_status_word
uint32
|
Low power status word. Bit position
corresponds to the USRP motherboard/
transmitter. A ‘1’ indicates low power
occurred at least once during integration
|
main_antenna_count
uint32
|
Number of main array antennas
|
num_samps
uint32
|
Number of samples in the sampling
period. Each sequence has its own
sampling period. Will also be provided
as the last data_dimension value.
|
num_sequences
int64
|
Number of sampling periods (equivalent to
number sequences transmitted) in the
integration time.
|
num_slices
int64
|
Number of slices used simultaneously in
this record by the experiment. If more
than 1, data should exist in another file
for this time period for the other slice.
|
rx_center_freq
float64
|
Center frequency of the sampled data
in kHz.
|
rx_sample_rate
float64
|
Sampling rate of the samples in this
file’s data in Hz.
|
samples_data_type
unicode
|
C data type of the samples, provided for
user friendliness. = ‘complex float’
|
scan_start_marker
bool
|
Designates if the record is the first in
a scan (scan is defined by the
experiment).
|
scheduling_mode
unicode
|
The mode being run during this time
period (ex. ‘common’, ‘special’,
‘discretionary’).
|
sqn_timestamps
[float64, ]
|
A list of GPS timestamps corresponding to
the beginning of transmission for each
sampling period in the integration time.
These timestamps come from the USRP
driver and the USRPs are GPS disciplined
and synchronized using the Octoclock.
Provided in seconds since epoch.
|
station
unicode
|
Three-letter radar identifier.
|
FIELD NAME
type
[dimensions]
|
description |
|---|---|
agc_status_word
uint32
[num_records]
|
AGC status word. Bit position
corresponds to the USRP motherboard/
transmitter. A ‘1’ indicates an agc fault
occurred at least once during integration
|
averaging_method
unicode
|
A string describing the averaging method.
Default is ‘mean’ but an experiment can
set this to ‘median’ to get the median of
all sequences in an integration period,
and other methods to combine all
sequences in an integration period could
be added in the future.
|
beam_azms
float64
[num_records x
max_num_beams]
|
A list of the beam azimuths for each beam
in degrees off boresite. Note that this
is padded with zeroes for any record
which has num_beams less than the
max_num_beams. The num_beams field should
be used to read the correct number of
beams for each record.
|
beam_nums
uint32
[num_records x
max_num_beams]
|
A list of beam numbers used in this slice
in this record. Note that this is padded
with zeroes for any record which has
num_beams less than the max_num_beams.
The num_beams field should be used to
read the correct number of beams for each
record.
|
blanked_samples
uint32
[num_records x
max_num_blanked_samples ]
|
Samples that should be blanked because
they occurred during transmission times,
given by sample number (index into
decimated data). Can differ from the
pulses array due to multiple slices in a
single sequence and can differ from
record to record if a new slice is added.
|
borealis_git_hash
unicode
|
Identifies the version of Borealis that
made this data. Contains git commit hash
characters. Typically begins with the
latest git tag of the software.
|
cfs_freq
float32
[] or [num_freqs]
|
Frequencies sampled during the clear
frequency search centered around
baseband.
|
cfs_masks
bool
[] or [num_freqs]
|
Masks for each cfs slice indicating which
frequencies cannot be selected for the
slice tx freq.
|
cfs_noise
float32
[] or [num_cfs_slices,
num_freqs]
|
Power measurements for each frequency in
the clear frequency search analysis.
A separate power array is generated for
each cfs slice.
|
cfs_range
float32
[] or [num_cfs_slices, 2]
|
The range of frequencies scanned by the
clear frequency search for each cfs slice
in the experiment. If not set in the
experiment, all cfs parameters will be []
|
data_descriptors
bytes
[4]
|
Denotes what each data dimension
(in main_acfs, intf_acfs, xcfs)
represents. = ‘num_records’,
‘max_num_beams’, ‘num_ranges’, ‘num_lags’
|
data_normalization_factor
float32
|
Scale of all the filters used,
multiplied, for a total scale to
normalize the data by.
|
experiment_comment
unicode
|
Comment provided in experiment about the
experiment as a whole.
|
experiment_id
int16
|
Number used to identify the experiment.
|
experiment_name
unicode
|
Name of the experiment file.
|
first_range
float32
|
Distance to use for first range in km.
|
first_range_rtt
float32
|
Round trip time of flight to first range
in microseconds.
|
freq
uint32
|
The frequency used for this experiment,
in kHz. This is the frequency the data
has been filtered to.
|
gps_locked
bool
[num_records]
|
Designates if the local GPS had a lock
during the entire integration period.
False if it unlocked at least once.
|
gps_to_system_time_diff
float32
[num_records]
|
The max time difference between box_time
GPS time) and system time (NTP) during the
integration. Negative when GPS time is
ahead of system time.
|
int_time
float32
[num_records]
|
Integration time in seconds.
|
intf_acfs
complex64
[num_records x
max_num_beams x
num_ranges x
num_lags]
|
Interferometer array correlations. Note
that records that do not have num_beams =
max_num_beams will have padded zeros. The
num_beams array should be used to
determine the correct number of beams to
read for the record.
|
intf_antenna_count
uint32
|
Number of interferometer array antennas
|
lags
uint32
[number of lags, 2]
|
The lags created from two pulses in the
pulses array. Values have to be from
pulses array. The lag number is lag[1] -
lag[0] for each lag pair.
|
lp_status_word
uint32
[num_records]
|
Low power status word. Bit position
corresponds to the USRP motherboard/
transmitter. A ‘1’ indicates low power
occurred at least once during integration
|
main_acfs
complex64
[num_records x
max_num_beams x
num_ranges x
num_lags]
|
Main array correlations. Note
that records that do not have num_beams =
max_num_beams will have padded zeros. The
num_beams array should be used to
determine the correct number of beams to
read for the record.
|
main_antenna_count
uint32
|
Number of main array antennas
|
noise_at_freq
float64
[num_records x
max_num_sequences]
|
Noise at the receive frequency, with
dimension = number of sequences.
20191114: not currently implemented and
filled with zeros. Still a TODO. Note
that records that do not have
num_sequences = max_num_sequences will
have padded zeros. The num_sequences
array should be used to determine the
correct number of sequences to read for
the record.
|
num_beams
uint32
[num_records]
|
The number of beams calculated for each
record. Allows the user to correctly read
the data up to the correct number and
remove the padded zeros in the data
array.
|
num_blanked_samples
uint32
[num_records]
|
The number of blanked samples for each
record.
|
num_sequences
int64
[num_records]
|
Number of sampling periods (equivalent to
number sequences transmitted) in the
integration time for each record. Allows
the user to correctly read the data up to
the correct number and remove the padded
zeros in the data array.
|
num_slices
int64
[num_records]
|
Number of slices used simultaneously in
the record by the experiment. If more
than 1, data should exist in another file
for the same time period as that record
for the other slice.
|
pulses
uint32
[number of pulses]
|
The pulse sequence in units of the
tau_spacing.
|
range_sep
float32
|
Range gate separation (conversion from
time (1/rx_sample_rate) to equivalent
distance between samples), in km.
|
rx_sample_rate
float64
|
Sampling rate of the samples in this
file’s data in Hz.
|
samples_data_type
unicode
|
C data type of the samples, provided for
user friendliness. = ‘complex float’
|
scan_start_marker
bool
[num_records]
|
Designates if the record is the first in
a scan (scan is defined by the
experiment).
|
scheduling_mode
unicode
|
The mode being run during this time
period (ex. ‘common’, ‘special’,
‘discretionary’).
|
slice_comment
unicode
|
Additional text comment that describes
the slice written in this file.
|
slice_id
uint32
|
The slice id of this file.
|
slice_interfacing
unicode
[num_records]
|
The interfacing of this slice to
other slices for each record. String
representation of the python dictionary
of {slice : interface_type, … }. Can
differ between records if slices updated.
|
sqn_timestamps
float64
[num_records x
max_num_sequences]
|
A list of GPS timestamps corresponding to
the beginning of transmission for each
sampling period in the integration time.
These timestamps come back from the USRP
driver and the USRPs are GPS disciplined
and synchronized using the Octoclock.
Provided in seconds since epoch.
Note that records that do not have
num_sequences = max_num_sequences will
have padded zeros. The num_sequences
array should be used to determine the
correct number of sequences to read for
the record.
|
station
unicode
|
Three-letter radar identifier.
|
tau_spacing
uint32
|
The minimum spacing between pulses in
microseconds. Spacing between pulses is
always a multiple of this.
|
tx_antenna_phases
complex64
[num_records x
num_main_antennas]
|
The complex phase for each antenna for
transmission, normalized such that full-
power has magnitude 1.
|
tx_pulse_len
uint32
|
Length of the transmit pulse in
microseconds.
|
xcfs
complex64
[num_records x
max_num_beams x
num_ranges x
num_lags]
|
Cross correlations of interferometer to
main array. Note
that records that do not have num_beams =
max_num_beams will have padded zeros. The
num_beams array should be used to
determine the correct number of beams to
read for the record.
|
Field name
type
|
description |
|---|---|
agc_status_word
uint32
|
AGC status word. Bit position
corresponds to the USRP motherboard/
transmitter. A ‘1’ indicates an agc fault
occurred at least once during integration
|
averaging_method
unicode
|
A string describing the averaging method.
Default is ‘mean’ but an experiment can
set this to ‘median’ to get the median of
all sequences in an integration period,
and other methods to combine all
sequences in an integration period could
be added in the future.
|
beam_azms
[float64, ]
|
A list of the beam azimuths for each
beam in degrees off boresite.
|
beam_nums
[uint32, ]
|
A list of beam numbers used in this slice
in this record.
|
blanked_samples
[uint32, ]
|
Samples that should be blanked because
they occurred during transmission times,
given by sample number (index into
decimated data). Can differ from the
pulses array due to multiple slices in a
single sequence.
|
borealis_git_hash
unicode
|
Identifies the version of Borealis that
made this data. Contains git commit hash
characters. Typically begins with the
latest git tag of the software.
|
data_descriptors
[bytes, ]
|
Denotes what each data dimension
(in main_acfs, intf_acfs, xcfs)
represents. (‘num_beams, ‘num_ranges’,
‘num_lags’)
|
data_dimensions
[uint32, ]
|
The dimensions of the acf of xcf
datasets. Dimensions correspond to
data_descriptors.
|
data_normalization_factor
float32
|
Scale of all the filters used, multiplied
for a total scale to normalize the data
by.
|
experiment_comment
unicode
|
Comment provided in experiment about the
experiment as a whole.
|
experiment_id
int16
|
Number used to identify the experiment.
|
experiment_name
unicode
|
Name of the experiment file.
|
first_range
float32
|
Distance to use for first range in km.
|
first_range_rtt
float32
|
Round trip time of flight to first range
in microseconds.
|
freq
uint32
|
The frequency used for this experiment,
in kHz. This is the frequency the data
has been filtered to.
|
gps_locked
bool
|
Designates if the local GPS had a lock
during the entire integration period.
|
gps_to_system_time_diff
float32
|
The max time difference between box_time
GPS time) and system time (NTP) during the
integration. Negative when GPS time is
ahead of system time.
|
int_time
float32
|
Integration time in seconds.
|
intf_acfs
[complex64, ]
|
Interferometer array correlations.
|
intf_antenna_count
uint32
|
Number of interferometer array antennas
|
lags
[[uint32, ], ]
|
The lags created from two pulses in the
pulses array. Dimensions are number of
lags x 2. Values have to be from pulses
array. The lag number is lag[1] - lag[0]
for each lag pair.
|
lp_status_word
uint32
|
Low power status word. Bit position
corresponds to the USRP motherboard/
transmitter. A ‘1’ indicates low power
occurred at least once during integration
|
main_acfs
[complex64, ]
|
Main array correlations.
|
main_antenna_count
uint32
|
Number of main array antennas
|
noise_at_freq
[float64, ]
|
Noise at the receive frequency, with
dimension = number of sequences.
20191114: not currently implemented and
filled with zeros. Still a TODO.
|
num_sequences
int64
|
Number of sampling periods (equivalent to
number sequences transmitted) in the
integration time.
|
num_slices
int64
|
Number of slices used simultaneously in
this record by the experiment. If more
than 1, data should exist in another file
for this time period for the other slice.
|
pulses
[uint32, ]
|
The pulse sequence in units of the
tau_spacing.
|
range_sep
float32
|
Range gate separation (conversion from
time (1/rx_sample_rate) to equivalent
distance between samples), in km.
|
rx_sample_rate
float64
|
Sampling rate of the samples in this
file’s data in Hz.
|
samples_data_type
unicode
|
C data type of the samples, provided for
user friendliness. = ‘complex float’
|
scan_start_marker
bool
|
Designates if the record is the first in
a scan (scan is defined by the
experiment).
|
scheduling_mode
unicode
|
The mode being run during this time
period (ex. ‘common’, ‘special’,
‘discretionary’).
|
slice_comment
unicode
|
Additional text comment that describes
the slice written in this file.
|
slice_id
uint32
|
The slice id of this file.
|
slice_interfacing
unicode
|
The interfacing of this slice to
other slices. String representation of
the python dictionary of
{slice : interface_type, … }
|
sqn_timestamps
[float64, ]
|
A list of GPS timestamps corresponding to
the beginning of transmission for each
sampling period in the integration time.
These timestamps come from the USRP
driver and the USRPs are GPS disciplined
and synchronized using the Octoclock.
Provided in seconds since epoch.
|
station
unicode
|
Three-letter radar identifier.
|
tau_spacing
uint32
|
The minimum spacing between pulses in
microseconds. Spacing between pulses is
always a multiple of this.
|
tx_antenna_phases
[complex64, ]
|
The complex phase for each antenna for
transmission, normalized such that full-
power has magnitude 1.
|
tx_pulse_len
uint32
|
Length of the transmit pulse in
microseconds.
|
xcfs
[complex64, ]
|
Cross correlations of interferometer to
main array.
|
FIELD NAME
type
[dimensions]
|
description |
|---|---|
agc_status_word
uint32
[num_records]
|
AGC status word. Bit position
corresponds to the USRP motherboard/
transmitter. A ‘1’ indicates an agc fault
occurred at least once during integration
|
antenna_arrays_order
bytes
[num_antenna_arrays]
|
States what order the data is in and
describes the data layout for the
num_antenna_arrays data dimension
|
beam_azms
float64
[num_records x
max_num_beams]
|
A list of the beam azimuths for each beam
in degrees off boresite. Note that this
is padded with zeroes for any record
which has num_beams less than the
max_num_beams. The num_beams field should
be used to read the correct number of
beams for each record.
|
beam_nums
uint32
[num_records x
max_num_beams]
|
A list of beam numbers used in this slice
in this record. Note that this is padded
with zeroes for any record which has
num_beams less than the max_num_beams.
The num_beams field should be used to
read the correct number of beams for each
record.
|
blanked_samples
uint32
[num_records x
max_num_blanked_samples ]
|
Samples that should be blanked because
they occurred during transmission times,
given by sample number (index into
decimated data). Can differ from the
pulses array due to multiple slices in a
single sequence and can differ from
record to record if a new slice is added.
|
borealis_git_hash
unicode
|
Identifies the version of Borealis that
made this data. Contains git commit hash
characters. Typically begins with the
latest git tag of the software.
|
cfs_freq
float32
[] or [num_freqs]
|
Frequencies sampled during the clear
frequency search centered around
baseband.
|
cfs_masks
bool
[] or [num_freqs]
|
Masks for each cfs slice indicating which
frequencies cannot be selected for the
slice tx freq.
|
cfs_noise
float32
[] or [num_cfs_slices,
num_freqs]
|
Power measurements for each frequency in
the clear frequency search analysis.
A separate power array is generated for
each cfs slice.
|
cfs_range
float32
[] or [num_cfs_slices, 2]
|
The range of frequencies scanned by the
clear frequency search for each cfs slice
in the experiment. If not set in the
experiment, all cfs parameters will be []
|
data
complex64
[num_records x
num_antenna_arrays x
max_num_sequences x
max_num_beams x
num_samps]
|
A set of samples (complex float) at given
sample rate. Note that records that do not
have num_sequences = max_num_sequences or
num_beams = max_num_beams will have
padded zeros. The num_sequences and
num_beams arrays should be used to
determine the correct number of sequences
and beams to read for the record.
|
data_descriptors
bytes
[5]
|
Denotes what each data dimension
represents. = ‘num_records’,
‘num_antenna_arrays’,
‘max_num_sequences’, ‘max_num_beams’,
‘num_samps’
|
data_normalization_factor
float32
|
Scale of all the filters used,
multiplied, for a total scale to
normalize the data by.
|
experiment_comment
unicode
|
Comment provided in experiment about the
experiment as a whole.
|
experiment_id
int16
|
Number used to identify the experiment.
|
experiment_name
unicode
|
Name of the experiment file.
|
first_range
float32
|
Distance to use for first range in km.
|
first_range_rtt
float32
|
Round trip time of flight to first range
in microseconds.
|
freq
uint32
|
The frequency used for this experiment,
in kHz. This is the frequency the data
has been filtered to.
|
gps_locked
bool
[num_records]
|
Designates if the local GPS had a lock
during the entire integration period.
False if it unlocked at least once.
|
gps_to_system_time_diff
float32
[num_records]
|
The max time difference between box_time
GPS time) and system time (NTP) during the
integration. Negative when GPS time is
ahead of system time.
|
int_time
float32
[num_records]
|
Integration time in seconds.
|
intf_antenna_count
uint32
|
Number of interferometer array antennas
|
lags
uint32
[number of lags, 2]
|
The lags created from two pulses in the
pulses array. Values have to be from
pulses array. The lag number is lag[1] -
lag[0] for each lag pair.
|
lp_status_word
uint32
[num_records]
|
Low power status word. Bit position
corresponds to the USRP motherboard/
transmitter. A ‘1’ indicates low power
occurred at least once during integration
|
main_antenna_count
uint32
|
Number of main array antennas
|
noise_at_freq
float64
[num_records x
max_num_sequences]
|
Noise at the receive frequency, with
dimension = number of sequences.
20191114: not currently implemented and
filled with zeros. Still a TODO. Note
that records that do not have
num_sequences = max_num_sequences will
have padded zeros. The num_sequences
array should be used to determine the
correct number of sequences to read for
the record.
|
num_beams
uint32
[num_records]
|
The number of beams calculated for each
record. Allows the user to correctly read
the data up to the correct number and
remove the padded zeros in the data
array.
|
num_blanked_samples
uint32
[num_records]
|
The number of blanked samples for each
record.
|
num_ranges
uint32
|
Number of ranges to calculate
correlations for.
|
num_samps
uint32
|
Number of samples in the sampling
period. Each sequence has its own
sampling period. Will also be provided
as the last data_dimension value.
|
num_sequences
int64
[num_records]
|
Number of sampling periods (equivalent to
number sequences transmitted) in the
integration time for each record. Allows
the user to correctly read the data up to
the correct number and remove the padded
zeros in the data array.
|
num_slices
int64
[num_records]
|
Number of slices used simultaneously in
the record by the experiment. If more
than 1, data should exist in another file
for the same time period as that record
for the other slice.
|
pulse_phase_offset
float32
[] or [num pulses x 1]
|
For pulse encoding phase, in degrees
offset. Contains one phase offset per
pulse in pulses
|
pulses
uint32
[number of pulses]
|
The pulse sequence in units of the
tau_spacing.
|
range_sep
float32
|
Range gate separation (conversion from
time (1/rx_sample_rate) to equivalent
distance between samples), in km.
|
rx_sample_rate
float64
|
Sampling rate of the samples in this
file’s data in Hz.
|
samples_data_type
unicode
|
C data type of the samples, provided for
user friendliness. = ‘complex float’
|
scan_start_marker
bool
[num_records]
|
Designates if the record is the first in
a scan (scan is defined by the
experiment).
|
scheduling_mode
unicode
|
The mode being run during this time
period (ex. ‘common’, ‘special’,
‘discretionary’).
|
slice_comment
unicode
|
Additional text comment that describes
the slice written in this file. The slice
number of this file is provided in the
filename.
|
slice_id
uint32
|
The slice id of this file.
|
slice_interfacing
unicode
[num_records]
|
The interfacing of this slice to
other slices for each record. String
representation of the python dictionary
of {slice : interface_type, … }. Can
differ between records if slices updated.
|
sqn_timestamps
float64
[num_records x
max_num_sequences]
|
A list of GPS timestamps corresponding to
the beginning of transmission for each
sampling period in the integration time.
These timestamps come back from the USRP
driver and the USRPs are GPS disciplined
and synchronized using the Octoclock.
Provided in seconds since epoch.
Note that records do not have
num_sequences = max_num_sequences will
have padded zeros. The num_sequences
array should be used to determine the
correct number of sequences to read for
the record.
|
station
unicode
|
Three-letter radar identifier.
|
tau_spacing
uint32
|
The minimum spacing between pulses in
microseconds. Spacing between pulses is
always a multiple of this.
|
tx_antenna_phases
complex64
[num_records x
num_main_antennas]
|
The complex phase for each antenna for
transmission, normalized such that full-
power has magnitude 1.
|
tx_pulse_len
uint32
|
Length of the transmit pulse in
microseconds.
|
Field name
type
|
description |
|---|---|
agc_status_word
uint32
|
AGC status word. Bit position
corresponds to the USRP motherboard/
transmitter. A ‘1’ indicates an agc fault
occurred at least once during integration
|
antenna_arrays_order
[bytes, ]
|
States what order the data is in and
describes the data layout for the
num_antenna_arrays data dimension
|
beam_azms
[float64, ]
|
A list of the beam azimuths for each
beam in degrees off boresite.
|
beam_nums
[uint32, ]
|
A list of beam numbers used in this slice
in this record.
|
blanked_samples
[uint32, ]
|
Samples that should be blanked because
they occurred during transmission times,
given by sample number (index into
decimated data). Can differ from the
pulses array due to multiple slices in a
single sequence.
|
borealis_git_hash
unicode
|
Identifies the version of Borealis that
made this data. Contains git commit hash
characters. Typically begins with the
latest git tag of the software.
|
data
[complex64, ]
|
A contiguous set of samples (complex
float) at given sample rate. Dimensions
match that of data_dimensions field.
|
data_descriptors
[bytes, ]
|
Denotes what each data dimension
represents. = ‘num_antenna_arrays’,
‘num_sequences’, ‘num_beams’, ‘num_samps’
for bfiq
|
data_dimensions
[uint32, ]
|
The dimensions of the data.
Dimensions correspond to
data_descriptors.
|
data_normalization_factor
float32
|
Scale of all the filters used, multiplied
for a total scale to normalize the data
by.
|
experiment_comment
unicode
|
Comment provided in experiment about the
experiment as a whole.
|
experiment_id
int16
|
Number used to identify the experiment.
|
experiment_name
unicode
|
Name of the experiment file.
|
first_range
float32
|
Distance to use for first range in km.
|
first_range_rtt
float32
|
Round trip time of flight to first range
in microseconds.
|
freq
uint32
|
The frequency used for this experiment,
in kHz. This is the frequency the data
has been filtered to.
|
gps_locked
bool
|
Designates if the local GPS had a lock
during the entire integration period.
|
gps_to_system_time_diff
float32
|
The max time difference between box_time
GPS time) and system time (NTP) during the
integration. Negative when GPS time is
ahead of system time.
|
int_time
float32
|
Integration time in seconds.
|
intf_antenna_count
uint32
|
Number of interferometer array antennas
|
lags
[[uint32, ], ]
|
The lags created from two pulses in the
pulses array. Dimensions are number of
lags x 2. Values have to be from pulses
array. The lag number is lag[1] - lag[0]
for each lag pair.
|
lp_status_word
uint32
|
Low power status word. Bit position
corresponds to the USRP motherboard/
transmitter. A ‘1’ indicates low power
occurred at least once during integration
|
main_antenna_count
uint32
|
Number of main array antennas
|
noise_at_freq
[float64, ]
|
Noise at the receive frequency, with
dimension = number of sequences.
20191114: not currently implemented and
filled with zeros. Still a TODO.
|
num_ranges
uint32
|
Number of ranges to calculate
correlations for.
|
num_samps
uint32
|
Number of samples in the sampling
period. Each sequence has its own
sampling period. Will also be provided
as the last data_dimension value.
|
num_sequences
int64
|
Number of sampling periods (equivalent to
number sequences transmitted) in the
integration time.
|
num_slices
int64
|
Number of slices used simultaneously in
this record by the experiment. If more
than 1, data should exist in another file
for this time period for the other slice.
|
pulse_phase_offset
[float32, ]
|
For pulse encoding phase, in degrees
offset. Contains one phase offset per
pulse in pulses, or none.
|
pulses
[uint32, ]
|
The pulse sequence in units of the
tau_spacing.
|
range_sep
float32
|
Range gate separation (conversion from
time (1/rx_sample_rate) to equivalent
distance between samples), in km.
|
rx_sample_rate
float64
|
Sampling rate of the samples in this
file’s data in Hz.
|
samples_data_type
unicode
|
C data type of the samples, provided for
user friendliness. = ‘complex float’
|
scan_start_marker
bool
|
Designates if the record is the first in
a scan (scan is defined by the
experiment).
|
scheduling_mode
unicode
|
The mode being run during this time
period (ex. ‘common’, ‘special’,
‘discretionary’).
|
slice_comment
unicode
|
Additional text comment that describes
the slice written in this file.
|
slice_id
uint32
|
The slice id of this file.
|
slice_interfacing
unicode
|
The interfacing of this slice to
other slices. String representation of
the python dictionary of
{slice : interface_type, … }
|
sqn_timestamps
[float64, ]
|
A list of GPS timestamps corresponding to
the beginning of transmission for each
sampling period in the integration time.
These timestamps come from the USRP
driver and the USRPs are GPS disciplined
and synchronized using the Octoclock.
Provided in seconds since epoch.
|
station
unicode
|
Three-letter radar identifier.
|
tau_spacing
uint32
|
The minimum spacing between pulses in
microseconds. Spacing between pulses is
always a multiple of this.
|
tx_antenna_phases
[complex64, ]
|
The complex phase for each antenna for
transmission, normalized such that full-
power has magnitude 1.
|
tx_pulse_len
uint32
|
Length of the transmit pulse in
microseconds.
|
FIELD NAME
type
[dimensions]
|
description |
|---|---|
agc_status_word
uint32
[num_records]
|
AGC status word. Bit position
corresponds to the USRP motherboard/
transmitter. A ‘1’ indicates an agc fault
occurred at least once during integration
|
antenna_arrays_order
bytes
[num_antennas]
|
States what order the data is in and
describes the data layout for the
num_antennas data dimension
Antennas are recorded main array
ascending and then interferometer array
ascending
|
beam_azms
float64
[num_records x
max_num_beams]
|
A list of the beam azimuths for each beam
in degrees off boresite. Note that this
is padded with zeroes for any record
which has num_beams less than the
max_num_beams. The num_beams field should
be used to read the correct number of
beams for each record.
|
beam_nums
uint32
[num_records x
max_num_beams]
|
A list of beam numbers used in this slice
in this record. Note that this is padded
with zeroes for any record which has
num_beams less than the max_num_beams.
The num_beams field should be used to
read the correct number of beams for each
record.
|
blanked_samples
uint32
[num_records x
max_num_blanked_samples ]
|
Samples that should be blanked because
they occurred during transmission times,
given by sample number (index into
decimated data). Can differ from the
pulses array due to multiple slices in a
single sequence and can differ from
record to record if a new slice is added.
|
borealis_git_hash
unicode
|
Identifies the version of Borealis that
made this data. Contains git commit hash
characters. Typically begins with the
latest git tag of the software.
|
cfs_freq
float32
[] or [num_freqs]
|
Frequencies sampled during the clear
frequency search centered around
baseband.
|
cfs_masks
bool
[] or [num_freqs]
|
Masks for each cfs slice indicating which
frequencies cannot be selected for the
slice tx freq.
|
cfs_noise
float32
[] or [num_cfs_slices,
num_freqs]
|
Power measurements for each frequency in
the clear frequency search analysis.
A separate power array is generated for
each cfs slice.
|
cfs_range
float32
[] or [num_cfs_slices, 2]
|
The range of frequencies scanned by the
clear frequency search for each cfs slice
in the experiment. If not set in the
experiment, all cfs parameters will be []
|
data
complex64
[num_records x
num_antennas x
max_num_sequences x
num_samps]
|
A set of samples (complex float) at given
sample rate. Note that records that do not
have num_sequences = max_num_sequences
will have padded zeros. The num_sequences
array should be used to determine the
correct number of sequences to read for
the record.
|
data_descriptors
bytes
[4]
|
Denotes what each data dimension
represents. = ‘num_records’,
‘num_antennas’, ‘max_num_sequences’,
‘num_samps’
|
data_normalization_factor
float32
|
Scale of all the filters used,
multiplied, for a total scale to
normalize the data by.
|
experiment_comment
unicode
|
Comment provided in experiment about the
experiment as a whole.
|
experiment_id
int16
|
Number used to identify the experiment.
|
experiment_name
unicode
|
Name of the experiment file.
|
first_range
float32
|
Distance to use for first range in km.
|
first_range_rtt
float32
|
Round trip time of flight to first range
in microseconds.
|
freq
uint32
|
The frequency used for this experiment,
in kHz. This is the frequency the data
has been filtered to.
|
gps_locked
bool
[num_records]
|
Designates if the local GPS had a lock
during the entire integration period.
False if it unlocked at least once.
|
gps_to_system_time_diff
float32
[num_records]
|
The max time difference between box_time
(GPS time) and system time (NTP) during
the integration. Negative when GPS time
is ahead of system time.
|
int_time
float32
[num_records]
|
Integration time in seconds.
|
intf_antenna_count
uint32
|
Number of interferometer array antennas
|
lags
uint32
[number of lags, 2]
|
The lags created from two pulses in the
pulses array. Values have to be from
pulses array. The lag number is lag[1] -
lag[0] for each lag pair.
|
lp_status_word
uint32
[num_records]
|
Low power status word. Bit position
corresponds to the USRP motherboard/
transmitter. A ‘1’ indicates low power
occurred at least once during integration
|
main_antenna_count
uint32
|
Number of main array antennas
|
noise_at_freq
float64
[num_records x
max_num_sequences]
|
Noise at the receive frequency, with
dimension = number of sequences.
20191114: not currently implemented and
filled with zeros. Still a TODO. Note
that records that do not have
num_sequences = max_num_sequences will
have padded zeros. The num_sequences
array should be used to determine the
correct number of sequences to read for
the record.
|
num_beams
uint32
[num_records]
|
The number of beams to calculate for each
record.
|
num_blanked_samples
uint32
[num_records]
|
The number of blanked samples for each
record.
|
num_ranges
uint32
|
Number of ranges to calculate
correlations for.
|
num_samps
uint32
|
Number of samples in the sampling
period. Each sequence has its own
sampling period. Will also be provided
as the last data_dimension value.
|
num_sequences
int64
[num_records]
|
Number of sampling periods (equivalent to
number sequences transmitted) in the
integration time for each record. Allows
the user to correctly read the data up to
the correct number and remove the padded
zeros in the data array.
|
num_slices
int64
[num_records]
|
Number of slices used simultaneously in
the record by the experiment. If more
than 1, data should exist in another file
for the same time period as that record
for the other slice.
|
pulse_phase_offset
float32
[] or [num pulses x 1]
|
For pulse encoding phase, in degrees
offset. Contains one phase offset per
pulse in pulses
|
pulses
uint32
[number of pulses]
|
The pulse sequence in units of the
tau_spacing.
|
range_sep
float32
|
Range gate separation (conversion from
time (1/rx_sample_rate) to equivalent
distance between samples), in km.
|
rx_sample_rate
float64
|
Sampling rate of the samples in this
file’s data in Hz.
|
samples_data_type
unicode
|
C data type of the samples, provided for
user friendliness. = ‘complex float’
|
scan_start_marker
bool
[num_records]
|
Designates if the record is the first in
a scan (scan is defined by the
experiment).
|
scheduling_mode
unicode
|
The mode being run during this time
period (ex. ‘common’, ‘special’,
‘discretionary’).
|
slice_comment
unicode
|
Additional text comment that describes
the slice written in this file. The slice
number of this file is provided in the
filename.
|
slice_id
uint32
|
The slice id of this file.
|
slice_interfacing
unicode
[num_records]
|
The interfacing of this slice to
other slices for each record. String
representation of the python dictionary
of {slice : interface_type, … }. Can
differ between records if slices updated.
|
sqn_timestamps
float64
[num_records x
max_num_sequences]
|
A list of GPS timestamps corresponding to
the beginning of transmission for each
sampling period in the integration time.
These timestamps come back from the USRP
driver and the USRPs are GPS disciplined
and synchronized using the Octoclock.
Provided in seconds since epoch.
Note that records that do not have
num_sequences = max_num_sequences will
have padded zeros. The num_sequences
array should be used to determine the
correct number of sequences to read for
the record.
|
station
unicode
|
Three-letter radar identifier.
|
tau_spacing
uint32
|
The minimum spacing between pulses in
microseconds. Spacing between pulses is
always a multiple of this.
|
tx_antenna_phases
complex64
[num_records x
num_main_antennas]
|
The complex phase for each antenna for
transmission, normalized such that full-
power has magnitude 1.
|
tx_pulse_len
uint32
|
Length of the transmit pulse in
microseconds.
|
Field name
type
|
description |
|---|---|
agc_status_word
uint32
|
AGC status word. Bit position
corresponds to the USRP motherboard/
transmitter. A ‘1’ indicates an agc fault
occurred at least once during integration
|
antenna_arrays_order
[bytes, ]
|
States what order the data is in and
describes the data layout for the
num_antennas data dimension. Antennas are
recorded main array ascending and then
interferometer array ascending.
|
beam_azms
[float64, ]
|
A list of the beam azimuths for each
beam in degrees off boresite.
|
beam_nums
[uint32, ]
|
A list of beam numbers used in this slice
in this record.
|
blanked_samples
[uint32, ]
|
Samples that should be blanked because
they occurred during transmission times,
given by sample number (index into
decimated data). Can differ from the
pulses array due to multiple slices in a
single sequence.
|
borealis_git_hash
unicode
|
Identifies the version of Borealis that
made this data. Contains git commit hash
characters. Typically begins with the
latest git tag of the software.
|
data
[complex64, ]
|
A contiguous set of samples (complex
float) at given sample rate. Dimensions
match data_dimensions field.
|
data_descriptors
[bytes, ]
|
Denotes what each data dimension
represents. = ‘num_antennas’,
‘num_sequences’, ‘num_samps’ for
antennas_iq
|
data_dimensions
[uint32, ]
|
The dimensions of the data.
Dimensions correspond to
data_descriptors.
|
data_normalization_factor
float32
|
Scale of all the filters used, multiplied
for a total scale to normalize the data
by.
|
experiment_comment
unicode
|
Comment provided in experiment about the
experiment as a whole.
|
experiment_id
int16
|
Number used to identify the experiment.
|
experiment_name
unicode
|
Name of the experiment file.
|
first_range
float32
|
Distance to use for first range in km.
|
first_range_rtt
float32
|
Round trip time of flight to first range
in microseconds.
|
freq
uint32
|
The frequency used for this experiment,
in kHz. This is the frequency the data
has been filtered to.
|
gps_locked
bool
|
Designates if the local GPS had a lock
during the entire integration period.
|
gps_to_system_time_diff
float32
|
The max time difference between box_time
GPS time) and system time (NTP) during the
integration. Negative when GPS time is
ahead of system time.
|
int_time
float32
|
Integration time in seconds.
|
intf_antenna_count
uint32
|
Number of interferometer array antennas
|
lags
[[uint32, ], ]
|
The lags created from two pulses in the
pulses array. Dimensions are number of
lags x 2. Values have to be from pulses
array. The lag number is lag[1] - lag[0]
for each lag pair.
|
lp_status_word
uint32
|
Low power status word. Bit position
corresponds to the USRP motherboard/
transmitter. A ‘1’ indicates low power
occurred at least once during integration
|
main_antenna_count
uint32
|
Number of main array antennas
|
noise_at_freq
[float64, ]
|
Noise at the receive frequency, with
dimension = number of sequences.
20191114: not currently implemented and
filled with zeros. Still a TODO.
|
num_ranges
uint32
|
Number of ranges to calculate
correlations for.
|
num_samps
uint32
|
Number of samples in the sampling
period. Each sequence has its own
sampling period. Will also be provided
as the last data_dimension value.
|
num_sequences
int64
|
Number of sampling periods (equivalent to
number sequences transmitted) in the
integration time.
|
num_slices
int64
|
Number of slices used simultaneously in
this record by the experiment. If more
than 1, data should exist in another file
for this time period for the other slice.
|
pulse_phase_offset
[float32, ]
|
For pulse encoding phase, in degrees
offset. Contains one phase offset per
pulse in pulses, or none.
|
pulses
[uint32, ]
|
The pulse sequence in units of the
tau_spacing.
|
range_sep
float32
|
Range gate separation (conversion from
time (1/rx_sample_rate) to equivalent
distance between samples), in km.
|
rx_sample_rate
float64
|
Sampling rate of the samples in this
file’s data in Hz.
|
samples_data_type
unicode
|
C data type of the samples, provided for
user friendliness. = ‘complex float’
|
scan_start_marker
bool
|
Designates if the record is the first in
a scan (scan is defined by the
experiment).
|
scheduling_mode
unicode
|
The mode being run during this time
period (ex. ‘common’, ‘special’,
‘discretionary’).
|
slice_comment
unicode
|
Additional text comment that describes
the slice written in this file.
|
slice_id
uint32
|
The slice id of this file.
|
slice_interfacing
unicode
|
The interfacing of this slice to
other slices. String representation of
the python dictionary of
{slice : interface_type, … }
|
sqn_timestamps
[float64, ]
|
A list of GPS timestamps corresponding to
the beginning of transmission for each
sampling period in the integration time.
These timestamps come from the USRP
driver and the USRPs are GPS disciplined
and synchronized using the Octoclock.
Provided in seconds since epoch.
|
station
unicode
|
Three-letter radar identifier.
|
tau_spacing
uint32
|
The minimum spacing between pulses in
microseconds. Spacing between pulses is
always a multiple of this.
|
tx_antenna_phases
[complex64, ]
|
The complex phase for each antenna for
transmission, normalized such that full-
power has magnitude 1.
|
tx_pulse_len
uint32
|
Length of the transmit pulse in
microseconds.
|
The pyDARNio format class for this format is BorealisRawrf found in the borealis_formats.
The rawrf format is intended to hold high bandwidth, non-filtered raw data from every antenna.
This format is only produced in a site-style, record by record format and is only available to be produced on request. Please note that this format can cause radar operating delays and may reduce number of averages in an integration, for example.
Site files are produced by the Borealis code package and have the data in a record by record style format. In site files, the hdf5 group names (ie record names) are given as the timestamp in ms past epoch of the first sequence or sampling period recorded in the record.
The naming convention of the rawrf site-structured files are:
[YYYYmmDD].[HHMM].[SS].[station_id].rawrf.hdf5.site
For example:
20191105.1400.02.sas.rawrf.hdf5.site
This is the file that began writing at 14:00:02 UT on November 5 2019 at the Saskatoon site, and it provides data the experiment that ran at that time. Since rawrf is not filtered, this data does not need a slice identifier because it contains all the samples being taken at that time. Some familiarity with the experiment may be necessary to understand the data, or some access to the other file types produced concurrently. This is primarily a debug format for engineering purposes and should only be produced for special cases.
These files are often bzipped after they are produced.
The file fields under the record name in rawrf site files are:
Field name
type
|
description |
|---|---|
agc_status_word
uint32
|
AGC status word. Bit position
corresponds to the USRP motherboard/
transmitter. A ‘1’ indicates an agc fault
occurred at least once during integration
|
blanked_samples
uint32
[number of blanked
samples]
|
Samples that should be blanked because
they occurred during transmission times,
given by sample number (index into
decimated data). Can differ from the
pulses array due to multiple slices in a
single sequence.
|
borealis_git_hash
unicode
|
Identifies the version of Borealis that
made this data. Contains git commit hash
characters. Typically begins with the
latest git tag of the software.
|
data
[complex64, ]
|
A contiguous set of samples (complex
float) at given sample rate. Needs to be
reshaped by data_dimensions to be
correctly read.
|
data_descriptors
[bytes, ]
|
Denotes what each data dimension
represents. = ‘num_sequences’,
‘num_antennas’, ‘num_samps’ for
rawrf
|
data_dimensions
[uint32, ]
|
The dimensions of the data.
Dimensions correspond to
data_descriptors.
|
experiment_comment
unicode
|
Comment provided in experiment about the
experiment as a whole.
|
experiment_id
int16
|
Number used to identify the experiment.
|
experiment_name
unicode
|
Name of the experiment file.
|
gps_locked
bool
|
Designates if the local GPS had a lock
during the entire integration period.
|
gps_to_system_time_diff
float32
|
The max time difference between box_time
GPS time) and system time (NTP) during the
integration. Negative when GPS time is
ahead of system time.
|
int_time
float32
|
Integration time in seconds.
|
intf_antenna_count
uint32
|
Number of interferometer array antennas
|
lp_status_word
uint32
|
Low power status word. Bit position
corresponds to the USRP motherboard/
transmitter. A ‘1’ indicates low power
occurred at least once during integration
|
main_antenna_count
uint32
|
Number of main array antennas
|
num_samps
uint32
|
Number of samples in the sampling
period. Each sequence has its own
sampling period. Will also be provided
as the last data_dimension value.
|
num_sequences
int64
|
Number of sampling periods (equivalent to
number sequences transmitted) in the
integration time.
|
num_slices
int64
|
Number of slices used simultaneously in
this record by the experiment. If more
than 1, data should exist in another file
for this time period for the other slice.
|
rx_center_freq
float64
|
Center frequency of the sampled data
in kHz.
|
rx_sample_rate
float64
|
Sampling rate of the samples in this
file’s data in Hz.
|
samples_data_type
unicode
|
C data type of the samples, provided for
user friendliness. = ‘complex float’
|
scan_start_marker
bool
|
Designates if the record is the first in
a scan (scan is defined by the
experiment).
|
scheduling_mode
unicode
|
The mode being run during this time
period (ex. ‘common’, ‘special’,
‘discretionary’).
|
sqn_timestamps
[float64, ]
|
A list of GPS timestamps corresponding to
the beginning of transmission for each
sampling period in the integration time.
These timestamps come from the USRP
driver and the USRPs are GPS disciplined
and synchronized using the Octoclock.
Provided in seconds since epoch.
|
station
unicode
|
Three-letter radar identifier.
|
tx_antenna_phases
[complex64, ]
|
The complex phase for each antenna for
transmission, normalized such that full-
power has magnitude 1.
|