Data Write

The data_write package contains the utilities to parse protobuf packets containing antennas_iq data, bfiq data, rawacf data, etc and write that data to HDF5 or JSON files.

data_write package

This package contains utilities to parse protobuf packets containing antennas_iq data, bfiq data, rawacf data, etc. and write that data to HDF5 or JSON files.

copyright

2017 SuperDARN Canada

class src.data_write.DataWrite(data_write_options)

Bases: object

This class contains the functions used to write out processed data to files.

Parameters

data_write_options (Options) – The options parsed from config file

output_data(write_bfiq, write_antenna_iq, write_raw_rf, write_tx, file_ext, aveperiod_meta, data_parsing, rt_dw, write_rawacf=True)

Parse through samples and write to file.

A file will be created using the file extension for each requested data product.

Parameters

• write_bfiq (bool) – Should beamformed IQ be written to file?

• write_antenna_iq (bool) – Should pre-beamformed IQ be written to file?

• write_raw_rf (bool) – Should raw rf samples be written to file?

• write_tx (bool) – Should the generated tx samples and metadata be written to file?

• file_ext (str) – Type of file extension to use

• aveperiod_meta (AveperiodMetadataMessage) – Metadata from radar control about averaging period

• data_parsing (ParseData) – All parsed and concatenated data from averaging period

• rt_dw (dict) – Pair of socket and iden for RT purposes.

• write_rawacf (bool, optional) – Should rawacfs be written to file? Defaults to True.

static write_dmap_file(filename, slice_data)

Write out data to a dmap file. If the file already exists it will be overwritten.

Parameters

• filename (str) – The path to the file to write out

• slice_data (SliceData) – Data to write out to the dmap file.

static write_hdf5_file(filename, slice_data, dt_str, file_type)

Write out data to an HDF5 file. If the file already exists it will be overwritten.

Parameters

• filename (str) – The path to the file to write out

• slice_data (SliceData) – Data to write out to the HDF5 file.

• dt_str (str) – A datetime timestamp of the first transmission time in the record

• file_type (str) – Type of file to write.

static write_json_file(filename, slice_data, file_type)

Write out data to a json file. If the file already exists it will be overwritten.

Parameters

• filename (str) – The path to the file to write out

• slice_data (SliceData) – Data to write out to the JSON file.

• file_type (str) – The type of file to write.

class src.data_write.ParseData(agc_status_word: int = 0, antenna_iq_accumulator: dict = <factory>, antenna_iq_available: bool = False, bfiq_accumulator: dict = <factory>, bfiq_available: bool = False, intfacfs_available: bool = False, gps_locked: bool = True, gps_to_system_time_diff: float = 0.0, intfacfs_accumulator: dict = <factory>, lp_status_word: int = 0, mainacfs_accumulator: dict = <factory>, mainacfs_available: bool = False, options: ~utils.options.Options = None, output_sample_rate: float = 0.0, rawrf_available: bool = False, rawrf_locations: list[str] = <factory>, rawrf_num_samps: int = 0, rx_rate: float = 0.0, slice_ids: set = <factory>, timestamps: list[float] = <factory>, xcfs_accumulator: dict = <factory>, xcfs_available: bool = False)

Bases: object

This class is for aggregating data during an averaging period.

agc_status_word: int = 0

antenna_iq_accumulator: dict

antenna_iq_available: bool = False

bfiq_accumulator: dict

bfiq_available: bool = False

gps_locked: bool = True

gps_to_system_time_diff: float = 0.0

intfacfs_accumulator: dict

intfacfs_available: bool = False

lp_status_word: int = 0

mainacfs_accumulator: dict

mainacfs_available: bool = False

numpify_arrays()

Consolidates data for each data type to one array.

In parse_[type](), new data arrays are appended to a list for speed considerations. This function converts these lists into numpy arrays.

options: Options = None

output_sample_rate: float = 0.0

parse_antenna_iq()

Parses out any pre-beamformed IQ if available. Runs on every ProcessedSequenceMessage (contains all sampling period data).

parse_bfiq()

Parses out any possible beamformed IQ data from the message. Runs on every ProcessedSequenceMessage (contains all sampling period data).

parse_correlations()

Parses out the possible correlation (acf/xcf) data from the message. Runs on every new ProcessedSequenceMessage (contains all sampling period data). The expectation value is calculated at the end of an averaging period by a different function.

processed_data: ProcessedSequenceMessage

rawrf_available: bool = False

rawrf_locations: list[str]

rawrf_num_samps: int = 0

rx_rate: float = 0.0

sequence_num: int

slice_ids: set

timestamps: list[float]

update(data)

Parses the message and updates the accumulator fields with the new data.

Parameters

data (ProcessedSequenceMessage) – Processed sequence from rx_signal_processing module.

xcfs_accumulator: dict

xcfs_available: bool = False

class src.data_write.SliceData

Bases: object

This class defines all fields that need to be written by any type of data file. The ‘groups’ metadata lists the applicable file types for each field.

agc_status_word: int

antenna_arrays_order: list[str]

averaging_method: str

beam_azms: list[float]

beam_nums: list[int]

blanked_samples: ndarray

borealis_git_hash: str

data: ndarray

data_descriptors: list[str]

data_dimensions: ndarray

data_normalization_factor: float

decimated_tx_samples: list

dm_rate: list[int]

experiment_comment: str

experiment_id: int

experiment_name: str

first_range: int

first_range_rtt: float

freq: float

gps_locked: bool

gps_to_system_time_diff: float

int_time: float

intf_acfs: ndarray

intf_antenna_count: int

lags: ndarray

lp_status_word: int

main_acfs: ndarray

main_antenna_count: int

noise_at_freq: ndarray

num_ranges: int

num_samps: int

num_sequences: int

num_slices: int

pulse_phase_offset: ndarray

pulse_sample_start: list[float]

pulse_timing_us: list[float]

pulses: ndarray

range_sep: int

rx_center_freq: float

rx_sample_rate: float

samples_data_type: str

scan_start_marker: bool

scheduling_mode: str

slice_comment: str

slice_id: int

slice_interfacing: dict

sqn_timestamps: list[float]

station: str

tau_spacing: float

tx_antenna_phases: list[complex]

tx_center_freq: list[float]

tx_pulse_len: float

tx_rate: list[float]

tx_samples: list

classmethod type_fields(file_type: str)

Returns a list of names for all fields which belong in ‘file_type’ files.

xcfs: ndarray

src.data_write.main()