data_write package¶
The data_write package contains the utilities to parse protobuf packets containing antennas_iq data, bfiq data, rawacf data, etc and write that data to HDF5 or JSON files.
Submodules¶
experiment_prototype¶
This is the base module for all experiments. An experiment will only run if it inherits from this class.
- copyright
2018 SuperDARN Canada
- author
Marci Detwiller
- class experiment_prototype.experiment_prototype.ExperimentPrototype[source]¶
Bases:
object
The base class for all experiments.
A prototype experiment class composed of metadata, including experiment slices (exp_slice) which are dictionaries of radar parameters. Basic, traditional experiments will be composed of a single slice. More complicated experiments will be composed of multiple slices that interface in one of four pre-determined ways, as described under interface_types.
This class is used via inheritance to create experiments.
Some variables shouldn’t be changed by the experiment, and their properties do not have setters. Some variables can be changed in the init of your experiment, and can also be modified in-experiment by the class method ‘update’ in your experiment class. These variables have been given property setters.
The following are the user-modifiable attributes of the ExperimentPrototype that are used to make an experiment:
xcf: boolean for cross-correlation data. A default can be set here for slices, but any slice can override this setting with the xcf slice key.
acf: boolean for auto-correlation data on main array. A default can be set here for slices, but any slice can override this setting with the acf slice key.
acfint: boolean for auto-correlation data on interferometer array. A default can be set here for slices, but any slice can override this setting with the acfint slice key.
slice_dict: modifiable only using the add_slice, edit_slice, and del_slice methods.
interface: modifiable using the add_slice, edit_slice, and del_slice methods, or by updating the interface dict directly.
Other parameters are set in the init and cannot be modified after instantiation.
- property acf¶
The default auto-correlation flag boolean.
This provides the default for slices where this key isn’t specified.
- property acfint¶
The default interferometer autocorrelation boolean.
This provides the default for slices where this key isn’t specified.
- add_slice(exp_slice, interfacing_dict={})[source]¶
Add a slice to the experiment.
- Parameters
exp_slice – a slice (dictionary of slice_keys) to add to the experiment.
interfacing_dict – dictionary of type {slice_id : INTERFACING , … } that defines how this slice interacts with all the other slices currently in the experiment.
- Raises
ExperimentException if slice is not a dictionary or if there are errors in setup_slice.
- Returns
the slice_id of the new slice that was just added.
- build_scans()[source]¶
Build the scan information, which means creating the Scan, AveragingPeriod, and Sequence instances needed to run this experiment.
Will be run by experiment handler, to build iterable objects for radar_control to use. Creates scan_objects in the experiment for identifying which slices are in the scans.
- check_new_slice_interfacing(interfacing_dict)[source]¶
Checks that the new slice plays well with its siblings (has interfacing that is resolvable). If so, returns a new dictionary with all interfacing values set.
The interfacing assumes that the interfacing_dict given by the user defines the closest interfacing of the new slice with a slice. For example, if the slice is to be PULSE combined with slice 0, the interfacing dict should provide this information. If only ‘SCAN’ interfacing with slice 1 is provided, then that will be assumed to be the closest and therefore the interfacing with slice 0 will also be ‘SCAN’.
If no interfacing_dict is provided for a slice, the default is to do ‘SCAN’ type interfacing for the new slice with all other slices.
- Parameters
interfacing_dict – the user-provided interfacing dict, which may be empty or incomplete. If empty, all interfacing is assumed to be = ‘SCAN’ type. If it contains something, we ensure that the interfacing provided makes sense with the values already known for its closest sibling.
- Returns
full interfacing dictionary.
- Raises
ExperimentException if invalid interface types provided or if interfacing can not be resolved.
- check_slice(exp_slice)[source]¶
Check the slice for errors.
This is the first test of the dictionary in the experiment done to ensure values in this slice make sense. This is a self-check to ensure the parameters (for example, txfreq, antennas) are appropriate. All fields should be full at this time (whether filled by the user or given default values in set_slice_defaults). This was built to be useable at any time after setup. :param: exp_slice: a slice to check :raise: ExperimentException: When necessary parameters do not exist or = None (would have to have been overridden by the user for this, as defaults all set when this runs).
- check_slice_minimum_requirements(exp_slice)[source]¶
Check the required slice keys.
Check for the minimum requirements of the slice. The following keys are always required: “pulse_sequence”, “tau_spacing”, “pulse_len”, “num_ranges”, “first_range”, (one of “intt” or “intn”), “beam_angle”, and “beam_order”. This function may modify the values in this slice dictionary to ensure that it is able to be run and that the values make sense.
- Parameters
exp_slice – slice to check.
- check_slice_specific_requirements(exp_slice)[source]¶
Set the specific slice requirements depending.
Check the requirements for the specific slice type as identified by the identifiers rxonly and clrfrqflag. The keys that need to be checked depending on these identifiers are “txfreq”, “rxfreq”, and “clrfrqrange”. This function may modify these keys.
- Parameters
exp_slice – the slice to check, before adding to the experiment.
- property comment_string¶
A string related to the experiment, to be placed in the experiment’s files.
This is read-only once established in instantiation.
- property cpid¶
This experiment’s CPID (control program ID, a term that comes from ROS).
The CPID is read-only once established in instantiation. It may be modified at runtime by the set_scheduling_mode function, to set it to a negative value during discretionary time.
- property decimation_scheme¶
The decimation scheme, of type DecimationScheme from the filtering module. Includes all filtering and decimating information for the signal processing module.
This is read-only once established in instantiation.
- del_slice(remove_slice_id)[source]¶
Remove a slice from the experiment.
- Parameters
remove_slice_id – the id of the slice you’d like to remove.
- Returns
a copy of the removed slice.
- Raises
exception if remove_slice_id does not exist in the slice dictionary.
- edit_slice(edit_slice_id, **kwargs)[source]¶
Edit a slice.
A quick way to edit a slice. In reality this is actually adding a new slice and deleting the old one. Useful for quick changes. Note that using this function will remove the slice_id that you are changing and will give it a new id. It will account for this in the interfacing dictionary.
- Parameters
edit_slice_id – the slice id of the slice to be edited.
kwargs – dictionary of slice parameter to slice value that you want to change.
- Returns new_slice_id
the new slice id of the edited slice, or the edit_slice_id if no change has occurred due to failure of new slice parameters to pass experiment checks.
- Raises
exceptions if the edit_slice_id does not exist in slice dictionary or the params or values do not make sense.
- property experiment_name¶
The experiment class name.
- get_scan_slice_ids()[source]¶
Organize the slice_ids by scan.
Take my own interfacing and get info on how many scans and which slices make which scans. Return a list of lists where each inner list contains the slices that are in an averagingperiod that is inside this scan. ie. len(nested_slice_list) = # of averagingperiods in this scan, len(nested_slice_list[0]) = # of slices in the first averagingperiod, etc.
:return list of lists. The list has one element per scan. Each element is a list of slice_ids signifying which slices are combined inside that scan. The list returned could be of length 1, meaning only one scan is present in the experiment.
- get_slice_interfacing(slice_id)[source]¶
Check the experiment’s interfacing dictionary for all interfacing that pertains to a given slice, and return the interfacing information in a dictionary. :param slice_id: Slice ID to search the interface dictionary for. :return: interfacing dictionary for the slice.
- property interface¶
The dictionary of interfacing for the experiment slices.
Interfacing should be set up for any slice when it gets added, ie. in add_slice, except for the first slice added. The dictionary of interfacing is setup as:
[(slice_id1, slice_id2) : INTERFACING_TYPE, (slice_id1, slice_id3) : INTERFACING_TYPE, …]
for all current slice_ids.
- property new_slice_id¶
The next unique slice id that is available to this instance of the experiment.
This gets incremented each time it is called to ensure it returns a unique ID each time.
- property num_slices¶
The number of slices currently in the experiment.
Will change after methods add_slice or del_slice are called.
- property options¶
The config options for running this experiment.
These cannot be set or removed, but are specified in the config.ini, hdw.dat, and restrict.dat files.
- property output_rx_rate¶
The output receive rate of the data, Hz.
This is read-only once established in instantiation.
- property rx_bandwidth¶
The receive bandwidth for this experiment, in Hz.
This is read-only once established in instantiation.
- property rx_maxfreq¶
The maximum receive frequency.
This is the maximum tx frequency possible in this experiment (maximum given by the center frequency and sampling rate), as license doesn’t matter for receiving. The maximum is slightly less than that allowed by the center frequency and rxrate, to stay away from the edges of the possible receive band where the signal may be distorted.
- property rx_minfreq¶
The minimum receive frequency.
This is the minimum rx frequency possible in this experiment (minimum given by the center frequency and sampling rate) - license doesn’t restrict receiving. The minimum is slightly more than that allowed by the center frequency and rxrate, to stay away from the edges of the possible receive band where the signal may be distorted.
- property rxctrfreq¶
The receive center frequency that USRP is tuned to (kHz).
- property rxrate¶
The receive bandwidth for this experiment, or the receive sampling rate (of I and Q samples) In Hz.
This is read-only once established in instantiation.
- property scan_objects¶
The list of instances of class Scan for use in radar_control.
These cannot be modified by the user, but are created using the slice dictionary.
- property scheduling_mode¶
Return the scheduling mode time type that this experiment is running in. Types are listed in possible_scheduling_modes. Initialized to ‘unknown’ until set by the experiment handler.
- set_slice_defaults(exp_slice)[source]¶
Set up defaults in case of some parameters being left blank.
- Parameters
exp_slice – slice to set defaults of
- Returns slice_with_defaults
updated slice
- static set_slice_identifiers(exp_slice)[source]¶
Set the hidden slice keys to determine how to run the slice.
This function sets up internal identifier flags ‘clrfrqflag’ and ‘rxonly’ in the slice so that we know how to properly set up the slice and know which keys in the slice must be specified and which are unnecessary. If these keys are ever written by the user, they will be rewritten here.
- Parameters
exp_slice – slice in which to set identifiers
- setup_slice(exp_slice)[source]¶
Check slice for errors and set defaults of optional keys.
Before adding the slice, ensure that the internal parameters are set, remove unnecessary keys and check values of keys that are needed, and set defaults of keys that are optional.
The following are always able to be defaulted, so are optional: “tx_antennas”, “rx_main_antennas”, “rx_int_antennas”, “pulse_phase_offset”, “scanboundflag”, “scanbound”, “acf”, “xcf”, “acfint”, “wavetype”, “seqoffset”, “averaging_method”
The following are always required for processing acf, xcf, and acfint which we will assume we are always doing: “pulse_sequence”, “tau_spacing”, “pulse_len”, “num_ranges”, “first_range”, “intt”, “intn”, “beam_angle”, “beam_order”
The following are required depending on slice type: “txfreq”, “rxfreq”, “clrfrqrange”
- Param
exp_slice: a slice to setup
- Returns
complete_slice : a checked slice with all defaults
- slice_beam_directions_mapping(slice_id)[source]¶
A mapping of the beam directions in the given slice id.
- Parameters
slice_id – id of the slice to get beam directions for.
- Returns mapping
enumeration mapping dictionary of beam number to beam direction(s) in degrees off boresight.
- property slice_dict¶
The dictionary of slices.
The slice dictionary can be updated in add_slice, edit_slice, and del_slice. The slice dictionary is a dictionary of dictionaries that looks like:
{ slice_id1 : {slice_key1 : x, slice_key2 : y, …}, slice_id2 : {slice_key1 : x, slice_key2 : y, …}, …}
- property slice_ids¶
The list of slice ids that are currently available in this experiment.
This can change when add_slice, edit_slice, and del_slice are called.
- property slice_keys¶
The list of slice keys available.
This cannot be updated. These are the keys in the current ExperimentPrototype slice_keys dictionary (the parameters available for slices).
- property transmit_metadata¶
A dictionary of config options and experiment-set values that cannot change in the experiment, that will be used to build pulse sequences.
- property tx_bandwidth¶
The transmission sample rate to the DAC (Hz), and the transmit bandwidth.
This is read-only once established in instantiation.
- property tx_maxfreq¶
The maximum transmit frequency.
This is the maximum tx frequency possible in this experiment (either maximum in our license or maximum given by the center frequency, and sampling rate). The maximum is slightly less than that allowed by the center frequency and txrate, to stay away from the edges of the possible transmission band where the signal is distorted.
- property tx_minfreq¶
The minimum transmit frequency.
This is the minimum tx frequency possible in this experiment (either minimum in our license or minimum given by the center frequency and sampling rate). The minimum is slightly more than that allowed by the center frequency and txrate, to stay away from the edges of the possible transmission band where the signal is distorted.
- property txctrfreq¶
The transmission center frequency that USRP is tuned to (kHz).
- property txrate¶
The transmission sample rate to the DAC (Hz).
This is read-only once established in instantiation.
- property xcf¶
The default cross-correlation flag boolean.
This provides the default for slices where this key isn’t specified.
These are used by the build_scans method (called from the experiment_handler every time the experiment is run). If set by the user, the values will be overwritten and therefore ignored.
- experiment_prototype.experiment_prototype.interface_types = ('SCAN', 'INTTIME', 'INTEGRATION', 'PULSE')¶
The types of interfacing available for slices in the experiment.
Interfacing in this case refers to how two or more components are meant to be run together. The following types of interfacing are possible:
1. SCAN. The scan by scan interfacing allows for slices to run a scan of one slice, followed by a scan of the second. The scan mode of interfacing typically means that the slice will cycle through all of its beams before switching to another slice.
There are no requirements for slices interfaced in this manner.
2. INTTIME. This type of interfacing allows for one slice to run its integration period (also known as integration time or averaging period), before switching to another slice’s integration period. This type of interface effectively creates an interleaving scan where the scans for multiple slices are run ‘at the same time’, by interleaving the integration times.
- Slices which are interfaced in this manner must share:
the same SCANBOUND value.
3. INTEGRATION. Integration interfacing allows for pulse sequences defined in the slices to alternate between each other within a single integration period. It’s important to note that data from a single slice is averaged only with other data from that slice. So in this case, the integration period is running two slices and can produce two averaged datasets, but the sequences (integrations) within the integration period are interleaved.
- Slices which are interfaced in this manner must share:
the same SCANBOUND value.
the same INTT or INTN value.
the same BEAM_ORDER length (scan length)
4. PULSE. Pulse interfacing allows for pulse sequences to be run together concurrently. Slices will have their pulse sequences summed together so that the data transmits at the same time. For example, slices of different frequencies can be mixed simultaneously, and slices of different pulse sequences can also run together at the cost of having more blanked samples. When slices are interfaced in this way the radar is truly transmitting and receiving the slices simultaneously.
- Slices which are interfaced in this manner must share:
the same SCANBOUND value.
the same INTT or INTN value.
the same BEAM_ORDER length (scan length)
- experiment_prototype.experiment_prototype.slice_key_set = frozenset({'acf', 'acfint', 'averaging_method', 'beam_angle', 'beam_order', 'clrfrqrange', 'comment', 'cpid', 'first_range', 'intn', 'intt', 'iwavetable', 'lag_table', 'num_ranges', 'pulse_len', 'pulse_phase_offset', 'pulse_sequence', 'qwavetable', 'range_sep', 'rx_int_antennas', 'rx_main_antennas', 'rxfreq', 'scanbound', 'seqoffset', 'slice_id', 'tau_spacing', 'tx_antennas', 'txfreq', 'wavetype', 'xcf'})¶
These are the keys that are set by the user when initializing a slice. Some are required, some can be defaulted, and some are set by the experiment and are read-only.
Slice Keys Required by the User
- pulse_sequence required
The pulse sequence timing, given in quantities of tau_spacing, for example normalscan = [0, 14, 22, 24, 27, 31, 42, 43].
- tau_spacing required
multi-pulse increment (mpinc) in us, Defines minimum space between pulses.
- pulse_len required
length of pulse in us. Range gate size is also determined by this.
- num_ranges required
Number of range gates.
- first_range required
first range gate, in km
- intt required or intn required
duration of an integration, in ms. (maximum)
- intn required or intt required
number of averages to make a single integration, only used if intt = None.
- beam_angle required
list of beam directions, in degrees off azimuth. Positive is E of N. The beam_angle list length = number of beams. Traditionally beams have been 3.24 degrees separated but we don’t refer to them as beam -19.64 degrees, we refer as beam 1, beam 2. Beam 0 will be the 0th element in the list, beam 1 will be the 1st, etc. These beam numbers are needed to write the beam_order list. This is like a mapping of beam number (list index) to beam direction off boresight.
- beam_order required
beam numbers written in order of preference, one element in this list corresponds to one integration period. Can have lists within the list, resulting in multiple beams running simultaneously in the averaging period, so imaging. A beam number of 0 in this list gives us the direction of the 0th element in the beam_angle list. It is up to the writer to ensure their beam pattern makes sense. Typically beam_order is just in order (scanning W to E or E to W, ie. [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]. You can list numbers multiple times in the beam_order list, for example [0, 1, 1, 2, 1] or use multiple beam numbers in a single integration time (example [[0, 1], [3, 4]], which would trigger an imaging integration. When we do imaging we will still have to quantize the directions we are looking in to certain beam directions.
- clrfrqrange required or txfreq or rxfreq required
range for clear frequency search, should be a list of length = 2, [min_freq, max_freq] in kHz. Not currently supported.
- txfreq required or clrfrqrange or rxfreq required
transmit frequency, in kHz. Note if you specify clrfrqrange it won’t be used.
- rxfreq required or clrfrqrange or txfreq required
receive frequency, in kHz. Note if you specify clrfrqrange or txfreq it won’t be used. Only necessary to specify if you want a receive-only slice.
Defaultable Slice Keys
- acf defaults
flag for rawacf and generation. The default is False. If True, the following fields are also used: - averaging_method (default ‘mean’) - xcf (default True if acf is True) - acfint (default True if acf is True) - lagtable (default built based on all possible pulse combos) - range_sep (will be built by pulse_len to verify any provided value)
- acfint defaults
flag for interferometer autocorrelation data. The default is True if acf is True, otherwise False.
- averaging_method defaults
a string defining the type of averaging to be done. Current methods are ‘mean’ or ‘median’. The default is ‘mean’.
- comment defaults
a comment string that will be placed in the borealis files describing the slice. Defaults to empty string.
- lag_table defaults
used in acf calculations. It is a list of lags. Example of a lag: [24, 27] from 8-pulse normalscan. This defaults to a lagtable built by the pulse sequence provided. All combinations of pulses will be calculated, with both the first pulses and last pulses used for lag-0.
- pulse_phase_offset defaults
Allows phase shifting of pulses, enabling encoding of pulses. Default all zeros for all pulses in pulse_sequence. Pulses can be shifted with a single phase shift for each pulse or with a phase shift specified for each sample in the pulses of the slice.
- range_sep defaults
a calculated value from pulse_len. If already set, it will be overwritten to be the correct value determined by the pulse_len. Used for acfs. This is the range gate separation, in azimuthal direction, in km.
- rx_int_antennas defaults
The antennas to receive on in interferometer array, default is all antennas given max number from config.
- rx_main_antennas defaults
The antennas to receive on in main array, default is all antennas given max number from config.
- scanbound defaults
A list of seconds past the minute for integration times in a scan to align to. Defaults to None, not required.
- seqoffset defaults
offset in us that this slice’s sequence will begin at, after the start of the sequence. This is intended for PULSE interfacing, when you want multiple slice’s pulses in one sequence you can offset one slice’s sequence from the other by a certain time value so as to not run both frequencies in the same pulse, etc. Default is 0 offset.
- tx_antennas defaults
The antennas to transmit on, default is all main antennas given max number from config.
- xcf defaults
flag for cross-correlation data. The default is True if acf is True, otherwise False.
Read-only Slice Keys
- clrfrqflag read-only
A boolean flag to indicate that a clear frequency search will be done. Not currently supported.
- cpid read-only
The ID of the experiment, consistent with existing radar control programs. This is actually an experiment-wide attribute but is stored within the slice as well. This is provided by the user but not within the slice, instead when the experiment is initialized.
- rx_only read-only
A boolean flag to indicate that the slice doesn’t transmit, only receives.
- slice_id read-only
The ID of this slice object. An experiment can have multiple slices. This is not set by the user but instead set by the experiment when the slice is added. Each slice id within an experiment is unique. When experiments start, the first slice_id will be 0 and incremented from there.
- slice_interfacing read-only
A dictionary of slice_id : interface_type for each sibling slice in the experiment at any given time.
Not currently supported and will be removed
- wavetype defaults
string for wavetype. The default is SINE. Not currently supported.
- iwavetable defaults
a list of numeric values to sample from. The default is None. Not currently supported but could be set up (with caution) for non-SINE. Not currently supported.
- qwavetable defaults
a list of numeric values to sample from. The default is None. Not currently supported but could be set up (with caution) for non-SINE. Not currently supported.
experiment_exception¶
This is the exception that is raised when there are problems with the experiment that cannot be remedied using experiment_prototype methods.
- copyright
2018 SuperDARN Canada
- author
Marci Detwiller
list_tests¶
Basic tests for use in checking slices.
- copyright
2018 SuperDARN Canada
- author
Marci Detwiller