experiment_prototype package

The experiment_prototype package contains the building blocks of experiments, which includes the ExperimentPrototype base class, the scan_classes subpackage including the ScanClassBase classes, and the ExperimentException. There is also a list_tests module which is used by the ExperimentPrototype class.

Submodules

experiment_prototype

This is the base module for all experiments. An experiment will only run if it inherits from this class.

copyright

2018 SuperDARN Canada

author

Marci Detwiller

class experiment_prototype.experiment_prototype.ExperimentPrototype

Bases: object

The base class for all experiments.

A prototype experiment class composed of metadata, including experiment slices (exp_slice) which are dictionaries of radar parameters. Basic, traditional experiments will be composed of a single slice. More complicated experiments will be composed of multiple slices that interface in one of four pre-determined ways, as described under interface_types.

This class is used via inheritance to create experiments.

Some variables shouldn’t be changed by the experiment, and their properties do not have setters. Some variables can be changed in the init of your experiment, and can also be modified in-experiment by the class method ‘update’ in your experiment class. These variables have been given property setters.

The following are the user-modifiable attributes of the ExperimentPrototype that are used to make an experiment:

• xcf: boolean for cross-correlation data. A default can be set here for slices, but any slice can override this setting with the xcf slice key.

• acf: boolean for auto-correlation data on main array. A default can be set here for slices, but any slice can override this setting with the acf slice key.

• acfint: boolean for auto-correlation data on interferometer array. A default can be set here for slices, but any slice can override this setting with the acfint slice key.

• slice_dict: modifiable only using the add_slice, edit_slice, and del_slice methods.

• interface: modifiable using the add_slice, edit_slice, and del_slice methods, or by updating the interface dict directly.

Other parameters are set in the init and cannot be modified after instantiation.

property acf

The default auto-correlation flag boolean.

This provides the default for slices where this key isn’t specified.

property acfint

The default interferometer autocorrelation boolean.

This provides the default for slices where this key isn’t specified.

add_slice(exp_slice, interfacing_dict={})

Add a slice to the experiment.

Parameters

• exp_slice – a slice (dictionary of slice_keys) to add to the experiment.

• interfacing_dict – dictionary of type {slice_id : INTERFACING , … } that defines how this slice interacts with all the other slices currently in the experiment.

Raises

ExperimentException if slice is not a dictionary or if there are errors in setup_slice.

Returns

the slice_id of the new slice that was just added.

build_scans()

Build the scan information, which means creating the Scan, AveragingPeriod, and Sequence instances needed to run this experiment.

Will be run by experiment handler, to build iterable objects for radar_control to use. Creates scan_objects in the experiment for identifying which slices are in the scans.

check_new_slice_interfacing(interfacing_dict)

Checks that the new slice plays well with its siblings (has interfacing that is resolvable). If so, returns a new dictionary with all interfacing values set.

The interfacing assumes that the interfacing_dict given by the user defines the closest interfacing of the new slice with a slice. For example, if the slice is to be PULSE combined with slice 0, the interfacing dict should provide this information. If only ‘SCAN’ interfacing with slice 1 is provided, then that will be assumed to be the closest and therefore the interfacing with slice 0 will also be ‘SCAN’.

If no interfacing_dict is provided for a slice, the default is to do ‘SCAN’ type interfacing for the new slice with all other slices.

Parameters

interfacing_dict – the user-provided interfacing dict, which may be empty or incomplete. If empty, all interfacing is assumed to be = ‘SCAN’ type. If it contains something, we ensure that the interfacing provided makes sense with the values already known for its closest sibling.

Returns

full interfacing dictionary.

Raises

ExperimentException if invalid interface types provided or if interfacing can not be resolved.

check_slice(exp_slice)

Check the slice for errors.

This is the first test of the dictionary in the experiment done to ensure values in this slice make sense. This is a self-check to ensure the parameters (for example, txfreq, antennas) are appropriate. All fields should be full at this time (whether filled by the user or given default values in set_slice_defaults). This was built to be useable at any time after setup. :param: exp_slice: a slice to check :raise: ExperimentException: When necessary parameters do not exist or = None (would have to have been overridden by the user for this, as defaults all set when this runs).

check_slice_minimum_requirements(exp_slice)

Check the required slice keys.

Check for the minimum requirements of the slice. The following keys are always required: “pulse_sequence”, “tau_spacing”, “pulse_len”, “num_ranges”, “first_range”, (one of “intt” or “intn”), “beam_angle”, and “beam_order”. This function may modify the values in this slice dictionary to ensure that it is able to be run and that the values make sense.

Parameters

exp_slice – slice to check.

check_slice_specific_requirements(exp_slice)

Set the specific slice requirements depending.

Check the requirements for the specific slice type as identified by the identifiers rxonly and clrfrqflag. The keys that need to be checked depending on these identifiers are “txfreq”, “rxfreq”, and “clrfrqrange”. This function may modify these keys.

Parameters

exp_slice – the slice to check, before adding to the experiment.

property comment_string

A string related to the experiment, to be placed in the experiment’s files.

This is read-only once established in instantiation.

property cpid

This experiment’s CPID (control program ID, a term that comes from ROS).

The CPID is read-only once established in instantiation. It may be modified at runtime by the set_scheduling_mode function, to set it to a negative value during discretionary time.

property decimation_scheme

The decimation scheme, of type DecimationScheme from the filtering module. Includes all filtering and decimating information for the signal processing module.

This is read-only once established in instantiation.

del_slice(remove_slice_id)

Remove a slice from the experiment.

Parameters

remove_slice_id – the id of the slice you’d like to remove.

Returns

a copy of the removed slice.

Raises

exception if remove_slice_id does not exist in the slice dictionary.

edit_slice(edit_slice_id, **kwargs)

Edit a slice.

A quick way to edit a slice. In reality this is actually adding a new slice and deleting the old one. Useful for quick changes. Note that using this function will remove the slice_id that you are changing and will give it a new id. It will account for this in the interfacing dictionary.

Parameters

• edit_slice_id – the slice id of the slice to be edited.

• kwargs – dictionary of slice parameter to slice value that you want to change.

Returns new_slice_id

the new slice id of the edited slice, or the edit_slice_id if no change has occurred due to failure of new slice parameters to pass experiment checks.

Raises

exceptions if the edit_slice_id does not exist in slice dictionary or the params or values do not make sense.

property experiment_name

The experiment class name.

get_scan_slice_ids()

Organize the slice_ids by scan.

Take my own interfacing and get info on how many scans and which slices make which scans. Return a list of lists where each inner list contains the slices that are in an averagingperiod that is inside this scan. ie. len(nested_slice_list) = # of averagingperiods in this scan, len(nested_slice_list[0]) = # of slices in the first averagingperiod, etc.

:return list of lists. The list has one element per scan. Each element is a list of slice_ids signifying which slices are combined inside that scan. The list returned could be of length 1, meaning only one scan is present in the experiment.

get_slice_interfacing(slice_id)

Check the experiment’s interfacing dictionary for all interfacing that pertains to a given slice, and return the interfacing information in a dictionary. :param slice_id: Slice ID to search the interface dictionary for. :return: interfacing dictionary for the slice.

property interface

The dictionary of interfacing for the experiment slices.

Interfacing should be set up for any slice when it gets added, ie. in add_slice, except for the first slice added. The dictionary of interfacing is setup as:

[(slice_id1, slice_id2) : INTERFACING_TYPE, (slice_id1, slice_id3) : INTERFACING_TYPE, …]

for all current slice_ids.

property new_slice_id

The next unique slice id that is available to this instance of the experiment.

This gets incremented each time it is called to ensure it returns a unique ID each time.

property num_slices

The number of slices currently in the experiment.

Will change after methods add_slice or del_slice are called.

property options

The config options for running this experiment.

These cannot be set or removed, but are specified in the config.ini, hdw.dat, and restrict.dat files.

property output_rx_rate

The output receive rate of the data, Hz.

This is read-only once established in instantiation.

printing(msg)

property rx_bandwidth

The receive bandwidth for this experiment, in Hz.

This is read-only once established in instantiation.

property rx_maxfreq

The maximum receive frequency.

This is the maximum tx frequency possible in this experiment (maximum given by the center frequency and sampling rate), as license doesn’t matter for receiving. The maximum is slightly less than that allowed by the center frequency and rxrate, to stay away from the edges of the possible receive band where the signal may be distorted.

property rx_minfreq

The minimum receive frequency.

This is the minimum rx frequency possible in this experiment (minimum given by the center frequency and sampling rate) - license doesn’t restrict receiving. The minimum is slightly more than that allowed by the center frequency and rxrate, to stay away from the edges of the possible receive band where the signal may be distorted.

property rxctrfreq

The receive center frequency that USRP is tuned to (kHz).

property rxrate

The receive bandwidth for this experiment, or the receive sampling rate (of I and Q samples) In Hz.

This is read-only once established in instantiation.

property scan_objects

The list of instances of class Scan for use in radar_control.

These cannot be modified by the user, but are created using the slice dictionary.

property scheduling_mode

Return the scheduling mode time type that this experiment is running in. Types are listed in possible_scheduling_modes. Initialized to ‘unknown’ until set by the experiment handler.

self_check()

Check that the values in this experiment are valid.

Checks all slices.

set_slice_defaults(exp_slice)

Set up defaults in case of some parameters being left blank.

Parameters

exp_slice – slice to set defaults of

Returns slice_with_defaults

updated slice

static set_slice_identifiers(exp_slice)

Set the hidden slice keys to determine how to run the slice.

This function sets up internal identifier flags ‘clrfrqflag’ and ‘rxonly’ in the slice so that we know how to properly set up the slice and know which keys in the slice must be specified and which are unnecessary. If these keys are ever written by the user, they will be rewritten here.

Parameters

exp_slice – slice in which to set identifiers

setup_slice(exp_slice)

Check slice for errors and set defaults of optional keys.

Before adding the slice, ensure that the internal parameters are set, remove unnecessary keys and check values of keys that are needed, and set defaults of keys that are optional.

The following are always able to be defaulted, so are optional: “tx_antennas”, “rx_main_antennas”, “rx_int_antennas”, “pulse_phase_offset”, “scanboundflag”, “scanbound”, “acf”, “xcf”, “acfint”, “wavetype”, “seqoffset”, “averaging_method”

The following are always required for processing acf, xcf, and acfint which we will assume we are always doing: “pulse_sequence”, “tau_spacing”, “pulse_len”, “num_ranges”, “first_range”, “intt”, “intn”, “beam_angle”, “beam_order”

The following are required depending on slice type: “txfreq”, “rxfreq”, “clrfrqrange”

Param

exp_slice: a slice to setup

Returns

complete_slice : a checked slice with all defaults

slice_beam_directions_mapping(slice_id)

A mapping of the beam directions in the given slice id.

Parameters

slice_id – id of the slice to get beam directions for.

Returns mapping

enumeration mapping dictionary of beam number to beam direction(s) in degrees off boresight.

property slice_dict

The dictionary of slices.

The slice dictionary can be updated in add_slice, edit_slice, and del_slice. The slice dictionary is a dictionary of dictionaries that looks like:

{ slice_id1 : {slice_key1 : x, slice_key2 : y, …}, slice_id2 : {slice_key1 : x, slice_key2 : y, …}, …}

property slice_ids

The list of slice ids that are currently available in this experiment.

This can change when add_slice, edit_slice, and del_slice are called.

property slice_keys

The list of slice keys available.

This cannot be updated. These are the keys in the current ExperimentPrototype slice_keys dictionary (the parameters available for slices).

property transmit_metadata

A dictionary of config options and experiment-set values that cannot change in the experiment, that will be used to build pulse sequences.

property tx_bandwidth

The transmission sample rate to the DAC (Hz), and the transmit bandwidth.

This is read-only once established in instantiation.

property tx_maxfreq

The maximum transmit frequency.

This is the maximum tx frequency possible in this experiment (either maximum in our license or maximum given by the center frequency, and sampling rate). The maximum is slightly less than that allowed by the center frequency and txrate, to stay away from the edges of the possible transmission band where the signal is distorted.

property tx_minfreq

The minimum transmit frequency.

This is the minimum tx frequency possible in this experiment (either minimum in our license or minimum given by the center frequency and sampling rate). The minimum is slightly more than that allowed by the center frequency and txrate, to stay away from the edges of the possible transmission band where the signal is distorted.

property txctrfreq

The transmission center frequency that USRP is tuned to (kHz).

property txrate

The transmission sample rate to the DAC (Hz).

This is read-only once established in instantiation.

property xcf

The default cross-correlation flag boolean.

This provides the default for slices where this key isn’t specified.

experiment_prototype.experiment_prototype.hidden_key_set = frozenset({'clrfrqflag', 'rxonly', 'slice_interfacing'})

These are used by the build_scans method (called from the experiment_handler every time the experiment is run). If set by the user, the values will be overwritten and therefore ignored.

experiment_prototype.experiment_prototype.interface_types = ('SCAN', 'INTTIME', 'INTEGRATION', 'PULSE')

The types of interfacing available for slices in the experiment.

Interfacing in this case refers to how two or more components are meant to be run together. The following types of interfacing are possible:

1. SCAN. The scan by scan interfacing allows for slices to run a scan of one slice, followed by a scan of the second. The scan mode of interfacing typically means that the slice will cycle through all of its beams before switching to another slice.

There are no requirements for slices interfaced in this manner.

2. INTTIME. This type of interfacing allows for one slice to run its integration period (also known as integration time or averaging period), before switching to another slice’s integration period. This type of interface effectively creates an interleaving scan where the scans for multiple slices are run ‘at the same time’, by interleaving the integration times.

Slices which are interfaced in this manner must share:

• the same SCANBOUND value.

3. INTEGRATION. Integration interfacing allows for pulse sequences defined in the slices to alternate between each other within a single integration period. It’s important to note that data from a single slice is averaged only with other data from that slice. So in this case, the integration period is running two slices and can produce two averaged datasets, but the sequences (integrations) within the integration period are interleaved.

Slices which are interfaced in this manner must share:

• the same SCANBOUND value.

• the same INTT or INTN value.

• the same BEAM_ORDER length (scan length)

4. PULSE. Pulse interfacing allows for pulse sequences to be run together concurrently. Slices will have their pulse sequences summed together so that the data transmits at the same time. For example, slices of different frequencies can be mixed simultaneously, and slices of different pulse sequences can also run together at the cost of having more blanked samples. When slices are interfaced in this way the radar is truly transmitting and receiving the slices simultaneously.

Slices which are interfaced in this manner must share:

• the same SCANBOUND value.

• the same INTT or INTN value.

• the same BEAM_ORDER length (scan length)

experiment_prototype.experiment_prototype.slice_key_set = frozenset({'acf', 'acfint', 'averaging_method', 'beam_angle', 'beam_order', 'clrfrqrange', 'comment', 'cpid', 'first_range', 'intn', 'intt', 'iwavetable', 'lag_table', 'num_ranges', 'pulse_len', 'pulse_phase_offset', 'pulse_sequence', 'qwavetable', 'range_sep', 'rx_int_antennas', 'rx_main_antennas', 'rxfreq', 'scanbound', 'seqoffset', 'slice_id', 'tau_spacing', 'tx_antennas', 'txfreq', 'wavetype', 'xcf'})

These are the keys that are set by the user when initializing a slice. Some are required, some can be defaulted, and some are set by the experiment and are read-only.

Slice Keys Required by the User

pulse_sequence required

The pulse sequence timing, given in quantities of tau_spacing, for example normalscan = [0, 14, 22, 24, 27, 31, 42, 43].

tau_spacing required

multi-pulse increment (mpinc) in us, Defines minimum space between pulses.

pulse_len required

length of pulse in us. Range gate size is also determined by this.

num_ranges required

Number of range gates.

first_range required

first range gate, in km

intt required or intn required

duration of an integration, in ms. (maximum)

intn required or intt required

number of averages to make a single integration, only used if intt = None.

beam_angle required

list of beam directions, in degrees off azimuth. Positive is E of N. The beam_angle list length = number of beams. Traditionally beams have been 3.24 degrees separated but we don’t refer to them as beam -19.64 degrees, we refer as beam 1, beam 2. Beam 0 will be the 0th element in the list, beam 1 will be the 1st, etc. These beam numbers are needed to write the beam_order list. This is like a mapping of beam number (list index) to beam direction off boresight.

beam_order required

beam numbers written in order of preference, one element in this list corresponds to one integration period. Can have lists within the list, resulting in multiple beams running simultaneously in the averaging period, so imaging. A beam number of 0 in this list gives us the direction of the 0th element in the beam_angle list. It is up to the writer to ensure their beam pattern makes sense. Typically beam_order is just in order (scanning W to E or E to W, ie. [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]. You can list numbers multiple times in the beam_order list, for example [0, 1, 1, 2, 1] or use multiple beam numbers in a single integration time (example [[0, 1], [3, 4]], which would trigger an imaging integration. When we do imaging we will still have to quantize the directions we are looking in to certain beam directions.

clrfrqrange required or txfreq or rxfreq required

range for clear frequency search, should be a list of length = 2, [min_freq, max_freq] in kHz. Not currently supported.

txfreq required or clrfrqrange or rxfreq required

transmit frequency, in kHz. Note if you specify clrfrqrange it won’t be used.

rxfreq required or clrfrqrange or txfreq required

receive frequency, in kHz. Note if you specify clrfrqrange or txfreq it won’t be used. Only necessary to specify if you want a receive-only slice.

Defaultable Slice Keys

acf defaults

flag for rawacf and generation. The default is False. If True, the following fields are also used: - averaging_method (default ‘mean’) - xcf (default True if acf is True) - acfint (default True if acf is True) - lagtable (default built based on all possible pulse combos) - range_sep (will be built by pulse_len to verify any provided value)

acfint defaults

flag for interferometer autocorrelation data. The default is True if acf is True, otherwise False.

averaging_method defaults

a string defining the type of averaging to be done. Current methods are ‘mean’ or ‘median’. The default is ‘mean’.

comment defaults

a comment string that will be placed in the borealis files describing the slice. Defaults to empty string.

lag_table defaults

used in acf calculations. It is a list of lags. Example of a lag: [24, 27] from 8-pulse normalscan. This defaults to a lagtable built by the pulse sequence provided. All combinations of pulses will be calculated, with both the first pulses and last pulses used for lag-0.

pulse_phase_offset defaults

Allows phase shifting of pulses, enabling encoding of pulses. Default all zeros for all pulses in pulse_sequence. Pulses can be shifted with a single phase shift for each pulse or with a phase shift specified for each sample in the pulses of the slice.

range_sep defaults

a calculated value from pulse_len. If already set, it will be overwritten to be the correct value determined by the pulse_len. Used for acfs. This is the range gate separation, in azimuthal direction, in km.

rx_int_antennas defaults

The antennas to receive on in interferometer array, default is all antennas given max number from config.

rx_main_antennas defaults

The antennas to receive on in main array, default is all antennas given max number from config.

scanbound defaults

A list of seconds past the minute for integration times in a scan to align to. Defaults to None, not required.

seqoffset defaults

offset in us that this slice’s sequence will begin at, after the start of the sequence. This is intended for PULSE interfacing, when you want multiple slice’s pulses in one sequence you can offset one slice’s sequence from the other by a certain time value so as to not run both frequencies in the same pulse, etc. Default is 0 offset.

tx_antennas defaults

The antennas to transmit on, default is all main antennas given max number from config.

xcf defaults

flag for cross-correlation data. The default is True if acf is True, otherwise False.

Read-only Slice Keys

clrfrqflag read-only

A boolean flag to indicate that a clear frequency search will be done. Not currently supported.

cpid read-only

The ID of the experiment, consistent with existing radar control programs. This is actually an experiment-wide attribute but is stored within the slice as well. This is provided by the user but not within the slice, instead when the experiment is initialized.

rx_only read-only

A boolean flag to indicate that the slice doesn’t transmit, only receives.

slice_id read-only

The ID of this slice object. An experiment can have multiple slices. This is not set by the user but instead set by the experiment when the slice is added. Each slice id within an experiment is unique. When experiments start, the first slice_id will be 0 and incremented from there.

slice_interfacing read-only

A dictionary of slice_id : interface_type for each sibling slice in the experiment at any given time.

Not currently supported and will be removed

wavetype defaults

string for wavetype. The default is SINE. Not currently supported.

iwavetable defaults

a list of numeric values to sample from. The default is None. Not currently supported but could be set up (with caution) for non-SINE. Not currently supported.

qwavetable defaults

a list of numeric values to sample from. The default is None. Not currently supported but could be set up (with caution) for non-SINE. Not currently supported.

experiment_exception

This is the exception that is raised when there are problems with the experiment that cannot be remedied using experiment_prototype methods.

copyright

2018 SuperDARN Canada

author

Marci Detwiller

exception experiment_prototype.experiment_exception.ExperimentException(message, *args)

Bases: Exception

Is raised for the exception where an experiment cannot be run due to setup errors.

list_tests

Basic tests for use in checking slices.

copyright

2018 SuperDARN Canada

author

Marci Detwiller

experiment_prototype.list_tests.has_duplicates(list_to_check)

Check if the list has duplicate values.

Parameters

list_to_check – A list to check.

Returns

boolean True if duplicates exist, False if not.

experiment_prototype.list_tests.is_increasing(list_to_check)

Check if list is increasing.

Parameters

list_to_check – a list of numbers

Returns

boolean True if is increasing, False if not.

Subpackages

• experiment_prototype.scan_classes package
  • scan_class_base
  • scans
  • averaging_periods
  • sequences