Scheduler Code

local_scd_server.py

Monitors for new SWG files and adds the SWG info to the scd if there is an update.

copyright

2022 SuperDARN Canada

class scheduler.local_scd_server.SWG(scd_dir)[source]

Bases: object

Holds the data needed for processing a SWG file.

new_swg_file_available()[source]

Checks if a new swg file is uploaded via git.

Returns

True, if new git update is available.

Return type

bool

parse_swg_to_scd(modes, radar, first_run)[source]

Reads the new SWG file and parses into a set of parameters than can be used for borealis scheduling.

Parameters

  • modes (dict) – Holds the modes that correspond to the SWG requests.

  • radar (str) – Radar acronym.

  • first_run (bool) – Is this the first run? If so - start with current month, otherwise next month.

Returns

List of all the parsed parameters.

Return type

list

pull_new_swg_file()[source]

Uses git to grab the new scd updates.

scheduler.local_scd_server.main()[source]

remote_server.py

This process runs on the Borealis computer at each radar site. This process should be running in the background whenever the radar is on, doing the following:

  • On start up, it schedules Borealis based on the existing schedule (.scd) file for the respective site. This is done using the Linux at service and the atq command.

  • Using inotify, remote_server.py then watches the .scd file for the respective site for any changes. If the .scd file is modified, the scheduled Borealis runs are updated.

Logs are printed to stdout. Specific logs for each time the schedule is updated are also created in borealis_schedules/logs/ and are emailed to verify if any issues occur.

copyright

2019 SuperDARN Canada

scheduler.remote_server.convert_scd_to_timeline(scd_lines, time_of_interest)[source]

Creates a true timeline from the scd lines, accounting for priority and duration of each line. Will reorder and add breaks to lines to account for differing priorities and durations. Keep the same line format.

Line dict keys are:

  • timestamp(ms since epoch)

  • time(datetime)

  • duration(minutes)

  • prio(priority)

  • experiment

  • scheduling_mode

The true timeline queued_lines dictionary differs from the scd_lines list by the following:

  • duration is parsed, adding in events so that all event durations are equal to the next event’s start time, subtract the current event’s start time.

  • priority is parsed so that there is only ever one event at any time (no overlap)

  • therefore the only event in the true timeline with infinite duration is the last event.

  • the keys of the true timeline dict are the original scd_lines order of the lines (integer). This allows the preservation of which events in the true timeline were scheduled in the same original line. This can be useful for plotting (same color = same scd scheduled line). The items in queued_lines dict are lists of all of the events corresponding to that original line’s order. These events have the same keys as the lines in scd_lines.

Parameters

  • scd_lines (list) – List of sorted lines by timestamp and priority, scd lines to try convert to a timeline.

  • time_of_interest (Datetime) – The datetime holding the time of scheduling.

Returns

Tuple containing the following items:

  • queued_lines: Groups of entries belonging to the same experiment.

  • warnings: List of warnings produced by the function

Return type

tuple(list, list)

scheduler.remote_server.format_to_atq(dt, experiment, scheduling_mode, first_event_flag=False, kwargs='', embargo=False)[source]

Turns an experiment line from the scd into a formatted atq command.

Parameters

  • dt (datetime) – Datetime of the experiment

  • experiment (str) – The experiment to run

  • scheduling_mode (str) – The scheduling mode to run

  • first_event_flag (bool) – Flag to signal whether the experiment is the first to run (Default value = False)

  • kwargs (str) – String of keyword arguments to run steamed hams (Default value = ‘’)

  • embargo (bool) – Option to embargo the data (makes the CPID negative)

Returns

Formatted atq str.

Return type

str

scheduler.remote_server.timeline_to_atq(timeline, scd_dir, time_of_interest, site_id)[source]

Converts the created timeline to actual atq commands.

Log and backup the existing atq, remove old events and then schedule everything recent. The first entry should be the currently running event, so it gets scheduled immediately. This function only backs up the commands that have not run yet.

Parameters

  • timeline (list) – A list holding all timeline events.

  • scd_dir (str) – The directory with SCD files.

  • time_of_interest (Datetime) – The datetime holding the time of scheduling.

  • site_id (str) – Site identifier for logs.

Returns

output of the executed atq command

Return type

bytes

scd_utils.py

Utilities for working with scd files

copyright

2019 SuperDARN Canada

class scheduler.scd_utils.SCDUtils(scd_filename, site_id)[source]

Bases: object

Contains utilities for working with SCD files. SCD files are schedule files for Borealis.

add_line(yyyymmdd, hhmm, experiment, scheduling_mode, prio=0, duration='-', kwargs='', embargo=False)[source]

Adds a new line to the schedule.

Parameters

  • yyyymmdd (str) – year/month/day string.

  • hhmm (str) – hour/minute string.

  • experiment (str) – The experiment to run.

  • scheduling_mode (str) – The mode type running for this time period.

  • prio (int or str) – priority value. (Default value = 0)

  • duration (str) – duration to run for. (Default value = ‘-‘)

  • kwargs (str) – kwargs for the experiment instantiation. (Default value = ‘’)

  • embargo (bool) – flag for embargoing files. (Default value = False)

Raises

ValueError – If line parameters are invalid or if line is a duplicate.

create_line(yyyymmdd, hhmm, experiment, scheduling_mode, prio, duration, kwargs='', embargo=False)[source]

Creates a line dictionary from inputs, turning the date and time into a timestamp since epoch.

Parameters

  • yyyymmdd (str) – year/month/day string.

  • hhmm (str) – hour/minute string.

  • experiment (str) – The experiment to run.

  • scheduling_mode (str) – The type of scheduling mode.

  • prio (str or int) – priority value.

  • duration (str) – an optional duration to run for.

  • kwargs (str) – kwargs for the experiment instantiation. Default None

  • embargo (bool) – flag for embargoing files. (Default value = False)

Returns

Dict of line params.

Return type

dict

classmethod fmt_line(line_dict)[source]

Formats a dictionary with line info into a text line for file.

Parameters

line_dict (dict) – A dict that holds all the line info.

Returns

Formatted string.

Return type

str

get_relevant_lines(yyyymmdd, hhmm)[source]

Gets the currently scheduled and future lines given a supplied time. If the provided time is equal to a scheduled line time, it provides that line and all future lines. If the provided time is between schedule line times, it provides any lines in the schedule from the past that haven’t ended yet, plus the most recently timestamped infinite-duration line, plus all future lines. If the provided time is before any lines in the schedule, it provides all schedule lines.

Parameters

  • yyyymmdd (str) – year/month/day string.

  • hhmm (str) – hour/minute string.

Returns

List of relevant dicts of line info.

Return type

list(dict)

Raises

  • ValueError – If datetime could not be created from supplied arguments.

  • IndexError – If schedule file is empty

line_fmt = '{datetime} {duration} {prio} {experiment} {scheduling_mode} {embargo} {kwargs}'
read_scd()[source]

Read and parse the Borealis schedule file.

Returns

list of dicts containing schedule info

Return type

list(dict)

Raises

  • ValueError – if any lines have obvious errors

  • OSError – if SCD file cannot be opened

remove_line(yyyymmdd, hhmm, experiment, scheduling_mode, prio=0, duration='-', kwargs='', embargo=False)[source]

Removes a line from the schedule

Parameters

  • yyyymmdd (str) – year/month/day string.

  • hhmm (str) – hour/minute string.

  • experiment (str) – The experiment to run.

  • scheduling_mode (str) – The mode type running for this time period.

  • prio (int or str) – priority value. (Default value = 0)

  • duration (str) – an optional duration to run for. (Default value = ‘-‘)

  • kwargs (str) – kwargs for the experiment instantiation. (Default value = ‘’)

  • embargo (bool) – flag for embargoing files. (Default value = False)

Raises

ValueError – If line parameters are invalid or if line does not exist.

scd_dt_fmt = '%Y%m%d %H:%M'

String format for scd line

site_id

Default event to run if no other infinite duration line is scheduled

test_line(line_dict)[source]

Check validity of fields and run the line through experiment unit tests to check that the experiment will run.

Parameters

line_dict (dict) – Dictionary of parameters for the line (as returned from create_line())

write_scd(scd_lines)[source]

Creates SCD text lines and writes to file. Creates a backup of the old file before writing.

Raises:

PermissionError - When there are not sufficient permissions with the scd file FileNotFoundError - When the scd file doesn’t exist IsADirectoryError - When the scd file given is a directory

Parameters

scd_lines (list(dict)) – A list of dicts that contain the schedule line info.

scheduler.scd_utils.get_next_month_from_date(date=None)[source]

Finds the datetime of the next month.

Args

date - Default today. Datetime to get next month from

Returns:

TYPE: datetime object.