Scheduler Code

local_scd_server.py

Monitors for new SWG files and adds the SWG info to the scd if there is an update.

copyright

2022 SuperDARN Canada

class scheduler.local_scd_server.SWG(scd_dir)[source]

Bases: object

Holds the data needed for processing a SWG file.

new_swg_file_available()[source]

Checks if a new swg file is uploaded via git.

Returns

True, if new git update is available.

Return type

bool

parse_swg_to_scd(modes, radar, first_run)[source]

Reads the new SWG file and parses into a set of parameters than can be used for borealis scheduling.

Parameters

  • modes (dict) – Holds the modes that correspond to the SWG requests.

  • radar (str) – Radar acronym.

  • first_run (bool) – Is this the first run? If so - start with current month, otherwise next month.

Returns

List of all the parsed parameters.

Return type

list

pull_new_swg_file()[source]

Uses git to grab the new scd updates.

scheduler.local_scd_server.main()[source]

remote_server.py

This process runs on the Borealis computer at each radar site. This process should be running in the background whenever the radar is on, doing the following:

  • On start up, it schedules Borealis based on the existing schedule (.scd) file for the respective site. This is done using the Linux at service and the atq command.

  • Using inotify, remote_server.py then watches the .scd file for the respective site for any changes. If the .scd file is modified, the scheduled Borealis runs are updated.

Logs are printed to stdout. Specific logs for each time the schedule is updated are also created in borealis_schedules/logs/ and are emailed to verify if any issues occur.

copyright

2019 SuperDARN Canada

scheduler.remote_server.format_to_atq(event: ScheduleLine, first_event_flag=False)[source]

Turns an experiment line from the scd into a formatted atq command.

Parameters

  • event (ScheduleLine) – Schedule entry

  • first_event_flag (bool) – Flag to signal whether the experiment is the first to run (Default value = False)

Returns

Formatted atq str.

Return type

str

scheduler.remote_server.resolve_schedule(scd_lines, time_of_interest)[source]

Creates a true timeline from the scd lines, accounting for priority and duration of each line. Will reorder and add breaks to lines to account for differing priorities and durations. Keep the same line format.

Parameters

scd_lines (list) – List of all lines to schedule

Returns

All distinct scheduling events, in chronological order

Return type

list

scheduler.remote_server.timeline_to_atq(timeline, scd_dir, time_of_interest, site_id)[source]

Converts the created timeline to actual atq commands.

Log and backup the existing atq, remove old events and then schedule everything recent. The first entry should be the currently running event, so it gets scheduled immediately. This function only backs up the commands that have not run yet.

Parameters

  • timeline (list) – A list holding all timeline events.

  • scd_dir (str) – The directory with SCD files.

  • time_of_interest (Datetime) – The datetime holding the time of scheduling.

  • site_id (str) – Site identifier for logs.

Returns

output of the executed atq command

Return type

bytes

scd_utils.py

Utilities for working with scd files

copyright

2019 SuperDARN Canada

class scheduler.scd_utils.LineConfig[source]

Bases: object

This class configures pydantic options for ScheduleLine.

validate_assignment: Whether to run all validators for a field whenever field is changed (init or after init) extra: Whether to allow extra fields not defined when instantiating arbitrary_types_allowed: Whether to allow arbitrary types like user-defined classes (e.g. Options, DecimationScheme)

arbitrary_types_allowed = False
extra = 'allow'
validate_assignment = True
class scheduler.scd_utils.SCDUtils(scd_filename, site_id)[source]

Bases: object

Contains utilities for working with SCD files. SCD files are schedule files for Borealis.

add_line(yyyymmdd, hhmm, experiment, scheduling_mode, prio=0, duration='-', kwargs=None, embargo=False, rawacf_format=None)[source]

Adds a new line to the schedule.

Parameters

  • yyyymmdd (str) – year/month/day string.

  • hhmm (str) – hour/minute string.

  • experiment (str) – The experiment to run.

  • scheduling_mode (str) – The mode type running for this time period.

  • prio (int or str) – priority value. (Default value = 0)

  • duration (str) – duration to run for. (Default value = ‘-‘)

  • kwargs (list[str]) – kwargs for the experiment instantiation. (Default value = ‘’)

  • embargo (bool) – flag for embargoing files. (Default value = False)

  • rawacf_format (str) – File format to use when writing rawacf files.

Raises

ValueError – If line parameters are invalid or if line is a duplicate.

create_line(yyyymmdd, hhmm, experiment, scheduling_mode, prio, duration, kwargs, embargo=False, rawacf_format=None) → ScheduleLine[source]

Creates a line dictionary from inputs, turning the date and time into a timestamp since epoch.

Parameters

  • yyyymmdd (str) – year/month/day string.

  • hhmm (str) – hour/minute string.

  • experiment (str) – The experiment to run.

  • scheduling_mode (str) – The type of scheduling mode.

  • prio (str or int) – priority value.

  • duration (Union[str, dt.timedelta]) – a duration to run for.

  • kwargs (list[str]) – kwargs for the experiment instantiation.

  • embargo (bool) – flag for embargoing files. (Default value = False)

  • rawacf_format (str) – The file format to save rawacf files in.

Returns

Line details

Return type

ScheduleLine

get_relevant_lines(yyyymmdd, hhmm)[source]

Gets the currently scheduled and future lines given a supplied time. If the provided time is equal to a scheduled line time, it provides that line and all future lines. If the provided time is between schedule line times, it provides any lines in the schedule from the past that haven’t ended yet, plus the most recently timestamped infinite-duration line, plus all future lines. If the provided time is before any lines in the schedule, it provides all schedule lines.

Parameters

  • yyyymmdd (str) – year/month/day string.

  • hhmm (str) – hour/minute string.

Returns

List of relevant dicts of line info.

Return type

list(dict)

Raises

  • ValueError – If datetime could not be created from supplied arguments.

  • IndexError – If schedule file is empty

read_scd()[source]

Read and parse the Borealis schedule file.

Returns

list of dicts containing schedule info

Return type

list(dict)

Raises

  • ValueError – if any lines have obvious errors

  • OSError – if SCD file cannot be opened

remove_line(yyyymmdd, hhmm, experiment, scheduling_mode, prio=0, duration='-', kwargs=None, embargo=False, rawacf_format=None)[source]

Removes a line from the schedule

Parameters

  • yyyymmdd (str) – year/month/day string.

  • hhmm (str) – hour/minute string.

  • experiment (str) – The experiment to run.

  • scheduling_mode (str) – The mode type running for this time period.

  • prio (int or str) – priority value. (Default value = 0)

  • duration (str) – an optional duration to run for. (Default value = ‘-‘)

  • kwargs (list[str]) – kwargs for the experiment instantiation. (Default value = ‘’)

  • embargo (bool) – flag for embargoing files. (Default value = False)

  • rawacf_format (str) – File format to use when writing rawacf files.

Raises

ValueError – If line parameters are invalid or if line does not exist.

scd_dt_fmt = '%Y%m%d %H:%M'
site_id

Default event to run if no other infinite duration line is scheduled

write_scd(scd_lines)[source]

Creates SCD text lines and writes to file. Creates a backup of the old file before writing.

Raises:

PermissionError - When there are not sufficient permissions with the scd file FileNotFoundError - When the scd file doesn’t exist IsADirectoryError - When the scd file given is a directory

Parameters

scd_lines (list(dict)) – A list of dicts that contain the schedule line info.

class scheduler.scd_utils.ScheduleLine(timestamp: ~datetime.datetime, duration: ~typing.Union[str, ~datetime.timedelta], experiment: str, priority: int, scheduling_mode: ~typing.Literal['common', 'discretionary', 'special'], kwargs: list[str] = <factory>, embargo: bool = False, rawacf_format: ~typing.Optional[~typing.Literal['dmap', 'hdf5']] = None)[source]

Bases: object

classmethod check_duration(v: Union[str, timedelta]) → Union[str, timedelta][source]

Verifies duration is either ‘-‘, or a positive timedelta

check_inf_line_priority() → Self[source]
classmethod check_timestamp(v: datetime) → datetime[source]

Verifies v is a time-aware datetime object

duration: Union[str, timedelta]
embargo: bool = False
experiment: str
format_to_atq()[source]
classmethod from_str(line: str) → Self[source]

Parses a line from the schedule file

kwargs: list[str] = FieldInfo(annotation=NoneType, required=False, default_factory=list)
priority: int
rawacf_format: Optional[Literal['dmap', 'hdf5']] = None
scheduling_mode: Literal['common', 'discretionary', 'special']
test(site_id: str)[source]

Check validity of fields and run the line through experiment unit tests to check that the experiment will run.

Parameters

site_id (str) – Three-letter site code

timestamp: datetime
scheduler.scd_utils.get_next_month_from_date(date=None)[source]

Finds the datetime of the next month.

Args

date - Default today. Datetime to get next month from

Returns:

TYPE: datetime object.