Scheduler Code¶
local_scd_server.py¶
Monitors for new SWG files and adds the SWG info to the scd if there is an update.
- copyright
2022 SuperDARN Canada
- class scheduler.local_scd_server.SWG(scd_dir)[source]¶
Bases:
object
Holds the data needed for processing a SWG file.
- new_swg_file_available()[source]¶
Checks if a new swg file is uploaded via git.
- Returns
True, if new git update is available.
- Return type
remote_server.py¶
This process runs on the Borealis computer at each radar site. This process should be running in the background whenever the radar is on, doing the following:
On start up, it schedules Borealis based on the existing schedule (.scd) file for the respective site. This is done using the Linux at service and the atq command.
Using inotify, remote_server.py then watches the .scd file for the respective site for any changes. If the .scd file is modified, the scheduled Borealis runs are updated.
Logs are printed to stdout. Specific logs for each time the schedule is updated are also created in borealis_schedules/logs/ and are emailed to verify if any issues occur.
- copyright
2019 SuperDARN Canada
- scheduler.remote_server.convert_scd_to_timeline(scd_lines, time_of_interest)[source]¶
Creates a true timeline from the scd lines, accounting for priority and duration of each line. Will reorder and add breaks to lines to account for differing priorities and durations. Keep the same line format.
- Line dict keys are:
timestamp
duration (rounded down to minutes)
priority
experiment
scheduling_mode
The true timeline queued_lines dictionary differs from the scd_lines list by the following:
duration is parsed, adding in events so that all event durations are equal to the next event’s start time, subtract the current event’s start time.
priority is parsed so that there is only ever one event at any time (no overlap)
therefore the only event in the true timeline with infinite duration is the last event.
the keys of the true timeline dict are the original scd_lines order of the lines (integer). This allows the preservation of which events in the true timeline were scheduled in the same original line. This can be useful for plotting (same color = same scd scheduled line). The items in queued_lines dict are lists of all of the events corresponding to that original line’s order. These events have the same keys as the lines in scd_lines.
- Parameters
scd_lines (list) – List of sorted lines by timestamp and priority, scd lines to try convert to a timeline.
time_of_interest (Datetime) – The datetime holding the time of scheduling.
- Returns
Tuple containing the following items:
queued_lines: Groups of entries belonging to the same experiment.
warnings: List of warnings produced by the function
- Return type
- scheduler.remote_server.format_to_atq(dt, experiment, scheduling_mode, first_event_flag=False, kwargs='', embargo=False, rawacf_format=None)[source]¶
Turns an experiment line from the scd into a formatted atq command.
- Parameters
dt (datetime) – Datetime of the experiment
experiment (str) – The experiment to run
scheduling_mode (str) – The scheduling mode to run
first_event_flag (bool) – Flag to signal whether the experiment is the first to run (Default value = False)
kwargs (str) – String of keyword arguments to run steamed hams (Default value = ‘’)
embargo (bool) – Option to embargo the data (makes the CPID negative)
rawacf_format – File format to use when writing rawacf files.
rawacf_format – str
- Returns
Formatted atq str.
- Return type
- scheduler.remote_server.timeline_to_atq(timeline, scd_dir, time_of_interest, site_id)[source]¶
Converts the created timeline to actual atq commands.
Log and backup the existing atq, remove old events and then schedule everything recent. The first entry should be the currently running event, so it gets scheduled immediately. This function only backs up the commands that have not run yet.
- class scheduler.scd_utils.LineConfig[source]¶
Bases:
object
This class configures pydantic options for ScheduleLine.
validate_assignment: Whether to run all validators for a field whenever field is changed (init or after init) extra: Whether to allow extra fields not defined when instantiating arbitrary_types_allowed: Whether to allow arbitrary types like user-defined classes (e.g. Options, DecimationScheme)
- arbitrary_types_allowed = False¶
- extra = 'allow'¶
- validate_assignment = True¶
- class scheduler.scd_utils.SCDUtils(scd_filename, site_id)[source]¶
Bases:
object
Contains utilities for working with SCD files. SCD files are schedule files for Borealis.
- add_line(yyyymmdd, hhmm, experiment, scheduling_mode, prio=0, duration='-', kwargs=None, embargo=False, rawacf_format=None)[source]¶
Adds a new line to the schedule.
- Parameters
yyyymmdd (str) – year/month/day string.
hhmm (str) – hour/minute string.
experiment (str) – The experiment to run.
scheduling_mode (str) – The mode type running for this time period.
duration (str) – duration to run for. (Default value = ‘-‘)
kwargs (list[str]) – kwargs for the experiment instantiation. (Default value = ‘’)
embargo (bool) – flag for embargoing files. (Default value = False)
rawacf_format (str) – File format to use when writing rawacf files.
- Raises
ValueError – If line parameters are invalid or if line is a duplicate.
- create_line(yyyymmdd, hhmm, experiment, scheduling_mode, prio, duration, kwargs, embargo=False, rawacf_format=None) ScheduleLine [source]¶
Creates a line dictionary from inputs, turning the date and time into a timestamp since epoch.
- Parameters
yyyymmdd (str) – year/month/day string.
hhmm (str) – hour/minute string.
experiment (str) – The experiment to run.
scheduling_mode (str) – The type of scheduling mode.
duration (Union[str, dt.timedelta]) – a duration to run for.
kwargs (list[str]) – kwargs for the experiment instantiation.
embargo (bool) – flag for embargoing files. (Default value = False)
rawacf_format (str) – The file format to save rawacf files in.
- Returns
Line details
- Return type
- get_relevant_lines(yyyymmdd, hhmm)[source]¶
Gets the currently scheduled and future lines given a supplied time. If the provided time is equal to a scheduled line time, it provides that line and all future lines. If the provided time is between schedule line times, it provides any lines in the schedule from the past that haven’t ended yet, plus the most recently timestamped infinite-duration line, plus all future lines. If the provided time is before any lines in the schedule, it provides all schedule lines.
- Parameters
- Returns
List of relevant dicts of line info.
- Return type
- Raises
ValueError – If datetime could not be created from supplied arguments.
IndexError – If schedule file is empty
- read_scd()[source]¶
Read and parse the Borealis schedule file.
- Returns
list of dicts containing schedule info
- Return type
- Raises
ValueError – if any lines have obvious errors
OSError – if SCD file cannot be opened
- remove_line(yyyymmdd, hhmm, experiment, scheduling_mode, prio=0, duration='-', kwargs=None, embargo=False, rawacf_format=None)[source]¶
Removes a line from the schedule
- Parameters
yyyymmdd (str) – year/month/day string.
hhmm (str) – hour/minute string.
experiment (str) – The experiment to run.
scheduling_mode (str) – The mode type running for this time period.
duration (str) – an optional duration to run for. (Default value = ‘-‘)
kwargs (list[str]) – kwargs for the experiment instantiation. (Default value = ‘’)
embargo (bool) – flag for embargoing files. (Default value = False)
rawacf_format (str) – File format to use when writing rawacf files.
- Raises
ValueError – If line parameters are invalid or if line does not exist.
- scd_dt_fmt = '%Y%m%d %H:%M'¶
- site_id¶
Default event to run if no other infinite duration line is scheduled
- write_scd(scd_lines)[source]¶
Creates SCD text lines and writes to file. Creates a backup of the old file before writing.
- Raises:
PermissionError - When there are not sufficient permissions with the scd file FileNotFoundError - When the scd file doesn’t exist IsADirectoryError - When the scd file given is a directory
- class scheduler.scd_utils.ScheduleLine(timestamp: ~datetime.datetime, duration: ~typing.Union[str, ~datetime.timedelta], experiment: str, priority: int, scheduling_mode: ~typing.Literal['common', 'discretionary', 'special'], kwargs: list[str] = <factory>, embargo: bool = False, rawacf_format: ~typing.Optional[~typing.Literal['dmap', 'hdf5']] = None)[source]¶
Bases:
object
- classmethod check_duration(v: Union[str, timedelta]) Union[str, timedelta] [source]¶
Verifies duration is either ‘-‘, or a positive timedelta