central_scd_server

Automatically schedules new events from the SWG

usage: central_scd_server.py [-h] [--scd-dir SCD_DIR] [--force] [--first-run]

Named Arguments

--scd-dir

The scd working directory

--force

Force an update to the schedules for the next month

Default: False

--first-run

This will generate the first set of schedule files if running on a fresh directory. If the next month schedule is available, you will need to roll back the SWG schedule folder back to the last commit before running in continuous operation.

Default: False

make_atq.py

Schedules new SCD file entries

usage: make_atq.py [-h] scd_dir

Positional Arguments

scd_dir

The scd working directory

API

central_scd_server.py

Monitors for new SWG files and adds the SWG info to the scd if there is an update.

copyright:

2022 SuperDARN Canada

class scheduler.central_scd_server.SWG[source]

Bases: object

Holds the data needed for processing a SWG file.

__init__(scd_dir)[source]
new_swg_file_available()[source]

Checks if a new swg file is uploaded via git.

Returns:

True, if new git update is available.

Return type:

bool

parse_swg_to_scd(modes, radar, first_run)[source]

Reads the new SWG file and parses into a set of parameters than can be used for borealis scheduling.

Parameters:

  • modes (dict) – Holds the modes that correspond to the SWG requests.

  • radar (str) – Radar acronym.

  • first_run (bool) – Is this the first run? If so - start with current month, otherwise next month.

Returns:

List of all the parsed parameters.

Return type:

list

pull_new_swg_file()[source]

Uses git to grab the new scd updates.

scheduler.central_scd_server.main()[source]
scheduler.central_scd_server.parser()[source]

make_atq.py

This script takes the Borealis schedule file (ending in .scd) and converts the schedule into commands that are scheduled using the system at service.

Logs are printed to stdout. Specific logs for each time the schedule is updated are also created in borealis_schedules/logs/.

This script can be configured to run automatically when the .scd file is modified, using Linux services. Specifically, a .path and .service file are required, and templates for each given below. A pre-requisite for this setup is a file called .borealis.env defined in the user’s home directory, containing the following:

BOREALISPATH=/path/to/borealis/
RADAR_ID=sas  # or whatever you define it as
PYTHON_VERSION=3.12  # or whatever version you use

monitor-schedule.path:

[Unit]
Description="Monitor schedule file and update schedule when changed."

[Path]
PathChanged=/home/radar/borealis_schedules/lab.scd
Unit=schedule-radar.service

[Install]
WantedBy=multi-user.target

schedule-radar.service:

[Unit]
Description="Run script to schedule the Borealis radar"

[Service]
User=radar
Group=users
EnvironmentFile=/home/radar/.borealis.env
ExecStart=/bin/bash -c '${BOREALISPATH}/borealis_env${PYTHON_VERSION}/bin/python${PYTHON_VERSION} ${BOREALISPATH}/scheduler/make_atq.py /home/radar/borealis_schedules/'

[Install]
WantedBy=multi-user.target
scheduler.make_atq.backup_atq(time_of_interest, scd_dir, site_id)[source]

Creates a backup of the current atq schedule, saving it to [scd_dir]/atq_backup/YYYYmmdd-HHMM.[site_id].atq.

Parameters:

  • time_of_interest (dt.datetime) – Time that the schedule is being evaluated for.

  • scd_dir (str) – Path to the directory containing schedule files.

  • site_id (str) – Three-letter radar identifier, e.g. “sas”

scheduler.make_atq.make_schedule(scd_dir, site_id)[source]

Reads the schedule file and uses at to schedule all relevant lines from the schedule.

If the schedule is empty or contains invalid lines, it will send a Slack message alerting that the operation has failed.

Parameters:

  • scd_dir (str) – Path to the directory containing schedule files.

  • site_id (str) – Three-letter radar identifier, e.g. “sas”

scheduler.make_atq.parser()[source]
scheduler.make_atq.tee_print(msg, logfile)[source]

Prints to stdout and to a logfile.

Parameters:

msg (str)

scd_utils.py

Utilities for working with scd files

copyright:

2019 SuperDARN Canada

exception scheduler.scd_utils.ScheduleError[source]

Bases: Exception

Error in the schedule.

class scheduler.scd_utils.SCDUtils[source]

Bases: object

Contains utilities for working with SCD files. SCD files are schedule files for Borealis.

classmethod clear_atq()[source]

Remove all scheduled commands in the atq.

classmethod get_atq()[source]

Retrieve all scheduled commands in the atq.

classmethod timeline_to_atq(timeline)[source]

Converts the created timeline to actual atq commands.

Remove old events and then schedule everything recent. The first entry should be the currently running event, so it gets scheduled immediately.

Parameters:

timeline (list[ScheduleLine]) – A list holding all timeline events.

Returns:

output of the executed atq command

Return type:

str

static resolve_schedule(scd_lines, time_of_interest)[source]

Creates a true timeline from the scd lines, accounting for priority and duration of each line. Will reorder and add breaks to lines to account for differing priorities and durations. Keep the same line format.

Parameters:

  • scd_lines (list[ScheduleLine]) – All lines to schedule

  • time_of_interest (dt.datetime) – Time to use when finding all schedule lines that should run (i.e. now and in the future)

Returns:

All distinct scheduling events, in chronological order

Return type:

list[ScheduleLine]

__init__(scd_filename, site_id)[source]
Parameters:

  • scd_filename (str) – Schedule file name

  • site_id (str) – Three-letter radar site ID, for testing the experiment

add_line(
yyyymmdd,
hhmm,
experiment,
scheduling_mode,
prio=0,
duration='-',
kwargs=None,
embargo=False,
rawacf_format=None,
)[source]

Adds a new line to the schedule.

Parameters:

  • yyyymmdd (str) – year/month/day string.

  • hhmm (str) – hour/minute string.

  • experiment (str) – The experiment to run.

  • scheduling_mode (str) – The mode type running for this time period.

  • prio (int or str) – priority value. (Default value = 0)

  • duration (str) – duration to run for. (Default value = ‘-‘)

  • kwargs (list[str]) – kwargs for the experiment instantiation. (Default value = ‘’)

  • embargo (bool) – flag for embargoing files. (Default value = False)

  • rawacf_format (str) – File format to use when writing rawacf files.

Raises:

ValueError – If line parameters are invalid or if line is a duplicate.

create_line(
yyyymmdd,
hhmm,
experiment,
scheduling_mode,
prio,
duration,
kwargs,
embargo=False,
rawacf_format=None,
)[source]

Creates a line dictionary from inputs, turning the date and time into a timestamp since epoch.

Parameters:

  • yyyymmdd (str) – year/month/day string.

  • hhmm (str) – hour/minute string.

  • experiment (str) – The experiment to run.

  • scheduling_mode (str) – The type of scheduling mode.

  • prio (str or int) – priority value.

  • duration (Union[str, dt.timedelta]) – a duration to run for.

  • kwargs (list[str]) – kwargs for the experiment instantiation.

  • embargo (bool) – flag for embargoing files. (Default value = False)

  • rawacf_format (str) – The file format to save rawacf files in.

Returns:

Line details

Return type:

ScheduleLine

get_relevant_lines(time_of_interest)[source]

Gets the currently scheduled and future lines given a supplied time. If the provided time is equal to a scheduled line time, it provides that line and all future lines. If the provided time is between schedule line times, it provides any lines in the schedule from the past that haven’t ended yet, plus the most recently timestamped infinite-duration line, plus all future lines. If the provided time is before any lines in the schedule, it provides all schedule lines.

Parameters:

time_of_interest (dt.datetime) – Current time

Returns:

List of relevant lines.

Return type:

list[ScheduleLine]

Raises:

ScheduleError – If schedule file is empty.

parse_and_schedule(time_of_interest=None)[source]

Parses the schedule file, and schedules all relevant lines using at.

Parameters:

time_of_interest (dt.Datetime) – Current time.

Raises:

ScheduleError – if any of the relevant schedule lines are invalid, or the schedule file is empty.

read_scd()[source]

Read and parse the Borealis schedule file.

Returns:

list of dicts containing schedule info

Return type:

list(dict)

Raises:

  • ValueError – if any lines have obvious errors

  • OSError – if SCD file cannot be opened

remove_line(
yyyymmdd,
hhmm,
experiment,
scheduling_mode,
prio=0,
duration='-',
kwargs=None,
embargo=False,
rawacf_format=None,
)[source]

Removes a line from the schedule

Parameters:

  • yyyymmdd (str) – year/month/day string.

  • hhmm (str) – hour/minute string.

  • experiment (str) – The experiment to run.

  • scheduling_mode (str) – The mode type running for this time period.

  • prio (int or str) – priority value. (Default value = 0)

  • duration (str) – an optional duration to run for. (Default value = ‘-‘)

  • kwargs (list[str]) – kwargs for the experiment instantiation. (Default value = ‘’)

  • embargo (bool) – flag for embargoing files. (Default value = False)

  • rawacf_format (str) – File format to use when writing rawacf files.

Raises:

ValueError – If line parameters are invalid or if line does not exist.

write_scd(scd_lines)[source]

Creates SCD text lines and writes to file. Creates a backup of the old file before writing.

Raises:

PermissionError - When there are not sufficient permissions with the scd file FileNotFoundError - When the scd file doesn’t exist IsADirectoryError - When the scd file given is a directory

Parameters:

scd_lines (list(dict)) – A list of dicts that contain the schedule line info.

clear_command = "for i in `atq | awk '{print $1}'`;do atrm $i;done"

Clear the atq.

get_atq_cmd = '\n        for j in $(atq | sort -k6,6 -k3,3M -k4,4 -k5,5 | cut -f 1); do\n           atq |grep -P "^$j\t";\n           at -c "$j" | tail -n 2;\n        done\n    '

This command is basically: for j in atq job number, print job num, time and command More detail: sort the atq first by year, then month name (‘-M flag), then day of month Then hour, minute and second. Finally, just get the atq index (job #) in first column then, iterate through all jobs in the atq, list them to standard output, get the last 2 lines

scd_dt_fmt = '%Y%m%d %H:%M'

String format for parsing and writing datetimes

site_id

Default event to run if no other infinite duration line is scheduled

class scheduler.scd_utils.ScheduleLine[source]

Bases: object

classmethod check_duration(v)[source]

Verifies duration is either ‘-’, or a positive timedelta

Parameters:

v (str | timedelta)

Return type:

str | timedelta

classmethod check_timestamp(v)[source]

Verifies v is a time-aware datetime object

Parameters:

v (datetime)

Return type:

datetime

classmethod from_str(line)[source]

Parses a line from the schedule file

Parameters:

line (str)

Return type:

Self

__init__(*args, **kwargs)
Parameters:

  • __dataclass_self__ (PydanticDataclass)

  • args (Any)

  • kwargs (Any)

Return type:

None

check_inf_line_priority()[source]
Return type:

Self

format_to_atq(first_event_flag=False)[source]
test(site_id)[source]

Check validity of fields and run the line through experiment unit tests to check that the experiment will run.

Parameters:

site_id (str) – Three-letter site code

duration: str | timedelta
embargo: bool = False
experiment: str
kwargs: list[str] = FieldInfo(annotation=NoneType, required=False, default_factory=list)
priority: Annotated[int, FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=0), Le(le=20)])]
rawacf_format: Literal['dmap', 'hdf5'] | None = None
scheduling_mode: Literal['common', 'discretionary', 'special']
timestamp: datetime
scheduler.scd_utils.get_next_month_from_date(date=None)[source]

Finds the datetime of the next month.

Args

date - Default today. Datetime to get next month from

Returns:

TYPE: datetime object.