=============================== Starting and Stopping the Radar =============================== --------------- Manual Start-up --------------- To start the radar, there is a script called ``steamed_hams.py``. The name of this script is a reference to a scene in an episode of The Simpsons in which Principal Skinner claims there is an aurora happening in his house. Usage ~~~~~ .. argparse:: :module: scripts.steamed_hams :func: steamed_hams_parser :prog: steamed_hams.py An example invocation to run ``twofsound`` in ``release`` mode during ``common`` time would be:: $BOREALISPATH/scripts/steamed_hams.py twofsound release common .. note:: This script will kill the Borealis software if it is currently running, before it starts it anew. The experiment name must match to an experiment in the ``src/borealis_experiments`` folder, and does not include the ``.py`` extension. The scheduling mode is one of ``common``, ``special``, or ``discretionary``, depending upon the DARN-SWG schedule (see the scheduling working group page `here `_) The script will boot all the radar processes in a detached ``screen`` window that runs in the background. This window can be reattached in any terminal window locally or over ssh (``screen -r``) to track any outputs if needed. To start the radar without the optional ``realtime`` module, pass the flag ``--realtime-off`` and the module will not be run. For example:: /home/radar/borealis/scripts/steamed_hams.py normalscan release discretionary --realtime-off If starting the radar in normal operation according to the schedule, there is a helper script called ``scripts/start_radar.sh``. ------------------ Automated Start-up ------------------ In order to start the radar automatically, the script ``start_radar.sh`` should be added to a startup script of the Borealis computer. It can also be called manually by the non-root user (typically ``radar``). The scheduling Python script, ``make_atq.py``, is responsible for automating the control of the radar to follow the schedule, and is started via the ``start_radar.sh`` script with the appropriate arguments. This script should be added to the control computer boot-up scripts so that it generates a new set of scheduled commands. .. _automated-restarts: ------------------ Automated Restarts ------------------ Occasionally, the Borealis software stops due to some software or computer issue. To automatically restart the radar software when this occurs, and to avoid lengthy downtimes, the scripts ``restart_borealis.daemon`` and ``restart_borealis.py`` were created. ``restart_borealis.py`` finds the directory Borealis writes to and checks the file most recently written to. If the file hasn't been written to within a specified time period, the script assumes the radar has stopped running and tries to restart it using ``stop_radar.sh`` and ``start_radar.sh``. ``restart_borealis.daemon`` runs continuously, periodically executing ``restart_borealis.py``. If the radar is restarted consecutive times, an alert is sent to our group's Slack workspace to notify us that the radar likely has a problem requiring manual intervention. For more information on integrating Slack alerts, see `here `__. To set up the daemon using ``systemd``: #. Copy ``borealis/scripts/restart_borealis.daemon`` to ``/usr/local/bin/``. Ensure the file can be executed by the radar operating user (i.e. ``radar``). #. Copy ``borealis/scripts/restart_borealis.service`` to ``/usr/lib/systemd/system/``. Modify the user specified in the service file to match the radar operating user. The service file is shown below: .. literalinclude:: ../../scripts/restart_borealis.service :linenos: :language: ini :caption: restart_borealis.service #. Enable and start the service: :: sudo systemctl enable restart_borealis.service sudo systemctl start restart_borealis.service systemctl status restart_borealis.service Alternatively, ``restart_borealis.py`` can be run via ``crontab``, as shown below: :: */10 * * * * . $HOME/.profile; /usr/bin/python3 /home/radar/borealis/scripts/restart_borealis.py >> /home/radar/borealis/restart_log.txt 2>&1 ------------------ Stopping the Radar ------------------ There are several ways to stop the Borealis radar. They are ranked here from most acceptable to last-resort: #. Run the script ``stop_radar.sh`` from the Borealis ``scripts/`` directory. This script kills the scheduling server, removes all scheduled entries from the ``at`` queue and kills the screen session running the Borealis software modules. #. While viewing the screen session running the Borealis software modules, type ``ctrl-A, ctrl-\\``. This will kill the screen session and all software modules running within it. #. Restart the Borealis computer. **NOTE** In a normal circumstance, the Borealis software will start back up again once the computer reboots. #. Shut down the Borealis computer. .. _ups-power-outages: ------------------- UPS & Power Outages ------------------- To protect the Borealis computer from power outages and ensure the computer can safely turn off, the computer should be powered by an Uninterruptible Power Supply (UPS). Additionally, powering the Borealis hardware (N200s, octoclocks, and network equipment) with the UPS will mitigate potential radar restarts due to power brownouts or short power outages. In this scenario, the UPS should shut temporarily turn off the radar while the Borealis equipment is on battery power during the power outage (since the transmitters will be powered off). This can be done as follows: 1. Use ``apcupsd`` to communicate between the radar computer and the UPS. Follow the `APCUPSD User Manual `__ to install and configure for your setup. 2. Copy the ``offbattery`` and ``onbattery`` scripts from ``borealis/scripts/apcupsd/`` to ``/etc/apcupsd/``. These scripts will be executed when each event occurs on the UPS: 1. ``onbattery``: This occurs when the power outage starts. This script will schedule the radar to turn off via ``stop_radar.sh``, and stop the ``restart_borealis.service`` daemon so the radar doesn't restart during the power outage. 2. ``offbattery``: This occurs when the power outage ends. This script will cancel the scheduled ``stop_radar.sh`` script call, and restart the ``restart_borealis.service`` daemon.