Preemptable instances are cheaper than normal ones but can automatically disappear. To deal with it, you not only need to make sure your data pipeline saves your progress at regular checkpoints but also to get notified when this unfortunate event happens. This post will give you step-by-step instructions on how to set up Slack webhooks to push a notification to yourself when the machine is stopped.

When an instance is preempted on GCP, you have 30 seconds to execute a bash script. We will set this script up to create a POST message with curl to a Slack channel of your own. I will also show further settings that makes the setup automatic.

Create a Slack app and enable webhooks

  • Visit the following page: https://api.slack.com/apps
  • Create New App (top right)
  • Specify appname and workspace (You will see the messages coming from “appname”)
  • Activate incoming webhooks
  • Add new webhook to workspace
  • At the “‘appname’ is requesting permission to access the ‘workspace’ Slack workspace” select a channel or person to send the message to
  • Copy the entire curl into a terminal (anywhere) and execute it to test if it works.

And that’s it your webhook is created.

Set up the environment

When you are SSH into a remote machine, some of your environmental variables are copied over. We will set a variable to carry your webhook URL to the remote machine each time you log in, so you don’t need to store it anywhere on that machine.

Add the following line to your LOCAL .zshrc or .bashrc depending on which shell you are using (with the full URL of course from the previous point). It is very important that your variable starts with LC_; These variables are automatically copied to the remote machine. This, of course, has a finer control, but it is out of scope here, and by default on Linux this will work (see Sources below).

export LC_SLACK_URL="https://hooks.slack.com/services/..."

Log in to the remote machine and test it with echo $LC_SLACK_URL, it should show the full URL from above.

Create the shutdown script

The best place to put this is the Settings repo. There is a problem, though. When your machine is shut down the user that will be running the shutdown script will not be your user, therefore the environment variable not going to work, you need to use the actual string. This is not ideal but solvable.

  • Create a shutdown_script.sh in the repo with the following content:
curl -X POST -H 'Content-type: application/json' --data '{"text":"shutdown"}' SLACK_URL
  • In the make_settings.sh add the following two lines:
cp shutdown_script.sh ..
sed -i "s|SLACK_URL|${LC_SLACK_URL}|" ../shutdown_script.sh

This will copy the file and replace the placeholder with the actual URL.

Add the shutdown script to the instance

  • Go to the instances on GCP: VM instances
  • Select the relevant instance
  • Click EDIT on the top
  • At “Custom metadata” add the following key-value pair (substitute your login name):
    • key: shutdown-script
    • value: /home/your_username/shutdown_script.sh
  • Click SAVE at the bottom

And that’s it, I’d use the full path because again the script will not run under your user.

Use the webhook from python

Now that the webhook exists, it can be used to notify you if a checkpoint is reached in processing. The snippet below allows you to send a message to yourself from any python script:

import os
import requests
def slack_message(message):
    response = requests.post(
        url=os.environ['LC_SLACK_URL'],
        headers={'Content-type': 'application/json'},
        data=f'{{"text":"{message}"}}'
    )

There are many options to customise the above content: send messages to team channels, send different messages from different machines, create multiple webhooks, etc.

Join our Discord server for more tips and tricks.

Sources: