Rodrigo Asensio 5 years ago

1 min read

When you are not willing to pay for monitoring services like AppDynamics (80k /y ) or UptimeMonitor (monthly fee) you still have a cheap shell script based solution.

When you are not willing to pay for monitoring services like AppDynamics (80k /y ) or UptimeMonitor (monthly fee) you still have a cheap shell script based solution.

Shells scripts and Slack are your friends.

Scenario

You have a Tomcat webserver running and you want to take action if it dies and notify of the action to a group of people.

Tools

Crontab, Shell script and Slack

Step 1: Setup crontab

In your linux server add a new crontab crontab -e and edit the following check every minute.

*/1 * * * * /opt/scripts/check.sh

Now, restart your cron service

service crond restart

Step 2: Create the check

Create a new shell script in /opt/scripts/check.sh with the following code:

#! /bin/sh
SERVICE=/etc/init.d/tomcat8

if $SERVICE status | grep -q "not running"; then
    $SERVICE start
    /opt/scripts/slackpost.sh "https://hooks.slack.com/services/your-hook" "#monitor" "Tomcat Automatic Restart"
fi
if $SERVICE status | grep -q "stopped"; then
    $SERVICE start
    /opt/scripts/slackpost.sh "https://hooks.slack.com/services/your-hook" "#monitor" "Tomcat Automatic Restart"
fi

This will find if the service crashed or was not started and it will start it. After staring the service it will send a notification to your slack channel (you can strip this piece if no slack desired)

Step 3: Notify to Slack

Create another script in /opt/scripts/slackpost.sh with the following content:

#!/bin/bash

# Usage: slackpost "" "" ""

webhook_url=$1
if [[ $webhook_url == "" ]]
then
        echo "No webhook_url specified"
        exit 1
fi

shift
channel=$1
if [[ $channel == "" ]]
then
        echo "No channel specified"
        exit 1
fi

shift

text=$*

if [[ $text == "" ]]
then
        echo "No text specified"
        exit 1
fi

escapedText=$(echo $text | sed 's/"/\"/g' | sed "s/'/\'/g" )
json="{\"channel\": \"$channel\", \"text\": \"$escapedText\"}"

curl -s -d "payload=$json" "$webhook_url"

And finally chmod +x /opt/scripts/*sh to make both scripts runnables.

Happy coding!

The code can be found @ https://github.com/rasensio/poors-man-monitoring-and-recovery

Rodrigo Asensio

Rodrigo Asensio is Manager of Solution Architecture at Amazon Web Services. He has more than 20 years of experience designing and operating distributed solutions. He is currently responsible for a team in the Enterprise segment helping large clients accelerate their adoption of the cloud and optimize the utilization of their resources.