Logging SSH logins to Slack

I’m using Slack to alert and log a few things in my environment, and one of the things I use it for is to alert me if someone logs on via SSH to my public facing Jumphost.

For a good walkthrough on how to set up such a host, check out Tunnel all your remote connections through ssh with a linux jumpbox by Luca Dell’Oca.

My Ubuntu 16.04 Jumphost is set up to only accept Key-Based Authentication, to secure it as much as possible, but I would still like to get instant notification if someone logs into it interactively.

How to set up SSH login notification to Slack.

  1. screenshot-2016-10-06-13-04-51First of all, we need  an Incoming WebHook in Slack in order to receive the notifications.
    You configure those from the Apps & Integration menu item. This in turn opens up the Slack App Directory, find Build on the top right and then choose Make a Custom Integration.
  2. screenshot-2016-10-06-13-08-09One your are in the Build a Custom Integration section, find (or search) Incoming WebHooks and select that.
  3. Next up, define which Slack channel should be the integration point, and click on Add Incoming WebHooks integration.
  4. Copy the Webhook URL presented on the next screen
    Note: keep this one a secret, anyone with access to this URL will be able to post to your Slack channel.
  5. On my Ubuntu 16.04 Linux jumphost I’ve created a small bash script called /etc/ssh/notify.sh. This script utilizes curl  and the WebHook URL to post information directly to Slack. The script looks like this:notify.sh
    [cc lang=”bash” escaped=”true”]
    #!/bin/sh
    url=”https://hooks.slack.com/services/*********”
    channel=”#messages”
    host=”`hostname`”
    content=”\”attachments\”: [ { \”mrkdwn_in\”: [\”text\”, \”fallback\”], \”fallback\”: \”SSH login: $USER connected to \`$host\`\”, \”text\”: \”SSH login to \`$host\`\”, \”fields\”: [ { \”title\”: \”User\”, \”value\”: \”$USER\”, \”short\”: true }, { \”title\”: \”IP Address\”, \”value\”: \”$SSH_CLIENT\”, \”short\”: true } ], \”color\”: \”#F35A00\” } ]”
    curl -s -X POST –data-urlencode “payload={\”channel\”: \”$channel\”, \”mrkdwn\”: true, \”username\”: \”ssh-bot\”, $content, \”icon_emoji\”: \”:computer:\”}” $url
    /bin/bash
    [/cc]Replace the  the WebHook URL with your own from step 4 and which channel to post to and you should be ready to go.  This script logs the username and the IP address the connection comes from, and then posts it to the Slack WebHook with the help of curl.Note: I’ve chosen to include the WebHook name etc in the script itself, instead of via the WebHook definition on Slack, mostly since I don’t want to create a WebHook for all hosts I want logging from. With this setup, I can just change the username part of the curl command. It already logs the hostname, so this is pretty much superficial, but hey, that’s how I made it.
  6. chmod +x /etc/ssh/notify.sh to make it executable, and test it. If everything works as expected, you should see an immediate log entry in your chosen Slack channel.
  7. On order to make this script runs every time someone logs into the Jumphost, I added a ForceCommand to the end of my /etc/ssh/sshd_config file, like this:[cc lang=”bash” escaped=”true”]
    ForceCommand /etc/ssh/notify.sh
    [/cc]

And that’s it. A login via SSH on the Jumphost now looks like this in my Slack channel:

screenshot-2016-10-06-13-26-10

How awesome is that? Of course, this just scratches the surface of what is possible with Slack’s Incoming WebHooks, I’m using a similar approach for logging new devices discovered in phpmyipam but that’s for another post.

HomeLabOps via Slack

Just like Lior Kamrat I’ve set up my own private Slack for messaging and alerting from various services running both in my lab and some external facing services.  It’s only been running a few days, but so far it works brilliantly and helps me keep track.

So far, I’ve set up Slack alerting and/or integrations for the following items:

I’m sure I’ll add other things to this as time passes. I plan on publishing something on how I’ve hacked some of this together,  I just need to clean up the code a bit and make it ready for publication first.

 

 

Homelab: CloudFlare Dynamic DNS Update Script (cf-ddns.sh)

As a part of my Homelab project, I’ve created a proper bash script to provide dynamic DNS updates for external resources, via CloudFlare. More details on the reasoning behind it can be found in Using CloudFlare for Dynamic DNS, but since that was posted I’ve fleshed the script out quite a bit more.

The new and updated script can be found on GitHub: cf-ddns.sh. It now writes events to syslog when it runs, so you can use VMware Log Insight (or another log solution) to capture the events, and use it to track public IP changes if you want. I’ve also added a -f parameter to it, so you can force an IP update even if the values have not changed since the last run.

It’s pretty self explanatory, just replace your own values from CloudFlare for the variables, and if you want to update more than one record, just copy the update block and edit it for the extra entries.

Hopefully someone will find this useful.