Linux and Nagios

SSH Reverse Tunnel

posted by Tristan Self

If you need to SSH to a host but don't have direct SSH access, you can perform a reverse SSH tunnel. For this you need a host that will except inbound connections to work as a "jump host", it is possible without it, where you could SSH back to your client computer; however for the purposes of this example the setup is as follows:

1. TARGETSRV - Target host you want to connect to, you will start the SSH reverse tunnel from here.
2. JUMPHOST - The host you'll be connecting the reverse tunnel to from the TARGETSRV.
3. CLIENT - Your client computer that you'll be SSHing from to the JUMPHOST.

The first step is to start the SSH reverse tunnel from the TARGETSRV. Here we are creating a tunnel from port 22 on the TARGETSRV to port 19999 on the JUMPHOST over port 22 SSH.

# ssh -p 22 -R 19999:localhost:22 JUMPHOST -l <user>

Now SSH to the JUMPHOST from CLIENT and run the following command:

ssh -p 19999 -l <user>

Now you will have connected to the TARGETSRV down the reverse tunnel.Especially useful if your target host is behind a firewall where direct access is not possible.

Raspberry Pi Time Synch

posted 22 May 2017, 12:50 by Tristan Self   [ updated ]

The Raspberry Pi lacks a real time clock, which means that the time can drift, so much so that NTP fails to update the software clock because its skewed so much.

To resolve this you can do the following:

Add this to /etc/ntp.conf:

tinker panic 0

That will cause ntpd to sync despite the large clock offset.

sudo service ntpd stop
sudo ntpd -gq
sudo service ntpd start

Now run a "date" and check the time has synched as expected.

You can also try to create a cron job to force the synch to a remote server if the above still does not keep it in synch, Setting this every 5 minutes or 15 minutes interval should do the trick.

/etc/init.d/ntp stop
/usr/sbin/ntpdate -s
/etc/init.d/ntp start

NagiosXI check_ldap Error - Could not bind to the LDAP server

posted 4 Jan 2017, 03:25 by Tristan Self

When attempting to use the check_ldap plugin, I found that unsecured LDAP lookups on port 389/TCP worked fine, but attempting a secure lookup on 636 or using TLS failed.

Attempting a check_ldap check normally worked fine (i.e. to port 389), but attempting an LDAPS or LDAP TLS check failed with the following error:

# /usr/local/nagios/libexec/check_ldaps -H <HOSTNAME> -p 636 -S -a "(objectclass=organizationalUnit)" -b "dc=domain,dc=co,dc=uk" -3 -v
ldap_bind: Can't contact LDAP server (-1)
        additional info: TLS error -8172:Peer's certificate issuer has been marked as not trusted by the user.
Could not bind to the LDAP server

The check_ldap plugin makes use of OpenLDAP, the OpenLDAP package is installed as part of the NagiosXI installation because the plugins have dependencies on it but it is left in a non-configured state.

To resolve the problem on each node (wtgc-nagios-01 and wtgc-nagios-02) the following is required, firstly edit the file: /etc/openldap/ldap.conf and at the bottom of the file add the following line:


Then performing the check again gives the expected response:

# /usr/local/nagios/libexec/check_ldaps -H <HOSTNAME> -p 636 -S -a "(objectclass=organizationalUnit)" -b "dc=domain,dc=co,dc=uk" -3 -v
LDAP OK - 0.050 seconds response time|time=0.049688s;;;0.000000

Setup SSH Gateway on Raspberry Pi

posted 19 Dec 2016, 13:17 by Tristan Self   [ updated 19 Dec 2016, 13:24 ]

So looking for something to use your Raspberry Pi for? I had a Mk1 Raspberry Pi laying around and thought it might be good to set it up as a SSH gateway, so I could easily and securely access my home machines when out and about.

These instructions assume you have a Raspberry Pi which has Raspbian Jessie installed, i'm assuming you've also enabled SSH and have a named account created which you have already used to SSH to the Raspberry Pi over your network.

These instructions also get you to set SSH onto a non-standard port, setup two factor authentication using Google's authenticator and setup Fail2Ban to ensure hacking attempts can be blocked somewhat.

Assuming you've done this, you are ready to start, note you should make sure you run an apt-get update and apt-get upgrade to ensure that you are running the latest security updates.

SSH and Google Authenticator Configuration

1. Lets firstly disable root login via SSH.
sudo nano /etc/ssh/sshd_config

Changing the line PermitRootLogin to no.

2. Now within the same file change the port number from 22 to something non-standard like 2222 for example. Do this by changing the line Port 22 to read Port 2222.

3. At this point you should reboot the Raspberry Pi, then attempt to login on port 2222, don't go any further until you have got this working.

4. Now we need to install the Google Authenticator package.

# sudo apt-get install libpam-google-authenticator

5. Once installed you will then need to run the google authenticator, ensure you are logged in as the user you will be SSHing onto the box with to ensure you setup the authenticator configuration into your user's home area.

# google-authenticator

Answer yes to all the prompts in the wizard, you should also make a note of the scratch codes just in case you need them in future.

6. At the end you'll see a QR code, now you need to get out your smart phone and install the Google Authenticator App from your devices Application Store.

7. Now scan in the QR code using the App or enter the code manually. When done, your phone and your Raspberry Pi will be paired via the Google Authenticator to trust each other for authentication.

8. Finally we need to set SSH to use the google Authenticator as part of the authentication process, so you'll need to enter your username/password followed by the code generated by your phone to be able to logon.

9. Open the/etc/pam.d/sshd file with sudo nano /etc/pam.d/sshd add the following line to the end file:

auth required

10. Next open /etc/ssh/sshd_config then locate the ChallengeResponseAuthentication line (normally set to "no") and then set it to "yes".

11. Finally restart the SSH server.

# sudo /etc/init.d/sshd restart

Do not close the active ssh window you are working in. If something went wrong then you can quick debug it. Open a new ssh window instead and you should be prompted for a password followed by the code, all being well you should login fine!

Common Issue: Check that your timezone and time are set correctly on the Raspberry Pi, if they are not you will find your connection attempts will be rejected.

Tip: Check your logs for any errors with: sudo cat /var/log/auth.log | more

Also here you'll need to have punched a hole through your firewall on port 2222/TCP to the IP address of your Raspberry Pi, if you want to access it externally.

Fail2Ban Install and Configuration

We will now install Fail2Ban, and configure it to check on port 2222/TCP which we are now using for SSH, if more than two password failures are created, the IP address will be added to the block list permanently. The configuration for different services can be found in /etc/fail2ban/jail.conf. The default configuration only monitors SSH and bans the suspicious IP after 6 unsuccessful attempts for 600 seconds.

1. Firstly install fail2ban with

# apt-get install fail2ban

2. Then you need to edit the file /etc/fail2ban/jail.local, you may need to create it if you do not yet have it.

# vi /etc/fail2ban/jail.local

3. Once you are editing the file add the following:

banaction = iptables-allports
bantime = -1
maxretry = 2

This will block all IP traffic from a host forever if they get two failed connection attempts.

Then edit the file: /etc/fail2ban/jail.conf, and then add the following:

(within the DEFAULT section, add your local subnet e.g., stops you locking yourself out!)
ignoreip =

Then in the SSH section add the additional port 2222 to the line so it reads:

enabled  = true
port     = ssh,2222
filter   = sshd
logpath  = /var/log/auth.log
maxretry = 6

4. To avoid these blocks disappearing after a reboot, add the following to  /etc/fail2ban/action.d/iptables-allports.conf file to the actionstart section:

# cat /etc/fail2ban/ip.list-<name> | while read IP; do iptables -I fail2ban-<name> 1 -s $IP -j DROP; done

and following line to the actionban section:

# echo '<ip>/24' >> /etc/fail2ban/ip.list-<name>

These settings mean that any banned IP address is added to the  /etc/fail2ban/ip.list file and after restart of the Pi they are readded to the iptables firewall. Note this blocks the whole /24 subnet, you can omit this from the echo command above if you just want to block on an IP by IP basis.

Restart the fail2ban service:

# sudo service fail2ban restart

Overtime you'll see some IPs that are permanently banned. Check in iptables for the firewall block rules with:

# sudo iptables -L -n --line


Ubuntu, find used disk space.

posted 5 Sep 2016, 07:15 by Tristan Self

To easily find where your disk space is being used from the command line you can use the NCDU utilty.

To run this use:

# ncdu -x /

Check an SSL Certificate File in plain English

posted 7 Mar 2016, 05:13 by Tristan Self

You need to verify that a file cert.pem contains the certificate you expected, to do this you can use this command to view the certificate in plain text with expiry date etc visible.

# openssl x509 -in cert.pem -noout -text

Using check_snmp to check if a metric is within a range or not

posted 18 Feb 2016, 05:21 by Tristan Self

Sometimes a metric needs to fall between a range to be considered OK, and outside the range (i.e. above and below) for a warning, and outside that range (i.e. above and below) for a critical.

So for example monitoring the input voltage into a APC UPS to ensure it is within a certain range.

So lets say OK is between 220 and 240, WARNING would be between 210 and 219; and 241 to 250. With CRITICAL being less than 209 or greater than 251.

So something like:
< 220 and 240 < means OK
> 219 and < 241 means WARNING
> 209 and < 251 means CRITICAL.

To configure the Nagios check use the following, this example pulls out the input voltage from the UPS:

$USER1$/check_snmp -H -o . -C community -w 219:241 -c 209:251

So some examples if the input voltage is: 

230v = OK (within the range of 220 to 240)
218v = WARNING (below the range of 219 for warning)
255v = CRITICAL (above the range of 251 for warning)

Nagios check_http plugin get OK returned for an expected error code

posted 27 Nov 2015, 03:08 by Tristan Self

If you have a webserver that you want to check, but it requires authentication, you can still perform a check without seeing an error code. So for example, if you had a webserver where you wanted to check that the web server was responding on port 80, but didn't want to have the plugin login to the server you can use the -e (expected argument) in the check_http command. This argument tells the plugin to expect a different code from 200 (OK) if that is the normal response.

In this environment there was a web server (SharePoint) where it was wanted to be checked if the web service was there or not. Because it required authentication, the normal state returned would be 401, which normally would be a warning, but using the -e argument this can be set to be the normal response and therefore returning an OK. Only if the plugin returns anything else does a warning or critical get generated.

So for example, i've created a specific service check command, but you could just use it as an argument to be passed on a normal check_http command if you want.

define command{
    command_name    check_http_401
    command_line    $USER1$/check_http -H $HOSTADDRESS$ -e 401

So now you get a response such as:

One time reboot of Linux (Ubuntu)

posted 3 Nov 2015, 08:17 by Tristan Self

If you occasionally need to reboot a Linux box as a one off one evening you can perform the following commands using the at command. Note if you set a time that has just past it will perform the reboot the following day. So lets say its 16:15 and you want to schedule a reboot at 21:00 that evening, to do this you can do run these commands.

# at 21:00
at> shutdown -r now

Hit Ctrl + D

You'll now see its been created:
job 1 at Tue Nov 3 21:00:00 2015

To check on any at commands that are scheduled you can run:
# atq
1       Tue Nov  3 21:00:00 2015 a root

Monitor Status of Dell PowerVault TL4000 Quick and Dirty with Nagios

posted 17 Jul 2015, 06:32 by Tristan Self

You can monitor the Dell PowerVault TL4000 Tape Library using the Nagios check_snmp plugin. After setting the TL4000 to accept SNMP connections on v2 SNMP you can run the following command below. This queries for the status of the library as a rollup.

# ./check_snmp -H <IP Address> -C community -o . -r3
SNMP OK - 3 | iso.

So this will query the tape library on the IP address and return the status as a number. 3 = OK, anything we want to know about so using the -r expression means you can enter a regex expression. This is a simple one, if its not 3, its not OK and we want Nagios to report an error!

1-10 of 21