Diskless true SSH honeypot using Alpine Linux

October 21, 2014

The goal of this article was to set up a honeypot to detect unauthorized SSH authentication attempts in order to detect IP addresses that are targeting SSH services. To achieve this we set up a virtual system running Alpine Linux in diskless mode with OpenSSH active. In this mode the OS is installed in RAM only and thus any data and configuration changes are lost upon a system restart. However we take a snapshot of the system after configuring it so that it can be instantly recreated in the event of a power loss or system restart. The Alpine Linux system in turn is configured to send its SSH logs to AlienVault (OSSIM) for reporting and also to generate actionable alerts.

Why did we choose this setup? Alpine is very quick to install, minimalist (ex: man pages are not installed by default), has low resource usage, and is designed with security in mind. In our opinion it is a neat non-mainstream Linux distro and so this tutorial doubles as a brief introduction to Alpine.

Why did we use a full fledged Linux system as the honeypot instead of something that emulates it? The principal reasons were:

  1. We wanted the exposed SSH service to perfectly replicate what OpenSSH actually behaves like. One of the challenges with honeypots, honeynets, honeyports, honeyfiles, etc. is creating something that appears authentic and mimics its real counterpart including every possible subtle nuances in the event that an attacker decides to test the system in unanticipated ways to determine whether or not it is bait.
  2. Given that we are using an actual Linux distribution here, we can easily expand the scope and include other remote services (MySQL, FTP, Apache, etc.) to detect other types of authentication failures or attack traffic.
  3. We wanted something that was in active development and that an administrator could tailor and modify to suit their needs using their existing knowledge of Linux.
  4. AlienVault natively supports OpenSSH logs, making it easy to integrate the two. This article assumes you are familiar with using AlienVault.

Our Alpine configuration is diskless however you can easily modify the instructions below to make it a disk installation. We identify below where you would make this change. But given that this will serve as a honeypot and will be virtualized, we prefer a diskless setup.

While writing this article we originally kept expanding the scope and spending more and more time making additional configuration changes on Alpine, but this then conflicted with our desire to keep the list of instructions to a minimum. We wish to stress to readers that there are additional steps that one would want to implement as safety measures. There are also topics that we won't be discussing (ex: network segregation for both the honeypot and the VirtualBox server from the rest of the network, patch management of Alpine, AlienVault and VirtualBox) but that are necessary.

Before proceeding with this article you may wish to review our tutorial on how to build a CentOS 6 VirtualBox Server to host the Alpine system.


WARNING & DISCLAIMER


This should be fairly obvious but we will state it here: The instructions in this article discuss how to set up a honeypot system using a functional Linux system. Without additional security controls in place an attacker that compromises such a system could use it for their own nefarious purpose. Readers should make no assumptions that there are no mistakes, typos, omissions, or flaws. Don't deploy something such as this without considerable thought, sufficient testing, and without checking your local laws. Treat this article strictly as a possible starting point towards an idea that can be developed further. Expose to the internet at your own peril.


Setup Alpine Linux

Begin by downloading Alpine Linux Standard ISO (version 3.0.5 at this time of writing). Create a new VirtualBox virtual machine. We give our system 256 MB of RAM (you can use only 128 MB if you wish), and for the hard drive we select "Do not add a virtual hard drive" (disregard if you plan on installing to disk). After VirtualBox creates the entry, right click on it, select Settings, click on System and on the Motherboard tab, change the Pointing Device from USB to PS/2 Mouse, and uncheck Floppy and Hard Disk from the boot order (disregard the latter if you plan on installing to disk). Click on Storage and under Controller: IDE specify the Alpine ISO that you downloaded, click on Audio and uncheck Enable Audio, click on Network and change it from NAT to Bridged, click on USB and uncheck Enable USB Controller. Click OK to complete.

Boot up the system and after 15 seconds you will be presented with the prompt "localhost login". Type "root" without quotes and press enter. We now proceed to install Alpine:

localhost:/# setup-alpine
Select keyboard layout [none]: us
Select variant []: us

For the next step we give our system a name (it is better to use your standard naming convention) and also give it a static IP address. The DNS domain should match your environment.

Enter system hostname (short form, e.g. 'foo') [localhost]:

Available interfaces are: eth0
Which one do you want to initialize? [eth0]: 

Ip address for eth0 [dhcp]: 10.1.2.3
Netmask? [255.0.0.0]: 255.255.0.0
Gateway? (or 'none') [none]: 10.1.1.1

Do you want to do any manual network configuration? [no]

DNS domain name? (e.g. 'bar.com') []

DNS nameserver(s)? [] 10.1.1.2

New password: 

For timezone, it is best to specify the correct time zone:

Which timezone are you in? [UTC]

HTTP/FTP proxy URL [none]

For this next question you can specify "f" to search for and use the closest mirror, or you can manually select one:

Enter mirror number (1-14) or URL to add [f]:

Which SSH server? ('openssh', 'dropbear' or 'none') [openssh]:

Which NTP client to run? ('busybox', 'openntpd', 'chrony', or 'none') [chrony]: none

Below is where we specify diskless. If you want instead to install to disk, you would specify the correct device and you will be prompted to reboot after installation:

Which disk(s) would you like to use? [none]: none

Enter where to store configs ('floppy', 'usb', or 'none') [none]

Enter apk cache directory (or '?' or 'none') [/var/cache/apk]: none

localhost:~#

With the installation complete, we proceed to configure SSH and install rsyslog. Optionally if you do not like vi as your editor you can install nano. Use the command "apk add" to install packages:

localhost:~# apk add rsyslog
localhost:~# apk add nano
localhost:~# nano -w /etc/rsyslog.conf

Edit /etc/rsyslog.conf. For any entries that outputs to disk, comment them out (you can ignore this if you've installed Alpine to disk). Then add the following entry replacing 10.1.1.1 with the IP of your AlienVault sensor:

authpriv.* @10.1.1.1

Restart rsyslog to reflect the changes, and configure it to start automatically upon bootup (technically the latter isn't necessary in our diskless setup). You may get a message stating "option -c is no longer supported - ignored". This can be ignored.

localhost:~# /etc/init.d/rsyslog start
localhost:~# rc-update add rsyslog default

Modify sshd_config:

localhost:~# nano -w /etc/ssh/sshd_config

For the following entry:

#SyslogFacility AUTH

We change this to:

SyslogFacility AUTHPRIV

For the following entry:

#PermitRootLogin yes

We change this to:

PermitRootLogin no

Point sshd to a service banner by adding the following entry in sshd_config, making sure to create the file /etc/sshbanner with 644 permissions. Your banner should display a warning that access is restricted and monitored:

Banner /etc/sshbanner

Restart sshd once you are done. This should automatically make rsyslogd start as well:

localhost:~# /etc/init.d/sshd restart

Optionally you can install ACF (Alpine Configuration Framework) to provide a web interface over HTTPS for administrating Alpine as shown below, but we do not recommend this unless you restrict access to this port:

localhost:~# setup-acf

On the topic of package management in Alpine, you use the commands apk update and apk upgrade to upgrade all packages on a system. Refer to this page to learn how package management works on Alpine. If you are familiar with Debian or Gentoo, you can review this in order to quickly learn how to use Alpine Linux.

See login failures and alarms in AlienVault

As mentioned earlier this article assumes you are already familiar with AlienVault, and so for this aspect we will only list the steps at a high level.

You will want to create an entry in the AlienVault asset list for your Alpine system, confirm that you've enabled the correct plugins in AlienVault to detect SSH events, and have a Directive enabled that detects SSH attacks. With this in place, from a remote system generate some SSH authentication failures against Alpine to confirm that they show up in AlienVault under Analysis | Security Events:

A sufficient amount of SSH failures will generate alarms in Analysis | Alarms. You can modify the thresholds by going to Configuration | Threat Intelligence | Directives:



Active Response with AlienVault

The following is meant as a proof of concept on how you can use AlienVault to action specific system events. What we will setup is a way to automatically shut down the Alpine system in the event that somebody manages to successfully authenticate to it.

In the event of successful authentication on Alpine over SSH, the following syslog event is sent to AlienVault:

sshd[2324]: Accepted password for username from 10.1.1.5 port 45106 ssh2

AlienVault will be configured with a Policy that executes a script on the AlienVault server whenever a SSH authentication success event occurs on Alpine. The script will initiate a SSH connection to Alpine that immediately shuts it down via the halt command. Obviously this means that you will never be able to manage Alpine through SSH again (instead we do so through the VirtualBox console).

The username that will be created on Alpine Linux for this will be "cavalry". Below are the instructions:

In Alpine install sudo (apk add sudo) and create a new user called cavalry (adduser -g 'cavalry' cavalry) and give it a random password (write it down as you'll need it later). Manually edit the file /etc/group and add user cavalry to the wheel group by adding ,cavalry at the end. i.e. in the group file you should now have the following entry:

wheel:x:10:root,cavalry

Then enter the command visudo to edit the file /etc/sudoers, and give members of the wheel group the ability to execute a command without being prompted for a password. This is done by uncommenting the following entry:

## Same thing without a password                                             
%wheel ALL=(ALL) NOPASSWD: ALL

Next on AlienVault create a directory in /root that will store both the script and the RSA keypair for cavalry. When prompted for a passphrase, keep it blank:

alienvault:~# mkdir scripts
alienvault:~# cd scripts
alienvault:~/scripts# ssh-keygen -t rsa -b 3072 -f /root/scripts/cavalry_id_rsa

Create the script that will SSH to Alpine using public key authentication and halt the system. Call the script whatever you want (we use cavalry.sh), give it 750 permissions, and populate it with the following replacing 10.1.2.3 with the IP address of Alpine Linux:

#!/bin/bash 
ssh -i /root/scripts/cavalry_id_rsa cavalry@10.1.2.3 'sudo halt'

Copy the RSA public key to your Alpine system using the root account on alienvault, but connecting to Alpine as the user cavalry (not as root). Then try to SSH to Alpine as user cavalry using public key authentication to confirm that it works. Lastly enter sudo su to confirm that the wheel group configuration changes works (you should become root without being prompted for a password):

alienvault:~# ssh-copy-id -i /root/scripts/cavalry_id_rsa.pub cavalry@10.1.2.3
alienvault:~# ssh -i /root/scripts/cavalry_id_rsa cavalry@10.1.2.3
localhost:~$ sudo su
localhost:~#

Optionally, at this point you can take a snapshot of the current state of Alpine (while it is running) and try to manually execute the script /root/scripts/cavalry.sh on AlienVault as root to confirm that it is able to SSH to Alpine and shut it down.

With this complete we now simply need to configure the Action as well as the Policy on AlienVault. Take a final snapshot of your Alpine system (while it is running) if you haven't done so already as this will be the snapshot you will restore to whenever the system halts.

In AlienVault create an action that runs /root/scripts/cavalry.sh. This is done by clicking on Configuration | Threat Intelligence | Actions | New.

Next, create a policy (Configuration | Threat Intelligence | Policy) based on taxonomy that detects authentication successes on the host Alpine when occurring on destination port 22, for which the action is to runs cavalry.sh. Once saved, reload your policies by clicking the Reload Policies button. Your system is now ready.

Below is a demonstration: The top left window is the VirtualBox console showing Alpine Linux, the bottom right is a terminal of an Ubuntu system that passed authentication on Alpine as root. The moment root authenticates successfully (we temporarily enabled PermitRootLogin in sshd_config on Alpine) the Alpine Linux system halts and root gets disconnected. This occurs near instantaneously.

Below are the SSH logs that occurred on Alpine, as captured by OSSEC on AlienVault. Notice the SSH authentication success (via password) from 10.1.2.128, followed by the SSH authentication via public key from the AlienVault system coming in to shut down Alpine.

AV - Alert - "1413253365" --> RID: "5715"; RL: "3"; RG: 
"syslog,sshd,authentication_success,"; RC:
"SSHD authentication success."; USER: "None"; SRCIP: "10.1.2.128"; HOSTNAME: 
"localhost"; LOCATION:
"/var/log/auth.log"; EVENT: "[INIT]Oct 13 21:26:50 localhost sshd[6892]: Accepted 
password for cavalry from 10.1.2.128 port 47892 ssh2[END]"; 


AV - Alert - "1413253365" --> RID: "5715"; RL: "3"; RG: 
"syslog,sshd,authentication_success,"; RC:
"SSHD authentication success."; USER: "None"; SRCIP: "10.1.1.50"; HOSTNAME: 
"localhost"; LOCATION:
"/var/log/auth.log"; EVENT: "[INIT]Oct 13 21:26:51 localhost sshd[7064]: Accepted 
publickey for cavalry from 10.1.1.50 port 54499
ssh2: RSA 21:a5:42:d8:13:2c:fe:75:44:86:09:4d:02:4e:8b:37[END]";

Just to repeat our earlier statement, this was meant as a proof of concept to tie in AlienVault command-line actions based upon the detection of an event sent via syslog. It is not meant as a reliable mechanism for shutting down a honeypot. For that it would be better to have a script running locally on Alpine that detects this condition and performs the shutdown (below is a basic one). If you were to rely on the AlienVault action as your sole safety mechanism, at minimum we would recommend sending the SSH logs over TCP instead of UDP.

#!/bin/sh

while true
do
 NSTAT=`sudo ps -ef | grep "sshd:" | grep pts | wc -l`
 if [ "$NSTAT" -ge 1 ]; then
    sudo halt
 else
    sleep 1
 fi
done

It is important to mention that one anomaly we observed with the AlienVault action is that after sufficient authentication failures the AlienVault system sometimes incorrectly executes the cavalry.sh script even though no authentication success occured. We successfully replicated it while running a packet capture of syslog traffic and did not see any additional events that were sent from Alpine to AlienVault that would have matched the policy and warranted triggering the response. We experienced the same issue when using a Data Source Group instead of a Taxonomy for the AlienVault policy. Towards the end we felt that it was related to either the OSSEC interpretation of the SSH login failures, or the triggering of the SSH Bruteforce directives into alarms on AlienVault that initiates the policy action. When we disabled the SSH Bruteforce directives this behavior ceased. For us the jury is still out whether this is a feature or bug. The AlienVault version was 4.11.

Fingerprints

Below are some fingerprints of the Alpine system with the above configuration:

# nmap -sV -p 22 10.1.2.3

PORT   STATE SERVICE VERSION
22/tcp open  ssh     OpenSSH 6.6.1p1-hpn14v4 (protocol 2.0)


# nmap -O 10.1.2.3

Not shown: 999 closed ports
PORT   STATE SERVICE
22/tcp open  ssh
No exact OS matches for host (If you know what OS is running on it, [...]


nmap -O --osscan-guess 10.1.2.3

Not shown: 999 closed ports
PORT   STATE SERVICE
22/tcp open  ssh
Aggressive OS guesses: Linux 2.6.32 (91%), Linux 2.6.26 - 2.6.35 (90%),
Linux 2.6.22 (90%), Linux 3.0 - 3.9 (90%), Linux 2.6.23 - 2.6.38 (89%),
Linux 3.2 - 3.6 (89%), Linux 2.6.32 - 3.9 (87%), Linux 2.6.8 - 2.6.27 (87%),
Linux 2.6.18 - 2.6.22 (87%), Linux 3.0 (87%)
No exact OS matches for host (If you know what OS is running on it, [...]


# nmap -sO 10.1.2.3

Not shown: 249 closed protocols
PROTOCOL STATE         SERVICE
1        open          icmp
2        open|filtered igmp
6        open          tcp
17       open          udp
47       open|filtered gre
103      open|filtered pim
136      open|filtered udplite


# nmap --script ssh2-enum-algos -p22 10.1.2.3 

PORT   STATE SERVICE
22/tcp open  ssh
| ssh2-enum-algos: 
|   kex_algorithms (8)
|       curve25519-sha256@libssh.org
|       ecdh-sha2-nistp256
|       ecdh-sha2-nistp384
|       ecdh-sha2-nistp521
|       diffie-hellman-group-exchange-sha256
|       diffie-hellman-group-exchange-sha1
|       diffie-hellman-group14-sha1
|       diffie-hellman-group1-sha1
|   server_host_key_algorithms (4)
|       ssh-rsa
|       ssh-dss
|       ecdsa-sha2-nistp256
|       ssh-ed25519
|   encryption_algorithms (16)
|       aes128-ctr
|       aes192-ctr
|       aes256-ctr
|       arcfour256
|       arcfour128
|       aes128-gcm@openssh.com
|       aes256-gcm@openssh.com
|       chacha20-poly1305@openssh.com
|       aes128-cbc
|       3des-cbc
|       blowfish-cbc
|       cast128-cbc
|       aes192-cbc
|       aes256-cbc
|       arcfour
|       rijndael-cbc@lysator.liu.se
|   mac_algorithms (19)
|       hmac-md5-etm@openssh.com
|       hmac-sha1-etm@openssh.com
|       umac-64-etm@openssh.com
|       umac-128-etm@openssh.com
|       hmac-sha2-256-etm@openssh.com
|       hmac-sha2-512-etm@openssh.com
|       hmac-ripemd160-etm@openssh.com
|       hmac-sha1-96-etm@openssh.com
|       hmac-md5-96-etm@openssh.com
|       hmac-md5
|       hmac-sha1
|       umac-64@openssh.com
|       umac-128@openssh.com
|       hmac-sha2-256
|       hmac-sha2-512
|       hmac-ripemd160
|       hmac-ripemd160@openssh.com
|       hmac-sha1-96
|       hmac-md5-96
|   compression_algorithms (2)
|       none
|_      zlib@openssh.com


# ssh -v test@10.1.2.3
<snip>
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1-hpn14v4
debug1: match: OpenSSH_6.6.1p1-hpn14v4 pat OpenSSH_6.6.1* compat 0x04000000
debug1: Authentications that can continue: publickey,password,keyboard-interactive