Linux Security Software

Of major concern in the protection of your managed hosting services business is the security of your system. Each day your system is up and running, you’ll face attacks from unscrupulous hackers from around the world—and you’re vulnerable to those attacks even if you’re running on top-notch servers, such as a Red Hat Linux Server or your own managed dedicated server.

PortSentry is a component of Psionic Technologies’ TriSentry suite of attack detection tools, which includes PortSentry, HostSentry, and LogSentry. PortSentry may be installed on CentOS, FreeBSD, and RHEL operating systems.

PortSentry Basics

Simply put, PortSentry operates by detecting scans conducted on the host, determines the appropriate response to the scan, and carries out that response. Because of its basic operation method, PortSentry is most commonly classified as an “attack detection” tool.

Here’s a typical scenario that affects many managed web hosting businesses:

Let’s say, for example, that you’re running a Red Hat dedicated server without PortSentry. An attacker has targeted your system, but first needs to know which ports are open and available to attack. To determine that information, the attacker first runs a scan as a precursor to the attack; this helps the attacker learn what services you’re running and to pinpoint ports. When an available port is found, the attack on your system begins. Your business is now compromised. You may experience significant downtime, loss of data, loss of customers, or even the loss of your business.

Not so when you run PortSentry. This managed hosting solution is a tool that monitors both the TCP and the UDP ports on your system. Depending on how you’ve configured PortSentry, it can also respond appropriately to the identified scan, and with the 2.0 version, it can also detect stealth scans like those produced by Nmap.

Scans PortSentry is capable of detecting include:

Connect Scans: A connect scan is a full connection scan in which the entire TCP three-way handshake is completed. Because the target host may record the connection from a scanning IP address, these scans are the most obvious type.
SYN Scans: SYN Scans are sometimes called “half-open” scans and are stealthier than connect scans. With this scan only the first two steps of the TCP three-way handshake take place. A TCP SYN package is sent in a manner that suggests that it is requesting to open a full connection. This prompts the targeted system to respond by sending a SYN-ACK packet. The attacker then sends a TCP RST, or reset, packet back, which closes the connection. When the connection is closed, the host system doesn’t log the connection because it was never completed.
FIN Scans: FIN Scans are those that use packets with the TCP FIN flag set. FIN packets are usually only present during the closing sequence of a connection. When an unsolicited FIN packet is sent to a closed TCP port, the other side responds by sending an RST packet.
NULL Scans: This scan uses a packet without any of the TCP flags set. Like the FIN Scan, it should illicit an RST packet in return.
XMAS Scans: XMAS scans have the FIN, URG, and PUSH TCP flags set in the TCP header. Because these aren’t considered “normal” packets on the Internet or a local LAN, the closed port should respond with an RST packet.
FULL-XMAS Scans: A FULL-XMAS Scan is one in which the SYN, ACK, RST, FIN, URG, and PSH flags are set. Again, this is not a normal packet.
UDP Scan: This scan is detected by the presence of multiple UDP packets coming from a single IP address.

When PortSentry can’t identify a scan as any of the above types, a default case is triggered on the scan. This is one of PortSentry’s exceptional features, since many other security tools ignore scans they can’t identify and allow them to go through.

PortSentry also has the ability to determine when a packet is part of a scan versus normal traffic, which it does using two methods:

PortSentry ignores ports that are in use.
Users can specify a list of ports they want PortSentry to monitor. In cases where a port is specified and has a service bound to it, PortSentry will ignore the port.

Another feature offered by PortSentry is the intelligent monitoring of ports. For protocols like FTP, for which the client opens up ports in the ephemeral range (TCP ports greater than 1024) and the server connects back to the host, PortSentry examines the incoming connection. If the connection is destined for one of the ephemeral ports, PortSentry ignores it. When the connection is terminated, PortSentry begins to again monitor the port as before.

In addition to checking for TCP and UDP scans, PortSentry also checks the IP header length for those less than 5 (the minimum length specified in RFC 791) and notes those falling below the minimum. If any IP options have been set, PortSentry notes this as well. Although this may seem to some to be problematic, the use of options like source-routing, record-route, and timestamp can be used by an attacker for reconnaissance information.

Internal Workings

In a very basic way, PortSentry operates by monitoring ports for potentially suspicious packets and then responding when a scan has been detected. To do this, PortSentry must maintain state with all of the IP addresses connecting into a host.

PortSentry utilizes an internal state engine in the form of an array of IP addresses that enables it to determine if a host has previously connected to the system. The array is a two-dimensional table consisting of an IP address and a counter. When an attacker initiates a scan, PortSentry reviews the array to see if the attacker’s IP has been seen before. If it has, PortSentry increments the counter. This is done for every port, so if the attacker’s IP has been seen on any port, the counter is incremented. When the trigger value is reached, PortSentry reacts to the scan. How PortSentry reacts to a scan depends on how the user configures it.

Scanning Response Methods

There are three response methods that can be utilized by PortSentry. Those methods are:

Insert a Route in the Host’s Routing Table: By inserting a route in the host’s routing table, all traffic coming from the scanning IP is sent into a “bit bucket.” The host route tables re-route traffic from a scanning host to a non-existent IP address or network, which makes the system look like a black hole—traffic goes in, but nothing comes out.
Tie PortSentry to a Firewall Software Package: With version 2.0, PortSentry supports the following firewalls: ipfw, ipfilter, ipfwadm, ipchains, and iptables. PortSentry can be tied to a firewall so that a firewall rule can be used to block traffic from the scanning IP. With this method, PortSentry will detect a scan and add an appropriate rule to the firewall configuration blocking the IP address of the scanning host.
TCP Wrapper Rule: This method allows a TCP wrapper rule for the attacking IP to be added to the system’s /etc/hosts.deny file. While this doesn’t block a scan, it does prevent the attacker from connecting from the scanning host to services protected by TCP wrappers on the scanned host.

When PortSentry is properly configured and the log output is monitored, potential risks are significantly reduced.