This release is focused on stability and speed, solving some of the problems the 4.0.x series had.
Some “older” protocols have been removed (but may come back in rewritten form in future versions), making space for newer “protocols” like twitter or facebook.

I already submitted the new SlackBuild script and the packages (including one for ARMedslack) are being uploaded to my site.
And, while you’re updating ntop, check if your GeoIP and graphviz packages are up-to-date, as new versions have been released for these dependencies as well (and are available also on my site).

What is new
The complete ChangeLog for version 3.3.1 can be found here. Basically, it contains several improvements and bug-fixes “under the hood”, but it also has some new enhancements “on the outside”:

• A new and refreshing “exfoliation” theme
• The documentation link now goes to the on-line docs, always showing the newest information
• The “home” page shows links to current events from Nagios

What it looks like
Well, here are just two simple screenshots from Nagios running on my home network…

This is the new “home” screen of Nagios:

And this is the new “Tactical overview” (yes, I have 13 hosts at home):

Where to get it
As always, the new script has been submitted to SlackBuilds.org and the pre-built packages are available on my site.

Actually, this was a nice popularity test… I received *lots* of e-mails asking about an update for LibreOffice, which reached the stable stage with version 3.3.0.
Next in order of requests was Zabbix, for the new version 1.8.4.
And third in line was FreeCiv – the game.

These updates were long over due but they are all in the pending queue on SlackBuilds.org now.
As I write this I’m also uploading everything to my site, for those who can’t wait

I’ll do some more updates soonish, so keep following the news here…

Recently version 1.6 of lbench was released, a benchmark utility.
And last week version 2.9.0.3 of snort, the Intrusion Detection software, and version 0.5 of daq, the Data Acquisition libray, were published.

I updated and submitted my SlackBuilds scripts to SBo and put the pre-built packages for Slackware, Slackware64 and ARMedslack on my site.

I updated my SlackBuilds and installed the new versions on my server. I waited a bit before releasing the packages as there had been some reports on a new “Unknown record type” in Barnyard2, this time with record type 110. Read my previous posts here and here to learn more details about this problem.

For the “110″ record type the workaround is disabling the “inspect_gzip” option in the snort.conf flie, while the authors of Barnyard2 work on a new release.

For daq-0.3 and snort-2.9.0.1 I submitted the new SlackBuilds and pre-built packages for Slackware, Slackware64 and ARMedslack are available on my site.

ERROR: Unknown record type read: 110

I had not seen this one myself, but did some quick investigating and found out that it is a new record type in Snort 2.9.0
This new record type is called “Unified2_Extra_Data” and is used for gzip and XFF data.

Barnyard2 will need to be adapted to recognize these new types and this will hopefully happen soon…

ERROR: Unknown record type read: 104
Fatal Error, Quitting..

I started looking for information on the internet (Google is our friend…), and found out that at least I was not alone

Similar problems

Basically, I found two types of reports:

1. some with record types like 2148576734, 1697542248, 3569595041, etc.
2. others more like the error I received, with the unknown record type being either 104 (like mine) or 105

Error type 1 (apparently random record types)

These errors are caused by reading a log from Snort that is not in the “unified2″ format. Barnyard2 only uses the newer unified2 log format from Snort as input.
The solution is to check your Snort configuration file and change it to use the newer format.

Error type 2 (record types 104 and 105)

The first “solution” I found was removing two parameters in the Snort configuration file for the unified2 log format, mpls_event_types and vlan_event_types.
The default line in the Snort 2.9.0 configuration reads:

output unified2: filename merged.log, limit 128, nostamp, mpls_event_types, vlan_event_types

For Barnyard I already had to remove the “nostamp” parameter, as it prefers to work with the time-stamp extensions.
I do not use the mpls option in Snort, but the vlan ID can be useful…

To get Snort working, I removed the two parameters, but was not satisfied.

Finding a real solution – Start examining the source code!

After reading through the source code of Barnyard2 and of the new u2spewfoo utility that comes with Snort 2.9.0, I found out that these record types are from version 2 of the Unified2 record format.
In the Barnyard2 source file unified2.h the definitions can be found:

#define UNIFIED2_EVENT 1
#define UNIFIED2_PACKET 2
#define UNIFIED2_IDS_EVENT 7
#define UNIFIED2_EVENT_EXTENDED 66
#define UNIFIED2_PERFORMANCE 67
#define UNIFIED2_PORTSCAN 68
#define UNIFIED2_IDS_EVENT_IPV6 72
#define UNIFIED2_IDS_EVENT_MPLS 99
#define UNIFIED2_IDS_EVENT_IPV6_MPLS 100

//version 2
#define UNIFIED2_IDS_EVENT_V2 104
#define UNIFIED2_IDS_EVENT_IPV6_V2 105

So, if Barnyard2 knows about the record types, why can’t it process them?
Well, I found that in spi_unified2.c there was this piece of code:

int Unified2ReadRecord(void *sph)
{
        DEBUG_WRAP(DebugMessage(DEBUG_LOG,"Reading record type=%u (%u bytes)\n", ntohl(u2hdr.type), ntohl(u2hdr.length)););

        switch (ntohl(u2hdr.type))
        {
                case UNIFIED2_IDS_EVENT:
                        return Unified2ReadEventRecord(sph);
                        break;
                case UNIFIED2_IDS_EVENT_IPV6:
                        return Unified2ReadEvent6Record(sph);
                        break;
                case UNIFIED2_PACKET:
                        return Unified2ReadPacketRecord(sph);
                        break;
        default:
                        FatalError("Unknown record type read: %u\n", ntohl(u2hdr.type));
            break;
        }

        return -1;
}

This means that it will simply show an error message Unknown record type read: ... when it encounters anything different from type 7, 72 or 2. And that’s exactly what it does in our case

This gave me something to start working on…
I first wrote a quick patch to simply recognize and handle record type 104 and sent it to the development team to check if this was the way to go. They answered me that it was, so I completed the patch with record type 105 (which is for ip6) and included the necessary routines for the debugging mode.

This is the resulting patch:

--- barnyard2-1.8_orig/src/input-plugins/spi_unified2.c    2010-03-03 08:03:12.000000000 -0300
+++ barnyard2-1.8/src/input-plugins/spi_unified2.c    2010-10-09 23:31:17.000000000 -0300
@@ -62,11 +62,15 @@

 int Unified2ReadEventRecord(void *);
 int Unified2ReadEvent6Record(void *);
+int Unified2ReadEventRecordV2(void *);
+int Unified2ReadEvent6RecordV2(void *);
 int Unified2ReadPacketRecord(void *);

 void Unified2PrintCommonRecord(Unified2EventCommon *evt);
 void Unified2PrintEventRecord(Unified2Event *);
 void Unified2PrintEvent6Record(Unified2Event6 *evt);
+void Unified2PrintEventRecordV2(Unified2Event_v2 *);
+void Unified2PrintEvent6RecordV2(Unified2Event6_v2 *evt);
 void Unified2PrintPacketRecord(Unified2Packet *);

 /* restart/shutdown functions */
@@ -163,9 +167,15 @@
         case UNIFIED2_IDS_EVENT:
             return Unified2ReadEventRecord(sph);
             break;
+        case UNIFIED2_IDS_EVENT_V2:
+            return Unified2ReadEventRecordV2(sph);
+            break;
         case UNIFIED2_IDS_EVENT_IPV6:
             return Unified2ReadEvent6Record(sph);
             break;
+        case UNIFIED2_IDS_EVENT_IPV6_V2:
+            return Unified2ReadEvent6RecordV2(sph);
+            break;
         case UNIFIED2_PACKET:
             return Unified2ReadPacketRecord(sph);
             break;
@@ -221,6 +231,50 @@
     return -1;
 }

+int Unified2ReadEventRecordV2(void *sph)
+{
+    ssize_t                bytes_read;
+    int                 record_size;
+    Spooler                *spooler = (Spooler *)sph;
+
+    record_size = sizeof(Unified2Event_v2);
+
+    if(!spooler->record.data)
+    {
+        // SnortAlloc will FatalError if memory can't be assigned.
+        spooler->record.data=SnortAlloc(record_size);
+    }
+
+    if (spooler->offset < record_size)
+    {
+        /* in case we don't have it already */
+        bytes_read = read(spooler->fd, spooler->record.data + spooler->offset,
+                    record_size - spooler->offset);
+
+        if(bytes_read == -1)
+        {
+            LogMessage("ERROR: read error: %s\n", strerror(errno));
+            return BARNYARD2_FILE_ERROR;
+        }
+
+        if(bytes_read + spooler->offset != record_size)
+        {
+            spooler->offset += bytes_read;
+            return BARNYARD2_READ_PARTIAL;
+        }
+
+#ifdef DEBUG
+        Unified2PrintEventRecordV2((Unified2Event_v2 *)spooler->record.data);
+#endif
+
+        spooler->offset = 0;
+
+        return 0;
+    }
+
+    return -1;
+}
+
 int Unified2ReadEvent6Record(void *sph)
 {
     ssize_t                bytes_read;
@@ -265,6 +319,50 @@
     return -1;
 }

+int Unified2ReadEvent6RecordV2(void *sph)
+{
+    ssize_t                bytes_read;
+    int                 record_size;
+    Spooler                *spooler = (Spooler *)sph;
+
+    record_size = sizeof(Unified2Event6_v2);
+
+    if(!spooler->record.data)
+    {
+        /* SnortAlloc will FatalError if memory can't be assigned */
+        spooler->record.data=SnortAlloc(record_size);
+    }
+
+    if (spooler->offset < record_size)
+    {
+        /* in case we don't have it already */
+        bytes_read = read(spooler->fd, spooler->record.data + spooler->offset,
+                    record_size - spooler->offset);
+
+        if(bytes_read == -1)
+        {
+            LogMessage("ERROR: read error: %s\n", strerror(errno));
+            return BARNYARD2_FILE_ERROR;
+        }
+
+        if(bytes_read + spooler->offset != record_size)
+        {
+            spooler->offset += bytes_read;
+            return BARNYARD2_READ_PARTIAL;
+        }
+
+#ifdef DEBUG
+        Unified2PrintEvent6RecordV2((Unified2Event6_v2 *)spooler->record.data);
+#endif
+
+        spooler->offset = 0;
+
+        return 0;
+    }
+
+    return -1;
+}
+
 int Unified2ReadPacketRecord(void *sph)
 {
     ssize_t                bytes_read;
@@ -366,10 +464,66 @@
         "  packet_action      = %d\n", evt->packet_action););
 }

+void Unified2PrintEventRecordV2(Unified2Event_v2 *evt)
+{
+    char                sip4[INET_ADDRSTRLEN];
+    char                dip4[INET_ADDRSTRLEN];
+
+    if(evt == NULL)
+        return;
+
+    Unified2PrintEventCommonRecord((Unified2EventCommon *)evt);
+
+    inet_ntop(AF_INET, &(evt->ip_source), sip4, INET_ADDRSTRLEN);
+    DEBUG_WRAP(DebugMessage(DEBUG_LOG,
+        "  ip_source          = %s\n", sip4););
+
+    DEBUG_WRAP(DebugMessage(DEBUG_LOG,
+        "  sport_itype        = %d\n", ntohs(evt->sport_itype)););
+    inet_ntop(AF_INET, &(evt->ip_destination), dip4, INET_ADDRSTRLEN);
+    DEBUG_WRAP(DebugMessage(DEBUG_LOG,
+        "  ip_destination     = %s\n", dip4););
+    DEBUG_WRAP(DebugMessage(DEBUG_LOG,
+        "  dport_icode        = %d\n", ntohs(evt->dport_icode)););
+
+    DEBUG_WRAP(DebugMessage(DEBUG_LOG,
+        "  ip_protocol        = %d\n", evt->protocol););
+    DEBUG_WRAP(DebugMessage(DEBUG_LOG,
+        "  packet_action      = %d\n", evt->packet_action););
+}
+
 void Unified2PrintEvent6Record(Unified2Event6 *evt)
 {
     char                sip6[INET6_ADDRSTRLEN];
     char                dip6[INET6_ADDRSTRLEN];
+
+    if(evt == NULL)
+        return;
+
+    Unified2PrintEventCommonRecord((Unified2EventCommon *)evt);
+
+    inet_ntop(AF_INET6, &(evt->ip_source), sip6, INET6_ADDRSTRLEN);
+    DEBUG_WRAP(DebugMessage(DEBUG_LOG,
+        "  ip_source          = %s\n", sip6););
+
+    DEBUG_WRAP(DebugMessage(DEBUG_LOG,
+        "  sport_itype        = %d\n", ntohs(evt->sport_itype)););
+    inet_ntop(AF_INET6, &(evt->ip_destination), dip6, INET6_ADDRSTRLEN);
+    DEBUG_WRAP(DebugMessage(DEBUG_LOG,
+        "  ip_destination     = %s\n", dip6););
+    DEBUG_WRAP(DebugMessage(DEBUG_LOG,
+        "  dport_icode        = %d\n", ntohs(evt->dport_icode)););
+
+    DEBUG_WRAP(DebugMessage(DEBUG_LOG,
+        "  ip_protocol        = %d\n", evt->protocol););
+    DEBUG_WRAP(DebugMessage(DEBUG_LOG,
+        "  packet_action      = %d\n", evt->packet_action););
+}
+
+void Unified2PrintEvent6RecordV2(Unified2Event6_v2 *evt)
+{
+    char                sip6[INET6_ADDRSTRLEN];
+    char                dip6[INET6_ADDRSTRLEN];

     if(evt == NULL)
         return;

The patch can also be downloaded here.

The code could probably be optimized since there are many duplicate lines now, but I did not want to change too much.
I sent the patch to the Barnyard2 developers and they may commit it in their git repository.
It looks like this patch should work on the 1.9-beta version as well, but I have not tested this… YMMV!

I have been running the patched Barnyard2 for several hours now on my Snort server without problems, using the vlan_event_types parameter.

New dependencies

libdnet is a “simplified, portable interface to several low-level networking routines” and has a SlackBuild on SlackBuilds.org.
daq is a new “Data Acquisition library” that can use several sources for the packages to Snort. Its source is available on the Snort site. I wrote a simple SlackBuild script for it, that I submitted.

Changes to default configuration

The second change I encountered, was with the new default options in the snort.conf file.
By default, the “nostamp”, “mpls_event_types” and “vlan_event_types” are enabled in the “unified2″ log format, but Barnyard2 does not understand them.
On the last two it actually crashes with this error:

ERROR: Unknown record type read: 104
Fatal Error, Quitting..

I sent an e-mail to the Barnyard2 mailing list, but I’m not sure if there will be an answer, as I found other requests that never received any response.
For now I decided to disable these options in the standard configuration.

I also needed to include the –enable-zlib option at the configuration stage of building Snort, as the new default is to use this option.

New unified2 tools

Snort 2.9.0 comes with two new tools:

u2spewfoo can read the binary unified2 logs and dump the output as text. This helped me to understand the problem I had with the “unknown event type” I encountered above.
u2boat is a Unified2 Binary Output & Alert Tool can convert the unified2 logs to standard tcpdump format.

Conclusion

So, after some struggling, I managed to update my Snort server and submitted the updated SlackBuild for version 2.9.0
As always, I also prepared the packages for Slackware and ARMedslack, that are available on my site.

Since the ChangeLog is quite short, I copy it here:

• Fixes problem where disabling all active hosts/services was not taking effect
• Fixes for compiler warnings
• Fixes for format errors in event handler logging
• Fixed incorrect info in sample nagios.cfg file for state_retention_file
• Fixed broker_event_handler() to return ERR if data is NULL

I already submitted the new SlackBuild script and the pre-built packages for Slackware (32/64/ARM) can be downloaded already from my site.

I have set up Snort a few times on a network to check for possible intrusion attempts. It is a very powerful tool to discover these attempts, based on rules that are frequently updated. Snort is a so-called “Intrusion Detection System” (=IDS), that can also be setup as a “Intrusion Prevention System”.
Recently I needed Snort again and decided to finally organize my notes and write down everything in a single document, so that I would spend less time if I ever needed it do this all over again.

Snort is powerful, but to effectively use it, you need some extra tools.
First of all, there is Barnyard2, a tool that reads Snorts binary logs (which are much faster than writing directly to a database)
Second, you will probably want an easy web interface to see the result remotely, for which I use BASE.

Preparations

You will need a working setup of your httpd server – with php – and MySQL.
I won’t explain how to do this, there are many tutorials available on the web. And – being quite honest – if you don’t know how to setup these programs, you’re probably not ready to setup or use an IDS

Installing and setting up Snort

This part is actually quite simple:

1) Get or build the Snort package and install it on your system
2) Download recent snort-rules (*) and install them manually or with my script (**)

(*) You will need to register for free or buy a subscription to get the rules file.
(**) I use a small script that is certainly not perfect, but you can download it and improve it as you wish.

3) Start Snort

You can use the following command to start Snort:

# /etc/rc.d/rc.snort start

You can check if Snort is running with this command:

# ps -ef | grep snort

Which should return something like:

 7874 ?        00:18:46 snort

Installing and setting up Barnyard2

The default setup of Snort creates binary logs in the “unified2″ format. This is the fastest way Snort can work, handling even heavy traffic.
Snort could also write directly to a database, but this takes some processor time and delays could cause packages to be missed.
The solution is to use a separate program to read the binary logs and save the information to a database for research, statistics, browsing, etc.

Enter Barnyard2 to the rescue!
Barnyard2 is based on the original Snort Barnyard program, but modified to read the newer “unified2″ log format.

1) Get or build the barnyard2 package and install it on the same system you installed Snort on

Now let’s do a basic test to check if Snort is identifying intrusions and if Barnyard2 is parsing the log.
Start Barnyard2 from the command line like this:

# barnyard2 -d /var/log/snort -f snort.log

The parameters we passed are:

• -d /var/log/snort – the standard directory where Snort stores its log files
• -f snort.log – the “base” name of the Snort log file (it is appended with a time-stamp)

Barnyard2 will read its configuration file and start processing the Snort log. If there are already events in the log, they will be displayed.

Let’s check if Snort is working by doing a port-scan from another computer on the same network (doing this from the computer Snort is installed on won’t have any effect, as it wont pass through the network adapter and Snort will never notice it).
You can do this with the following command:

# nmap -A <ip_address_of_snort_server>

Snort should detect the port scan, write the event in its log, and Barnyard2 should display it.
If all works fine, you can continue with the next step.

2) Create the database to store the events

You will need to set up the structure of the database:
(replace the <…> with your real passwords)

# cd /usr/doc/barnyard2-*/schemas
# mysql -p
Enter password: <your_mysql_root_password>

Now inside MySQL:

mysql> create database snort;
mysql> grant create,select,update,insert,delete on snort.* to snort@localhost;
mysql> set password for snort@localhost=PASSWORD('<your_mysql_snort_password>');
mysql> exit

Back at the prompt:

# mysql -p < create_mysql snort
Enter password: <your_mysql_root_password>


This will create the snort database and tables.

3) Configure Barnyard2

Open the /etc/barnyard2.conf file with your favorite editor (that’s VIm, of course…), go to the end of the file and edit one of the sample lines so that it looks like this:

output database: log, mysql, user=snort password=<your_mysql_snort_password> dbname=snort host=localhost

4) Start the Barnyard2 daemon

The package created with the SlackBuild script includes a file to start the daemon. use it like this:

# /etc/rc.d/rc.barnyard2 start

The script only parses the new events from the log, to avoid duplicate entries in the database.

You can now check your MySQL database to see if the entries are saved there. I personally use phpmysql to check the database, but other tools exist.
If needed, run the port-scan again.

If all is fine, we can now start installing BASE, the web front-end for Snort.

Installing and setting up BASE

BASE stands for Basic Analysis and Security Engine. It is a web-based tool to display and filter all events captured by Snort.

1) Install dependencies

BASE needs some other files to work properly. The first one is adodb, the PHP database abstraction library. A SlackBuild for this can be found on SlackBuilds.org or a pre-built package can be downloaded from my site.

You will also need a few modules from the PHP “pear” library. Use these commands to download & install them:

# pear install Image_Color-1.0.4
# pear install Image_Canvas-0.3.2
# pear install Image_Graph-0.7.2

2) Get or build BASE and install the package

3) Configure BASE

After installing the dependencies and the package, point your web browser to http://<your_snort_server>/base and the first configuration screen should show up:

For safety, the directory where the configuration file will be stored does not give write access to the web server, so after finishing the next steps, you will need to create this file by copying the parameters and pasting them into a text file on the console.

Click on “continue” and select your language in the next screen:

If you installed adodb from the standard SlackBuild or package, use /var/www/htdocs/adodb for its location, and click on continue.

The next screen will ask the parameters to connect to your database:

Since we set up the database on the same server as Snort and Barnyard2, use “localhost” as the database host.

After clicking on continue, screen number 3 will appear:

Select the check-box and choose whatever name you prefer for the administrator (I used “admin” in this example) and create a password and a full name (like “Administrator” in my example).

Click on continue again to go to the next step:

Here BASE will try to include some extra needed tables for the “Alert Groups” in the database we created before. Click on “Create BASE AG” and you should see this screen:

The messages show that everything went fine. The text that the “snort” user needs DELETE and UPDATE privileges in the database is nothing to worry about, as we took care of that when we installed Barnyard2 a few steps back.

Click on “step 5″ to continue to the next screen:

Copy the text of the configuration and paste it using your favorite text editor to create the file /var/www/htdocs/base/base_conf.php

After saving it, click on “click here to access your install” to enter your newly set-up BASE! The following screen should show up, asking for login and password:

Since we only have the administrator account for now, let’s use the login we created and the password we set. After clicking on “login”, we’re in the “home” screen of BASE:

We already have one captured event, so let’s see what it is by clicking on “listing” for today’s events:

So that event was our port-scan we tried earlier

Where to go from here

This concludes our basic Snort / Barnyard2 / BASE set-up and we now have a working IDS – Intrusion Detection System.

There are many more things to configure, but there are some great books out there teaching what you can do with Snort and how to configure it. I can personally recommend “The Snort Cookbook” and “Managing Security with Snort & IDS Tools”, both by O’Reilly but there are several other options.
And don’t forget: Google is your friend!

Links

For Snort, Barnyard2, adodb and BASE, I submitted SlackBuild scripts to SlackBuilds.org.
If you prefer pre-built packages for Slackware, Slackware64 or ARMedslack, you can find them on my site.

To read more about these programs, visit their sites:

• Snort
• Barnyard2
• adodb
• BASE