IDS: YaF SiLK untuk Menangkap Packet Flow
building a yaf+silk testing vm Mar 9, 2017
I am currently in the process of developing a collection of tools to provide network security monitoring and traffic analysis capabilities on a Raspberry Pi 3 called berry sense. After speaking with my mentor about this, he suggested I take a look at SiLK, a set of traffic analysis tools. Before any real development could begin, I needed to create a testing environment for these tools. This VM would be used solely for testing YaF+SiLK and generating output files that could later be used as template files for understanding how to process the data.
SiLK
The SiLK Suite is a set of traffic analysis tools developed by the CERT Network Situational Awareness Team (CERT NetSA).
From the main website:
The SiLK tool suite supports the efficient collection, storage, and analysis of network flow data, enabling network security analysts to rapidly query large historical traffic data sets.
The suite primarily consists of command-line tools for processing SiLK flow records generated by the SiLK Packing System. The packing system is comprised of daemon applications that collect flow data and convert them to a more space-efficient outputs binary format which can then be partitioned, sorted, and counted using the collection of tools provided. Flow data must be IPFIX flows generated by yaf or NetFlow v5 or v9 PDUs from a suitable router. The application we will be using is rwflowpack.
YaF
Silk collects records from sensors, typically routers. Since I had no suitable routers available, I would have to use yaf to generate IPFIX flows from the raw network traffic.
YaF (Yet Another Flowmeter) “processes packet data from pcap dumpfiles as generated by tcpdump or via live capture from an interface using pcap into bidirectional flows, then exports those flows to IPFIX Collecting Processes or in an IPFIX-based file format”.
YaF will be used to process saved pcap files from scheduled captures or do live captures and send them to rwflowpack so we can then further process the data using the other SiLK tools.
Testing Environment Setup
I decided to begin testing YaF and SiLK by installing both to a virtual machine.
Tools
Virtualbox 5.1.14 silk-3.14.0 yaf-2.8.4
Virtual Machine
OS: Ubuntu Server 16.04.2 RAM: 1GB Network Adapters enp0s3 - NAT adapter (192.168.56.0/24) enp0s8 - Host-only adapter (10.10.16.0/24)
YaF+SiLK Installation
After the OS was installed and configured, the next step was installing YaF and SiLK. Both packages are available for download from the following sites:
SiLK yaf
I began with the YaF installation first.
YaF
Dependencies
Before building YaF, there were a few dependencies that needed to be installed.
sudo apt-get install build-essential pkg-config libfixbuf3 libfixbuf3-dev libpcap0.8-dev
Installation
Once these dependencies were installed, I moved into the directory where I had extracted yaf and went through the standard build process.
./configure make sudo make install
This produced a successful build and installation.
SiLK
Dependencies
Most of SiLK’s dependencies were covered by YaF, so all that wsa left to install was python-dev.
sudo apt-get install python-dev
Installation
Before installing Silk, it was necessary to create a directory where it’s data would be stored. Following the documentation from the CERT site, I created /data as the storage directory.
sudo mkdir /data
Next, I changed into the directory where I had extracted SiLK and went through the standard build process again, passing the --with-python flag to ./configure to include Python support.
./configure --with-python make sudo make install
Installation completed successfully.
Per the documentation, to avoid exporting LD_LIBRARY_PATH each time SiLK is used, I added the following paths to ld.so.conf like so:
cat <<EOF >>silk.conf /usr/local/lib /usr/local/lib/silk EOF sudo mv silk.conf /etc/ld.so.conf.d/
Finally, I ran ldconfig to update changes:
sudo ldconfig
Configuring SiLK
First, I copied the default configuration from the directory where SiLK was extracted at site/twoway/silk.conf to the /data directory.
sudo cp site/twoway/silk.conf /data
Next, I created the sensors.conf file with the following settings:
probe S0 ipfix listen-on-port 18001 # you may need to allow this port through the firewall so that yaf can talk to it protocol tcp listen-as-host 127.0.0.1 end probe group my-network ipblocks 192.168.56.0/24 # address of the host-only adapter (change this to the appropriate address) ipblocks 10.0.2.0/24 # address of the NAT adapter end group sensor S0 ipfix-probes S0 internal-ipblocks @my-network external-ipblocks remainder end sensor
And saved this file to /data .
Configuring rwflowpack
I first made a copy of the default configuration stored at /usr/local/share/silk/etc/rwflowpack.conf and changed the these settings to the following values:
ENABLED=yes CREATE_DIRECTORIES=yes SENSOR_CONFIG=/data/sensors.conf SITE_CONFIG=/data/silk.conf LOG_TYPE=legacy LOG_DIR=/var/log/
And saved the file to /usr/local/etc/ .
Finally, I copied the start up script that was installed by SiLK to /etc/init.d, set it to start on boot, and started the service.
sudo cp /usr/local/share/silk/etc/init.d/rwflowpack /etc/init.d sudo sudo update-rc.d rwflowpack start 20 3 4 5 . sudo service rwflowpack start
Per default settings, rwflowpack listens on port 18001.
Conclusion I now have a working standalone flow collection and analysis test environment. I can now we use rwfiler and other tools provided SiLK to generate files then can later be process and formatted further with code. A future post will show the results of running some tests on the local machine and generating records.
YaF+SiLK Usage Example In order to begin capturing and generating records, YaF must be started. Here is an example:
sudo nohup yaf --silk --ipfix=tcp --live=pcap --out=127.0.0.1 --ipfix-port=18001 --in=eth0 --applabel --max-payload=384 &
NOTE: Change eth0 to the appropriate interface we want to capture on.
Now we can generate some traffic using ping or some other method and wait for the records to be flushed. It can take between 10-15 for the first records to be flushed. Coffee time!