FlowMon - Plugins for HTTP Monitoring

Introduction

This site is a homepage of FlowMon HTTP Input/Export Plugins.

Author

Tomáš Šíma - simatomas@mail.muni.cz

Petr Velan - velan@mail.muni.cz

Pavel Čeleda - celeda@mail.muni.cz

Downloads

Version 1.0, release date: December 20, 2012 flowmon-http-plugins-1.0.tar.gz

Papers about HTTP flow extensions

• Design and Evaluation of HTTP Protocol Parsers for IPFIX Measurement

Doxygen documentation

Doxygen generated code documentation is available here.

You can also generate documentation by

$ make doc

then you will find it in ./doc/html/index.html

Requirements

The plugin was tested on :

Intel(R) Xeon(R) CPU E31230 @ 3.20GHz
4x4GB RAM 1330Mhz
Centos 5

Compilation requires FlowMon development headers to be installed. The configure script checks for the header files and required utilities.

Installation

To compile and install plugin plugins, follow these steps:

$ tar -xpzf flowmon-input-http-1.0.tar.gz
$ cd flowmon-input-http-1.0
$ ./configure
$ make && make install

You can use :

$ ./configure --enable-benchmark
$ make && make install

To install HTTP input plugin in benchmark mode. You have to recompile plugins in order to switch between benchmark and normal mode.

Recompiling :

$ cd flowmon-input-http-1.0
$ make clean
$ ./configure
$ make && make install

Usage

Each plugin file must be loaded to flowmonexp using -X option. Running flowmonexp with -l option shows help for all available plugins.

Plugin	Plugin name	File containing plugin
HTTP input plugin	input-http	flowmon-input-http.so
HTTP process plugin	process-http	flowmon-input-http.so
HTTP export to PostgreSQL	export-http_postgres	flowmon-export-http_postgres.so
CSV export plugin	export-csv	flowmon-export-csv.so

Using all HTTP plugins, command examples, shortened tutorial

Install HTTP plugins as shown in "Install" section.
Then cd to directory with plugin installed.
If you are not sure where plugin files are :

$ find / | grep flowmon-input-http.so 

Then use (modify parameters) one example below :

silent, read from eth0, export to postgreSQL, get only HTTP traffic from port 80
$ sudo /usr/bin/flowmonexp -X ./flowmon-export-http_postgres.so -X ./flowmon-input-http.so -I input-http:filter=y,device=eth0,verbose=n -P process-http -E export-http_postgres:host=server.company.ccom,port=5432,dbname=http_flow,user=username,password=t0pSecr3t,connect_timeout=10,verbose=n

verbose(slower performance), read from eth0, export to postgreSQL, get only HTTP traffic from port 80
$ sudo /usr/bin/flowmonexp -X ./flowmon-export-http_postgres.so -X ./flowmon-input-http.so -I input-http:filter=y,device=eth0,verbose=y -P process-http -E export-http_postgres:host=server.company.ccom,port=5432,dbname=http_flow,user=username,password=t0pSecr3t,connect_timeout=10,verbose=y

silent, read from eth0, export to postgreSQL, get any HTTP traffic even on unusual ports
$ sudo /usr/bin/flowmonexp -X ./flowmon-export-http_postgres.so -X ./flowmon-input-http.so -I input-http:filter=n,device=eth0,verbose=n -P process-http -E export-http_postgres:host=server.company.ccom,port=5432,dbname=http_flow,user=username,password=t0pSecr3t,connect_timeout=10,verbose=n

silent, read from eth0, write into ./http.csv
$ sudo /usr/bin/flowmonexp -X ./flowmon-export-csv.so -X ./flowmon-input-http.so -I input-http:filter=y,device=eth0 -P process-http -E export-csv:file=./http.csv

silent, get only HTTP traffic on port 80 stored in packets.pcap, write into ./http.csv
$ sudo /usr/bin/flowmonexp -X ./flowmon-export-csv.so -X ./flowmon-input-http.so -I input-http:filter=y,file=./packets.pcap -P process-http -E export-csv:file=./http.csv

HTTP input and process plugin

Plugin parse HTTP headers and adds data about HTTP to flow.
HTTP process plugin must be used along with HTTP input plugin.

Value List

These tables shows relation between content of HTTP header and content of HTTP_x variable.
These variables are used in flowmonexp cache and also in database schema in HTTP export to PostgreSQL plugin.

Name of variable	HTTP header it refers to	data type
HTTP_USERAGENT	Useragent:	uint8_t
HTTP_METHOD	---	uint8_t
HTTP_DOMAIN	Host:	char[32]
HTTP_REFERER	Referer:	char[32]
HTTP_CONTENT_TYPE	Content_type:	uint8_t
HTTP_URL	---	char[32]
HTTP_STATUS	---	uint16_t
HTTP_HEADER_COUNT	---	uint16_t

When value is undefined, it contains 0 or empty string

HTTP_USERAGENT values :

Firefox	1
MSIE	2
Opera	3
Safari	4
Android	5
Chrome	6
Konqueror	7
Googlebot	8

HTTP_METHOD values :

GET	1
POST	2
HEAD	3
PUT	4
DELETE	5
TRACE	6
CONNECT	7

HTTP_DOMAIN values :

www.google.com
google.com
www.nsa.gov

HTTP_REFERER values :

www.google.com/index.html
google.com
www.nsa.gov/secret.php

HTTP_CONTENT_TYPE values :

application	1
text	2
image	3
message	4
model	5
multipart	6
audio	7
video	8

HTTP_URL values :

/
/index.php
/video.avi
/javascript.js

HTTP_STATUS

HTTP status code used in response.

HTTP_HEADER_COUNT

The number of HTTP headers encountered in all packets that belong to the flow record.

Input plugin parameters :

 * filter=y              <- use default filter in plugin. ( <==> Parse traffic coming from/to port 80, only unicast traffic in ipv4)
 * filter=n              <- dont use any filter

 * filterfile=filter.txt <- use filter in filter.txt

		filter.txt must contain just one line with string representing pcap_filter.

 * device=dev            <- dev is device to listen on.

 * file=/path/file.pcap  <- path to pcap file

 * benchmin=X

 * benchmax=Y

 * benchtick=int         <- Set number of packets to pass between each line of output of benchmarking mode
 * verbose=y/n           <- switch on/off verbose mode (default: verbose mode of flowmonexp)
 * debug=y/n             <- switch on/off debug mode (default: verbose mode of flowmonexp)
 

U must use '(device OR file) AND (filter OR filterfile)' combination of parameters in normal mode.
U must use '(filter OR filterfile) AND benchmin AND benchmax' combination of parameters in benchmark mode.
In benchmark mode, plugin cycle in <;X,Y> (<;benchmin,benchmax>) range through packets specified in benchmark_packets.c . 0<=X<=Y<40

Usage examples:

$ flowmonexp -X /path/flowmon-input-http.so -P process_http -I input-http:device=eth0,filter=y
$ flowmonexp -X /path/flowmon-input-http.so -P process_http -I input-http:file=/path/file.pcap,filter=n

CSV export plugin

Plugin export data into text file. CSV export plugin can be used with any input plugin.

CSV export plugin parameters

 * file=path/file.txt   <- file to write into
 * verbose=y/n          <- switch on/off verbose mode (default: verbose mode of flowmonexp)
 * debug=y/n           <- switch on/off debug mode (default: verbose mode of flowmonexp)

Usage example:

 $ flowmonexp -X /path/flowmon-export-csv.so -E export-csv:file=path/file.txt

Text file format :

	FLOW 0,value1_name value1, Getter2_name value2, Getter3_name value3...
	FLOW 1,value1_name value1, Getter2_name value2, Getter3_name value3...
	...

Values are written as hexadecimal dump from memory.

HTTP export PostgreSQL plugin

Plugin inserts parsed HTTP values into PostgreSQL database.

HTTP export PostgreSQL parameters :

 * verbose=y/n <- switch on/off verbose mode (default: verbose mode of flowmonexp)
 * debug=y/n   <- switch on/off debug mode (default: debug mode of flowmonexp)

Remote connection parameters

You can find more detailed description at http://www.postgresql.org/docs/8.4/static/libpq-connect.html

 * host=server.company.com <- hostname of server to connect to
 * port=5432               <- port to connect to
 * dbname=my_database      <- name of database
 * user=username           <- username
 * password=secret_pass    <- password
 * connect_timeout=10      <- timeout of connection specified in seconds
 * hostaddr=192.168.1.10   <- server address, can be used instead of 'host' parameter
 * sslcert=path/cert       <- filename of client SSL certificate
 * sslmode=Option          <- This option determines whether or with what priority a SSL TCP/IP connection will be negotiated with the server. There are six modes

Option	Description
disable	only try a non-SSL connection
allow	first try a non-SSL connection; if that fails, try an SSL connection
prefer (default)	first try an SSL connection; if that fails, try a non-SSL connection
require	only try an SSL connection. If a root CA file is present, verify the certificate in the same way as if verify-ca was specified
verify-ca	only try an SSL connection, and verify that the server certificate is issued by a trusted CA.
verify-full	only try an SSL connection, verify that the server certificate is issued by a trusted CA and that the server hostname matches that in the certificate.

 * sslkey=/path/key        <- This parameter specifies the location for the secret key used for the client certificate
 * sslrootcert=/path/cert  <- specifies the file name of the root SSL certificate
 * sslcrl=path/crl         <- specifies the file name of the SSL certificate revocation
 * insert_len=num          <- how many records to insert with each insert command, greater insert_len -> less insert commands -> faster inserting to database (default 10000)
 * insert_time=X           <- if X seconds passed since last insert into database, insert into database after processing next packet (default 60)
 * export_all=y            <- insert also flows that aren't HTTP into database

Understanding the output

In the table below, is typical example of HTTP request and answer.
First flow is HTTP GET request for web page, so the "HTTP_METHOD" "HTTP_DOMAIN" "HTTP_URL" "HTTP_USERAGENT" are filled.
Second flow is the answer from webserver, the "HTTP_CONTENT" field is filled and others are blank.
You may encounter flows with both "HTTP_CONTENT" and "HTTP_METHOD" equals "0".
These flows are unusual HTTP methods like PUT, TRACE or HEAD

SOURCE_IP inet	L4_PORT_SRC integer	DESTINATION_IP inet	L4_PORT_DST integer	FLOW_START_USEC timestamptz	FLOW_END_USEC timestamptz	L3_PROTO smallint	PACKETS bigint	BYTES bigint	HTTP_CONTENT_TYPE smallint	HTTP_USERAGENT smallint	HTTP_METHOD smallint	HTTP_DOMAIN character(32)	HTTP_URL character(32)	HTTP_REFERER character(32)
127.0.0.1	5388	173.194.44.241	80	2013-02-12 18:54:09.733244+01	2013-02-12 18:54:09.828393+01	4	3	1200	0	3	1	www.google.com	/index.html	www.referer.com
173.194.44.241	80	127.0.0.1	5388	2013-02-12 18:54:10.735244+01	2013-02-12 18:54:10.928493+01	4	7	11800	2	0	0

Table collumns description

SOURCE_IP <- IP adress of source of the flow
L4_PORT_SRC <- Port of the machine with SOURCE_IP
DESTINATION_IP <- IP of destination of the flow
L4_PORT_DST <- Port of the machine with DESTINATION_IP
FLOW_START_USEC <- Start of the flow, timestamp with timezone
FLOW_END_USEC <- End of the flow, timestamp with timezone
L3_PROTO <- IP protocol version, 4 for IPV4, 6 for IPV6
PACKETS <- Number of packets in flow
BYTES <- Number of bytes transferred
HTTP_CONTENT_TYPE <- For HTTP_* collumns, see "Value list" section in Input plugin description
HTTP_USERAGENT
HTTP_METHOD
HTTP_DOMAIN
HTTP_URL
HTTP_REFERER

Usage example:

sudo /usr/bin/flowmonexp -X ./flowmon-export-http_postgres.so -E export-http_postgres:host=server.company.com,port=5432,dbname=http_flow,user=exporter,password=t0pSecr3t,connect_timeout=10,verbose=y

Checking database content with psql

Connecting to PostgreSQL

$ export PAGER="less -RSX"
$ psql -h host.company.com -U user -d dbname -P pager=always

Check if table 'flows' exists

dbname> \lt

Show stored flows :

dbname> SELECT * from flows;

Delete all flows in table :

dbname> truncate flows;

Delete table

dbname> drop table flows;

Disconnect from database

dbname> \q

BSD License

Copyright (C) 2012 Masaryk University, Institute of Computer Science
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
   notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
   notice, this list of conditions and the following disclaimer in
   the documentation and/or other materials provided with the
   distribution.
3. Neither the name of the Masaryk University nor the names of its
   contributors may be used to endorse or promote products derived
   from this software without specific prior written permission.

This software is provided ``as is'', and any express or implied
warranties, including, but not limited to, the implied warranties of
merchantability and fitness for a particular purpose are disclaimed.
In no event shall the company or contributors be liable for any
direct, indirect, incidental, special, exemplary, or consequential
damages (including, but not limited to, procurement of substitute
goods or services; loss of use, data, or profits; or business
interruption) however caused and on any theory of liability, whether
in contract, strict liability, or tort (including negligence or
otherwise) arising in any way out of the use of this software, even
if advised of the possibility of such damage.