L7-filter Userspace Version HOWTO

What you need to get

Download the "l7-filter userspace version" and "Protocol definitions" packages from our Sourceforge project page.

Installation

Compile the program:

• Untar l7-filter-userspace-X.Y.tar.gz
• Run ./configure
• Run make
• Run make install as root

./configure will probably find that you need to install some libraries (libnetfilter_conntrack and libnetfilter_queue) from ftp.netfilter.org. Do so if prompted.

Now set up the protocol definitions (a.k.a. pattern files). These files tell l7-filter how protocol names correspond to regular expressions, for example "ftp" means "^220[\x09-\x0d -~]*ftp". Uncompress the "Protocol definitions" package and make the resulting directory /etc/l7-protocols.

Getting it going

First, a reminder: Just because you're using l7-filter, you don't need to do all of your packet classification using it. It's likely that what you want to accomplish can be at least partially done with less demanding classifiers, such as port matching. For instance, you can probably assume that traffic on TCP port 80 that isn't matched by any P2P patterns is HTTP; you don't need to actually use the HTTP pattern.

Kernel

For Linux 2.6.14 through 2.6.19.7, you simply need to have connection tracking and the connection tracking netlink interface enabled. I think that this is the default in most cases. Older versions of Linux don't have this interface and so will not work.

For Linux 2.6.20 and newer, Netfilter has new "Layer 3 Independent Connection tracking" which l7-filter is not yet compatible with (mostly due to lack of library support from libnetfilter_conntrack). While the old layer 3 dependent connection tracking is still available, it is not selected by default, so you will probably need to recompile your kernel with it. In the Linux kernel config, go to:

• Networking → Networking options → Network packet filtering framework (Netfilter) → Core Netfilter Configuration

Under "Netfilter connection tracking support", select "Layer 3 Dependent Connection tracking (OBSOLETE)". Then go to:

• Networking → Networking options → Network packet filtering framework → IP: Netfilter Configuration

and enable "Connection tracking netlink interface" (and probably most of the rest of the stuff on that page). Warning: the kernel guys like to rename these configuration options frequently, so they may not be called exactly what we show here.

Either way, you need either the module ip_conntrack_netlink or nf_conntrack_netlink or the same code compiled into your kernel.

l7-filter

Set up your l7-filter configuration file. It consists of pairs of protocol names and Netfilter mark numbers. l7-filter will mark packets that match each given protocol with the corresponding mark number. There is a sample in the l7-filter-userspace source tarball. For a list of valid protocols, see the protocols page. You can also add your own protocols. Marks are 32 bit integers, but l7-filter assigns special meaning to 0, 1 and 2, so yours must be at least 3.

Load the ip_conntrack_netlink module with modprobe ip_conntrack_netlink, or make sure it is compiled into your kernel.

Send traffic to l7-filter using one of:

iptables [specify table and chain] -j QUEUE
iptables [specify table and chain] -j NFQUEUE --queue-num [queue number]

NFQUEUE defaults to queue number 0 and so does l7-filter. So to send all traffic passing through your machine to l7-filter on queue number 0:

iptables -A FORWARD -j (NF)QUEUE

Now run l7-filter:

l7-filter -f [configuration file] -q [queue number]

For example, if you're using queue number 0 and your configuration file is l7-filter.conf:

l7-filter -f l7-filter.conf

In order to do its classification, l7-filter must be able to see all of the relevant traffic, so make sure you do not use a chain like OUTPUT which only gets traffic going in one direction. See this packet flow diagram for details. In some cases, l7-filter can sucessfully match even if it can only see one side of the connection, but in general, this won't work.

See man l7-filter for various ways of tweaking l7-filter's behavior.

What it does

l7-filter examines the application data in connections to determine what protocol is being used and sets Netfilter marks accordingly. It expects that packets it receives have no mark yet (that is, their mark is 0). It generally takes a few packets before l7-filter can identify what protocol is being used, so it gives packets from new, unidentified connections the special mark of 1. Eventually if l7-filter can't identify a connection, it will give up. In this case, it marks them with the special value 2. Otherwise, it marks packets with the marks specified in its configuration file.

l7-filter ACCEPTs all packets sent to it. This means that an iptables rule that sends packets to l7-filter always terminates the chain and sends the packet on. (This is not how we would like it to work, but the Netfilter libraries currently make any other approach very difficult.) So in order to use the marks that l7-filter sets, you need to use a chain that is downstream of it. See the next section of this document.

Making it useful

Just marking packets isn't very interesting. What you actually want is probably one of these three things: (1) accounting (2) controlling bandwidth use (3) blocking. To do these, you must set up rules to act on the marks that l7-filter applies to the packets. These can either be iptables (firewall) or tc (QoS) rules.

1. Accouting

If you just want to keep track of what's in use on your network, simply match on the mark in some chain that is downstream of where you are QUEUEing to l7-filter and don't use any -j option. For instance, if l7-filter acts in FORWARD, you can do, for instance:

iptables -t mangle -A POSTROUTING -m mark --mark 3

and then get statistics by using iptables -L. (See man iptables for details.)

2. Bandwidth Restriction

If l7-filter has marked, say, IMAP packets with 3, you can use tc ("traffic control", the userspace tool for Linux QoS, part of the iproute2 package) to control the bandwidth used by IMAP using commands like this:

tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 3 fw flowid 1:3

Did you understand that? You can try reading The Linux Advanced Routing and Traffic Control HOWTO for enlightenment. You should do this so that you have some idea what you're doing, but unfortunately, tc is incredibly obtuse and you're likely to wish you just had a canned script such as this one to work from. This will need to be modified if your setup is significantly different than mine, but it should provide a good starting point.

Be prudent when choosing the amount of bandwidth you allow each protocol. Restricting a protocol to an unusably low bandwidth can have similar consequences to blocking it (see next section).

3. Blocking

Don't. Here's why:

• l7-filter matching isn't foolproof: there may be both false positives (one protocol can look like another) and false negatives (applications can do obscure things that we didn't count on). Patterns that are known to regularly generate false positives are marked "overmatching" on the protocols page, but others may also do so occasionally.
• Almost every type of Internet traffic has legitimate uses. For instance, P2P protocols, while widely used to violate copyright, are also an efficient way to distribute open source software and legally free music.
• Programs can respond to being blocked by port-hopping, switching between TCP and UDP, opening a new connection for every trivial operation, using encryption, or employing other evasion tactics. Trying to block such protocols has consequences on two levels:
  1. In the case of port/protocol-hopping, you make it harder for yourself to identify protocols that already act this way.
  2. You encourage programmers to include these "features" in new programs, making it harder for everyone in the future. For example: In early 2006, Bittorrent started moving towards end-to-end encryption because many networks were either blocking it or severely restricting its bandwidth.
• l7-filter patterns are not generally designed with blocking in mind. We consider a protocol to be well identified if the identification is useful for controlling its bandwidth. This means, for instance, that for P2P applications, we do not focus on catching connections that are not downloads.
• Blocking with l7-filter provides no security, since any reasonably determined person can easily circumvent it.

Instead of dropping packets you don't like, we recommend using Linux QoS to restrict their bandwidth usage. If you insist on using l7-filter to drop packets, make sure you have investigated other options first, such as the features of your HTTP proxy (useful for worms).

More information

Dealing with FTP, IRC, etc.

Some protocols open child connections to transfer data. This version of l7-filter currently cannot handle this by itself. It will only correctly match the parent connection.

Other things that use Netfilter marks

If a packet already has a non-zero Netfilter mark when l7-filter gets it, it will pass it on with that same mark without trying to identify it. You can modify this behaviour using the -m and -c options. See man l7-filter.

Upgrading the protocol definitions

The protocol definitions are simple text files with a format described in the Pattern-HOWTO. They can be updated as a package or individually.

L7-filter only reads the definitions when it starts up, so if you update the protocol definitions, restart it.

Other things to know

• By default l7-filter looks at the first 10 packets or 12000 Bytes of each connection, whichever is smaller. This can be modified with command line switches. See man l7-filter.
• It's possible (although rare) for a connection to be matchable by more than one pattern. The patterns are tested in the order you specified in the configuration file. After a match is made, l7-filter does not continue testing that connection, so changing the order of your rules may change what happens.

Please see the FAQ for more information.