Logging Packet Drops in Firewalld

Up-Front Answer Summary:

The short answer is:  the feature to simply log all packet actions isn’t available yet.  See the link referenced below in the whole thing.

The longer answer is:  There may be a feasible work-around:

Your firewalld default configuration includes what are called “zones” which are mapped in a one-to-many relationship with network interfaces.  Each interface on your machine can only be mapped to a single zone, but a single zone can be mapped to many interfaces.  Each zone contains rules and services, the latter being configuration files which explain the ports used by various common services (like port 22 for ssh, which is an included service enabled by default on the public zone), and the former being manually written rules using the firewalld rich rules language.

The document quoted below demonstrates via a Red Hat employee working on the firewalld system that there is no easy way to log packet drops as of February 2014.  Unfortunately, the only logging options which seem to be provided by firewalld are debug logging and rich rule logging.  The former doesn’t appear to log packet handling actions even when maxed out (10 on a scale of 1-10) and the latter lacks the capability to be written to act as a general action logging rule, given that each rule must be bound to a particular element (be that a service, port, protocol, etc.) and actions taken against those elements.

One could, however, remove all services from the zone to be used (public is mapped to the primary Ethernet adapter on the machine by default) and replace them with rich rules of one’s own design.  Each rule could then be customized to include the log parameter and a prefix could be specified to inform the system administrator if a packet was accepted or dropped because of the rule.

Other than manually rebuilding the necessary zones (which isn’t actually that arduous given the likely acceptability of a default or lightly modified zone) will allow you to simply log packet handling.

Please correct me if I’m wrong, but that’s my finding after three hours of investigation.  If you’d like more information about the rich language and its syntax, see the Fedora Project’s documentation: https://fedoraproject.org/wiki/Features/FirewalldRichLanguage

The Whole Thing:

All over the Internet, users are complaining that firewalld doesn’t support a simple option to log all dropped (or accepted) packets.  As one can see from a February 2014 mailing list interaction between Red Hat employee Thomas Woerner and one Christian Lupien:

On 02/12/2014 09:47 AM, Christian Lupien wrote:
...
> I just recently converted my iptables rules to firewalld. Most of it was
> straightforward. However I had trouble with trying to log my rejected
> packets.
>
> My old /etc/sysconfig/iptables INPUT chain ended with
> -A INPUT -m limit --limit 6/hour --limit-burst 10 -j LOG
> -A INPUT -j REJECT --reject-with icmp-host-prohibited
>
> Is there a simple way to do this with firewalld?
>
No, there is no simple way at the moment. But I have "simple logging" on 
my todo list.
...
> Thanks
> Christian
>
> _______________________________________________
> firewalld-users mailing list
> firewalld-users at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/firewalld-users
>
Regards,
Thomas

So, how might one resolve this issue in the meanwhile?

I first experimented with the –debug option for the firewalld daemon, but nowhere could I locate documentation explaining the distinction between the given range (1-10) of debug logging levels.  The output is placed in /var/log/firewalld, and is easy enough to read.  Regardless of debug level, however, I could not see packet handling information when connecting to the virtual machine over the virtual network via SSH.  Modifying that debug value for the firewalld daemon isn’t as intuitive as one would hope, either.

Using systemd’s systemctl command-line interface, as one does with Fedora 20, there is no apparent way to load a daemon while specifying command line options.  Don’t try simply stopping firewalld in systemctl and then sneakily attempting to load the daemon manually outside of systemctl; systemd monitors inactive daemons and quickly returns them to their inactive states if attempts are made to start them.

What you must do is include the command line options intended to be passed to firewalld in the /etc/sysconfig/firewalld file (there’s a FIREWALLDARGS variable to define; just add “–debug”…I wish I knew how to easily fix the typo in the instructional comment: “possile,” that could be the start of my contribution to Fedora 20’s code base…ha).  Once you’ve done that, you want to execute:  systemctl reload firewalld.service.  This causes systemd to instruct the daemon to reload its configuration files.  The command can be confused with systemctl daemon-reload, but the latter is used to reload systemd units (configuration files for daemons, among other things), which we have not changed here.

Regardless of this change, however, the firewalld daemon did not seem to log events at any debug level other than 1 (all log entries were prefixed with DEBUG1), though I may have been incapable of generating them with packet transmission.  Executing systemctl status firewalld.service resulted in:

firewalld.service – firewalld – dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled)
Active: active (running) since Wed 2014-04-16 22:46:04 EDT; 8min ago
Main PID: 2380 (firewalld)
CGroup: /system.slice/firewalld.service
└─2380 /usr/bin/python -Es /usr/sbin/firewalld –nofork –nopid –debug

Regardless of the value I place in the variable definition in /etc/sysconfig/firewalld, I can’t get systemd to recognize it despite seeming to call that variable in the /usr/lib/systemd/system/firewalld.service file.  The –debug option is never called with a value, which means the level is 1 per the default.  So instead, I circumvented firewalld’s config file and edited /usr/lib/systemd/system/firewalld.service to include the line:

ExecStart=/usr/sbin/firewalld –nofork –debug=10

rather than

ExecStart=/usr/sbin/firewalld –nofork $FIREWALLD_ARGS

I then performed systemctl daemon-reload and then systemctl restart firewalld.service.  When I checked systemctl status firewalld.service, I saw:

firewalld.service – firewalld – dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled)
Active: active (running) since Wed 2014-04-16 23:02:03 EDT; 4min 26s ago
Process: 4206 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS)
Main PID: 4319 (firewalld)
CGroup: /system.slice/firewalld.service
└─4319 /usr/bin/python -Es /usr/sbin/firewalld –nofork –nopid –debug=10

Perfect.  Now, the /var/log/firewalld file shows DEBUG2 entries as well.  Unfortunately, though the debug level is assuredly set at its maximum capacity, this function simply doesn’t report packet handling events; I can log into the VM using SSH over the Ethernet device which is added to the public zone, which is specifically configured (per the default settings) to allow ssh traffic over port 22.  Indeed, I can successfully log in.

I found that debug level 2 is sufficient to log changes made to firewalld’s zones; when testing to ensure that the firewall was functional, I temporarily disabled ssh service support in the puiblic zone with the expected results from my host machine (a “no route to host” error).  I was then able to reconnect after enabling ssh service support for the public zone.  I noticed in the log the following entries:

2014-04-16 23:09:01 DEBUG2: firewall.core.ipXtables.ip4tables: /sbin/iptables -A IN_public_allow -t filter -m tcp -p tcp –dport 22 -m conntrack –ctstate NEW -j ACCEPT
2014-04-16 23:09:01 DEBUG2: firewall.core.ipXtables.ip6tables: /sbin/ip6tables -A IN_public_allow -t filter -m tcp -p tcp –dport 22 -m conntrack –ctstate NEW -j ACCEPT

The creation of the rules is logged.  Their removal doesn’t appear to be provided the same treatment, which is odd.

Regardless, I think I can say something which should be easily discernible from the firewalld documentation (it’s not; correct me if I’m wrong): debug logging does not include basic packet handling information.

Instead, I execute:

firewall-cmd –add-rich-rule=’rule service name=ssh log level=info’

Unfortunately, the syntax requirements stipulate that a service name must be provided, so it doesn’t appear there is any easy way to specify all permitted services, for example.  You can, however, specify the port value rather than the service name value, so you could easily log all TCP traffic occurring over the interfaces attached to the zone (which, in my configuration, is a single Ethernet adapter attached to the public zone).  Since I’m testing with ssh, I’m specifying that service here.

This causes firewalld to call the kernel to log the event with rsyslogd in a format resembling the following log entry (for an accepted packet):

Apr 16 23:24:17 FedoraVM kernel: IN=eth0 OUT= MAC=12:34:56:78:9A:BC:DE:F1:23:45:67:89:AB:CD SRC=192.168.143.1 DST=192.168.143.238 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=35627 DF PROTO=TCP SPT=45808 DPT=22 WINDOW=29200 RES=0x00 SYN URGP=0

and for an unaccepted packet:

IN=eth0 OUT= MAC=12:34:56:78:9A:BC:DE:F1:23:45:67:89:AB:CD SRC=192.168.143.1 DST=192.168.143.238 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=13454 DF PROTO=TCP SPT=45822 DPT=22 WINDOW=29200 RES=0x00 SYN URGP=0

Unfortunately, the kernel is only logging the encounter with the packet, and not the final decision to add or drop it.  What we really need is some way to order the logging so that it occurs only after the accept or drop phases.  We could then set up prefixes in the logs to tell us which action is occurring.  Unfortunately, as one can see from the firewalld.service(5) man page, there is no place for such an option in the service configuration file.

So, it looks like our capability ends with kernel logging demonstrating that a packet was observed, but what happens to it afterwards is not known.  Of course, we could simply remove all services from the zone and replace them with rich rules of our own creation.  Using the logging feature for each rule, we could then define useful prefixes for the kernel entries which explain whether the packet was accepted or dropped.

I may be wrong, but I’ve spent three hours on this and that’s the best conclusion at which I’ve arrived.  I’m surprised at such a basic issue with firewalld, so I can’t help but think I’m somehow wrong, but after three hours of investigation, it’s time to hit the sack.