Logging Packet Drops in Firewalld

Up-Front Answer Summary:

The short answer is:  the feature to simply log all packet actions isn’t available yet.  See the link referenced below in the whole thing.

The longer answer is:  There may be a feasible work-around:

Your firewalld default configuration includes what are called “zones” which are mapped in a one-to-many relationship with network interfaces.  Each interface on your machine can only be mapped to a single zone, but a single zone can be mapped to many interfaces.  Each zone contains rules and services, the latter being configuration files which explain the ports used by various common services (like port 22 for ssh, which is an included service enabled by default on the public zone), and the former being manually written rules using the firewalld rich rules language.

The document quoted below demonstrates via a Red Hat employee working on the firewalld system that there is no easy way to log packet drops as of February 2014.  Unfortunately, the only logging options which seem to be provided by firewalld are debug logging and rich rule logging.  The former doesn’t appear to log packet handling actions even when maxed out (10 on a scale of 1-10) and the latter lacks the capability to be written to act as a general action logging rule, given that each rule must be bound to a particular element (be that a service, port, protocol, etc.) and actions taken against those elements.

One could, however, remove all services from the zone to be used (public is mapped to the primary Ethernet adapter on the machine by default) and replace them with rich rules of one’s own design.  Each rule could then be customized to include the log parameter and a prefix could be specified to inform the system administrator if a packet was accepted or dropped because of the rule.

Other than manually rebuilding the necessary zones (which isn’t actually that arduous given the likely acceptability of a default or lightly modified zone) will allow you to simply log packet handling.

Please correct me if I’m wrong, but that’s my finding after three hours of investigation.  If you’d like more information about the rich language and its syntax, see the Fedora Project’s documentation: https://fedoraproject.org/wiki/Features/FirewalldRichLanguage

The Whole Thing:

All over the Internet, users are complaining that firewalld doesn’t support a simple option to log all dropped (or accepted) packets.  As one can see from a February 2014 mailing list interaction between Red Hat employee Thomas Woerner and one Christian Lupien:

On 02/12/2014 09:47 AM, Christian Lupien wrote:
> I just recently converted my iptables rules to firewalld. Most of it was
> straightforward. However I had trouble with trying to log my rejected
> packets.
> My old /etc/sysconfig/iptables INPUT chain ended with
> -A INPUT -m limit --limit 6/hour --limit-burst 10 -j LOG
> -A INPUT -j REJECT --reject-with icmp-host-prohibited
> Is there a simple way to do this with firewalld?
No, there is no simple way at the moment. But I have "simple logging" on 
my todo list.
> Thanks
> Christian
> _______________________________________________
> firewalld-users mailing list
> firewalld-users at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/firewalld-users

So, how might one resolve this issue in the meanwhile?

I first experimented with the –debug option for the firewalld daemon, but nowhere could I locate documentation explaining the distinction between the given range (1-10) of debug logging levels.  The output is placed in /var/log/firewalld, and is easy enough to read.  Regardless of debug level, however, I could not see packet handling information when connecting to the virtual machine over the virtual network via SSH.  Modifying that debug value for the firewalld daemon isn’t as intuitive as one would hope, either.

Using systemd’s systemctl command-line interface, as one does with Fedora 20, there is no apparent way to load a daemon while specifying command line options.  Don’t try simply stopping firewalld in systemctl and then sneakily attempting to load the daemon manually outside of systemctl; systemd monitors inactive daemons and quickly returns them to their inactive states if attempts are made to start them.

What you must do is include the command line options intended to be passed to firewalld in the /etc/sysconfig/firewalld file (there’s a FIREWALLDARGS variable to define; just add “–debug”…I wish I knew how to easily fix the typo in the instructional comment: “possile,” that could be the start of my contribution to Fedora 20’s code base…ha).  Once you’ve done that, you want to execute:  systemctl reload firewalld.service.  This causes systemd to instruct the daemon to reload its configuration files.  The command can be confused with systemctl daemon-reload, but the latter is used to reload systemd units (configuration files for daemons, among other things), which we have not changed here.

Regardless of this change, however, the firewalld daemon did not seem to log events at any debug level other than 1 (all log entries were prefixed with DEBUG1), though I may have been incapable of generating them with packet transmission.  Executing systemctl status firewalld.service resulted in:

firewalld.service – firewalld – dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled)
Active: active (running) since Wed 2014-04-16 22:46:04 EDT; 8min ago
Main PID: 2380 (firewalld)
CGroup: /system.slice/firewalld.service
└─2380 /usr/bin/python -Es /usr/sbin/firewalld –nofork –nopid –debug

Regardless of the value I place in the variable definition in /etc/sysconfig/firewalld, I can’t get systemd to recognize it despite seeming to call that variable in the /usr/lib/systemd/system/firewalld.service file.  The –debug option is never called with a value, which means the level is 1 per the default.  So instead, I circumvented firewalld’s config file and edited /usr/lib/systemd/system/firewalld.service to include the line:

ExecStart=/usr/sbin/firewalld –nofork –debug=10

rather than

ExecStart=/usr/sbin/firewalld –nofork $FIREWALLD_ARGS

I then performed systemctl daemon-reload and then systemctl restart firewalld.service.  When I checked systemctl status firewalld.service, I saw:

firewalld.service – firewalld – dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled)
Active: active (running) since Wed 2014-04-16 23:02:03 EDT; 4min 26s ago
Process: 4206 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS)
Main PID: 4319 (firewalld)
CGroup: /system.slice/firewalld.service
└─4319 /usr/bin/python -Es /usr/sbin/firewalld –nofork –nopid –debug=10

Perfect.  Now, the /var/log/firewalld file shows DEBUG2 entries as well.  Unfortunately, though the debug level is assuredly set at its maximum capacity, this function simply doesn’t report packet handling events; I can log into the VM using SSH over the Ethernet device which is added to the public zone, which is specifically configured (per the default settings) to allow ssh traffic over port 22.  Indeed, I can successfully log in.

I found that debug level 2 is sufficient to log changes made to firewalld’s zones; when testing to ensure that the firewall was functional, I temporarily disabled ssh service support in the puiblic zone with the expected results from my host machine (a “no route to host” error).  I was then able to reconnect after enabling ssh service support for the public zone.  I noticed in the log the following entries:

2014-04-16 23:09:01 DEBUG2: firewall.core.ipXtables.ip4tables: /sbin/iptables -A IN_public_allow -t filter -m tcp -p tcp –dport 22 -m conntrack –ctstate NEW -j ACCEPT
2014-04-16 23:09:01 DEBUG2: firewall.core.ipXtables.ip6tables: /sbin/ip6tables -A IN_public_allow -t filter -m tcp -p tcp –dport 22 -m conntrack –ctstate NEW -j ACCEPT

The creation of the rules is logged.  Their removal doesn’t appear to be provided the same treatment, which is odd.

Regardless, I think I can say something which should be easily discernible from the firewalld documentation (it’s not; correct me if I’m wrong): debug logging does not include basic packet handling information.

Instead, I execute:

firewall-cmd –add-rich-rule=’rule service name=ssh log level=info’

Unfortunately, the syntax requirements stipulate that a service name must be provided, so it doesn’t appear there is any easy way to specify all permitted services, for example.  You can, however, specify the port value rather than the service name value, so you could easily log all TCP traffic occurring over the interfaces attached to the zone (which, in my configuration, is a single Ethernet adapter attached to the public zone).  Since I’m testing with ssh, I’m specifying that service here.

This causes firewalld to call the kernel to log the event with rsyslogd in a format resembling the following log entry (for an accepted packet):

Apr 16 23:24:17 FedoraVM kernel: IN=eth0 OUT= MAC=12:34:56:78:9A:BC:DE:F1:23:45:67:89:AB:CD SRC= DST= LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=35627 DF PROTO=TCP SPT=45808 DPT=22 WINDOW=29200 RES=0x00 SYN URGP=0

and for an unaccepted packet:

IN=eth0 OUT= MAC=12:34:56:78:9A:BC:DE:F1:23:45:67:89:AB:CD SRC= DST= LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=13454 DF PROTO=TCP SPT=45822 DPT=22 WINDOW=29200 RES=0x00 SYN URGP=0

Unfortunately, the kernel is only logging the encounter with the packet, and not the final decision to add or drop it.  What we really need is some way to order the logging so that it occurs only after the accept or drop phases.  We could then set up prefixes in the logs to tell us which action is occurring.  Unfortunately, as one can see from the firewalld.service(5) man page, there is no place for such an option in the service configuration file.

So, it looks like our capability ends with kernel logging demonstrating that a packet was observed, but what happens to it afterwards is not known.  Of course, we could simply remove all services from the zone and replace them with rich rules of our own creation.  Using the logging feature for each rule, we could then define useful prefixes for the kernel entries which explain whether the packet was accepted or dropped.

I may be wrong, but I’ve spent three hours on this and that’s the best conclusion at which I’ve arrived.  I’m surprised at such a basic issue with firewalld, so I can’t help but think I’m somehow wrong, but after three hours of investigation, it’s time to hit the sack.

This entry was posted in Information Technology and tagged , , , , . Bookmark the permalink.

11 Responses to Logging Packet Drops in Firewalld

  1. Harri says:

    well, i’ve spent more like three days trying to get sanity on firewalld logging capabilities on Fedora 19. Your post is easy to read and thorough over the subject thanks. I have come to same conclusion with it. Anyway firewalld seems to work as configured the logging/lack of logging of packet action is the weakness.
    Question: would you know other Linux distro where there is another firewall implementation which provides zone based handling + logging capability of actions + mature enough for versy small business needs, graphical UI web or native gui.
    br, Harri

    • So, I’m not sure what exactly you’re trying to accomplish with the “zone based handling,” but you can actually install and use iptables on Fedora or CentOS/RHEL 7 systems if you absolutely need another firewall package. While iptables isn’t zone-based, per se, I would expect that you should be able to configure it to cover just about every scenario covered by firewalld’s zone-based architecture. As far as a GUI interface, however, I’m not quite sure what you might find – I use the command line, myself. I think it’s worth pointing out that you can actually configure firewalld to log just about everything you need logged as long as you replace the service-based configuration with rich rules of your own design, as well. That’s probably the easiest path forward for you if you need the zone-based design of firewalld and the GUI provided with it.

  2. I have come to the same conclusion about the lack of packet filter logging with both RHEL 7 and Fedora 21. FirewallD simply does not support the same level of logging that the old iptables-service did. As you stated, you can enable the ‘old’ IP Tables easily enough with systemctl enable iptables, and disable FirewallD via systemctl. I would suggest masking FirewallD as well. Then you can use RHEL6 style iptables commands to build out the firewall as required.

    From there, its you can use a standardized label, e.g. [IP Tables] [Action=Accept] or [IP Tables] [Action=Drop] for the log entries. The standardized label will allow rsyslog to filter the messages, and keep the log messages out of the main /var/log/messages.

    I have implemented the above configuration on a RHEL 7 workstation, with the logs going to /var/logs/iptables/iptables.log, with the logs being analyzed by Elasticsearch.

  3. cdcdcd says:

    firewall-cmd –add-rich-rule=’rule family=”ipv4″ source address=”XX.XX.XX.XXX” log prefix=”DROP: ” level=”info” reject’

    ==> reject and logs all input from source address (for all services)

  4. It’s bad that the firewalld doesn’t have an easy way to log dropped packets. One more reason to stay with iptables??

    Anyway, a workaround I ended up using was to use tcpdump. Since I knew the source IP address range of the user who are having trouble connecting to my machine, I simply did..

    $ tcpdump -n src net

    This showed all the packets that my client was requesting (assumed to be dropped by firewalld), and I’ve used this info to troubleshoot & adjust firewalld rules to allow connection.

  5. nnsense says:

    It seems to me that still MUCH should be done to make this firewalld worth to use.
    Actually I don’t understand how is possible RedHat (and family) choose to switch so early.
    This service still has known bugs (one, and seems not fixed, is interfaces going out from defined –permanent zones after reboot…), its firewall-cmd has the most confusing (and LONG) syntax ever (try firewall-cmd –list-all: you would expect a list of all enabled configurations, maybe the list of zones with enabled interfaces.. Nope. It lists only the default. Even if you’re using internal), and because its limits, it has THREE set of rules: own, direct and rich… If you want real control you must anyway learn iptables.. and It has TWO configs, runtime and permanent… And, still NO simple logging.. I will do the RHCE exam on this thing and.. guess? RedHat allows iptables 😀

    • Yeah, firewalld has a lot of issues, I agree. It certainly seems vastly inferior to iptables. The fact that it has been developed and integrated into a current, prevalent, and important software suite makes me think I’m missing some advantage it has over iptables, but I haven’t yet figured it out. I guess, in some ways, it’s a bit more n00b-friendly, but that seems a small advantage for so many apparent technical inferiorities.

      Oh well, gotta learn it and use it, especially if you’re attempting the LFCE on CentOS 7! I wouldn’t advise reliance on ignorant guesswork for something as fundamental as the system’s firewall software. It’d be a shame to have such a surmountable inadequacy in your skillset kill your entire examination, especially when other, more difficult areas such as dynamic routing and whatnot are potentially involved.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s