Because this seems to be a common question on the Internet, I thought I might answer it based on recent experience: AIX does not support iptables.
Built into AIX is IPsec support. Part of the IPsec support packages is a filter feature. While it sounds as though it might only apply to IPsec configurations, it does allow for non-IPsec traffic filtering and it can be implemented independently of any IPsec considerations. IBM Developerworks has a nice article describing its configuration for use in a firewall scenario.
Additionally, one will find IPFilter present on the AIX Expansion Pack. IPFilter is a cross-platform firewall software package currently employed by the likes of FreeBSD, OpenBSD, and even Solaris. It has been ported to AIX and is offered in the Expansion Pack (which I hear is unsupported, though I have not verified that for myself) by IBM.
FreeBSD.com actually sports some of the most informative documentation regarding IPFilter, and IBM has enough documentation on their infocenter site to help you to use IPFilter for your standard iptables-esque firewall needs in AIX, but some of the best AIX-specific information from IBM is in their IPFilter FAQ.
I personally found IPFilter to be very easy to use and the rules to be very human readable, which makes it attractive. This is not the case with the built-in IPsec Filter (though it’s not horrible). After examining both options closely, my choice of IPFilter seems somewhat rationally arbitrary, but that’s what I went with based on my knee-jerk reaction to the documentation I examined at the onset of my project.
So hopefully that puts the issue to rest for any interested/confused parties (I saw lots of confusion around the web regarding the capabilities of the IPsec Filter and its relationship with iptables). Let me know if you have any questions!