Troubleshooting Apache httpd and mod_wsgi with Anaconda Python 3.5 on RHEL/CentOS 6

Man, I have spent many hours over the last few days figuring out why, exactly, I am having so much trouble getting mod_wsgi to do so much as present a simple test application.  The issues I have encountered have seemed downright unavoidable for a reasonably secure implementation, but I’m sure I’m doing something really obvious and stupid for a system administrator more well versed in httpd and the module than I.

Nonetheless, I thought I might post these issues up here and the solutions I found.  If you happen to be such a superior system administrator and you could kindly educate me regarding my failures as described below (since I’m sure there are superior solutions to be had), please do!

System Configuration
  • RHEL 6.8
  • httpd 2.2
  • mod_wsgi 4.5.6 compiled with:
  • anaconda 4.1.1 (Python 3.5)

I’m using Red Hat Enterprise Linux 6.8 (fully up to date) with the standard httpd package delivered and supported for that distribution (so httpd 2.2).  I compiled mod_wsgi from source, so I’m using the latest and greatest (4.5.6) and I performed that compilation using Python 3.5 through Continuum.io’s Anaconda software (not my choice; seems like an ok service they provide, though).

Initial Trouble

When I tried to start the httpd daemon using the init script and Upstart service command provided by RHEL, the attempt failed and I found in /var/log/httpd/error_log (after increasing to info level verbosity):

Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
[info] mod_wsgi (pid=3786): Python home /opt/anaconda3.
[info] mod_wsgi (pid=3786): Initializing Python.
[info] mod_wsgi (pid=3787): Starting process 'sampleapp' with uid=48, gid=48 and threads=15.
[info] mod_wsgi (pid=3787): Python home /opt/anaconda3.
[info] mod_wsgi (pid=3787): Initializing Python.
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ImportError: No module named 'encodings'

And so it begins…

Solution 1: Directing Apache to the Anaconda Python Resources

Though it appears from the log and from the documentation on mod_wsgi that I was correctly establishing the configuration directive necessary to point httpd to the correct location for Python resources, the error seems pretty clearly indicative of a failure in that respect.  After using every damn WSGI directive (WSGIPythonHome, WSGIPythonPath, WSGIDaemonProcess with the python-home or python-path options…) which seemed plausibly related, all to no avail, I changed tactics.

I ruled out permission and SELinux issues through a variety of standard means.  I tried running Apache as root without SELinux enabled just to make sure this couldn’t possibly be the problem.  Just to be 100% sure and confirm for myself my own rudimentary assessment capabilities, I even `chmod -R 777`’d /opt/anaconda3 (not before taking a backup of it which I then restored after the test, of course).  It was clear that httpd could access the files and do whatever it pleased with them; for some reason, despite seeming to acknowledge clear direction, even, it just refused to find them.

So I decided I’d just have to stack trace httpd and see what, exactly, is going on.  In doing so, I noticed that the httpd daemon would seem to successfully locate the /opt/anaconda3 directory (as expected).  In performing the initial module load, it would successfully open the necessary Python libraries.

open("/opt/anaconda3/lib/libpython3.5m.so.1.0", O_RDONLY) = 5

w00t.  No issues so far.

However, despite the fact that mod_wsgi logs an info event indicating that it has properly recognized the WSGIPythonHome directive I am providing (which is accurate; it is equivalent with the sys.prefix variable, as the documentation indicates it should be), the startup process invariably devolves into searching what appears to be a default series of locations for the Python resources (/usr, for example).

From the stack trace (my user name converted to “myusername”):

stat("/sbin/python3", 0x7fffa0fba600)   = -1 ENOENT (No such file or directory)
stat("/bin/python3", 0x7fffa0fba600)    = -1 ENOENT (No such file or directory)
stat("/usr/sbin/python3", 0x7fffa0fba600) = -1 ENOENT (No such file or directory)
stat("/usr/bin/python3", 0x7fffa0fba600) = -1 ENOENT (No such file or directory)
readlink("", 0x7fffa0fa5520, 4096)      = -1 ENOENT (No such file or directory)
open("pyvenv.cfg", O_RDONLY)            = -1 ENOENT (No such file or directory)
open("pyvenv.cfg", O_RDONLY)            = -1 ENOENT (No such file or directory)
stat("Modules/Setup", 0x7fffa0fa64a0)   = -1 ENOENT (No such file or directory)
getcwd("/home/myusername", 4096)         = 16
stat("/home/myusername/lib/python3.5/os.py", 0x7fffa0fa64a0) = -1 ENOENT (No such file or directory)
stat("/home/myusername/lib/python3.5/os.pyc", 0x7fffa0fa63d0) = -1 ENOENT (No such file or directory)
stat("/home/myusername/lib/python3.5/os.py", 0x7fffa0fa64a0) = -1 ENOENT (No such file or directory)
stat("/home/myusername/lib/python3.5/os.pyc", 0x7fffa0fa63d0) = -1 ENOENT (No such file or directory)
stat("/home/lib/python3.5/os.py", 0x7fffa0fa64a0) = -1 ENOENT (No such file or directory)
stat("/home/lib/python3.5/os.pyc", 0x7fffa0fa63d0) = -1 ENOENT (No such file or directory)
stat("/home/ilan/minonda/envs/_build/lib/python3.5/os.py", 0x7fffa0fa63d0) = -1 ENOENT (No such file or directory)
stat("/home/ilan/minonda/envs/_build/lib/python3.5/os.pyc", 0x7fffa0fa6320) = -1 ENOENT (No such file or directory)
write(2, "Could not find platform independ"..., 55) = 55
stat("pybuilddir.txt", 0x7fffa0fa1440)  = -1 ENOENT (No such file or directory)
getcwd("/home/myusername", 4096)         = 16
stat("/home/myusername/lib/python3.5/lib-dynload", 0x7fffa0fa5520) = -1 ENOENT (No such file or directory)
stat("/home/myusername/lib/python3.5/lib-dynload", 0x7fffa0fa5520) = -1 ENOENT (No such file or directory)
stat("/home/lib/python3.5/lib-dynload", 0x7fffa0fa5520) = -1 ENOENT (No such file or directory)
stat("/home/ilan/minonda/envs/_build/lib/python3.5/lib-dynload", 0x7fffa0fa5520) = -1 ENOENT (No such file or directory)
write(2, "Could not find platform dependen"..., 58) = 58
write(2, "Consider setting $PYTHONHOME to "..., 57) = 57

If you’ll note, the search becomes so desperate, in fact, that it includes some weird searches to subdirectories and resources in the non-existant /home/ilan directory, which looks like it must be somehow inadvertently included information on behalf of the designer of Anaconda, as I found a forum post signed “ilan” from him, it seems.

Eventually, the search process fails and httpd proclaims its inability to locate the resources.

I noticed, however, as you may have noticed as well, that the search pattern being run by httpd/mod_wsgi looks like it includes an attempt to locate Python resources using the current working directory.  So… I changed my current working directory to /opt/anaconda3 and, voila, if I start the httpd process from within the Python Home location, it works! GLORY! IT WORKS!

But… I can’t exactly change the httpd default working directory to solve this problem; that’s crazy.  However, you also might have noticed in my stack trace that one of the steps Apache was conducting in searching for the Python resources consisted of attempts to read a pyvenv.cfg file in the current working directory.

I managed to look up the simple syntax for such a file, and it is something like this:

home = /opt/anaconda3
include-system-site-packages = true
version = 3.5.2

In fact, it is exactly like that in my case. If I place it in the root of the file tree (/pyvenv.cfg). That actually fixes the problem.

Amazing. I don’t know why this is required. I intend to ask Mr. Graham Dumpleton who seems to be quite helpful and active in troubleshooting end users’ issues for his module.

Solution 2: The Dumpleton-Provided Test WSGI Script Fails to Run with Python 3.5

Once the httpd process was up and running successfully with mod_wsgi, I hurriedly tested the situation with Graham’s test file. Unfortunately, I was met with failure:  a 500 Internal Server error in the browser, and this:

[error] [client 192.168.1.18] TypeError: sequence of byte string values expected, value of type str found

It appears the problem is that, using Python 3, the WSGI application must return a byte string output value, so one can make use of this test script for Python 3, instead:

def application(environ, start_response):

    status = '200 OK'
    output = b'Hello World!'

    response_headers = [('Content-type', 'text/plain'),
                        ('Content-Length', str(len(output)))]

    start_response(status, response_headers)
 
    return [output]

Thank you, Interwebs!

Solution 3: Fixing CFUNCTYPE Memory Errors

So then, I tried to execute my client’s actual Python module, and I got this:

[Sat Sep 03 12:58:52 2016] [error] [client 192.168.1.18] mod_wsgi (pid=44501): Target WSGI script '/var/www/wsgi-scripts/scriptname.wsgi' cannot be loaded as Python module.
[Sat Sep 03 12:58:52 2016] [error] [client 192.168.1.18] mod_wsgi (pid=44501): Exception occurred processing WSGI script '/var/www/wsgi-scripts/scriptname.wsgi'.
[Sat Sep 03 12:58:52 2016] [error] [client 192.168.1.18] Traceback (most recent call last):
...
[Sat Sep 03 12:58:52 2016] [error] [client 192.168.1.18]   File "/opt/anaconda3/lib/python3.5/site-packages/pandas/__init__.py", line 13, in <module>
[Sat Sep 03 12:58:52 2016] [error] [client 192.168.1.18]     __import__(dependency)
[Sat Sep 03 12:58:52 2016] [error] [client 192.168.1.18]   File "/opt/anaconda3/lib/python3.5/site-packages/numpy/__init__.py", line 111, in <module>
[Sat Sep 03 12:58:52 2016] [error] [client 192.168.1.18]     import ctypes
[Sat Sep 03 12:58:52 2016] [error] [client 192.168.1.18]   File "/opt/anaconda3/lib/python3.5/ctypes/__init__.py", line 537, in <module>
[Sat Sep 03 12:58:52 2016] [error] [client 192.168.1.18]     _reset_cache()
[Sat Sep 03 12:58:52 2016] [error] [client 192.168.1.18]   File "/opt/anaconda3/lib/python3.5/ctypes/__init__.py", line 276, in _reset_cache
[Sat Sep 03 12:58:52 2016] [error] [client 192.168.1.18]     CFUNCTYPE(c_int)(lambda: None)
[Sat Sep 03 12:58:52 2016] [error] [client 192.168.1.18] MemoryError

As it turns out, ctypes attempts to execute code within the /tmp directory, and SELinux (as well as other common security policies such as those involving mounting /tmp with noexec) prevent that from occurring. Fixing those relevant issues resolved that matter.

The SELinux Boolean in question here is httpd_tmp_exec.  By default, it is disabled, preventing httpd from executing code in /tmp, and this can be changed with `setsebool -P httpd_tmp_exec 1`.  I don’t do that lightly, since that’s a great security feature for httpd, but in this case, it seems necessary.

That’s All For Now

I hope it prevents someone else out there from spending many hours on the issue.  I intend to chase this down further with the mod_wsgi developer, as I wrote above.  If I get a response and figure this out, I’ll certainly update the post.

Advertisements
This entry was posted in Information Technology and tagged , , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s