The Midnight Oil: Jailing CentOS 6 in FreeBSD 10.2 Edition

Introduction

CentOS 6!  In a jail on the FreeBSD 10.2 kernel!  THAT’S EFFING AWESOME.

Inspired by the recent release of FreeBSD 10.2 and the accompanying announcement that the Application Binary Interface Linux compatibility version has been upgraded to 2.6.18, I thought an attempt at running the CentOS 6 userland (which relies on Linux kernel version 2.6.32) in a FreeBSD 10.2 jail might be warranted.

Of course, it’s CentOS 5 that runs on Linux kernel 2.6.18, so CentOS 5 should work very well in a jail, but I thought I might try pushing the envelope given the extant proof of concept (which itself likely needs updating for use in FreeBSD 10.2) for running CentOS 5 userland in FreeBSD 8 jails. To my knowledge, there has been no previous successful demonstration of the capacity to host the CentOS 6 userland in a FreeBSD jail.

CentOS 6 runs on Linux kernel version 2.6.32, so it is quite possible that some requisite kernel functionality will be absent from our FreeBSD kernel’s ABI. If there prove to be insurmountable obstacles, downgrading to the CentOS 5 userland should be a reasonable option as, theoretically, the CentOS 5 userland should operate without a problem with the FreeBSD 10.2 kernel’s ABI.  The incentive to run CentOS 6 rather than 5 is the standard OS fare: though alive, CentOS 5 is significantly older with a noticeable lack of current features when compared with CentOS 6, and its end of life will come in 2017, with CentOS 6 alive and well until 2020.

All that said, I just got this up and running after some hours spent cobbling together instructions from the various sources to which I link below, filling in the gaps, and then troubleshooting some issues (one of which is listed at the bottom for anyone else who encounters it), and it works!  I am sure there are limitations, and I want to see how sophisticated this solution can get (wouldn’t it be cool if we could run CentOS 6 software offerings in FreeBSD 10.2 jails with confidence in their functionality and stability?), but it works at a rudimentary level at this point.

I am most pleased.

Process

1)  Configure Linux Binary Compatibility  and load additional required modules:

FreeBSD10# kldload linux fdescfs linprocfs linsysfs tmpfs
FreeBSD10# kldstat
Id Refs Address            Size     Name
 1   21 0xffffffff80200000 179ddb0  kernel
 2    1 0xffffffff81a11000 641b     nullfs.ko
 3    1 0xffffffff81a18000 56aa     fdescfs.ko
 4    3 0xffffffff81a1e000 43c7c    linux.ko
 5    1 0xffffffff81a62000 9da0     linprocfs.ko
 6    1 0xffffffff81a6c000 55ad     linsysfs.ko
 7    1 0xffffffff81a72000 a77f     tmpfs.ko

FreeBSD10# portsnap fetch
FreeBSD10# portsnap extract
FreeBSD10# cd /usr/ports/emulators
FreeBSD10# make install clean
===>  linux_base-f10-10_9 has known vulnerabilities:
linux_base-f10-10_9 is vulnerable:
glibc -- gethostbyname buffer overflow
CVE: CVE-2015-0235
WWW: https://vuxml.FreeBSD.org/freebsd/0765de84-a6c1-11e4-a0c1-c485083ca99c.html

1 problem(s) in the installed packages found.
=> Please update your ports tree and try again.
=> Note: Vulnerable ports are marked as such even if there is no update available.
=> If you wish to ignore this vulnerability rebuild with 'make DISABLE_VULNERABILITIES=yes'
*** Error code 1

Stop.
make: stopped in /usr/ports/emulators/linux_base-f10
FreeBSD10# portsnap update
Ports tree is already up to date.
FreeBSD10# make DISABLE_VULNERABILITIES=yes install clean #YOLO lab machine

2)  Acquire CentOS 6 tarball (Got the idea from this guy):

FreeBSD10# pkg install wget
...
FreeBSD10# wget http://download.openvz.org/template/precreated/centos-6-x86.tar.gz #FreeBSD only supports 32 bit emulation

3)  Unpack the tarball into the centos6 root directory.

FreeBSD10#: tar xf centos-6-x86.tar.gz -C /usr/jails/centos6

4)  Mount the necessary virtual file systems into the jail’s root directory:

FreeBSD10# cd /usr/jails/centos6
FreeBSD10# mount -t linprocfs linprocfs proc
FreeBSD10# mount -t linsysfs linsysfs sys  
FreeBSD10# mount -t devfs devfs dev
FreeBSD10# mount                        
/dev/vtbd0p2 on / (ufs, local, journaled soft-updates)
devfs on /dev (devfs, local, multilabel)
linprocfs on /usr/jails/centos6/proc (linprocfs, local)
linsysfs on /usr/jails/centos6/sys (linsysfs, local)
devfs on /usr/jails/centos6/dev (devfs, local, multilabel)

5)  Configure the jail in /etc/rc.conf:

FreeBSD10# vim /etc/rc.conf

jail_enable="YES"
jail_list="centos6"

#cloned_interfaces="lo2"
#jail_centos6_interface="lo2"

jail_centos6_interface="em0"
jail_centos6_ip="127.0.2.1"
jail_centos6_hostname="centos6.dianshan"
jail_centos6_set_hostname_allow="YES"

jail_centos6_devfs_enable="YES"
jail_centos6_fdescfs_enable="YES"
jail_centos6_procfs_enable="YES"
jail_centos6_rootdir="/usr/jails/centos6"
jail_centos6_exec_start="/etc/init.d/rc 3"
jail_centos6_flags="-l -u root"

jail_centos6_fstab="/usr/jails/fstab_centos"
:wq

6)  Write some basic CentOS configuration files:

FreeBSD10# cp /etc/resolv.conf /usr/jails/centos6/etc/resolv.conf
FreeBSD10# echo "NETWORKING=yes" >> etc/sysconfig/network
FreeBSD10# echo "linproc /jails/centos/proc linprocfs rw 0 0" >> /usr/jails/fstab_centos6
FreeBSD10# echo "192.168.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 centos6 centos6.dianshan" >> /usr/jails/centos6/etc/hosts

That last command probably mishandles the loopback creation in the jail; I’ll most likely be revising it later.

7) Create the password database for the jail:

FreeBSD10# cd /usr/jails/centos6/etc 
FreeBSD10# echo "root::0:0::0:0:Charlie &:/root:/bin/bash" > master.passwd
FreeBSD10# pwd_mkdb -d ./ -p master.passwd
pwd_mkdb: warning, unknown root shell

That last remark is ok; pwd_mkdb is just informing us that the host FreeBSD system doesn’t know what /bin/bash is.  No matter, since our CentOS 6 system in the jail will know.

8)  chroot into the centos6 directory and prepare the environment (bless the author of this post):

FreeBSD10# chroot /usr/jails/centos6 /bin/bash
[root@FreeBSD10 bin]# cd /etc                      #OMG how great is this?
[root@FreeBSD10 etc]# pwconv
[root@FreeBSD10 etc]# grpconv                      #OMG it worked
[root@FreeBSD10 etc]# passwd                       #OMG it wasn't a fluke
Changing password for user root.
New password: 
BAD PASSWORD: it is based on a dictionary word     #YOLO
Retype new password: 
passwd: all authentication tokens updated successfully.
[root@FreeBSD10 etc]# touch /etc/fstab
[root@FreeBSD10 etc]# touch /etc/mtab
[root@FreeBSD10 etc]# cd /sbin
[root@FreeBSD10 sbin]# mv consoletype consoletype.orig
[root@FreeBSD10 sbin]# ln -s /bin/true consoletype
[root@FreeBSD10 sbin]# cd /bin
[root@FreeBSD10 bin]# mv umount umount.prev
[root@FreeBSD10 bin]# ln -s /bin/true umount
[root@FreeBSD10 bin]# chkconfig --list
crond           0:off   1:off   2:on    3:on    4:on    5:on    6:off
htcacheclean    0:off   1:off   2:off   3:off   4:off   5:off   6:off
httpd           0:off   1:off   2:off   3:on    4:off   5:off   6:off
ip6tables       0:off   1:off   2:on    3:off   4:on    5:on    6:off
iptables        0:off   1:off   2:on    3:on    4:on    5:on    6:off
modules_dep     0:off   1:off   2:on    3:on    4:on    5:on    6:off
named           0:off   1:off   2:off   3:off   4:off   5:off   6:off
netconsole      0:off   1:off   2:off   3:off   4:off   5:off   6:off
netfs           0:off   1:off   2:off   3:off   4:on    5:on    6:off
network         0:off   1:off   2:on    3:on    4:on    5:on    6:off
nmb             0:off   1:off   2:off   3:off   4:off   5:off   6:off
nscd            0:off   1:off   2:off   3:off   4:off   5:off   6:off
portreserve     0:off   1:off   2:on    3:off   4:on    5:on    6:off
quota_nld       0:off   1:off   2:off   3:off   4:off   5:off   6:off
rdisc           0:off   1:off   2:off   3:off   4:off   5:off   6:off
restorecond     0:off   1:off   2:off   3:off   4:off   5:off   6:off
rpcbind         0:off   1:off   2:on    3:off   4:on    5:on    6:off
rsyslog         0:off   1:off   2:on    3:on    4:on    5:on    6:off
saslauthd       0:off   1:off   2:off   3:on    4:off   5:off   6:off
sendmail        0:off   1:off   2:on    3:on    4:on    5:on    6:off
smb             0:off   1:off   2:off   3:off   4:off   5:off   6:off
snmpd           0:off   1:off   2:off   3:off   4:off   5:off   6:off
snmptrapd       0:off   1:off   2:off   3:off   4:off   5:off   6:off
sshd            0:off   1:off   2:on    3:on    4:on    5:on    6:off
udev-post       0:off   1:on    2:on    3:off   4:on    5:on    6:off
winbind         0:off   1:off   2:off   3:off   4:off   5:off   6:off
xinetd          0:off   1:off   2:off   3:on    4:on    5:on    6:off

xinetd based services:
        chargen-dgram:  off
        chargen-stream: off
        daytime-dgram:  off
        daytime-stream: off
        discard-dgram:  off
        discard-stream: off
        echo-dgram:     off
        echo-stream:    off
        finger:         off
        ntalk:          off
        rsync:          off
        talk:           off
        tcpmux-server:  off
        time-dgram:     off
        time-stream:    off
[root@FreeBSD10 bin]# chkconfig httpd off
[root@FreeBSD10 bin]# chkconfig ip6tables off
[root@FreeBSD10 bin]# exit         
exit                                               #ZOOOMG so awesome

9)  And start the jail

FreeBSD10# /etc/rc.d/jail start centos6
Starting jails: centos6.dianshan.
/etc/rc.d/jail: WARNING: Per-jail configuration via jail_* variables  is obsolete.  Please consider to migrate to /etc/jail.conf.

IT LIIIIIIIIIIIIIIIIIIIIIVES!!!!

10)  Connect in!

FreeBSD10# jls              
   JID  IP Address      Hostname                      Path
     7  192.168.122.200 centos6.dianshan              /usr/jails/centos6
FreeBSD10# jexec 7 /bin/bash
[root@centos6 /]#

omg.  It’s up and running.

That is awesome.

Troubleshooting Tips

If you try to start your jail and you run into output that looks like this:

FreeBSD10# /etc/rc.d/jail start centos6
Starting jails: cannot start jail  "centos6.dianshan": 
4
jail: centos6: getpwnam root: No such file or directory
jail: centos6: /bin/sh /etc/rc.d/rc 3: failed
.
/etc/rc.d/jail: WARNING: Per-jail configuration via jail_* variables  is obsolete.  Please consider to migrate to /etc/jail.conf.
FreeBSD10#

You have likely messed up your password database.  Try revisiting the steps above and recreating the password database after seeding it with the proper root line in master.passwd.  The relevant steps are below:

FreeBSD10# cd /usr/jails/centos6/etc 
FreeBSD10# echo "root::0:0::0:0:Charlie &:/root:/bin/bash" > master.passwd
FreeBSD10# pwd_mkdb -d ./ -p master.passwd
pwd_mkdb: warning, unknown root shell
FreeBSD10# chroot /usr/jails/centos6 /bin/bash
[root@FreeBSD10 bin]# cd /etc                      #OMG how great is this?
[root@FreeBSD10 etc]# pwconv
[root@FreeBSD10 etc]# grpconv                      #OMG it worked
[root@FreeBSD10 etc]# passwd                       #OMG it wasn't a fluke
Changing password for user root.
New password: 
BAD PASSWORD: it is based on a dictionary word     #YOLO
Retype new password: 
passwd: all authentication tokens updated successfully.

That should recreate the thing and, in our case, hopefully rectify your problem.  Let me know if you get stuck.

Postscript

I’ve been reading this, and despite its typos and unpolished quality, it tracks a pleasantly edifying route through the FreeBSD kernel and operating system for the purpose of understanding FreeBSD’s emulation capabilities.

Advertisements
This entry was posted in Information Technology and tagged , , , . Bookmark the permalink.

7 Responses to The Midnight Oil: Jailing CentOS 6 in FreeBSD 10.2 Edition

  1. Vikash says:

    Thanks for this quide

    i tried to take it one step further:

    [root@centos /]# yum install wget
    Traceback (most recent call last):
    File “/usr/bin/yum”, line 4, in
    import yum
    File “/usr/lib/python2.6/site-packages/yum/__init__.py”, line 47, in
    import tempfile
    File “/usr/lib/python2.6/tempfile.py”, line 34, in
    from random import Random as _Random
    File “/usr/lib/python2.6/random.py”, line 873, in
    _inst = Random()
    File “/usr/lib/python2.6/random.py”, line 96, in __init__
    self.seed(x)
    File “/usr/lib/python2.6/random.py”, line 110, in seed
    a = long(_hexlify(_urandom(16)), 16)
    RuntimeError: Failed to read %zi bytes from /dev/urandom

    any ideas ?

  2. dmehler937 says:

    Hello,

    I’m following this post and trying to start centos6 in a freebsd 10.2 jail.

    The only thing I’m getting is a repeating message in the log:

    cat /proc/cmdline no such file or directory

    and nothing else. Needless to say the jail does not start.

    Any ideas?

    Thanks.

    • Hey, sorry for the long delay, but could you show me the output of mount from within the jail? We should see linprocfs mounted, and I’m guessing it’s not. If it is, we’ll try an ls -l /proc (from within the jail) to see what’s going on in there.

  3. Kavitha says:

    hi
    i have some issuses getting centos 7.2 on freebsd ?Have you tried centos 7.2 version of i386 on FreeBSD10.1

    • I have not tried CentOS 7.2 yet – it relies on Kernel 3.10, so it’s not as likely that it will work. The BSD kernel’s ABI is only just now officially at a level where CentOS 5’s kernel is fully supported.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s