Introduction
CentOS 6! In a jail on the FreeBSD 10.2 kernel! THAT’S EFFING AWESOME.
Inspired by the recent release of FreeBSD 10.2 and the accompanying announcement that the Application Binary Interface Linux compatibility version has been upgraded to 2.6.18, I thought an attempt at running the CentOS 6 userland (which relies on Linux kernel version 2.6.32) in a FreeBSD 10.2 jail might be warranted.
Of course, it’s CentOS 5 that runs on Linux kernel 2.6.18, so CentOS 5 should work very well in a jail, but I thought I might try pushing the envelope given the extant proof of concept (which itself likely needs updating for use in FreeBSD 10.2) for running CentOS 5 userland in FreeBSD 8 jails. To my knowledge, there has been no previous successful demonstration of the capacity to host the CentOS 6 userland in a FreeBSD jail.
CentOS 6 runs on Linux kernel version 2.6.32, so it is quite possible that some requisite kernel functionality will be absent from our FreeBSD kernel’s ABI. If there prove to be insurmountable obstacles, downgrading to the CentOS 5 userland should be a reasonable option as, theoretically, the CentOS 5 userland should operate without a problem with the FreeBSD 10.2 kernel’s ABI. The incentive to run CentOS 6 rather than 5 is the standard OS fare: though alive, CentOS 5 is significantly older with a noticeable lack of current features when compared with CentOS 6, and its end of life will come in 2017, with CentOS 6 alive and well until 2020.
All that said, I just got this up and running after some hours spent cobbling together instructions from the various sources to which I link below, filling in the gaps, and then troubleshooting some issues (one of which is listed at the bottom for anyone else who encounters it), and it works! I am sure there are limitations, and I want to see how sophisticated this solution can get (wouldn’t it be cool if we could run CentOS 6 software offerings in FreeBSD 10.2 jails with confidence in their functionality and stability?), but it works at a rudimentary level at this point.
I am most pleased.
Process
1) Configure Linux Binary Compatibility and load additional required modules:
FreeBSD10# kldload linux fdescfs linprocfs linsysfs tmpfs FreeBSD10# kldstat Id Refs Address Size Name 1 21 0xffffffff80200000 179ddb0 kernel 2 1 0xffffffff81a11000 641b nullfs.ko 3 1 0xffffffff81a18000 56aa fdescfs.ko 4 3 0xffffffff81a1e000 43c7c linux.ko 5 1 0xffffffff81a62000 9da0 linprocfs.ko 6 1 0xffffffff81a6c000 55ad linsysfs.ko 7 1 0xffffffff81a72000 a77f tmpfs.ko FreeBSD10# portsnap fetch FreeBSD10# portsnap extract FreeBSD10# cd /usr/ports/emulators FreeBSD10# make install clean ===> linux_base-f10-10_9 has known vulnerabilities: linux_base-f10-10_9 is vulnerable: glibc -- gethostbyname buffer overflow CVE: CVE-2015-0235 WWW: https://vuxml.FreeBSD.org/freebsd/0765de84-a6c1-11e4-a0c1-c485083ca99c.html 1 problem(s) in the installed packages found. => Please update your ports tree and try again. => Note: Vulnerable ports are marked as such even if there is no update available. => If you wish to ignore this vulnerability rebuild with 'make DISABLE_VULNERABILITIES=yes' *** Error code 1 Stop. make: stopped in /usr/ports/emulators/linux_base-f10 FreeBSD10# portsnap update Ports tree is already up to date. FreeBSD10# make DISABLE_VULNERABILITIES=yes install clean #YOLO lab machine
2) Acquire CentOS 6 tarball (Got the idea from this guy):
FreeBSD10# pkg install wget ... FreeBSD10# wget http://download.openvz.org/template/precreated/centos-6-x86.tar.gz #FreeBSD only supports 32 bit emulation
3) Unpack the tarball into the centos6 root directory.
FreeBSD10#: tar xf centos-6-x86.tar.gz -C /usr/jails/centos6
4) Mount the necessary virtual file systems into the jail’s root directory:
FreeBSD10# cd /usr/jails/centos6 FreeBSD10# mount -t linprocfs linprocfs proc FreeBSD10# mount -t linsysfs linsysfs sys FreeBSD10# mount -t devfs devfs dev FreeBSD10# mount /dev/vtbd0p2 on / (ufs, local, journaled soft-updates) devfs on /dev (devfs, local, multilabel) linprocfs on /usr/jails/centos6/proc (linprocfs, local) linsysfs on /usr/jails/centos6/sys (linsysfs, local) devfs on /usr/jails/centos6/dev (devfs, local, multilabel)
5) Configure the jail in /etc/rc.conf:
FreeBSD10# vim /etc/rc.conf jail_enable="YES" jail_list="centos6" #cloned_interfaces="lo2" #jail_centos6_interface="lo2" jail_centos6_interface="em0" jail_centos6_ip="127.0.2.1" jail_centos6_hostname="centos6.dianshan" jail_centos6_set_hostname_allow="YES" jail_centos6_devfs_enable="YES" jail_centos6_fdescfs_enable="YES" jail_centos6_procfs_enable="YES" jail_centos6_rootdir="/usr/jails/centos6" jail_centos6_exec_start="/etc/init.d/rc 3" jail_centos6_flags="-l -u root" jail_centos6_fstab="/usr/jails/fstab_centos" :wq
6) Write some basic CentOS configuration files:
FreeBSD10# cp /etc/resolv.conf /usr/jails/centos6/etc/resolv.conf FreeBSD10# echo "NETWORKING=yes" >> etc/sysconfig/network FreeBSD10# echo "linproc /jails/centos/proc linprocfs rw 0 0" >> /usr/jails/fstab_centos6 FreeBSD10# echo "192.168.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 centos6 centos6.dianshan" >> /usr/jails/centos6/etc/hosts
That last command probably mishandles the loopback creation in the jail; I’ll most likely be revising it later.
7) Create the password database for the jail:
FreeBSD10# cd /usr/jails/centos6/etc FreeBSD10# echo "root::0:0::0:0:Charlie &:/root:/bin/bash" > master.passwd FreeBSD10# pwd_mkdb -d ./ -p master.passwd pwd_mkdb: warning, unknown root shell
That last remark is ok; pwd_mkdb is just informing us that the host FreeBSD system doesn’t know what /bin/bash is. No matter, since our CentOS 6 system in the jail will know.
8) chroot into the centos6 directory and prepare the environment (bless the author of this post):
FreeBSD10# chroot /usr/jails/centos6 /bin/bash [root@FreeBSD10 bin]# cd /etc #OMG how great is this? [root@FreeBSD10 etc]# pwconv [root@FreeBSD10 etc]# grpconv #OMG it worked [root@FreeBSD10 etc]# passwd #OMG it wasn't a fluke Changing password for user root. New password: BAD PASSWORD: it is based on a dictionary word #YOLO Retype new password: passwd: all authentication tokens updated successfully. [root@FreeBSD10 etc]# touch /etc/fstab [root@FreeBSD10 etc]# touch /etc/mtab [root@FreeBSD10 etc]# cd /sbin [root@FreeBSD10 sbin]# mv consoletype consoletype.orig [root@FreeBSD10 sbin]# ln -s /bin/true consoletype [root@FreeBSD10 sbin]# cd /bin [root@FreeBSD10 bin]# mv umount umount.prev [root@FreeBSD10 bin]# ln -s /bin/true umount [root@FreeBSD10 bin]# chkconfig --list crond 0:off 1:off 2:on 3:on 4:on 5:on 6:off htcacheclean 0:off 1:off 2:off 3:off 4:off 5:off 6:off httpd 0:off 1:off 2:off 3:on 4:off 5:off 6:off ip6tables 0:off 1:off 2:on 3:off 4:on 5:on 6:off iptables 0:off 1:off 2:on 3:on 4:on 5:on 6:off modules_dep 0:off 1:off 2:on 3:on 4:on 5:on 6:off named 0:off 1:off 2:off 3:off 4:off 5:off 6:off netconsole 0:off 1:off 2:off 3:off 4:off 5:off 6:off netfs 0:off 1:off 2:off 3:off 4:on 5:on 6:off network 0:off 1:off 2:on 3:on 4:on 5:on 6:off nmb 0:off 1:off 2:off 3:off 4:off 5:off 6:off nscd 0:off 1:off 2:off 3:off 4:off 5:off 6:off portreserve 0:off 1:off 2:on 3:off 4:on 5:on 6:off quota_nld 0:off 1:off 2:off 3:off 4:off 5:off 6:off rdisc 0:off 1:off 2:off 3:off 4:off 5:off 6:off restorecond 0:off 1:off 2:off 3:off 4:off 5:off 6:off rpcbind 0:off 1:off 2:on 3:off 4:on 5:on 6:off rsyslog 0:off 1:off 2:on 3:on 4:on 5:on 6:off saslauthd 0:off 1:off 2:off 3:on 4:off 5:off 6:off sendmail 0:off 1:off 2:on 3:on 4:on 5:on 6:off smb 0:off 1:off 2:off 3:off 4:off 5:off 6:off snmpd 0:off 1:off 2:off 3:off 4:off 5:off 6:off snmptrapd 0:off 1:off 2:off 3:off 4:off 5:off 6:off sshd 0:off 1:off 2:on 3:on 4:on 5:on 6:off udev-post 0:off 1:on 2:on 3:off 4:on 5:on 6:off winbind 0:off 1:off 2:off 3:off 4:off 5:off 6:off xinetd 0:off 1:off 2:off 3:on 4:on 5:on 6:off xinetd based services: chargen-dgram: off chargen-stream: off daytime-dgram: off daytime-stream: off discard-dgram: off discard-stream: off echo-dgram: off echo-stream: off finger: off ntalk: off rsync: off talk: off tcpmux-server: off time-dgram: off time-stream: off [root@FreeBSD10 bin]# chkconfig httpd off [root@FreeBSD10 bin]# chkconfig ip6tables off [root@FreeBSD10 bin]# exit exit #ZOOOMG so awesome
9) And start the jail
FreeBSD10# /etc/rc.d/jail start centos6 Starting jails: centos6.dianshan. /etc/rc.d/jail: WARNING: Per-jail configuration via jail_* variables is obsolete. Please consider to migrate to /etc/jail.conf.
IT LIIIIIIIIIIIIIIIIIIIIIVES!!!!
10) Connect in!
FreeBSD10# jls JID IP Address Hostname Path 7 192.168.122.200 centos6.dianshan /usr/jails/centos6 FreeBSD10# jexec 7 /bin/bash [root@centos6 /]#
omg. It’s up and running.
That is awesome.
Troubleshooting Tips
If you try to start your jail and you run into output that looks like this:
FreeBSD10# /etc/rc.d/jail start centos6 Starting jails: cannot start jail "centos6.dianshan": 4 jail: centos6: getpwnam root: No such file or directory jail: centos6: /bin/sh /etc/rc.d/rc 3: failed . /etc/rc.d/jail: WARNING: Per-jail configuration via jail_* variables is obsolete. Please consider to migrate to /etc/jail.conf. FreeBSD10#
You have likely messed up your password database. Try revisiting the steps above and recreating the password database after seeding it with the proper root line in master.passwd. The relevant steps are below:
FreeBSD10# cd /usr/jails/centos6/etc FreeBSD10# echo "root::0:0::0:0:Charlie &:/root:/bin/bash" > master.passwd FreeBSD10# pwd_mkdb -d ./ -p master.passwd pwd_mkdb: warning, unknown root shell FreeBSD10# chroot /usr/jails/centos6 /bin/bash [root@FreeBSD10 bin]# cd /etc #OMG how great is this? [root@FreeBSD10 etc]# pwconv [root@FreeBSD10 etc]# grpconv #OMG it worked [root@FreeBSD10 etc]# passwd #OMG it wasn't a fluke Changing password for user root. New password: BAD PASSWORD: it is based on a dictionary word #YOLO Retype new password: passwd: all authentication tokens updated successfully.
That should recreate the thing and, in our case, hopefully rectify your problem. Let me know if you get stuck.
Postscript
I’ve been reading this, and despite its typos and unpolished quality, it tracks a pleasantly edifying route through the FreeBSD kernel and operating system for the purpose of understanding FreeBSD’s emulation capabilities.
Engineering Walden 
Thanks for this quide
i tried to take it one step further:
[root@centos /]# yum install wget
Traceback (most recent call last):
File “/usr/bin/yum”, line 4, in
import yum
File “/usr/lib/python2.6/site-packages/yum/__init__.py”, line 47, in
import tempfile
File “/usr/lib/python2.6/tempfile.py”, line 34, in
from random import Random as _Random
File “/usr/lib/python2.6/random.py”, line 873, in
_inst = Random()
File “/usr/lib/python2.6/random.py”, line 96, in __init__
self.seed(x)
File “/usr/lib/python2.6/random.py”, line 110, in seed
a = long(_hexlify(_urandom(16)), 16)
RuntimeError: Failed to read %zi bytes from /dev/urandom
any ideas ?
Hmm..looks like it’s failing to draw output from /dev/urandom. That probably won’t be too hard to resolve – I’ll see what I can do when I get a chance. If you get to it before me, let me know!
found my problem
devfs was not mounted
thanks
Hello,
I’m following this post and trying to start centos6 in a freebsd 10.2 jail.
The only thing I’m getting is a repeating message in the log:
cat /proc/cmdline no such file or directory
and nothing else. Needless to say the jail does not start.
Any ideas?
Thanks.
Hey, sorry for the long delay, but could you show me the output of
mountfrom within the jail? We should see linprocfs mounted, and I’m guessing it’s not. If it is, we’ll try anls -l /proc(from within the jail) to see what’s going on in there.hi
i have some issuses getting centos 7.2 on freebsd ?Have you tried centos 7.2 version of i386 on FreeBSD10.1
I have not tried CentOS 7.2 yet – it relies on Kernel 3.10, so it’s not as likely that it will work. The BSD kernel’s ABI is only just now officially at a level where CentOS 5’s kernel is fully supported.