Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 4 Oct 2006 23:56:17 -0600
From:      "Chad Leigh -- Shire.Net LLC" <>
To:        FreeBSD Questions <>
Cc:        Chad Leigh <>
Subject:   apache in "strange" jail getting permissions errors
Message-ID:  <>

Next in thread | Raw E-Mail | Index | Archive | Help

Content-Transfer-Encoding: 7bit
Content-Type: text/plain;


I run a jails (have for the past 18 months in deployment and a year  
before that in testing) with the following setup, currently on 6.1- 
RELEASE.  This works fine and I have no issues.  I am trying to work  
on a new set up that is giving me issues.

Here is the set up that works:

I create a master jail that I do not "boot".

I create an md(4) memory backed disk using a regular disk file (vnode  
mode) as backings tore.  I install the basic directories found in /  
in this md device (no files, just the directories).  I then populate  
the /etc and /var as well as some special ones (/local and /stubs for  
example) that are used for jail-specific stuff (/local for all the  
"customer" stuff and /stubs for jail specific system stuff like a  
place for /usr/local to link in to).  I then use nullfs mounts to  
mount from the master jail, the /lib /bin /libexec /usr and /sbin  
directories in read only fashion.  I can then boot the jail and it  
runs find.  I also have a /usr/public which is read only place for  
ports to install in to.  I have appropriate links out of the read  
only directories into local per jail (per md(4) space) directories.   
What I just described all works fine and I can upgrade jails really  
easily as I just have to upgrade the master jail installation, watch  
for any important etc changes which have to be done individually, and  
update my one set of ports [and apps built from source without ports]  
in the /usr/public and all the jails get all the changes.

What I want to do is use my Solaris 10 server with 1.7TB ZFS file  
system exported through NFS as the root for each jail, with the same  
nullfs mounts as used above in the md(4) version.  This actually  
works in my test jail (but I have not tried to run any applications  
inside -- just boot it and log in and do basic shell things).  I  
tried last night to move an existing jail that runs apache2 for my  
dad and a few of his family genealogy sites into such a nfs backed  
jail.  Apache through fits and I saw that you have to have some local  
space for LockFile, SSLMutex, and a few other mod specific things.   
So what I did is set up a local directory on the FBSD system with the  
normal / directories as I do above in the md(4) way of doing things  
and left a local directory for the apache stuff. I then used nullfs  
to mount the same dirs as above and then nfs to mount the rest from  
the ZFS system.  It looks like this

solaris-i3:/local/jails/leigh/etc                  2.0G    439M     
1.6G    21%    /local/jails/leigh/etc
solaris-i3:/local/jails/leigh/home                 2.0G    439M     
1.6G    21%    /local/jails/leigh/home
solaris-i3:/local/jails/leigh/local                2.0G    439M     
1.6G    21%    /local/jails/leigh/local
solaris-i3:/local/jails/leigh/log                  2.0G    439M     
1.6G    21%    /local/jails/leigh/log
solaris-i3:/local/jails/leigh/root                 2.0G    439M     
1.6G    21%    /local/jails/leigh/root
solaris-i3:/local/jails/leigh/space                2.0G    439M     
1.6G    21%    /local/jails/leigh/space
solaris-i3:/local/jails/leigh/stubs                2.0G    439M     
1.6G    21%    /local/jails/leigh/stubs
solaris-i3:/local/jails/leigh/var                  2.0G    439M     
1.6G    21%    /local/jails/leigh/var
/local/jails/master/bin                            66G     59G     
2.4G    96%    /local/jails/leigh/bin
/local/jails/master/lib                            66G     59G     
2.4G    96%    /local/jails/leigh/lib
/local/jails/master/libexec                        66G     59G     
2.4G    96%    /local/jails/leigh/libexec
/local/jails/master/sbin                           66G     59G     
2.4G    96%    /local/jails/leigh/sbin
/local/jails/master/usr                            66G     59G     
2.4G    96%    /local/jails/leigh/usr
procfs                                            4.0K    4.0K       
0B   100%    /local/jails/leigh/proc
devfs                                             1.0K    1.0K       
0B   100%    /local/jails/leigh/dev

This boots fine, and apache no longer gives a fit (nothing in  
error_log) as I tell it to put its lock files and stuff in /tmp/ 
scratch which is a local space not nullfs mounted nor nfs mounted.

I can log in to the jail and do shell things.  I can change over to  
my dad's shell account and do things (read files and write new files  
or change files) as his user and group inside of the nfs mounted / 
local space.  There are NO permission problems (the nfs is mounted  
with maproot=root [or the equivalent solaris way of exporting it]  
etc) either root or the user shell account to do things inside the  
nfs mounted space.

However, apache, which runs as the user and group of my dad's login  
account since he is the sole user of this jail and sole user of  
apacge, cannot read any of the website files.  The exact same apache  
config file when using the md(4) backed space with all the same files  
and permissions, has no problems.

Here are some examples from the virtual host error files:

[Wed Oct 04 12:53:19 2006] [error] [client] (13) 
Permission denied: file permissions deny server access: /local/web/
[Wed Oct 04 12:53:19 2006] [error] [client] File does  
not exist: /local/web/
[Wed Oct 04 12:53:21 2006] [error] [client] (13) 
Permission denied: file permissions deny server access: /local/web/
[Wed Oct 04 12:53:21 2006] [error] [client] File does  
not exist: /local/web/
[Wed Oct 04 12:54:55 2006] [crit] [client] (13)Permission  
denied: /local/web/  
pcfg_openfile: unable to check htaccess file, ensure it is readable,  
[Wed Oct 04 12:54:56 2006] [crit] [client] (13)Permission  
denied: /local/web/  
pcfg_openfile: unable to check htaccess file, ensure it is readable,  
[Wed Oct 04 12:54:56 2006] [crit] [client] (13)Permission  
denied: /local/web/  
pcfg_openfile: unable to check htaccess file, ensure it is readable,  
[Wed Oct 04 12:54:58 2006] [crit] [client] (13)Permission  
denied: /local/web/  
pcfg_openfile: unable to check htaccess file, ensure it is readable,  
[Wed Oct 04 12:55:15 2006] [crit] [client] (13)Permission  
denied: /local/web/  
pcfg_openfile: unable to check htaccess file, ensure it is readable

Again, the user that apache is running as can access the files R/W no  
problem.   I have confirmed this by running a shell as the same user  
(and the same apache when running on the md(4) back jail with the  
same local directorys tructure with same user and permissions etc  
runs fine).  I also confirmed the permissions of everything  
visually.  user and group are r(+x as appropriate) including all dirs.

I am at a loss on why, when the apache virtual host document roots  
are living on an nfs mounted space this would happen.  (Again, the  
LockFile and other things like that [SSLMutex, mod)rewrite stuff] is  
set to a local space that is not nfs mounted and apache is not giving  
and crazy errors to indicate otherwise in the apache wide error_log  
like it did when I first started my attempts).

Any ideas or help would be appreciated.

In case anyone wants to know why I want to do this jail on nfs thing:

1)  I want to get rid of the md(4) backed devices as they seem to not  
be completely stable -- once in a while I get a lost I/O that leads  
to hanging the server. I cannot prove it is related to md(4) but it  
always starts in relation to a jail on the md(4) device (and when I  
try and do an ls of the file for example, once it starts, the ls of  
that file will hang while other ls won't).

2)  More importantly, I don't want a jail to be tied to a specific HW  
server.  I want to be able to move a jail easily around HW servers as  
needed.  For example, if a specific server were to develop HW  
problems, I could easily shut it down and bring up its allotment of  
jails on another server or set of servers with very minimal downtime  
to my customers.  Right now, with the md(4) backed devices, the jails  
live where their storage backing is and each server has their own set  
of disks/storage.  If a server dies, I cannot easily bring up all my  
jails on another server without HW hacking.  I know I have a single  
point of failure with the Solaris server and its large disk array but  
it is easier to throw money at a single file server and have a lot of  
smaller minimal front end servers than have to buy larger beefier  
front end servers in multiples.  The solaris server has 2 areca raid  
6 arrays (1 installed and 1 to be installed later this Fall) mirrored  
together using ZFS, has a 2+1 redundant power supply that will be  
spread across 2 circuits, battery backed raid arrays, etc. is on  
hospital grade power with UPS etc. with extra spare parts to quickly  
replace things that fail.


Chad Leigh -- Shire.Net LLC
Your Web App and Email hosting provider
chad at


Want to link to this message? Use this URL: <>