Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 15 May 2013 12:22:12 -0400
From:      Outback Dingo <outbackdingo@gmail.com>
To:        Martin Simmons <martin@lispworks.com>
Cc:        freebsd-fs@freebsd.org
Subject:   Re: FreeBSD 9-RELEASE zpool Out of File Descriptors ??
Message-ID:  <CAKYr3zxS-3HxEFiLGA84R9FXOw75fRcA=yGxkxLQRnSyBZd0SA@mail.gmail.com>
In-Reply-To: <201305151617.r4FGHRGo031542@higson.cam.lispworks.com>
References:  <CAKYr3zwer=P=GhW9WyxrMpLv4%2BAkxBfJkuZk99RTHtRW%2B-%2B24g@mail.gmail.com> <op.ww4kn4wx8527sy@ronaldradial.versatec.local> <CAKYr3zznxT7iL01ts1Cs9Eur-QkkK5mkKnb=qKPG-2UX%2BraqEw@mail.gmail.com> <op.ww4k7gmr8527sy@ronaldradial.versatec.local> <CAKYr3zznHFWJTiM5B980s7-2JDU99Dk2svf_%2BaUhhMZngFkayA@mail.gmail.com> <1CE05B33-7639-4F7F-9FD6-78B27D31E186@my.gd> <CAKYr3zwhcyto4ZPOs9_1EJx1N6jRF35E=RJvfw-3FsCesbdhkA@mail.gmail.com> <201305151617.r4FGHRGo031542@higson.cam.lispworks.com>

next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, May 15, 2013 at 12:17 PM, Martin Simmons <martin@lispworks.com>wrote:

> >>>>> On Wed, 15 May 2013 11:56:50 -0400, Outback Dingo said:
> >
> > On Wed, May 15, 2013 at 11:49 AM, Fleuriot Damien <ml@my.gd> wrote:
> >
> > >
> > > On May 15, 2013, at 2:46 PM, Outback Dingo <outbackdingo@gmail.com>
> wrote:
> > >
> > > > On Wed, May 15, 2013 at 8:34 AM, Ronald Klop <
> > > ronald-freebsd8@klop.yi.org>wrote:
> > > >
> > > >> On Wed, 15 May 2013 14:27:10 +0200, Outback Dingo <
> > > outbackdingo@gmail.com>
> > > >> wrote:
> > > >>
> > > >> On Wed, May 15, 2013 at 8:22 AM, Ronald Klop <
> > > ronald-freebsd8@klop.yi.org
> > > >>>> **wrote:
> > > >>>
> > > >>> On Wed, 15 May 2013 14:13:32 +0200, Outback Dingo <
> > > >>>> outbackdingo@gmail.com>
> > > >>>> wrote:
> > > >>>>
> > > >>>> So it seems a new deployment we just built with zfsonroot mirror
> and a
> > > >>>>
> > > >>>>> 48TB
> > > >>>>> master pool is already out of File Descriptors???
> > > >>>>>
> > > >>>>>  pool: master
> > > >>>>> state: ONLINE
> > > >>>>> status: The pool is formatted using a legacy on-disk format.  The
> > > pool
> > > >>>>> can
> > > >>>>>        still be used, but some features are unavailable.
> > > >>>>> action: Upgrade the pool using 'zpool upgrade'.  Once this is
> done,
> > > the
> > > >>>>>        pool will no longer be accessible on software that does
> not
> > > >>>>> support
> > > >>>>> feature
> > > >>>>>        flags.
> > > >>>>>  scan: none requested
> > > >>>>> config:
> > > >>>>>
> > > >>>>>        NAME                      STATE     READ WRITE CKSUM
> > > >>>>>        master                    ONLINE       0     0     0
> > > >>>>>          raidz3-0                ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN01  ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN03  ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN04  ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN05  ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN07  ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN08  ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN09  ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN10  ONLINE       0     0     0
> > > >>>>>          raidz3-1                ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN11  ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN12  ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN13  ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN14  ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN15  ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN16  ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN17  ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN18  ONLINE       0     0     0
> > > >>>>>          raidz3-2                ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN19  ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN20  ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN21  ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN22  ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN23  ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN24  ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN26  ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN27  ONLINE       0     0     0
> > > >>>>>          raidz3-3                ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN29  ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN30  ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN32  ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN33  ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN35  ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN36  ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN37  ONLINE       0     0     0
> > > >>>>>            multipath/SATA_LUN38  ONLINE       0     0     0
> > > >>>>>        logs
> > > >>>>>          multipath/SATA_LUN06    ONLINE       0     0     0
> > > >>>>>        cache
> > > >>>>>          multipath/SATA_LUN02    ONLINE       0     0     0
> > > >>>>>
> > > >>>>> errors: No known data errors
> > > >>>>>  pool: tank
> > > >>>>> state: ONLINE
> > > >>>>> status: The pool is formatted using a legacy on-disk format.  The
> > > pool
> > > >>>>> can
> > > >>>>>        still be used, but some features are unavailable.
> > > >>>>> action: Upgrade the pool using 'zpool upgrade'.  Once this is
> done,
> > > the
> > > >>>>>        pool will no longer be accessible on software that does
> not
> > > >>>>> support
> > > >>>>> feature
> > > >>>>>        flags.
> > > >>>>>  scan: none requested
> > > >>>>> config:
> > > >>>>>
> > > >>>>>        NAME        STATE     READ WRITE CKSUM
> > > >>>>>        tank        ONLINE       0     0     0
> > > >>>>>          mirror-0  ONLINE       0     0     0
> > > >>>>>            da34p3  ONLINE       0     0     0
> > > >>>>>            da35p3  ONLINE       0     0     0
> > > >>>>>
> > > >>>>> errors: No known data errors
> > > >>>>>
> > > >>>>> while compiling a kernel, buildworld worked and installed ok
> > > >>>>>
> > > >>>>> lex -t
> > > >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/****
> > > >>>>> aicasm/aicasm_scan.l
> > > >>>>>
> > > >>>>> aicasm_scan.c
> > > >>>>>>
> > > >>>>>> lex -t  -Pmm
> > > >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/****
> > > >>>>>
> > > >>>>> aicasm/aicasm_macro_scan.l
> > > >>>>>
> > > >>>>> aicasm_macro_scan.c
> > > >>>>>>
> > > >>>>>> rm -f .depend_aicasm
> > > >>>>> mkdep -f .depend_aicasm -a    -I.
> > > >>>>> -I/usr/src/sys/modules/****aic7xxx/aicasm/../../../dev/****
> > > >>>>> aic7xxx/aicasm
> > > >>>>> -std=gnu99
> > > >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/****
> > > >>>>> aicasm/aicasm.c
> > > >>>>> /usr/src/sys/modules/aic7xxx/****aicasm/../../../dev/aic7xxx/****
> > > >>>>>
> > > >>>>> aicasm/aicasm_symbol.c
> > > >>>>> aicasm_gram.c aicasm_macro_gram.c aicasm_scan.c
> aicasm_macro_scan.c
> > > >>>>> Out of file descriptors
> > > >>>>> *** [.depend_aicasm] Error code 2
> > > >>>>>
> > > >>>>> Stop in /usr/src/sys/modules/aic7xxx/****aicasm.
> > > >>>>>
> > > >>>>> *** [buildkernel] Error code 1
> > > >>>>>
> > > >>>>> Stop in /usr/src.
> > > >>>>> *** [buildkernel] Error code 1
> > > >>>>>
> > > >>>>>
> > > >>>>
> > > >>>> Do you know what a file descriptor is?
> > > >>>>
> > > >>>> There is something about it here:
> http://www.freebsd.org/doc/en/****<;
> > > http://www.freebsd.org/doc/en/**>;
> > > >>>> books/handbook/configtuning-****kernel-limits.html<http://www.**
> > > >>>>
> freebsd.org/doc/en/books/**handbook/configtuning-kernel-**limits.html
> > > <
> > >
> http://www.freebsd.org/doc/en/books/handbook/configtuning-kernel-limits.html
> > > >>(search
> > > >>>> for 'file descriptor')
> > > >>>> Or more info in here: https://www.google.nl/search?****<;
> > > https://www.google.nl/search?**>;
> > > >>>> q=freebsd+Out+of+file+****descriptors<https://www.**
> > > >>>> google.nl/search?q=freebsd+**Out+of+file+descriptors<
> > > https://www.google.nl/search?q=freebsd+Out+of+file+descriptors>;
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>> It mainly says that you have more files open than your system is
> > > >>>> configured to allow. This is often produced by a bug in a program
> > > which
> > > >>>> does not close some files properly.
> > > >>>>
> > > >>>>
> > > >>>> Yes I do.... but this is a brand new zfsonroot with barely any
> data on
> > > >>> it
> > > >>> and
> > > >>>
> > > >>> sysctl -a | grep kern.openfiles
> > > >>> kern.openfiles: 68
> > > >>> root@:/master/builder # sysctl -a | grep kern.maxfiles
> > > >>> kern.maxfiles: 24600
> > > >>> kern.maxfilesperproc: 11095
> > > >>>
> > > >>> the openfiles and maxfiles seems to be plenty, but im out of
> > > descriptors,
> > > >>> i
> > > >>> used to see this back in the 4.x days when you could format
> > > >>> ufs with larger inodes, but zfs ??? really?
> > > >>>
> > > >>
> > > >>
> > > >> Is this kern.openfiles when idle or during your buildkernel?
> > > >> Did you check 'ulimit -a'?
> > > >> You say installworld worked ok while you are just now doing
> buildkernel.
> > > >> Are your kernel and world out of sync?
> > > >>
> > > >>
> > > > well, i was upgrading FreeBSD 9-RELEASE to CURRENT, did a make
> > > buildworld,
> > > > then a make installworld, when i went to do a make buildkernel
> > > > thats the error i got, so id say, yes it is "out of sync"
> > >
> > >
> > > Wow wow wow, hold on a sec here.
> > >
> > > You're installing your new world before the kernel ?
> > >
> > >
> > > I've had no trouble upgrading several boxes from 8-STABLE to
> 10.0-CURRENT,
> > > following the regular procedure as described at:
> > > http://www.freebsd.org/doc/handbook/makeworld.html
> > >
> > >
> > > Note that the correct order is:
> > > - buildworld (or kernel-toolchain if you only want to test the kernel
> and
> > > not do the whole world)
> > > - buildkernel
> > > - installkernel
> > > - reboot
> > > - mergemaster -p (preferably in single user, I've always run it
> multiuser
> > > w/o problems)
> > > - installworld
> > > - mergemaster
> > > - reboot
> > >
> > > You don't want to run the new world on the old kernel.
> > >
> >
> > yeah i know better, seems when i went to install the old kernel the build
> > had failed, i can get through it from here
> >
> > it was odd because the CURRENT kernel failed to build with a yyparse
> error,
> > until after i had completed a buildworld
>
> Taking a zfs snapshot of the system file systems before installing
> anything is
> a good idea.
>

yeah in hindsight, it is actually

>
> __Martin
>



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAKYr3zxS-3HxEFiLGA84R9FXOw75fRcA=yGxkxLQRnSyBZd0SA>