Skip site navigation (1)Skip section navigation (2)
Date:      Sun, 12 Dec 2010 22:40:12 +0200
From:      Andriy Gapon <avg@freebsd.org>
To:        =?x-viet-vps?Q?Martin_Matus=28ka?= <mm@freebsd.org>, freebsd-multimedia@freebsd.org
Subject:   ffmpeg and mmap
Message-ID:  <4D05332C.7010208@freebsd.org>

next in thread | raw e-mail | index | archive | help

I've been getting some crash dumps in libswscale.so code.
The stack trace is always like this:
#0  0x000000083a6abf10 in ?? ()
#1  0x000000080a717dc6 in hyscale_fast_MMX2
#2  0x000000080a71bd64 in swScale_MMX2
#3  0x000000080a71ebf9 in sws_scale
...

>From disassembling I've identified that the crash happens as soon as inline
assembly in hyscale_fast_MMX2 calls code pointed to by lumMmx2FilterCode pointer.

The following code in libswscale/utils.c, function sws_getContext() is of interest:

#if ARCH_X86 && (HAVE_MMX2 || CONFIG_RUNTIME_CPUDETECT)
// can't downscale !!!
        if (c->canMMX2BeUsed && (flags & SWS_FAST_BILINEAR)) {
            c->lumMmx2FilterCodeSize = initMMX2HScaler(      dstW, c->lumXInc,
NULL, NULL, NULL, 8);
            c->chrMmx2FilterCodeSize = initMMX2HScaler(c->chrDstW, c->chrXInc,
NULL, NULL, NULL, 4);

#ifdef MAP_ANONYMOUS
            c->lumMmx2FilterCode = mmap(NULL, c->lumMmx2FilterCodeSize,
PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
            c->chrMmx2FilterCode = mmap(NULL, c->chrMmx2FilterCodeSize,
PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
#elif HAVE_VIRTUALALLOC
            c->lumMmx2FilterCode = VirtualAlloc(NULL, c->lumMmx2FilterCodeSize,
MEM_COMMIT, PAGE_EXECUTE_READWRITE);
            c->chrMmx2FilterCode = VirtualAlloc(NULL, c->chrMmx2FilterCodeSize,
MEM_COMMIT, PAGE_EXECUTE_READWRITE);
#else
            c->lumMmx2FilterCode = av_malloc(c->lumMmx2FilterCodeSize);
            c->chrMmx2FilterCode = av_malloc(c->chrMmx2FilterCodeSize);
#endif

            if (!c->lumMmx2FilterCode || !c->chrMmx2FilterCode)
                goto fail;
            FF_ALLOCZ_OR_GOTO(c, c->hLumFilter   , (dstW
/8+8)*sizeof(int16_t), fail);
            FF_ALLOCZ_OR_GOTO(c, c->hChrFilter   , (c->chrDstW
/4+8)*sizeof(int16_t), fail);
            FF_ALLOCZ_OR_GOTO(c, c->hLumFilterPos, (dstW
/2/8+8)*sizeof(int32_t), fail);
            FF_ALLOCZ_OR_GOTO(c, c->hChrFilterPos,
(c->chrDstW/2/4+8)*sizeof(int32_t), fail);

            initMMX2HScaler(      dstW, c->lumXInc, c->lumMmx2FilterCode,
c->hLumFilter, c->hLumFilterPos, 8);
            initMMX2HScaler(c->chrDstW, c->chrXInc, c->chrMmx2FilterCode,
c->hChrFilter, c->hChrFilterPos, 4);

#ifdef MAP_ANONYMOUS
            mprotect(c->lumMmx2FilterCode, c->lumMmx2FilterCodeSize, PROT_EXEC |
PROT_READ);
            mprotect(c->chrMmx2FilterCode, c->chrMmx2FilterCodeSize, PROT_EXEC |
PROT_READ);
#endif
        } else
#endif /* ARCH_X86 && (HAVE_MMX2 || CONFIG_RUNTIME_CPUDETECT) */


I've noticed that in our port lumMmx2FilterCode and chrMmx2FilterCode buffers
are allocated using av_malloc() (which is a wrapper around libc malloc) and
that's the source of the problem - on amd64 heap memory is not executable.  So
the attempt to execute code in the heap buffer results in a page fault and in
the crash (SIGBUS).

The buffer should be actually allocated via mmap() and then get proper
permissions via mprotect().  That doesn't happen because MAP_ANONYMOUS is not
defined in that context.

MAP_ANONYMOUS/MAP_ANON is not defined, because it is not specified in POSIX and
it is under _BSD_VISIBLE in sys/mman.h, but the code at hand is compiled with
_POSIX_SOURCE defined.  On FreeBSD _POSIX_SOURCE implies !_BSD_VISIBLE.

To get around that issue _on Linux_, ffmpeg folks have put explicit _SVID_SOURCE
definition right into the libswscale/utils.c file:
#define _SVID_SOURCE //needed for MAP_ANONYMOUS

For much the same reason we need to add the following for FreeBSD (as hackish as
it is):
#define __BSD_VISIBLE 1

With that addition the code buffer is allocated properly and the code in it can
be executed.

P.S. this page talks about a similar issue:
http://ubuntuforums.org/showthread.php?t=1547726
-- 
Andriy Gapon



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4D05332C.7010208>