Date: Thu, 16 Jul 2015 18:42:26 +0300 From: Dmitry Sivachenko <trtrmitya@gmail.com> To: hackers@freebsd.org Subject: Strange memory management with mmap() Message-ID: <FDB6E0F9-A3FF-4194-83C1-A3121CBAE407@gmail.com>
next in thread | raw e-mail | index | archive | help
--Apple-Mail=_F134DC1F-8226-4CB0-B2E6-3B8F9A79A1BA
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=us-ascii
Hello!
I am using FreeBSD-10-stable and writing a program that uses large data =
file via mmap() in read only mode.
To be specific, I have 256GB RAM machine and typical size of data file =
is ~160GB (more than 1/2 of RAM and less that the whole RAM).
There is no other programs running during the test.
Consider the following use case: I have two files on disk. I mmap() the =
first one and prefetch data to RAM (touch every page of the file).
After that I expect all data to be cached in RAM and subsequent access =
will be fast.
Next I do munmap() on the first file, mmap() the second one and do the =
same test: prefetch data and expect it to be cached in RAM (and some of =
the pages belonging to the first file to be purged out, because =
size_of(file1)+size_of(file2) > size_of(RAM).
Please find my test program attached.
I run the program with 2 files provided via command line (both about =
160GB).
What I observe in real is:
-- before I run the program all RAM is in FREE state as reported by =
top(1).
-- after first prefetch() of the first file, all it's data goes to =
"Cache" state, RES column of the process remains the same (small)
-- second prefetch() works fast as expected, memory goes from Cache to =
Active state, RES column of the process grows up to match file size =
(SIZE=3D=3DRES now)
-- now first prefetch() for second file starts: the remaining Free =
memory goes to Cache state, Active size still equals to first file size.
-- second prefetch() for second file works as slow as first one, like if =
nothing was cached in memory during the first prefetch() run, RES column =
does not change.
Here is the output:
% /tmp/a.out file1.dat file2.dat
file1.dat... First prefault time: 1235.747351 seconds
Second prefault time: 74.893323 seconds
Ok.
file2.dat... First prefault time: 1316.405527 seconds
Second prefault time: 1311.491842 seconds
Ok.
I treat this like the bug somewhere in virtual memory management. Am I =
right or my expectations about how that test program should work are =
false?
Thanks in advance.
--Apple-Mail=_F134DC1F-8226-4CB0-B2E6-3B8F9A79A1BA
Content-Disposition: attachment;
filename=mmap_test.c
Content-Type: application/octet-stream;
name="mmap_test.c"
Content-Transfer-Encoding: 7bit
#include <err.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>
static void prefault(const char *buf, size_t s) {
volatile const char *p;
size_t page_size;
if (s == 0)
return;
page_size = sysconf(_SC_PAGESIZE);
for (p = buf; p < buf + s; p += page_size)
*p;
*(volatile const char *)(buf + s - 1);
}
int main(int argc, char* argv[]) {
if (argc < 2) {
fprintf(stderr, "Usage: %s <file> ...\n", argv[0]);
exit(0);
}
int i, fd;
struct stat st;
void *p;
struct timeval tv1, tv2;
for (i=1; i < argc; i++) {
printf("%s... ", argv[i]);
if ((fd = open(argv[i], O_RDONLY)) < 0)
err(1, "open");
if (fstat(fd, &st) != 0)
err(1, "fstat");
if (st.st_size > 0) {
if ((p = mmap(NULL, st.st_size, PROT_READ, MAP_NOCORE, fd, 0)) == MAP_FAILED)
err(1, "mmap");
gettimeofday(&tv1, NULL);
prefault(p, st.st_size);
gettimeofday(&tv2, NULL);
printf("First prefault time: %f seconds\n",
(double) (tv2.tv_usec - tv1.tv_usec) / 1000000 +
(double) (tv2.tv_sec - tv1.tv_sec));
gettimeofday(&tv1, NULL);
prefault(p, st.st_size);
gettimeofday(&tv2, NULL);
printf("Second prefault time: %f seconds\n",
(double) (tv2.tv_usec - tv1.tv_usec) / 1000000 +
(double) (tv2.tv_sec - tv1.tv_sec));
if (munmap(p, st.st_size) != 0)
err(1, "munmap");
}
close(fd);
printf("Ok.\n");
}
return 0;
}
--Apple-Mail=_F134DC1F-8226-4CB0-B2E6-3B8F9A79A1BA--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?FDB6E0F9-A3FF-4194-83C1-A3121CBAE407>