Date: Sat, 30 Jan 2010 10:45:23 GMT From: Mikolaj Golub <to.my.trociny@gmail.com> To: freebsd-gnats-submit@FreeBSD.org Subject: bin/143368: [patch] awk(1): number of open files is limited to small constant Message-ID: <201001301045.o0UAjNsN094092@www.freebsd.org> Resent-Message-ID: <201001301050.o0UAo1li051418@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 143368
>Category: bin
>Synopsis: [patch] awk(1): number of open files is limited to small constant
>Confidential: no
>Severity: non-critical
>Priority: low
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Sat Jan 30 10:50:01 UTC 2010
>Closed-Date:
>Last-Modified:
>Originator: Mikolaj Golub
>Release:
>Organization:
>Environment:
FreeBSD zhuzha.ua1 8.0-STABLE FreeBSD 8.0-STABLE #6: Sun Jan 24 21:36:17 EET 2010 root@zhuzha.ua1:/usr/obj/usr/src/sys/GENERIC i386
>Description:
The maximal number of open files in awk(1) is limited by FOPEN_MAX constant defined for awk in usr.bin/awk/Makefile:
CFLAGS+= -DHAS_ISBLANK -I. -I${AWKSRC} -DFOPEN_MAX=64
This might be low for today needs and today hardware allows much higher values.
NetBSD has fixed their version of awk making the array of files dynamically allocated
http://www.netbsd.org/cgi-bin/query-pr-single.pl?number=37205
It would be nice to do the same in FreeBSD.
>How-To-Repeat:
kopusha:~% awk 'BEGIN {for (i=1; i <= 2000; ++i){print "test" > ("/tmp/file_" i)}}'
awk: /tmp/file_62 makes too many open files
source line number 1
>Fix:
See the attached patch adopted from NetBSD (PR/37205: Aleksey Cheusov: nawk: a number of open files is limited to very small constant. Make the array of files dynamically allocated.).
Patch attached with submission follows:
diff -ru contrib/one-true-awk.orig/run.c contrib/one-true-awk/run.c
--- contrib/one-true-awk.orig/run.c 2007-06-05 18:33:51.000000000 +0300
+++ contrib/one-true-awk/run.c 2010-01-30 12:27:38.000000000 +0200
@@ -1613,28 +1613,36 @@
FILE *fp;
const char *fname;
int mode; /* '|', 'a', 'w' => LE/LT, GT */
-} files[FOPEN_MAX] ={
- { NULL, "/dev/stdin", LT }, /* watch out: don't free this! */
- { NULL, "/dev/stdout", GT },
- { NULL, "/dev/stderr", GT }
-};
+} *files;
+size_t nfiles;
void stdinit(void) /* in case stdin, etc., are not constants */
{
+ nfiles = FOPEN_MAX;
+ files = calloc(nfiles, sizeof(*files));
+ if (files == NULL)
+ FATAL("can't allocate file memory for %zu files", nfiles);
files[0].fp = stdin;
+ files[0].fname = "/dev/stdin";
+ files[0].mode = LT;
files[1].fp = stdout;
+ files[1].fname = "/dev/stdout";
+ files[1].mode = GT;
files[2].fp = stderr;
+ files[2].fname = "/dev/stderr";
+ files[2].mode = GT;
}
FILE *openfile(int a, const char *us)
{
const char *s = us;
- int i, m;
+ size_t i;
+ int m;
FILE *fp = 0;
if (*s == '\0')
FATAL("null file name in print or getline");
- for (i=0; i < FOPEN_MAX; i++)
+ for (i = 0; i < nfiles; i++)
if (files[i].fname && strcmp(s, files[i].fname) == 0) {
if (a == files[i].mode || (a==APPEND && files[i].mode==GT))
return files[i].fp;
@@ -1644,11 +1652,19 @@
if (a == FFLUSH) /* didn't find it, so don't create it! */
return NULL;
- for (i=0; i < FOPEN_MAX; i++)
- if (files[i].fp == 0)
+ for (i = 0; i < nfiles; i++)
+ if (files[i].fp == NULL)
break;
- if (i >= FOPEN_MAX)
- FATAL("%s makes too many open files", s);
+ if (i >= nfiles) {
+ struct files *nf;
+ size_t nnf = nfiles + FOPEN_MAX;
+ nf = realloc(files, nnf * sizeof(*nf));
+ if (nf == NULL)
+ FATAL("cannot grow files for %s and %zu files", s, nnf);
+ (void)memset(&nf[nfiles], 0, FOPEN_MAX * sizeof(*nf));
+ nfiles = nnf;
+ files = nf;
+ }
fflush(stdout); /* force a semblance of order */
m = a;
if (a == GT) {
@@ -1674,9 +1690,9 @@
const char *filename(FILE *fp)
{
- int i;
+ size_t i;
- for (i = 0; i < FOPEN_MAX; i++)
+ for (i = 0; i < nfiles; i++)
if (fp == files[i].fp)
return files[i].fname;
return "???";
@@ -1685,13 +1701,14 @@
Cell *closefile(Node **a, int n)
{
Cell *x;
- int i, stat;
+ size_t i;
+ int stat;
n = n;
x = execute(a[0]);
getsval(x);
stat = -1;
- for (i = 0; i < FOPEN_MAX; i++) {
+ for (i = 0; i < nfiles; i++) {
if (files[i].fname && strcmp(x->sval, files[i].fname) == 0) {
if (ferror(files[i].fp))
WARNING( "i/o error occurred on %s", files[i].fname );
@@ -1715,9 +1732,10 @@
void closeall(void)
{
- int i, stat;
+ size_t i;
+ int stat;
- for (i = 0; i < FOPEN_MAX; i++) {
+ for (i = 0; i < nfiles; i++) {
if (files[i].fp) {
if (ferror(files[i].fp))
WARNING( "i/o error occurred on %s", files[i].fname );
@@ -1733,9 +1751,9 @@
void flush_all(void)
{
- int i;
+ size_t i;
- for (i = 0; i < FOPEN_MAX; i++)
+ for (i = 0; i < nfiles; i++)
if (files[i].fp)
fflush(files[i].fp);
}
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201001301045.o0UAjNsN094092>