Date: Sun, 10 Feb 2008 15:02:41 +0000 From: Poul-Henning Kamp <phk@phk.freebsd.dk> To: arch@freebsd.org Subject: Fifolog - a circular file for embedded systems Message-ID: <76888.1202655761@critter.freebsd.dk>
next in thread | raw e-mail | index | archive | help
On NanoBSD and similar systems running on flash devices, logfiles are a problem due to the write-wear on the flash devices. As part of a SCADA task for the Danish Air Traffic Control, I wrote a facility that uses a file or disk partition as a round robin storage device for log information. The data is compressed with zlib to reduce the size. But when you compress data, you have seeking problem, therefore fifologs have periodic synchronization points where the compression is reset and a timestamp is stored uncompressed (default: 1 min) This makes it possible to hunt for a specific timestamp using a modified binary search, thus eliminating the need to read and uncompress half the logfile on average. In addition to this, a second timer will force a write of the compressed data at higher frequency (default:10s) to put a strict upper bound on the amount of data which can be lost in case of a crash. (This timer is obviously a trade-off between wearing out the flash device and loosing information about a crash, but if you are debugging FreeBSD crashes, this is not the right tool anyway.) I have started to extract the fifolog from the SCADA application because I need it as a free standing facility in another contract, and that got me wondering if it should be included in FreeBSD as a general tool ? Using it with syslogd is trivial, just initialize a fifolog somewhere: fifolog_create -s 10m /var/log/syslog.fifolog add this line to /etc/syslog.conf: *.* |fifolog_writer /var/log/syslog.fifolog restart syslogd & and you're done. To read your logfile: fifolog_reader -B "17 hours ago" -E "15 hours ago" /var/log/syslog.fifolog In difference from the current /var/logs + newsyslog(8) a fifolog has a constant storage requirement and you always have access to as much log information as will fit in that amount of storage. The complete fifolog tools sources: http://phk.freebsd.dk/patch/fifolog.tgz The manual page is included below. Comments, code reviews and opinions on the suitability of this for inclusion in FreeBSDs base system are most welcome. Poul-Henning FIFOLOG(1) FreeBSD General Commands Manual FIFOLOG(1) NAME fifolog_create -- Initialize storage for fifolog fifolog_write -- Write data to fifolog fifolog_read -- Seek and extract data from fifolog SYNOPSIS fifolog_create [-l record-size] [-r record-count] [-s size] file fifolog_reader [-t] [-b tstart] [-B Tstart] [-e tend] [-E Tend] [-o ofile] [-R regexp] [-T timefmt] file fifolog_writer [-w write-rate] [-s sync-rate] [-z compression] file DESCRIPTION Fifologs are designed to provide a round-robin circular storage for recording text and binary information to permanent storage in a bounded and predictable fashion, time and space wise. A fifolog can be stored either directly on a disk partition or in a regu- lar filesystem file. The input data stream is encoded and compressed and marked up with time- stamps before it is written to storage, such that it is possible to seek out a particular time interval in the stored data, without having to decompress all of the logfile. Writes happen whenever the output buffer is filled with compressed data or when either of two timers expire forcing the partially filled buffer to be written. The first and faster timer just forces available data to be written but does not flush and reset the compression dictionary. This timer is intended to minimize the amount of logdata lost in RAM in case of a crash and by default it fires 10 seconds after the previous write. The second and slower timer forces a full flush and reset of the compres- sion engine and forces the next record written to be a synchronization point with its own timestamp, making it possible to initiate reading from that record. By default this timer fires a minute after the previous sync. The fifolog_create program is used to initialize the first sector of a disk device or filesystem file to make it a fifolog and should be called only once. Running fifolog_create on an existing fifolog will reset it so that fifolog_reader and fifolog_writer will not see the previous contents. (The previos contents is not physically erased, and with a bit of hand- work, all but the first record can be easily recovered). If the file does not already exist fifolog_create will attempt to create and ftruncate(3) it to the specified size, defaulting to 86400 records if no -r argument is specified. fifolog_writer will read standard input and write it to the end of the fifolog according to the parameters given. fifolog_reader will retrieve records from the fifolog according to the specified parameters and write them either to stdout or the file speci- fied with -o. It is possible to specify a start and end time to limit the amount of data fifolog_reader will report. The lower-case variants -b and -e take a time_t value, whereas the upper-case variants -B and -E take human red- able specifications such as "1 hour ago". The -t argument forces timestamps to be formatted as "YYYYMMDDhhmmss" instead of as time_t, and -T allows the specification of a strftime(3) formatting string. Finally, records can be filtered such that only records matching the (REG_BASIC) regular expression specified with -R is output. IMPLEMENTATION NOTES The data stored in the fifolog consists of three layers, an outher layer that allows searches to synchronization points based on timestamps with- out having to decompress and decode the actual contents, a compression layer implemented with zlib(3) and an inner serialization and timestamp- ing layer. The exact encoding is described in the fifolog.h file. Fifolog is particularly well suited for use on Flash based media, where it results in much less write-wear, than a filesystem with regular log- files and newsyslog(8) etc. EXAMPLES Create a fifolog with 1024*1024 records of 512 bytes: fifolog_create -s 10m /tmp/fifolog Write a single record to this file: date | fifolog_writer /tmp/fifolog Read it back with human readable timestamps: fifolog_reader -t /tmp/fifolog One particular useful use of fifolog_writer is with syslogd(8) using a line such as this in /etc/syslog.conf(5): *.* |fifolog_writer /var/log/syslog_fifolog HISTORY The fifolog tools have been liberated from an open source SCADA applica- tions called "measured", which monitors and controls remote radio naviga- tion transmitters for the Danish Air Traffic Control system. AUTHORS The fifolog tools were written by Poul-Henning Kamp FreeBSD 8.0 Feb 9, 2008 FreeBSD 8.0 -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?76888.1202655761>