From owner-freebsd-ports@FreeBSD.ORG Fri Mar 25 21:13:06 2011 Return-Path: Delivered-To: ports@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CEE7F1065674; Fri, 25 Mar 2011 21:13:06 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id BB2308FC1D; Fri, 25 Mar 2011 21:13:05 +0000 (UTC) Received: by wyf23 with SMTP id 23so1660440wyf.13 for ; Fri, 25 Mar 2011 14:13:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=cUnF30A0EiLnPiiKXB5G6jlDPdyudcLaEK7bevmXRFQ=; b=A1ezVe+GKW7IOEhYg2qyMXgNcQlo3ClnlM8A08uIqnk5CE/5RlcXJ3IOJs5U4t+UhF VgRer4r9yZDWeJ5+zal3ts61+qtKKWe9aGNPxT6tovYqR3XnWYMh2NV31Y5eqFpPCxBq g1I9SQ3wNuXWOfYNupRafPc170L0UgKbm0XKA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=UyXO6GCdmv0L5MS74jNS2SxWJd7vAaF2XV710xxa4n/hM3FRfKDapdWUvqa6dvu/30 0j1TjnjNZZsPorvz243K5hu+JXFm8fntsyDUfpKvYvx4gYjU9Zsw0S+JAYq1VPUDVmBt NHITgi+vxqBfyhUUhCpHu6rXjCpxdp24L+Kyw= MIME-Version: 1.0 Received: by 10.217.7.66 with SMTP id z44mr919649wes.100.1301086033606; Fri, 25 Mar 2011 13:47:13 -0700 (PDT) Received: by 10.216.173.142 with HTTP; Fri, 25 Mar 2011 13:47:13 -0700 (PDT) In-Reply-To: <20110325211439.00004dda@unknown> References: <20110325101111.GA36840@azathoth.lan> <20110325150653.21132ej6abxmjpgk@webmail.leidinger.net> <20110325153814.20287h1594npcu80@webmail.leidinger.net> <20110325153520.GB23861@gahrfit.gahr.ch> <20110325211439.00004dda@unknown> Date: Fri, 25 Mar 2011 13:47:13 -0700 Message-ID: From: Garrett Cooper To: Alexander Leidinger Content-Type: text/plain; charset=ISO-8859-1 Cc: Daroussin , current@freebsd.org, Julien Laffaye , ports@freebsd.org, hackers@freebsd.org, gahr@freebsd.org, Baptiste@freebsd.org Subject: Re: [ECFT] pkgng 0.1-alpha1: a replacement for pkg_install X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Mar 2011 21:13:06 -0000 On Fri, Mar 25, 2011 at 1:14 PM, Alexander Leidinger wrote: > On Fri, 25 Mar 2011 16:35:21 +0100 Pietro Cerutti > wrote: > >> On 2011-Mar-25, 15:03, Julien Laffaye wrote: >> > >>> What about DB corruption/loss? Do you keep >> > >>> the /var/db/pkg//xxx files even with pkgng and only >> > >>> use the DB as a way to speed up some work (so >> > >>> the DB corruption just requires to run pkg2ng), or are you lost >> > >>> of the DB is >> > >>> lost? >> > >>> >> > >> >> > >> Nothing is done about DB corruption/loss, I am not sure we need >> > >> to do something. >> > >> Maybe. >> > > >> > > I would say "for sure". Info: In Solaris 10 sqlite is used for >> > > the service managenemt framework (SMF). It is possible that the >> > > DB is corrupt in some bad situations. In this case you have to >> > > rebuild the DB (script provided, been there, had to use it). >> > >> > If sqlite is properly used with transactions, it is very hard to >> > corrupt the database. But if hardware lies to us and say that the > > And as I told above, I even had such a case (more than once), and the > hardware was not buggy. What do you want to tell in this case, "life > sucks, reinstall everything"? If you use binary packages, pulling down everything should be trivial, fast, and easy to install. If you're using ports, well then things are going to be slow as expected. >> > data is on disk whereas it isnt... what can we do? > > Sometimes you have to stay with broken hardware. Sometimes you have to go buy new parts? Playing with broken hardware is like playing with fire -- sometimes you'll get burned if it goes out of commission during critical operations. I would be more concerned about overall system operation than having a packaging system that can handle all error conditions that should be rightfully handled by various kernel subsystems. If the kernel's doing it's job, then the packaging manager can do its job as well. >> > Another potential problem is fsync(), but if it is broken on FreeBSD >> > we want to fix it! >> > >> > BTW, the goal is to only have the database and not the flat files. >> > If you are paranoid about power outage, use something like zfs >> > snapshots... > > There are more FS than only ZFS (personally I use ZFS, and I have > snapshots, but this is not a good solution for this problem). A lot of filesystems feature snapshot'ing, including UFS. If you aren't smart enough to back up your data you're toast if the data is gone. I would be more concerned about the program getting killed, not getting properly cleaned up, etc as this is something that the package manager frontend (or whatever the official name is) should catch and fail gracefully with. Things need to follow an ACID methodology and be recoverable in the event that it can't be ACID, or it's no better than pkg_install/ports currently is if it's caught in the middle of a critical operation today installing or removing software. If SQLite can't deliver this level of ACID-like capability, then pkg_install needs to be redesigned. > As I told already, if it isn't automatic, nearly nobody will use it. > And the package management stuff has to be automatic, no freshman will > think about setting up a snapshot script when he starts to use > packages/ports. I'd just provide an export command to print out a (JSON?) version of the information, and move on. None of the other major packaging systems out there that I know of use flat files for this data, and I would rather not make it automatic because it's an unnecessary performance hit. If the user feels the need for backing up his/her data they will. If not, they're SoL in the event of a crash. >> No need to look for strange scenarios, I'm surely going to sudo rm -f >> the file more sooner than later, so... maybe just save a copy? > > A copy or two would be enough, but it has to be done automatically, and > once a day is not enough. A copy after each X modifications maybe > (for suitable definitions of X and 'modifications'). Please see my comment above. There's no reason why this belongs in a packaging system (you can add it as an external tool, but the point is to avoid architectural mistakes that leaked into the old pkg_install over the period of 10 years or so). Thanks, -Garrett PS Sorry for being so hardnosed on this, but I want something that's fast and correct, instead of something bloated, slow, half-baked, harder to test, etc. pkg_install gets executed enough times during a port upgrade that having something more streamlined for most usecases is the only way to go, and there's enough code that doesn't get executed on a regular basis that has no business being in pkg_install.