From owner-freebsd-hackers@FreeBSD.ORG Sun Sep 7 00:17:40 2014 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D0F15861; Sun, 7 Sep 2014 00:17:40 +0000 (UTC) Received: from pp2.rice.edu (proofpoint2.mail.rice.edu [128.42.201.101]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 92CD11812; Sun, 7 Sep 2014 00:17:39 +0000 (UTC) Received: from pps.filterd (pp2.rice.edu [127.0.0.1]) by pp2.rice.edu (8.14.5/8.14.5) with SMTP id s870HVvN011858; Sat, 6 Sep 2014 19:17:31 -0500 Received: from mh3.mail.rice.edu (mh3.mail.rice.edu [128.42.199.10]) by pp2.rice.edu with ESMTP id 1p7nyb0a6m-1; Sat, 06 Sep 2014 19:17:30 -0500 X-Virus-Scanned: by amavis-2.7.0 at mh3.mail.rice.edu, auth channel Received: from 108-254-203-201.lightspeed.hstntx.sbcglobal.net (108-254-203-201.lightspeed.hstntx.sbcglobal.net [108.254.203.201]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) (Authenticated sender: alc) by mh3.mail.rice.edu (Postfix) with ESMTPSA id 66C824013D; Sat, 6 Sep 2014 19:17:30 -0500 (CDT) Message-ID: <540BA416.7010106@rice.edu> Date: Sat, 06 Sep 2014 19:17:26 -0500 From: Alan Cox User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Konstantin Belousov , Pieter de Goeje Subject: Re: mmap MAP_NOSYNC regression in 10.x References: <540903FF.6010602@degoeje.nl> <20140905080633.GM2737@kib.kiev.ua> <5409A4F8.6020204@degoeje.nl> <20140905123826.GP2737@kib.kiev.ua> In-Reply-To: <20140905123826.GP2737@kib.kiev.ua> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 kscore.is_bulkscore=0 kscore.compositescore=0 circleOfTrustscore=0 compositescore=0.248919945447816 urlsuspect_oldscore=0.0298999927260837 suspectscore=3 recipient_domain_to_sender_totalscore=0 phishscore=0 bulkscore=0 kscore.is_spamscore=0 recipient_to_sender_totalscore=0 recipient_domain_to_sender_domain_totalscore=498 rbsscore=0.248919945447816 spamscore=0 recipient_to_sender_domain_totalscore=0 urlsuspectscore=0.9 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=7.0.1-1402240000 definitions=main-1409070002 Cc: alc@freebsd.org, hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Sep 2014 00:17:40 -0000 On 09/05/2014 07:38, Konstantin Belousov wrote: > On Fri, Sep 05, 2014 at 01:56:40PM +0200, Pieter de Goeje wrote: >> Thanks, works for me! > I realized that the patch contains yet another bug. The oflags page > flags update is protected by the exclusive vm object lock, which is only > held in shared mode on the fast path. Below is the fixed patch, where > I take the page lock around setting VPO_NOSYNC (exclusive lock owners > cannot race with fast path since we own shared lock, and two parallel > fast path execution would be handled by the page lock). Suppose that the page is clean and two threads are executing this code concurrently. One's map entry has MAP_NOSYNC set, and the other's doesn't. Let's call these threads NOSYNC and SYNC, respectively. Suppose that the thread SYNC is slightly ahead. It has already performed "m->oflags &= ~VPO_NOSYNC;" and now it's about to perform "vm_page_dirty(fs.m);". However, just before the thread SYNC calls vm_page_dirty(), the thread NOSYNC evaluates "m->dirty == 0", which is still true, and it performs "m->oflags |= VPO_NOSYNC; " This can't happen on the slow path. That is, a fault by a thread without MAP_NOSYNC set on its map entry will reliably clear VPO_NOSYNC. The best course of action may be to fall back to the slow path if you actually need to change VPO_NOSYNC's state. Usually, you won't need to. > diff --git a/sys/vm/vm_fault.c b/sys/vm/vm_fault.c > index 30b0456..f327ab0 100644 > --- a/sys/vm/vm_fault.c > +++ b/sys/vm/vm_fault.c > @@ -174,6 +174,54 @@ unlock_and_deallocate(struct faultstate *fs) > } > } > > +static void > +vm_fault_dirty(vm_map_entry_t entry, vm_page_t m, vm_prot_t prot, > + vm_prot_t fault_type, int fault_flags, boolean_t set_wd) > +{ > + > + if (((prot & VM_PROT_WRITE) != 0 || > + (fault_flags & VM_FAULT_DIRTY) != 0) && > + (m->oflags & VPO_UNMANAGED) == 0) { > + if (set_wd) > + vm_object_set_writeable_dirty(m->object); > + > + /* > + * If this is a NOSYNC mmap we do not want to set VPO_NOSYNC > + * if the page is already dirty to prevent data written with > + * the expectation of being synced from not being synced. > + * Likewise if this entry does not request NOSYNC then make > + * sure the page isn't marked NOSYNC. Applications sharing > + * data should use the same flags to avoid ping ponging. > + */ > + if (entry->eflags & MAP_ENTRY_NOSYNC) { > + if (m->dirty == 0) { > + if (!set_wd) > + vm_page_lock(m); > + m->oflags |= VPO_NOSYNC; > + if (!set_wd) > + vm_page_unlock(m); > + } > + } else { > + m->oflags &= ~VPO_NOSYNC; > + } > + > + /* > + * If the fault is a write, we know that this page is being > + * written NOW so dirty it explicitly to save on > + * pmap_is_modified() calls later. > + * > + * Also tell the backing pager, if any, that it should remove > + * any swap backing since the page is now dirty. > + */ > + if (((fault_type & VM_PROT_WRITE) != 0 && > + (fault_flags & VM_FAULT_CHANGE_WIRING) == 0) || > + (fault_flags & VM_FAULT_DIRTY) != 0) { > + vm_page_dirty(m); > + vm_pager_page_unswapped(m); > + } > + } > +} > + > /* > * TRYPAGER - used by vm_fault to calculate whether the pager for the > * current object *might* contain the page. > @@ -321,11 +369,8 @@ RetryFault:; > vm_page_hold(m); > vm_page_unlock(m); > } > - if ((fault_type & VM_PROT_WRITE) != 0 && > - (m->oflags & VPO_UNMANAGED) == 0) { > - vm_page_dirty(m); > - vm_pager_page_unswapped(m); > - } > + vm_fault_dirty(fs.entry, m, prot, fault_type, fault_flags, > + FALSE); > VM_OBJECT_RUNLOCK(fs.first_object); > if (!wired) > vm_fault_prefault(&fs, vaddr, 0, 0); > @@ -898,42 +943,7 @@ vnode_locked: > if (hardfault) > fs.entry->next_read = fs.pindex + faultcount - reqpage; > > - if (((prot & VM_PROT_WRITE) != 0 || > - (fault_flags & VM_FAULT_DIRTY) != 0) && > - (fs.m->oflags & VPO_UNMANAGED) == 0) { > - vm_object_set_writeable_dirty(fs.object); > - > - /* > - * If this is a NOSYNC mmap we do not want to set VPO_NOSYNC > - * if the page is already dirty to prevent data written with > - * the expectation of being synced from not being synced. > - * Likewise if this entry does not request NOSYNC then make > - * sure the page isn't marked NOSYNC. Applications sharing > - * data should use the same flags to avoid ping ponging. > - */ > - if (fs.entry->eflags & MAP_ENTRY_NOSYNC) { > - if (fs.m->dirty == 0) > - fs.m->oflags |= VPO_NOSYNC; > - } else { > - fs.m->oflags &= ~VPO_NOSYNC; > - } > - > - /* > - * If the fault is a write, we know that this page is being > - * written NOW so dirty it explicitly to save on > - * pmap_is_modified() calls later. > - * > - * Also tell the backing pager, if any, that it should remove > - * any swap backing since the page is now dirty. > - */ > - if (((fault_type & VM_PROT_WRITE) != 0 && > - (fault_flags & VM_FAULT_CHANGE_WIRING) == 0) || > - (fault_flags & VM_FAULT_DIRTY) != 0) { > - vm_page_dirty(fs.m); > - vm_pager_page_unswapped(fs.m); > - } > - } > - > + vm_fault_dirty(fs.entry, fs.m, prot, fault_type, fault_flags, TRUE); > vm_page_assert_xbusied(fs.m); > > /* > diff --git a/sys/vm/vm_page.h b/sys/vm/vm_page.h > index f12b76c..a45648d 100644 > --- a/sys/vm/vm_page.h > +++ b/sys/vm/vm_page.h > @@ -147,7 +147,7 @@ struct vm_page { > uint16_t hold_count; /* page hold count (P) */ > uint16_t flags; /* page PG_* flags (P) */ > uint8_t aflags; /* access is atomic */ > - uint8_t oflags; /* page VPO_* flags (O) */ > + uint8_t oflags; /* page VPO_* flags (OM) */ > uint8_t queue; /* page queue index (P,Q) */ > int8_t psind; /* pagesizes[] index (O) */ > int8_t segind; > @@ -163,8 +163,9 @@ struct vm_page { > /* > * Page flags stored in oflags: > * > - * Access to these page flags is synchronized by the lock on the object > - * containing the page (O). > + * Access to these page flags is synchronized by the exclusive lock on > + * the object containing the page, or combination of shared object > + * lock and the page lock (OM). > * > * Note: VPO_UNMANAGED (used by OBJT_DEVICE, OBJT_PHYS and OBJT_SG) > * indicates that the page is not under PV management but From owner-freebsd-hackers@FreeBSD.ORG Sun Sep 7 05:51:28 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1D0BD881; Sun, 7 Sep 2014 05:51:28 +0000 (UTC) Received: from message.langara.bc.ca (message.mylangara.bc.ca [142.35.159.25]) by mx1.freebsd.org (Postfix) with ESMTP id 0135313D7; Sun, 7 Sep 2014 05:51:27 +0000 (UTC) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-disposition: inline Content-type: text/plain; charset=us-ascii Received: from langara.bc.ca ([127.0.0.1]) by message.langara.bc.ca (Sun Java(tm) System Messaging Server 6.3-6.03 (built Mar 14 2008; 32bit)) with ESMTP id <0NBI00KEBO9LA5A0@message.langara.bc.ca>; Sat, 06 Sep 2014 22:51:21 -0700 (PDT) Received: from [23.16.76.141] by message.langara.bc.ca (mshttpd); Sun, 07 Sep 2014 05:51:21 +0000 (GMT) From: Steven Stewart-Gallus To: Benjamin Kaduk Message-id: Date: Sun, 07 Sep 2014 05:51:21 +0000 (GMT) X-Mailer: Sun Java(tm) System Messenger Express 6.3-6.03 (built Mar 14 2008; 32bit) Content-language: en Subject: Re: Can anyone help clarify details about the FreeBSD system call interface? X-Accept-Language: en Priority: normal In-reply-to: References: Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Sep 2014 05:51:28 -0000 Okay, I've fixed up my patches and want to submit them. Obviously, I'll CC the conversation here but how should I submit the patches? I find it strange that the developer's handbook doesn't cover this case. Perhaps that is another thing that needs patching. I know I should message freebsd-doc and CC freebsd-afs but is there any formal procedure or something? Thank you, Steven Stewart-Gallus From owner-freebsd-hackers@FreeBSD.ORG Sun Sep 7 07:10:58 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 77760641 for ; Sun, 7 Sep 2014 07:10:58 +0000 (UTC) Received: from mail.turbocat.net (heidi.turbocat.net [88.198.202.214]) (using TLSv1.1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 347A61BEE for ; Sun, 7 Sep 2014 07:10:57 +0000 (UTC) Received: from laptop015.home.selasky.org (cm-176.74.213.204.customer.telag.net [176.74.213.204]) (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 241591FE027; Sun, 7 Sep 2014 09:10:53 +0200 (CEST) Message-ID: <540C04F8.2000703@selasky.org> Date: Sun, 07 Sep 2014 09:10:48 +0200 From: Hans Petter Selasky User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.1.0 MIME-Version: 1.0 To: Poul-Henning Kamp Subject: Re: What to do if USB stack seems dead References: <1455331.JKPkSxLmbq@quad> <540B7AC4.9060504@selasky.org> <33743.1410072134@critter.freebsd.dk> In-Reply-To: <33743.1410072134@critter.freebsd.dk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Maxim V FIlimonov X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Sep 2014 07:10:58 -0000 On 09/07/14 08:42, Poul-Henning Kamp wrote: > -------- > In message <540B7AC4.9060504@selasky.org>, Hans Petter Selasky writes: >> On 09/06/14 21:57, Maxim V FIlimonov wrote: >>> Lately, I've been heavily experimenting with different USB devices (for >>> instance, USB to TTL one, but that's another story), and at a moment I >>> encountered that the system doesn't react on new USB devices connected. The >>> connected devices work fine, though. The question is: what can I do in such a >>> case if I don't want to reboot my box? >>> >> >> Hi, >> >> If a TTY port is not closed, it will prevent other USB devices on that >> particular USB controller from enumerating. > > I'm pretty sure I have seen this "dead USB" thing on > > FreeBSD ni.freebsd.dk 11.0-CURRENT FreeBSD 11.0-CURRENT #3 r270830: > > without any USB/TTY ports being involved at all... > Hi, If it is a USB SW problem, you should see that the USB explore / RootHub threads are stuck. Else it is a USB HW problem. --HPS From owner-freebsd-hackers@FreeBSD.ORG Sun Sep 7 07:52:20 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 12D474FA; Sun, 7 Sep 2014 07:52:20 +0000 (UTC) Received: from mail-wi0-x234.google.com (mail-wi0-x234.google.com [IPv6:2a00:1450:400c:c05::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7C5A41F09; Sun, 7 Sep 2014 07:52:19 +0000 (UTC) Received: by mail-wi0-f180.google.com with SMTP id ex7so1206361wid.7 for ; Sun, 07 Sep 2014 00:52:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=YoZqge4e93rT53taGs4SzlkbrgM/1CFuMlrteasRkaM=; b=RJYiouOL81NqNFDu0IWZz/xCwiRBhh5utXIKb+bJWb1BBhYWprGbsdvDA1EIj5n2TC SWpl/y415pn+PeAedf8ROOMcqxavRLw9oIAFzshY5pYNMTRPx0n5naQBxyS4ZlmNBgN5 xXXU+Ul/JcHEoanCUcfUaKKz4YWdDYpB2T5wDe8M4gUR2n2qkqflEbXZb3wbJ7FPqBGg fJvlsGydCg9Y5Kp1qYtvQ2m02iTj8UpMwcbvBO4XMYhoB8Oi0DPQGH19Hcbf5Bo66Eso OIMKeMjKrtTGHL7JjY0tzKaaFrCvdu+iC8eRYC4F2frygeHFrsszT1rs1XP7uOM29dPj TnIA== MIME-Version: 1.0 X-Received: by 10.194.89.200 with SMTP id bq8mr26318233wjb.52.1410076337382; Sun, 07 Sep 2014 00:52:17 -0700 (PDT) Received: by 10.180.104.66 with HTTP; Sun, 7 Sep 2014 00:52:17 -0700 (PDT) In-Reply-To: References: Date: Sun, 7 Sep 2014 09:52:17 +0200 Message-ID: Subject: Re: Can anyone help clarify details about the FreeBSD system call interface? From: =?UTF-8?Q?Fernando_Apestegu=C3=ADa?= To: Steven Stewart-Gallus Content-Type: text/plain; charset=UTF-8 Cc: Benjamin Kaduk , FreeBSD Hackers X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Sep 2014 07:52:20 -0000 On Sun, Sep 7, 2014 at 7:51 AM, Steven Stewart-Gallus wrote: > Okay, I've fixed up my patches and want to submit them. Obviously, > I'll CC the conversation here but how should I submit the patches? I > find it strange that the developer's handbook doesn't cover this > case. Perhaps that is another thing that needs patching. I know I > should message freebsd-doc and CC freebsd-afs but is there any formal > procedure or something? The standard procedure would be to file a PR: https://www.freebsd.org/support/bugreports.html That way it doesn't get lost in a mail thread. Cheers. > > Thank you, > Steven Stewart-Gallus > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" From owner-freebsd-hackers@FreeBSD.ORG Sun Sep 7 18:01:13 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1643578D; Sun, 7 Sep 2014 18:01:13 +0000 (UTC) Received: from mail-wi0-x22d.google.com (mail-wi0-x22d.google.com [IPv6:2a00:1450:400c:c05::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 398C5188A; Sun, 7 Sep 2014 18:01:12 +0000 (UTC) Received: by mail-wi0-f173.google.com with SMTP id cc10so1566155wib.12 for ; Sun, 07 Sep 2014 11:01:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:cc:content-type; bh=AG7jPUP5GOQW453Rv6J5WZXUBb6dtuexfMIe+WQdf8U=; b=V4aqyjpR8xRPAt3yH1S+zGcWHlCWWFWgei+aDgHbaqOTqroWr52uA7xNPPaMIqNR+W Yu/+PsK3BE5nTWb/RNbMSMEkc/UoX4Eip+B2pbZmT6ROaxDGAOmlrS+o1yEA5rbCIcns IuzMuvpo2NwbpNxEs4/Ml0FNGM84RYRVrsX6twlIFm1nNal6Xe7Qms0JlVBN3e7LT6Vi xozMAw4N22ho/pwOPgFhCCOdAXCokMybxLq718348vm0tR0BLk+yujNnE7IEYi5t9ymY fo+HhDQGKG2jEZjadx22DAB5eTjVli2H5sG8WhtWw15AgaEV4lKoYhfqMwJWf5YGp9k1 RtLQ== MIME-Version: 1.0 X-Received: by 10.194.19.165 with SMTP id g5mr29180956wje.65.1410112870408; Sun, 07 Sep 2014 11:01:10 -0700 (PDT) Received: by 10.180.87.228 with HTTP; Sun, 7 Sep 2014 11:01:10 -0700 (PDT) Date: Sun, 7 Sep 2014 22:01:10 +0400 Message-ID: Subject: Android Emulator for FreeBSD + FreeBSD/ARM port From: Alexander Tarasikov To: "freebsd-arm@freebsd.org" , freebsd-hackers@freebsd.org, freebsd-emulation@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: Gavin Atkinson X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Sep 2014 18:01:13 -0000 Hi, FreeBSD crowd! During summer as part of GSoC program I was working on porting FreeBSD to the Android Emulator. Besides, I have ported Android Emulator to run natively on x86_64 as opposed to using linuxulator for 32-bit support. As for the kernel side, I have implemented the support for the following hardware: *IRQ, Timer, UART *MMC *Ethernet *Framebuffer (using NEWCONS) It allows to mount rootfs and boot up to login prompt with raspberry pi sd card image. As for the emulator, it's a bit complicated. FreeBSD boots fine if you launch the emulator on Linux or Mac OS X. I have fixed some parts of the build system and headers to make it compile and pass the tests on FreeBSD. Unfortunately, the GUI doesn't work right now and only console output (virtual UART) works. If anyone is interested, I would like that you read my blog post, review my kernel commit and suggest ways to improve and push the changes upstream. As for the Emulator, I think we can get it running with GUI and everything because it just has too many "#ifdef __linux__" stuff around the code which should compile and run on FreeBSD as well, we just need to gradually uncomment each of these sections. Relevant links: [kernel code commit] https://github.com/astarasikov/freebsd/commit/514be1f08a552f54abef83232498559c72e1d33b [README, compilation instructions] https://github.com/astarasikov/freebsd/blob/android_goldfish_arm_master/sys/arm/goldfish/README [Emulator with patches] https://github.com/astarasikov/qemu/tree/l-preview-freebsd [blog] http://allsoftwaresucks.blogspot.ru/2014/09/gsoc-results-or-how-i-nearly-wasted.html -- Regards, Alexander From owner-freebsd-hackers@FreeBSD.ORG Sun Sep 7 20:12:52 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5CAF3C99; Sun, 7 Sep 2014 20:12:52 +0000 (UTC) Received: from message.langara.bc.ca (message.langara.bc.ca [142.35.159.25]) by mx1.freebsd.org (Postfix) with ESMTP id 3DFE415B4; Sun, 7 Sep 2014 20:12:51 +0000 (UTC) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-disposition: inline Content-type: text/plain; charset=us-ascii Received: from langara.bc.ca ([127.0.0.1]) by message.langara.bc.ca (Sun Java(tm) System Messaging Server 6.3-6.03 (built Mar 14 2008; 32bit)) with ESMTP id <0NBJ007PSS5EZ830@message.langara.bc.ca>; Sun, 07 Sep 2014 13:12:50 -0700 (PDT) Received: from [216.13.187.194] by message.langara.bc.ca (mshttpd); Sun, 07 Sep 2014 20:12:50 +0000 (GMT) From: Steven Stewart-Gallus To: Steven Stewart-Gallus Message-id: Date: Sun, 07 Sep 2014 20:12:50 +0000 (GMT) X-Mailer: Sun Java(tm) System Messenger Express 6.3-6.03 (built Mar 14 2008; 32bit) Content-language: en Subject: Re: Can anyone help clarify details about the FreeBSD system call interface? X-Accept-Language: en Priority: normal In-reply-to: References: Cc: freebsd-afs@freebsd.org, freebsd-hackers@freebsd.org, freebsd-doc@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Sep 2014 20:12:52 -0000 Okay I've submitted a patch and I'm CCing freebsd-doc and freebsd-afs. Thank you, Steven Stewart-Gallus From owner-freebsd-hackers@FreeBSD.ORG Sun Sep 7 20:32:50 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 50B184CE; Sun, 7 Sep 2014 20:32:50 +0000 (UTC) Received: from mail-qg0-x22f.google.com (mail-qg0-x22f.google.com [IPv6:2607:f8b0:400d:c04::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D29EA18EC; Sun, 7 Sep 2014 20:32:49 +0000 (UTC) Received: by mail-qg0-f47.google.com with SMTP id i50so310928qgf.34 for ; Sun, 07 Sep 2014 13:32:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=qjD/4mZXcqXbD3kRE4IBtfypR4RkfZJMeb+AvZdW7hA=; b=pO1by0zuUmKP+MGh5rTmgAnihkJkSf/CfejD9VLFVUYvbD41tdPJdz3KAY/6BYlUZW S1RYKfYh4BJ4HwzdqPcvhCH8b0tcPVMbv72Vkm66OxTb/XXEBzRUklbLNX6huUyIbPJJ jlpG/fQoMQWusCEpWWnkPkRZCg5fWMo+4bpGI7HDDipV73Qz16ZTwjQnJtsIhafCt8qg ULA9TGHrncuwsezw53bgytOw2Bl4673en6ifdFTUbgy1PEP46joVcUocQWOsuWoDSVf3 av0Xz7MPr9JsGLz1u4kT+QvQg9zVcO2STdVS5Nc9ayWqrH6niqhKlZQLkid13fbNCjwd h4jA== MIME-Version: 1.0 X-Received: by 10.224.151.69 with SMTP id b5mr36549971qaw.37.1410121968853; Sun, 07 Sep 2014 13:32:48 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.39.139 with HTTP; Sun, 7 Sep 2014 13:32:48 -0700 (PDT) In-Reply-To: References: Date: Sun, 7 Sep 2014 13:32:48 -0700 X-Google-Sender-Auth: xUycw52RJnT7I_ozjXhKuDhMrQA Message-ID: Subject: Re: Android Emulator for FreeBSD + FreeBSD/ARM port From: Adrian Chadd To: Alexander Tarasikov Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-hackers@freebsd.org" , "freebsd-arm@freebsd.org" , Gavin Atkinson , freebsd-emulation@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Sep 2014 20:32:50 -0000 On 7 September 2014 11:01, Alexander Tarasikov wrote: > Hi, FreeBSD crowd! > > During summer as part of GSoC program I was working on porting FreeBSD > to the Android Emulator. Cool! > Besides, I have ported Android Emulator to run natively on x86_64 as opposed to > using linuxulator for 32-bit support. Double cool! > As for the kernel side, I have implemented the support for the > following hardware: > *IRQ, Timer, UART > *MMC > *Ethernet > *Framebuffer (using NEWCONS) Triple cool! > It allows to mount rootfs and boot up to login prompt with raspberry > pi sd card image. > > As for the emulator, it's a bit complicated. FreeBSD boots fine if you > launch the emulator on Linux or Mac OS X. I have fixed some parts of > the build system and headers to make it compile and pass the tests on > FreeBSD. Unfortunately, the GUI doesn't work right now and only > console output (virtual UART) works. > > If anyone is interested, I would like that you read my blog post, > review my kernel commit and suggest ways to improve and push the > changes upstream. As for the Emulator, I think we can get it running > with GUI and everything because it just has too many "#ifdef > __linux__" stuff around the code which should compile and run on > FreeBSD as well, we just need to gradually uncomment each of these > sections. > > Relevant links: > [kernel code commit] > https://github.com/astarasikov/freebsd/commit/514be1f08a552f54abef83232498559c72e1d33b > [README, compilation instructions] > https://github.com/astarasikov/freebsd/blob/android_goldfish_arm_master/sys/arm/goldfish/README > [Emulator with patches] > https://github.com/astarasikov/qemu/tree/l-preview-freebsd > [blog] http://allsoftwaresucks.blogspot.ru/2014/09/gsoc-results-or-how-i-nearly-wasted.html This work is awesome. I'll see if I can spin it up! -a From owner-freebsd-hackers@FreeBSD.ORG Mon Sep 8 08:01:14 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 296AE261; Mon, 8 Sep 2014 08:01:14 +0000 (UTC) Received: from mail-oi0-x230.google.com (mail-oi0-x230.google.com [IPv6:2607:f8b0:4003:c06::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DC9B1159D; Mon, 8 Sep 2014 08:01:13 +0000 (UTC) Received: by mail-oi0-f48.google.com with SMTP id a141so9464350oig.21 for ; Mon, 08 Sep 2014 01:01:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=SZxmwXPLy6d/Rt6knVWUn14QJw5L0fNRhAJAgRT+fEI=; b=kGDnw0P1DCpH3E82jau/+IveWoq/4eMjKOurHdOre7WBNdrqZZnQboJFMsIBB27HtR /EcsWNFm+ptRKJHE1CyZxys4P2swB+Z0kBEKPtGoW9PCv54Rl8o9o/MBkjsitExu6+dj uffKlzUWPE00sFqL7WSp42alzW15kI6WaDc4zj7bRamskOak1OmS8KaIzGWNOjYe4WvA nNp68msMVk8RDx5ilA1QKMss8rzuaiq0kGUipinwbA0gXMRSwpiDu8rKfg1JVjduyXBF uVJyRh0KFIer1PfPqAzELpIj9yQa68P7q3shuKojVBA9EyJRWGkWDfDIsIs1GlQjOYNu Umjg== MIME-Version: 1.0 X-Received: by 10.60.145.143 with SMTP id su15mr12411304oeb.58.1410163273240; Mon, 08 Sep 2014 01:01:13 -0700 (PDT) Sender: tomek.cedro@gmail.com Received: by 10.202.48.15 with HTTP; Mon, 8 Sep 2014 01:01:13 -0700 (PDT) Received: by 10.202.48.15 with HTTP; Mon, 8 Sep 2014 01:01:13 -0700 (PDT) Date: Mon, 8 Sep 2014 10:01:13 +0200 X-Google-Sender-Auth: 9GwNDKd37UTNkHAX3YOVlpTWZkc Message-ID: Subject: FreeBSD and WiDi / Miracast / WiFi Direct HDMI streaming From: CeDeROM To: freebsd-hackers@freebsd.org, FreeBSD Questions Mailing List Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Sep 2014 08:01:14 -0000 Hello world! :-) I am considering to buy Netgear PTV3000 [1] for wireless audio-video streaming HDMI adapter. Does FreeBSD support any of the WiDi/Miracast/WiFiDirect medium? It seems very very interesting portable multiplatform solution :-) Best regards :-) Tomek [1] http://www.netgear.com/home/products/connected-entertainment/wireless-display-adapters/PTV3000.aspx -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info From owner-freebsd-hackers@FreeBSD.ORG Mon Sep 8 08:41:37 2014 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B1223BD8; Mon, 8 Sep 2014 08:41:37 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 39A291AAD; Mon, 8 Sep 2014 08:41:37 +0000 (UTC) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id s888fRYr028076 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 8 Sep 2014 11:41:27 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua s888fRYr028076 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id s888fQRT028075; Mon, 8 Sep 2014 11:41:26 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 8 Sep 2014 11:41:26 +0300 From: Konstantin Belousov To: Alan Cox Subject: Re: mmap MAP_NOSYNC regression in 10.x Message-ID: <20140908084126.GX2737@kib.kiev.ua> References: <540903FF.6010602@degoeje.nl> <20140905080633.GM2737@kib.kiev.ua> <5409A4F8.6020204@degoeje.nl> <20140905123826.GP2737@kib.kiev.ua> <540BA416.7010106@rice.edu> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="cZMJGra2+uNVq8Ts" Content-Disposition: inline In-Reply-To: <540BA416.7010106@rice.edu> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: alc@freebsd.org, Pieter de Goeje , hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Sep 2014 08:41:37 -0000 --cZMJGra2+uNVq8Ts Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Sep 06, 2014 at 07:17:26PM -0500, Alan Cox wrote: > On 09/05/2014 07:38, Konstantin Belousov wrote: > > On Fri, Sep 05, 2014 at 01:56:40PM +0200, Pieter de Goeje wrote: > >> Thanks, works for me! > > I realized that the patch contains yet another bug. The oflags page > > flags update is protected by the exclusive vm object lock, which is only > > held in shared mode on the fast path. Below is the fixed patch, where > > I take the page lock around setting VPO_NOSYNC (exclusive lock owners > > cannot race with fast path since we own shared lock, and two parallel > > fast path execution would be handled by the page lock). >=20 >=20 > Suppose that the page is clean and two threads are executing this code > concurrently. One's map entry has MAP_NOSYNC set, and the other's > doesn't. Let's call these threads NOSYNC and SYNC, respectively. >=20 > Suppose that the thread SYNC is slightly ahead. It has already > performed "m->oflags &=3D ~VPO_NOSYNC;" and now it's about to perform > "vm_page_dirty(fs.m);". However, just before the thread SYNC calls > vm_page_dirty(), the thread NOSYNC evaluates "m->dirty =3D=3D 0", which is > still true, and it performs "m->oflags |=3D VPO_NOSYNC; " >=20 > This can't happen on the slow path. That is, a fault by a thread > without MAP_NOSYNC set on its map entry will reliably clear VPO_NOSYNC. As I understand things, it is indeed not possible on the slow path, due to PG_RW only set from pmap_enter(), am I right ? I.e. this is another place where the rule 'no PG_RW without PG_M' is important. Let me formulate my question another way: what are the guarantees we provide to the applications when the same page is mapped with and without MAP_NOSYNC simultaneously ? Is it contractually guaranteed that any write from !MAP_NOSYNC entry triggers write in the syncer activity period ? >=20 > The best course of action may be to fall back to the slow path if you > actually need to change VPO_NOSYNC's state. Usually, you won't need to. >=20 Let me first try to improve the original patch to handle MAP_ENTRY_NOSYNC on fast path as well. It seems to be one of the cases when the parallel faults are actually useful. One more note: the previous patch handled m->oflags inconsistency for setting VPO_NOSYNC operation, but missed the clear one line later. I think that increasing the page lock to cover also the vm_page_dirty() would fix the race you described, and the second manipulation with oflags. diff --git a/sys/vm/vm_fault.c b/sys/vm/vm_fault.c index 30b0456..944b479 100644 --- a/sys/vm/vm_fault.c +++ b/sys/vm/vm_fault.c @@ -174,6 +174,70 @@ unlock_and_deallocate(struct faultstate *fs) } } =20 +static void +vm_fault_dirty(vm_map_entry_t entry, vm_page_t m, vm_prot_t prot, + vm_prot_t fault_type, int fault_flags, boolean_t set_wd) +{ + boolean_t need_dirty; + + if (((prot & VM_PROT_WRITE) =3D=3D 0 && + (fault_flags & VM_FAULT_DIRTY) =3D=3D 0) || + (m->oflags & VPO_UNMANAGED) !=3D 0) + return; + + VM_OBJECT_ASSERT_LOCKED(m->object); + + need_dirty =3D ((fault_type & VM_PROT_WRITE) !=3D 0 && + (fault_flags & VM_FAULT_CHANGE_WIRING) =3D=3D 0) || + (fault_flags & VM_FAULT_DIRTY) !=3D 0; + + if (set_wd) + vm_object_set_writeable_dirty(m->object); + else + /* + * If two callers of vm_fault_dirty() with set_wd =3D=3D + * FALSE, one for the map entry with MAP_ENTRY_NOSYNC + * flag set, other with flag clear, race, it is + * possible for the no-NOSYNC thread to see m->dirty + * !=3D 0 and not clear VPO_NOSYNC. Take vm_page lock + * around manipulation of VPO_NOSYNC and + * vm_page_dirty() call, to avoid the race and keep + * m->oflags consistent. + */ + vm_page_lock(m); + + /* + * If this is a NOSYNC mmap we do not want to set VPO_NOSYNC + * if the page is already dirty to prevent data written with + * the expectation of being synced from not being synced. + * Likewise if this entry does not request NOSYNC then make + * sure the page isn't marked NOSYNC. Applications sharing + * data should use the same flags to avoid ping ponging. + */ + if ((entry->eflags & MAP_ENTRY_NOSYNC) !=3D 0) { + if (m->dirty =3D=3D 0) { + m->oflags |=3D VPO_NOSYNC; + } + } else { + m->oflags &=3D ~VPO_NOSYNC; + } + + /* + * If the fault is a write, we know that this page is being + * written NOW so dirty it explicitly to save on + * pmap_is_modified() calls later. + * + * Also tell the backing pager, if any, that it should remove + * any swap backing since the page is now dirty. + */ + if (need_dirty) + vm_page_dirty(m); + if (!set_wd) + vm_page_unlock(m); + if (need_dirty) + vm_pager_page_unswapped(m); +} + /* * TRYPAGER - used by vm_fault to calculate whether the pager for the * current object *might* contain the page. @@ -321,11 +385,8 @@ RetryFault:; vm_page_hold(m); vm_page_unlock(m); } - if ((fault_type & VM_PROT_WRITE) !=3D 0 && - (m->oflags & VPO_UNMANAGED) =3D=3D 0) { - vm_page_dirty(m); - vm_pager_page_unswapped(m); - } + vm_fault_dirty(fs.entry, m, prot, fault_type, fault_flags, + FALSE); VM_OBJECT_RUNLOCK(fs.first_object); if (!wired) vm_fault_prefault(&fs, vaddr, 0, 0); @@ -898,42 +959,7 @@ vnode_locked: if (hardfault) fs.entry->next_read =3D fs.pindex + faultcount - reqpage; =20 - if (((prot & VM_PROT_WRITE) !=3D 0 || - (fault_flags & VM_FAULT_DIRTY) !=3D 0) && - (fs.m->oflags & VPO_UNMANAGED) =3D=3D 0) { - vm_object_set_writeable_dirty(fs.object); - - /* - * If this is a NOSYNC mmap we do not want to set VPO_NOSYNC - * if the page is already dirty to prevent data written with - * the expectation of being synced from not being synced. - * Likewise if this entry does not request NOSYNC then make - * sure the page isn't marked NOSYNC. Applications sharing - * data should use the same flags to avoid ping ponging. - */ - if (fs.entry->eflags & MAP_ENTRY_NOSYNC) { - if (fs.m->dirty =3D=3D 0) - fs.m->oflags |=3D VPO_NOSYNC; - } else { - fs.m->oflags &=3D ~VPO_NOSYNC; - } - - /* - * If the fault is a write, we know that this page is being - * written NOW so dirty it explicitly to save on=20 - * pmap_is_modified() calls later. - * - * Also tell the backing pager, if any, that it should remove - * any swap backing since the page is now dirty. - */ - if (((fault_type & VM_PROT_WRITE) !=3D 0 && - (fault_flags & VM_FAULT_CHANGE_WIRING) =3D=3D 0) || - (fault_flags & VM_FAULT_DIRTY) !=3D 0) { - vm_page_dirty(fs.m); - vm_pager_page_unswapped(fs.m); - } - } - + vm_fault_dirty(fs.entry, fs.m, prot, fault_type, fault_flags, TRUE); vm_page_assert_xbusied(fs.m); =20 /* diff --git a/sys/vm/vm_page.h b/sys/vm/vm_page.h index f12b76c..a45648d 100644 --- a/sys/vm/vm_page.h +++ b/sys/vm/vm_page.h @@ -147,7 +147,7 @@ struct vm_page { uint16_t hold_count; /* page hold count (P) */ uint16_t flags; /* page PG_* flags (P) */ uint8_t aflags; /* access is atomic */ - uint8_t oflags; /* page VPO_* flags (O) */ + uint8_t oflags; /* page VPO_* flags (OM) */ uint8_t queue; /* page queue index (P,Q) */ int8_t psind; /* pagesizes[] index (O) */ int8_t segind; @@ -163,8 +163,9 @@ struct vm_page { /* * Page flags stored in oflags: * - * Access to these page flags is synchronized by the lock on the object - * containing the page (O). + * Access to these page flags is synchronized by the exclusive lock on + * the object containing the page, or combination of shared object + * lock and the page lock (OM). * * Note: VPO_UNMANAGED (used by OBJT_DEVICE, OBJT_PHYS and OBJT_SG) * indicates that the page is not under PV management but --cZMJGra2+uNVq8Ts Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJUDWu2AAoJEJDCuSvBvK1B+bwP/2jAQVhay1dGGP8PjHH3zqFk zkgPe1zKrKJvybXe76fCy9yPxxcIyp1lLONqYSjenkvPVigvFPwPdKo37H/H5Lyr oK+6f/LGSOx0m2Vpd1h5HKtYfnuBEs3W/etFX9ZacZpR3EpTj7BTZaShzqFKNjKP 8nhT5XkJV1YItn8Rjft5f9MC8UhwG+fGkZ+UXjmQ1elSE4e6HhLFzfgQ8mfM8Zls p0PnZ9qVp35M+tqOklkEMhzb+pIgORL7t2eqqWms5Y+VDfDdtFv956Pc8O/N5zB8 6zmaKfKaDNRnD1cS3HAFgHbozLz/KOZMphDO88yymkdIhakld5cusPYJeWz09MXm rD9Bg9bfvpMjo6TOPZ7DV40CfO3u1SC4SLEt/Czimt5ivKdTKrIXVJCCqY+JOdN3 2b8WkvMcIgMTe6z7SDr4g+j7jReJtKv8eBpQMfA5uKpf/zxtyBEx4RiYkCjaIaXL KBYYcc8eZjZ4C0JyVcnpsWSdvCADiwlCf9W2wwFRbMRr3Fh9cHLmpJLTGzJYtRF3 VU4kQjTMrQt91j/tfh84vDR/EvFtMLyxCLo1X/MrHCQG5/SrzlGtLwZFqDoJXV+E 3E6izBTdqeWz6Pel4XtNperaMGEIFyPRpCkhVvMH3FhGpBkh9ba+qFxlLiJfpsa6 tS/iofuRSvZd2wzLyDAC =1VQ0 -----END PGP SIGNATURE----- --cZMJGra2+uNVq8Ts-- From owner-freebsd-hackers@FreeBSD.ORG Mon Sep 8 16:41:31 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E1758905; Mon, 8 Sep 2014 16:41:31 +0000 (UTC) Received: from mail-oa0-x232.google.com (mail-oa0-x232.google.com [IPv6:2607:f8b0:4003:c02::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9E9CE1881; Mon, 8 Sep 2014 16:41:31 +0000 (UTC) Received: by mail-oa0-f50.google.com with SMTP id o6so10763393oag.37 for ; Mon, 08 Sep 2014 09:41:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=zmB242tGei/GzjTSTFowwwg6ppmfplFh3zBH/Getw1o=; b=hedgjMkJ1sLillAMSQYR3gN9IAzy+MLrCel4cvb7FnbLRMZZz6BEBg3OPiHnzi3mjQ f9vQFkQzUuu1k3OZgm7VYG4NCbMObkUv9NQOav9wCBk/LcYT3Zbgl9DA+JL7z+nM7lNV 88JvOvThU6Og97/n4J5t8fDAhLsTvlnOWpggbT53KDrcTN6EJLfbzUW/nUd5AFHTH4OJ 9ydZkTXrSwDuqbfDwLhMJOkJCS7iWIl8a+vw9MVW55nr0hpChJLua8aUBp1Q2wnFibYD 0Bou7/WFlp8Qn6U5CziqGF4UIyQq+qtvKnUVatJm+H1Mr13rRVcwbmJOhsiGTnfL0tmJ uHug== MIME-Version: 1.0 X-Received: by 10.182.29.200 with SMTP id m8mr32984966obh.13.1410194490881; Mon, 08 Sep 2014 09:41:30 -0700 (PDT) Sender: tomek.cedro@gmail.com Received: by 10.202.48.15 with HTTP; Mon, 8 Sep 2014 09:41:30 -0700 (PDT) In-Reply-To: References: Date: Mon, 8 Sep 2014 18:41:30 +0200 X-Google-Sender-Auth: Y8xEgx-KyJc4OMPLLot9OGPSGw8 Message-ID: Subject: Re: FreeBSD and WiDi / Miracast / WiFi Direct HDMI streaming From: CeDeROM To: Waitman Gobble Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-hackers@freebsd.org" , FreeBSD Questions Mailing List X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Sep 2014 16:41:32 -0000 On Mon, Sep 8, 2014 at 5:39 PM, Waitman Gobble wrote: > Looks like openwfd could be modified to build on FreeBSD. > Waitman Tanks! Nice hint! :-) -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info From owner-freebsd-hackers@FreeBSD.ORG Mon Sep 8 17:09:08 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CF79DDB2 for ; Mon, 8 Sep 2014 17:09:08 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A6C041BD7 for ; Mon, 8 Sep 2014 17:09:08 +0000 (UTC) Received: from ralph.baldwin.cx (pool-173-70-85-31.nwrknj.fios.verizon.net [173.70.85.31]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 1F192B923; Mon, 8 Sep 2014 13:09:07 -0400 (EDT) From: John Baldwin To: freebsd-hackers@freebsd.org Subject: Re: Tool to access ZFS/NFSv4 alternate data streams on FreeBSD? Date: Mon, 08 Sep 2014 11:55:13 -0400 Message-ID: <2154953.PmiECqQQIi@ralph.baldwin.cx> User-Agent: KMail/4.10.5 (FreeBSD/10.0-STABLE; KDE/4.10.5; amd64; ; ) In-Reply-To: <1538621043.33014113.1409955190712.JavaMail.root@uoguelph.ca> References: <1538621043.33014113.1409955190712.JavaMail.root@uoguelph.ca> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 08 Sep 2014 13:09:07 -0400 (EDT) Cc: Richard Yao , Rick Macklem , Lionel Cons , Jan Bramkamp X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Sep 2014 17:09:08 -0000 On Friday, September 05, 2014 06:13:10 PM Rick Macklem wrote: > Lionel Cons wrote: > > On 5 September 2014 19:26, Richard Yao wrote: > > > On 09/05/2014 11:35 AM, Jan Bramkamp wrote: > > >> On 05.09.2014 16:25, Lionel Cons wrote:> Is there any tool which > > >> can be > > >> used to access ZFS and NFSv4 alternate > > >> > > >>> data streams on FreeBSD? > > >> > > >> Are you looking for lsextattr(8) and getextattr(8)? > > >> _______________________________________________ > > >> freebsd-hackers@freebsd.org mailing list > > >> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > > >> To unsubscribe, send any mail to > > >> "freebsd-hackers-unsubscribe@freebsd.org" > > > > > > Do you mean Solaris extended attributes? Those tend to be called > > > resource forks on other platforms. Unifying extended attributes and > > > resource forks was clever. > > > > Yes, they are also called resource forks, or alternate data streams. > > The attribute files which can be accessed via O_XATTR or cd -@ > > file/dir on newer ksh/ksh93/bash revisions. > > For FreeBSD's NFSv4 the answer is definitely no. Because the Linux/FreeBSD > style setextattr() assumes an atomic replacement of the extended attribute, > it is not semantically compatible (ie. cannot be accurately emulated) by > resource forks. > > I do not know of any work for ZFS on FreeBSD w.r.t. this, but I'm not a > ZFS guy. Does the NFSv4 protocol support resource forks as a separate entity from EAs though? Presumably O_XATTR would turn into a new VOP (VOP_OPENFORK() or some such), it wouldn't be shoehorned into the EA APIs. -- John Baldwin From owner-freebsd-hackers@FreeBSD.ORG Mon Sep 8 17:09:09 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 19DBDDB3; Mon, 8 Sep 2014 17:09:09 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E2B531BD8; Mon, 8 Sep 2014 17:09:08 +0000 (UTC) Received: from ralph.baldwin.cx (pool-173-70-85-31.nwrknj.fios.verizon.net [173.70.85.31]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id B11E6B962; Mon, 8 Sep 2014 13:09:07 -0400 (EDT) From: John Baldwin To: freebsd-hackers@freebsd.org Subject: Re: O_XATTR support in FreeBSD? Date: Mon, 08 Sep 2014 11:52:50 -0400 Message-ID: <14246348.Ehh2xp5iLB@ralph.baldwin.cx> User-Agent: KMail/4.10.5 (FreeBSD/10.0-STABLE; KDE/4.10.5; amd64; ; ) In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 08 Sep 2014 13:09:07 -0400 (EDT) Cc: Rick Macklem , Cedric Blancher , Richard Yao , Pedro Giffuni , Lionel Cons , Jordan Hubbard X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Sep 2014 17:09:09 -0000 On Friday, September 05, 2014 04:24:02 PM Lionel Cons wrote: > On 1 December 2013 23:05, Lionel Cons wrote: > > On 27 November 2013 05:52, Tim Kientzle wrote: > >> On Nov 26, 2013, at 1:51 AM, Cedric Blancher wrote: > >>> 5. Support for tar and pax is already there. Its described in > >>> Solaris's fsattr man page, they use a extended header with filename > >>> /dev/null (to prevent older tar versions from tripping over the new > >>> headers) and then have a named attribute header which describes the > >>> attributes names and flags. > >> > >> There are quite a few alternative approaches for storing > >> extended attributes in tar and pax files. > > > > But this discussion is *not* about extended attributes, this > > discussion is about Alternate Data Streams. Unfortunately the O_XATTR > > discussion somehow started to cover the Linux "extended attribute > > system", which is utterly useless in the intended use cases (as said, > > no access through normal POSIX read(), write(), mmap(), no unlimited > > size, no sparse data support (aka SEEK_HOLE, SEEK_DATA) etc etc). > > > > Lionel > > What is the status of O_XATTR (alias alternate data stream support) in > FreeBSD? We run more and more into the trouble that these kind of > streams are in use but cannot be accessed or processed on FreeBSD. I don't think anyone is even looking at it. I didn't quite follow the earlier mail about this and some concrete examples probably would help. Can you do 'open("/path/to/foo@forkname", O_XATTR)' to open the "forkname" fork of the "/path/to/foo" file? -- John Baldwin From owner-freebsd-hackers@FreeBSD.ORG Mon Sep 8 17:59:53 2014 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0D0E8ECA; Mon, 8 Sep 2014 17:59:53 +0000 (UTC) Received: from pp1.rice.edu (proofpoint1.mail.rice.edu [128.42.201.100]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C05BC1200; Mon, 8 Sep 2014 17:59:52 +0000 (UTC) Received: from pps.filterd (pp1.rice.edu [127.0.0.1]) by pp1.rice.edu (8.14.5/8.14.5) with SMTP id s88HvKPv008141; Mon, 8 Sep 2014 12:59:44 -0500 Received: from mh3.mail.rice.edu (mh3.mail.rice.edu [128.42.199.10]) by pp1.rice.edu with ESMTP id 1p8w07ggb7-1; Mon, 08 Sep 2014 12:59:44 -0500 X-Virus-Scanned: by amavis-2.7.0 at mh3.mail.rice.edu, auth channel Received: from 108-254-203-201.lightspeed.hstntx.sbcglobal.net (108-254-203-201.lightspeed.hstntx.sbcglobal.net [108.254.203.201]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) (Authenticated sender: alc) by mh3.mail.rice.edu (Postfix) with ESMTPSA id E9A15403F5; Mon, 8 Sep 2014 12:59:43 -0500 (CDT) Message-ID: <540DEE8F.5080005@rice.edu> Date: Mon, 08 Sep 2014 12:59:43 -0500 From: Alan Cox User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Konstantin Belousov Subject: Re: mmap MAP_NOSYNC regression in 10.x References: <540903FF.6010602@degoeje.nl> <20140905080633.GM2737@kib.kiev.ua> <5409A4F8.6020204@degoeje.nl> <20140905123826.GP2737@kib.kiev.ua> <540BA416.7010106@rice.edu> <20140908084126.GX2737@kib.kiev.ua> In-Reply-To: <20140908084126.GX2737@kib.kiev.ua> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 kscore.is_bulkscore=0 kscore.compositescore=0 circleOfTrustscore=0 compositescore=0.248919945447816 urlsuspect_oldscore=0.0298999927260837 suspectscore=3 recipient_domain_to_sender_totalscore=0 phishscore=0 bulkscore=0 kscore.is_spamscore=0 recipient_to_sender_totalscore=0 recipient_domain_to_sender_domain_totalscore=498 rbsscore=0.248919945447816 spamscore=0 recipient_to_sender_domain_totalscore=0 urlsuspectscore=0.9 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=7.0.1-1402240000 definitions=main-1409080147 Cc: alc@freebsd.org, Pieter de Goeje , hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Sep 2014 17:59:53 -0000 On 09/08/2014 03:41, Konstantin Belousov wrote: > On Sat, Sep 06, 2014 at 07:17:26PM -0500, Alan Cox wrote: >> On 09/05/2014 07:38, Konstantin Belousov wrote: >>> On Fri, Sep 05, 2014 at 01:56:40PM +0200, Pieter de Goeje wrote: >>>> Thanks, works for me! >>> I realized that the patch contains yet another bug. The oflags page >>> flags update is protected by the exclusive vm object lock, which is only >>> held in shared mode on the fast path. Below is the fixed patch, where >>> I take the page lock around setting VPO_NOSYNC (exclusive lock owners >>> cannot race with fast path since we own shared lock, and two parallel >>> fast path execution would be handled by the page lock). >> >> Suppose that the page is clean and two threads are executing this code >> concurrently. One's map entry has MAP_NOSYNC set, and the other's >> doesn't. Let's call these threads NOSYNC and SYNC, respectively. >> >> Suppose that the thread SYNC is slightly ahead. It has already >> performed "m->oflags &= ~VPO_NOSYNC;" and now it's about to perform >> "vm_page_dirty(fs.m);". However, just before the thread SYNC calls >> vm_page_dirty(), the thread NOSYNC evaluates "m->dirty == 0", which is >> still true, and it performs "m->oflags |= VPO_NOSYNC; " >> >> This can't happen on the slow path. That is, a fault by a thread >> without MAP_NOSYNC set on its map entry will reliably clear VPO_NOSYNC. > As I understand things, it is indeed not possible on the slow path, due > to PG_RW only set from pmap_enter(), am I right ? I.e. this is another > place where the rule 'no PG_RW without PG_M' is important. Yes, it's not possible, but I'm a little confused by the rest of your question, specifically, the statement "no PG_RW without PG_M". Did you actually mean "no PG_M without PG_RW"? > Let me formulate my question another way: what are the guarantees we > provide to the applications when the same page is mapped with and > without MAP_NOSYNC simultaneously ? Is it contractually guaranteed that > any write from !MAP_NOSYNC entry triggers write in the syncer activity > period ? Yes, that is the intent. However, I can think of at least one case where the existing code doesn't work as intended. Suppose that the first fault on a !MAP_NOSYNC entry is triggered by a read access. Then, vm_fault() won't call vm_page_dirty(), but it will nonetheless install a mapping in the pmap that allows write access. Now, suppose this same process writes to the page. Finally, suppose that the second fault happens on a MAP_NOSYNC entry. That fault will see a clean page, i.e., m->dirty == 0, and set VPO_NOSYNC on the page, even though the first faulting process that wants the page sync'ed has dirtied the page. >> The best course of action may be to fall back to the slow path if you >> actually need to change VPO_NOSYNC's state. Usually, you won't need to. >> > Let me first try to improve the original patch to handle > MAP_ENTRY_NOSYNC on fast path as well. It seems to be one of the cases > when the parallel faults are actually useful. I think it may be time to take a step back, decide what semantics we really want, and see if there is a better way of implementing those semantics. The current approach based on toggling VPO_NOSYNC only really works for the simplest cases. > One more note: the previous patch handled m->oflags inconsistency for > setting VPO_NOSYNC operation, but missed the clear one line later. > I think that increasing the page lock to cover also the vm_page_dirty() > would fix the race you described, and the second manipulation with > oflags. > > diff --git a/sys/vm/vm_fault.c b/sys/vm/vm_fault.c > index 30b0456..944b479 100644 > --- a/sys/vm/vm_fault.c > +++ b/sys/vm/vm_fault.c > @@ -174,6 +174,70 @@ unlock_and_deallocate(struct faultstate *fs) > } > } > > +static void > +vm_fault_dirty(vm_map_entry_t entry, vm_page_t m, vm_prot_t prot, > + vm_prot_t fault_type, int fault_flags, boolean_t set_wd) > +{ > + boolean_t need_dirty; > + > + if (((prot & VM_PROT_WRITE) == 0 && > + (fault_flags & VM_FAULT_DIRTY) == 0) || > + (m->oflags & VPO_UNMANAGED) != 0) > + return; > + > + VM_OBJECT_ASSERT_LOCKED(m->object); > + > + need_dirty = ((fault_type & VM_PROT_WRITE) != 0 && > + (fault_flags & VM_FAULT_CHANGE_WIRING) == 0) || > + (fault_flags & VM_FAULT_DIRTY) != 0; > + > + if (set_wd) > + vm_object_set_writeable_dirty(m->object); > + else > + /* > + * If two callers of vm_fault_dirty() with set_wd == > + * FALSE, one for the map entry with MAP_ENTRY_NOSYNC > + * flag set, other with flag clear, race, it is > + * possible for the no-NOSYNC thread to see m->dirty > + * != 0 and not clear VPO_NOSYNC. Take vm_page lock > + * around manipulation of VPO_NOSYNC and > + * vm_page_dirty() call, to avoid the race and keep > + * m->oflags consistent. > + */ > + vm_page_lock(m); > + > + /* > + * If this is a NOSYNC mmap we do not want to set VPO_NOSYNC > + * if the page is already dirty to prevent data written with > + * the expectation of being synced from not being synced. > + * Likewise if this entry does not request NOSYNC then make > + * sure the page isn't marked NOSYNC. Applications sharing > + * data should use the same flags to avoid ping ponging. > + */ > + if ((entry->eflags & MAP_ENTRY_NOSYNC) != 0) { > + if (m->dirty == 0) { > + m->oflags |= VPO_NOSYNC; > + } > + } else { > + m->oflags &= ~VPO_NOSYNC; > + } > + > + /* > + * If the fault is a write, we know that this page is being > + * written NOW so dirty it explicitly to save on > + * pmap_is_modified() calls later. > + * > + * Also tell the backing pager, if any, that it should remove > + * any swap backing since the page is now dirty. > + */ > + if (need_dirty) > + vm_page_dirty(m); > + if (!set_wd) > + vm_page_unlock(m); > + if (need_dirty) > + vm_pager_page_unswapped(m); > +} > + > /* > * TRYPAGER - used by vm_fault to calculate whether the pager for the > * current object *might* contain the page. > @@ -321,11 +385,8 @@ RetryFault:; > vm_page_hold(m); > vm_page_unlock(m); > } > - if ((fault_type & VM_PROT_WRITE) != 0 && > - (m->oflags & VPO_UNMANAGED) == 0) { > - vm_page_dirty(m); > - vm_pager_page_unswapped(m); > - } > + vm_fault_dirty(fs.entry, m, prot, fault_type, fault_flags, > + FALSE); > VM_OBJECT_RUNLOCK(fs.first_object); > if (!wired) > vm_fault_prefault(&fs, vaddr, 0, 0); > @@ -898,42 +959,7 @@ vnode_locked: > if (hardfault) > fs.entry->next_read = fs.pindex + faultcount - reqpage; > > - if (((prot & VM_PROT_WRITE) != 0 || > - (fault_flags & VM_FAULT_DIRTY) != 0) && > - (fs.m->oflags & VPO_UNMANAGED) == 0) { > - vm_object_set_writeable_dirty(fs.object); > - > - /* > - * If this is a NOSYNC mmap we do not want to set VPO_NOSYNC > - * if the page is already dirty to prevent data written with > - * the expectation of being synced from not being synced. > - * Likewise if this entry does not request NOSYNC then make > - * sure the page isn't marked NOSYNC. Applications sharing > - * data should use the same flags to avoid ping ponging. > - */ > - if (fs.entry->eflags & MAP_ENTRY_NOSYNC) { > - if (fs.m->dirty == 0) > - fs.m->oflags |= VPO_NOSYNC; > - } else { > - fs.m->oflags &= ~VPO_NOSYNC; > - } > - > - /* > - * If the fault is a write, we know that this page is being > - * written NOW so dirty it explicitly to save on > - * pmap_is_modified() calls later. > - * > - * Also tell the backing pager, if any, that it should remove > - * any swap backing since the page is now dirty. > - */ > - if (((fault_type & VM_PROT_WRITE) != 0 && > - (fault_flags & VM_FAULT_CHANGE_WIRING) == 0) || > - (fault_flags & VM_FAULT_DIRTY) != 0) { > - vm_page_dirty(fs.m); > - vm_pager_page_unswapped(fs.m); > - } > - } > - > + vm_fault_dirty(fs.entry, fs.m, prot, fault_type, fault_flags, TRUE); > vm_page_assert_xbusied(fs.m); > > /* > diff --git a/sys/vm/vm_page.h b/sys/vm/vm_page.h > index f12b76c..a45648d 100644 > --- a/sys/vm/vm_page.h > +++ b/sys/vm/vm_page.h > @@ -147,7 +147,7 @@ struct vm_page { > uint16_t hold_count; /* page hold count (P) */ > uint16_t flags; /* page PG_* flags (P) */ > uint8_t aflags; /* access is atomic */ > - uint8_t oflags; /* page VPO_* flags (O) */ > + uint8_t oflags; /* page VPO_* flags (OM) */ > uint8_t queue; /* page queue index (P,Q) */ > int8_t psind; /* pagesizes[] index (O) */ > int8_t segind; > @@ -163,8 +163,9 @@ struct vm_page { > /* > * Page flags stored in oflags: > * > - * Access to these page flags is synchronized by the lock on the object > - * containing the page (O). > + * Access to these page flags is synchronized by the exclusive lock on > + * the object containing the page, or combination of shared object > + * lock and the page lock (OM). > * > * Note: VPO_UNMANAGED (used by OBJT_DEVICE, OBJT_PHYS and OBJT_SG) > * indicates that the page is not under PV management but From owner-freebsd-hackers@FreeBSD.ORG Mon Sep 8 18:34:43 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A54D8E07; Mon, 8 Sep 2014 18:34:43 +0000 (UTC) Received: from dmz-mailsec-scanner-4.mit.edu (dmz-mailsec-scanner-4.mit.edu [18.9.25.15]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2DD0917D6; Mon, 8 Sep 2014 18:34:42 +0000 (UTC) X-AuditID: 1209190f-f79aa6d000005b45-ae-540df6c1f31b Received: from mailhub-auth-2.mit.edu ( [18.7.62.36]) (using TLS with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by dmz-mailsec-scanner-4.mit.edu (Symantec Messaging Gateway) with SMTP id 99.5C.23365.1C6FD045; Mon, 8 Sep 2014 14:34:41 -0400 (EDT) Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) by mailhub-auth-2.mit.edu (8.13.8/8.9.2) with ESMTP id s88IYenn013670; Mon, 8 Sep 2014 14:34:40 -0400 Received: from multics.mit.edu (system-low-sipb.mit.edu [18.187.2.37]) (authenticated bits=56) (User authenticated as kaduk@ATHENA.MIT.EDU) by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id s88IYcmg025555 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Mon, 8 Sep 2014 14:34:39 -0400 Received: (from kaduk@localhost) by multics.mit.edu (8.12.9.20060308) id s88IYbDj006507; Mon, 8 Sep 2014 14:34:37 -0400 (EDT) Date: Mon, 8 Sep 2014 14:34:37 -0400 (EDT) From: Benjamin Kaduk To: John Baldwin Subject: Re: Tool to access ZFS/NFSv4 alternate data streams on FreeBSD? In-Reply-To: <2154953.PmiECqQQIi@ralph.baldwin.cx> Message-ID: References: <1538621043.33014113.1409955190712.JavaMail.root@uoguelph.ca> <2154953.PmiECqQQIi@ralph.baldwin.cx> User-Agent: Alpine 1.10 (GSO 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFuplleLIzCtJLcpLzFFi42IRYrdT0T34jTfE4MdFMYvtm/8xWuw9cp3Z ovlRL5PFw2XXmBxYPGZ8ms/isXPWXXaP35v3MgUwR3HZpKTmZJalFunbJXBlHFlzlrmgkaVi Q/91xgbGPuYuRk4OCQETiQl7PzJB2GISF+6tZ+ti5OIQEpjNJPHt1XUWCGcDo8T9vc1QzkEm iRuL57ODtAgJ1EvMfdcE1s4ioCUxe+ZJFhCbTUBFYuabjUCjODhEBJQkpn5TAwkzC5RKnJ+5 nRHEFhbwkGhccxdsDKeAkcT1DRfBLuIVcJRoWvCZFWJ8vsSBCdfARooK6Eis3j+FBaJGUOLk zCcsEDO1JJZP38YygVFwFpLULCSpBYxMqxhlU3KrdHMTM3OKU5N1i5MT8/JSi3RN9HIzS/RS U0o3MYLCmFOSfwfjt4NKhxgFOBiVeHgTrvKECLEmlhVX5h5ilORgUhLl3faRN0SILyk/pTIj sTgjvqg0J7X4EKMEB7OSCO/Ty0A53pTEyqrUonyYlDQHi5I471trq2AhgfTEktTs1NSC1CKY rAwHh5IEb89XoEbBotT01Iq0zJwShDQTByfIcB6g4fzfQIYXFyTmFmemQ+RPMRpztDS97WXi WNf5rZ9JiCUvPy9VSpx3zhegUgGQ0ozSPLhpsFT0ilEc6Dlh3hUgS3mAaQxu3iugVUxAqyYF g60qSURISTUwtouJzuNhKOHbYnBWTflrmEpci6xT4uQlJk9KBFe2eNgcP6JQtW3TroPl7aLe Dxu7by6MuXaf84siv+Myc4Ow/X+/yLy3/hBYGWV31ZTPvjpk6m3XWMM//rs6l156cVwlw6Ux bxVLWu61OGXhN7WaIpvWc12yEl7t9u9Kb+tBff0dIcv38CgosRRnJBpqMRcVJwIASRSDiyAD AAA= Cc: freebsd-hackers@freebsd.org, Rick Macklem , Lionel Cons X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Sep 2014 18:34:43 -0000 On Mon, 8 Sep 2014, John Baldwin wrote: > On Friday, September 05, 2014 06:13:10 PM Rick Macklem wrote: > > > > I do not know of any work for ZFS on FreeBSD w.r.t. this, but I'm not a > > ZFS guy. > > Does the NFSv4 protocol support resource forks as a separate entity from EAs > though? Presumably O_XATTR would turn into a new VOP (VOP_OPENFORK() or some I think it does not, but this stuff is rather confusing to try and follow. (I only participate in nfsv4@ietf.org a tiny little bit.) -Ben From owner-freebsd-hackers@FreeBSD.ORG Sun Sep 7 06:42:23 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BFA2D1A6 for ; Sun, 7 Sep 2014 06:42:23 +0000 (UTC) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) by mx1.freebsd.org (Postfix) with ESMTP id 8477219B3 for ; Sun, 7 Sep 2014 06:42:23 +0000 (UTC) Received: from critter.freebsd.dk (unknown [192.168.60.3]) by phk.freebsd.dk (Postfix) with ESMTP id 71E9F1578; Sun, 7 Sep 2014 06:42:16 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.14.9/8.14.9) with ESMTP id s876gEHM033744; Sun, 7 Sep 2014 06:42:15 GMT (envelope-from phk@phk.freebsd.dk) To: Hans Petter Selasky Subject: Re: What to do if USB stack seems dead In-reply-to: <540B7AC4.9060504@selasky.org> From: "Poul-Henning Kamp" References: <1455331.JKPkSxLmbq@quad> <540B7AC4.9060504@selasky.org> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <33742.1410072134.1@critter.freebsd.dk> Content-Transfer-Encoding: quoted-printable Date: Sun, 07 Sep 2014 06:42:14 +0000 Message-ID: <33743.1410072134@critter.freebsd.dk> X-Mailman-Approved-At: Mon, 08 Sep 2014 20:28:05 +0000 Cc: freebsd-hackers@freebsd.org, Maxim V FIlimonov X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Sep 2014 06:42:23 -0000 -------- In message <540B7AC4.9060504@selasky.org>, Hans Petter Selasky writes: >On 09/06/14 21:57, Maxim V FIlimonov wrote: >> Lately, I've been heavily experimenting with different USB devices (for >> instance, USB to TTL one, but that's another story), and at a moment I >> encountered that the system doesn't react on new USB devices connected.= The >> connected devices work fine, though. The question is: what can I do in = such a >> case if I don't want to reboot my box? >> > >Hi, > >If a TTY port is not closed, it will prevent other USB devices on that = >particular USB controller from enumerating. I'm pretty sure I have seen this "dead USB" thing on = FreeBSD ni.freebsd.dk 11.0-CURRENT FreeBSD 11.0-CURRENT #3 r270830: without any USB/TTY ports being involved at all... -- = Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe = Never attribute to malice what can adequately be explained by incompetence= . From owner-freebsd-hackers@FreeBSD.ORG Mon Sep 8 15:39:25 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B7E538D5; Mon, 8 Sep 2014 15:39:25 +0000 (UTC) Received: from mail-la0-x232.google.com (mail-la0-x232.google.com [IPv6:2a00:1450:4010:c03::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 12ED01E4F; Mon, 8 Sep 2014 15:39:24 +0000 (UTC) Received: by mail-la0-f50.google.com with SMTP id ty20so2817237lab.37 for ; Mon, 08 Sep 2014 08:39:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=gTeBojhxJIumREFY8IeDhPU+oBdiSm06s0cfq69Gx9M=; b=y6TY6ym3ElrZQbK55HzddyZvFLjP6qMH3VNftkuC8KQxdhUOG0qVRDGT6Bj/9AFgQB 66q8uiC8UwGYVEj1bK9rtS2r3313qJ08ElUayU2vrucVT7dfUVUlHQBGIWPI/Vmi7EvX fQmaymJX7BT/+CeIYolDk6fdm5BAfR3Y49ajkmthk/KbTjlDWWNE0+XBVWsoD8d9cAiq tHC+fPyax22AgfkzoujGc6PYByInPliXV74rdzSFlDGmGLJ9bGarLJ3+R2TAUhkwDc8v 1HQjRbC4pgQ/rT0lZt1zrK3C7Xs6dIW2eBG9h0AjHuppUR5xGwLarAO8MyDvudP+4Sbd aXiw== MIME-Version: 1.0 X-Received: by 10.112.114.227 with SMTP id jj3mr28514095lbb.39.1410190762898; Mon, 08 Sep 2014 08:39:22 -0700 (PDT) Received: by 10.25.166.211 with HTTP; Mon, 8 Sep 2014 08:39:22 -0700 (PDT) Received: by 10.25.166.211 with HTTP; Mon, 8 Sep 2014 08:39:22 -0700 (PDT) In-Reply-To: References: Date: Mon, 8 Sep 2014 08:39:22 -0700 Message-ID: Subject: Re: FreeBSD and WiDi / Miracast / WiFi Direct HDMI streaming From: Waitman Gobble To: CeDeROM X-Mailman-Approved-At: Mon, 08 Sep 2014 20:29:39 +0000 Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-hackers@freebsd.org, freebsd-questions@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Sep 2014 15:39:25 -0000 On Sep 8, 2014 1:01 AM, "CeDeROM" wrote: > > Hello world! :-) > > I am considering to buy Netgear PTV3000 [1] for wireless audio-video > streaming HDMI adapter. Does FreeBSD support any of the > WiDi/Miracast/WiFiDirect medium? It seems very very interesting portable > multiplatform solution :-) > > Best regards :-) > Tomek > > [1] > http://www.netgear.com/home/products/connected-entertainment/wireless-display-adapters/PTV3000.aspx > > -- > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > _______________________________________________ > freebsd-questions@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to " freebsd-questions-unsubscribe@freebsd.org" Looks like openwfd could be modified to build on FreeBSD. Waitman From owner-freebsd-hackers@FreeBSD.ORG Mon Sep 8 16:21:53 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 24B0DEC3; Mon, 8 Sep 2014 16:21:53 +0000 (UTC) Received: from st11p02mm-asmtp001.mac.com (st11p02mm-asmtpout001.mac.com [17.172.220.236]) (using TLSv1 with cipher DES-CBC3-SHA (168/168 bits)) (Client CN "smtp.me.com", Issuer "VeriSign Class 3 Extended Validation SSL SGC CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EB11514B3; Mon, 8 Sep 2014 16:21:52 +0000 (UTC) Received: from fukuyama.hsd1.ca.comcast.net (unknown [73.162.13.215]) by st11p02mm-asmtp001.mac.com (Oracle Communications Messaging Server 7u4-27.10(7.0.4.27.9) 64bit (built Jun 6 2014)) with ESMTPSA id <0NBL005O1C4D6P20@st11p02mm-asmtp001.mac.com>; Mon, 08 Sep 2014 16:21:51 +0000 (GMT) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.12.52,1.0.27,0.0.0000 definitions=2014-09-08_03:2014-09-08,2014-09-08,1970-01-01 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 suspectscore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=7.0.1-1402240000 definitions=main-1409080140 Content-type: text/plain; charset=us-ascii MIME-version: 1.0 (Mac OS X Mail 8.0 \(1973.6\)) Subject: Re: FreeBSD and WiDi / Miracast / WiFi Direct HDMI streaming From: Rui Paulo In-reply-to: Date: Mon, 08 Sep 2014 09:21:48 -0700 Content-transfer-encoding: 7bit Message-id: <042D853C-F703-4B76-AA28-632B5F55C8D3@me.com> References: To: CeDeROM X-Mailer: Apple Mail (2.1973.6) X-Mailman-Approved-At: Mon, 08 Sep 2014 20:29:49 +0000 Cc: freebsd-hackers@freebsd.org, FreeBSD Questions Mailing List X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Sep 2014 16:21:53 -0000 On Sep 8, 2014, at 01:01, CeDeROM wrote: > > Hello world! :-) > > I am considering to buy Netgear PTV3000 [1] for wireless audio-video > streaming HDMI adapter. Does FreeBSD support any of the > WiDi/Miracast/WiFiDirect medium? No, it's not supported. -- Rui Paulo From owner-freebsd-hackers@FreeBSD.ORG Mon Sep 8 20:40:13 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6888ED13 for ; Mon, 8 Sep 2014 20:40:13 +0000 (UTC) Received: from mho-02-ewr.mailhop.org (mho-02-ewr.mailhop.org [204.13.248.72]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3C52A194D for ; Mon, 8 Sep 2014 20:40:13 +0000 (UTC) Received: from [73.34.117.227] (helo=ilsoft.org) by mho-02-ewr.mailhop.org with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.72) (envelope-from ) id 1XR5jP-000Kuz-9C; Mon, 08 Sep 2014 20:40:11 +0000 Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by ilsoft.org (8.14.9/8.14.9) with ESMTP id s88Ke9JU022103; Mon, 8 Sep 2014 14:40:09 -0600 (MDT) (envelope-from ian@FreeBSD.org) X-Mail-Handler: Dyn Standard SMTP by Dyn X-Originating-IP: 73.34.117.227 X-Report-Abuse-To: abuse@dyndns.com (see http://www.dyndns.com/services/sendlabs/outbound_abuse.html for abuse reporting information) X-MHO-User: U2FsdGVkX1+IBWTMUXzEjIUcwDjmAbNp X-Authentication-Warning: paranoia.hippie.lan: Host revolution.hippie.lan [172.22.42.240] claimed to be [172.22.42.240] Subject: Re: What to do if USB stack seems dead From: Ian Lepore To: Hans Petter Selasky In-Reply-To: <540B7AC4.9060504@selasky.org> References: <1455331.JKPkSxLmbq@quad> <540B7AC4.9060504@selasky.org> Content-Type: text/plain; charset="us-ascii" Date: Mon, 08 Sep 2014 14:40:08 -0600 Message-ID: <1410208808.1150.406.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Sep 2014 20:40:13 -0000 On Sat, 2014-09-06 at 23:21 +0200, Hans Petter Selasky wrote: > On 09/06/14 21:57, Maxim V FIlimonov wrote: > > Lately, I've been heavily experimenting with different USB devices (for > > instance, USB to TTL one, but that's another story), and at a moment I > > encountered that the system doesn't react on new USB devices connected. The > > connected devices work fine, though. The question is: what can I do in such a > > case if I don't want to reboot my box? > > > > Hi, > > If a TTY port is not closed, it will prevent other USB devices on that > particular USB controller from enumerating. > > --HPS What does that mean, "is not closed"? You mean if I have any usb-serial adapter tty device open, other device insert/remove events aren't handled? Or that if a usb-serial device is removed while the device is open, that prevents other insert/remove events? -- Ian From owner-freebsd-hackers@FreeBSD.ORG Mon Sep 8 21:18:49 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C06BAC6F; Mon, 8 Sep 2014 21:18:49 +0000 (UTC) Received: from mail.turbocat.net (heidi.turbocat.net [88.198.202.214]) (using TLSv1.1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7B5A01DE0; Mon, 8 Sep 2014 21:18:49 +0000 (UTC) Received: from laptop015.home.selasky.org (cm-176.74.213.204.customer.telag.net [176.74.213.204]) (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id CFB121FE027; Mon, 8 Sep 2014 23:18:45 +0200 (CEST) Message-ID: <540E1D2E.3020401@selasky.org> Date: Mon, 08 Sep 2014 23:18:38 +0200 From: Hans Petter Selasky User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.1.0 MIME-Version: 1.0 To: Ian Lepore Subject: Re: What to do if USB stack seems dead References: <1455331.JKPkSxLmbq@quad> <540B7AC4.9060504@selasky.org> <1410208808.1150.406.camel@revolution.hippie.lan> In-Reply-To: <1410208808.1150.406.camel@revolution.hippie.lan> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Sep 2014 21:18:49 -0000 On 09/08/14 22:40, Ian Lepore wrote: > On Sat, 2014-09-06 at 23:21 +0200, Hans Petter Selasky wrote: >> On 09/06/14 21:57, Maxim V FIlimonov wrote: >>> Lately, I've been heavily experimenting with different USB devices (for >>> instance, USB to TTL one, but that's another story), and at a moment I >>> encountered that the system doesn't react on new USB devices connected. The >>> connected devices work fine, though. The question is: what can I do in such a >>> case if I don't want to reboot my box? >>> >> >> Hi, >> >> If a TTY port is not closed, it will prevent other USB devices on that >> particular USB controller from enumerating. >> >> --HPS > > What does that mean, "is not closed"? You mean if I have any usb-serial > adapter tty device open, other device insert/remove events aren't > handled? > > Or that if a usb-serial device is removed while the device is open, that > prevents other insert/remove events? > > -- Ian Yes, because we need to wait for some refcounting inside the TTY layer at detach when the TTY device is open at detach. It is not a USB stack problem. --HPS From owner-freebsd-hackers@FreeBSD.ORG Mon Sep 8 23:10:47 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 56886BCC; Mon, 8 Sep 2014 23:10:47 +0000 (UTC) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 07B9B2BF; Mon, 8 Sep 2014 23:10:46 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ArcEABU3DlSDaFve/2dsb2JhbABZg2BXBIJ4xnwKhnlTAYEreIQEAQEEAQEBICsgCxsOCgICDRkCKQEJJgYIBwQBHASIIQ2nXJVmAReBLI1QAQEbNAeCeYFTBZVwg3+EYopEiQmDfSEvB4EIOYEHAQEB X-IronPort-AV: E=Sophos;i="5.04,489,1406606400"; d="scan'208";a="152555800" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 08 Sep 2014 19:10:44 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 6B63B792A8; Mon, 8 Sep 2014 19:10:44 -0400 (EDT) Date: Mon, 8 Sep 2014 19:10:44 -0400 (EDT) From: Rick Macklem To: John Baldwin Message-ID: <755175739.33844219.1410217844431.JavaMail.root@uoguelph.ca> In-Reply-To: <2154953.PmiECqQQIi@ralph.baldwin.cx> Subject: Re: Tool to access ZFS/NFSv4 alternate data streams on FreeBSD? MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.209] X-Mailer: Zimbra 7.2.6_GA_2926 (ZimbraWebClient - FF3.0 (Win)/7.2.6_GA_2926) Cc: freebsd-hackers@freebsd.org, Richard Yao , Jordan Hubbard , Lionel Cons , Jan Bramkamp X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Sep 2014 23:10:47 -0000 John Baldwin wrote: > On Friday, September 05, 2014 06:13:10 PM Rick Macklem wrote: > > Lionel Cons wrote: > > > On 5 September 2014 19:26, Richard Yao wrote: > > > > On 09/05/2014 11:35 AM, Jan Bramkamp wrote: > > > >> On 05.09.2014 16:25, Lionel Cons wrote:> Is there any tool > > > >> which > > > >> can be > > > >> used to access ZFS and NFSv4 alternate > > > >> > > > >>> data streams on FreeBSD? > > > >> > > > >> Are you looking for lsextattr(8) and getextattr(8)? > > > >> _______________________________________________ > > > >> freebsd-hackers@freebsd.org mailing list > > > >> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > > > >> To unsubscribe, send any mail to > > > >> "freebsd-hackers-unsubscribe@freebsd.org" > > > > > > > > Do you mean Solaris extended attributes? Those tend to be > > > > called > > > > resource forks on other platforms. Unifying extended attributes > > > > and > > > > resource forks was clever. > > > > > > Yes, they are also called resource forks, or alternate data > > > streams. > > > The attribute files which can be accessed via O_XATTR or cd -@ > > > file/dir on newer ksh/ksh93/bash revisions. > > > > For FreeBSD's NFSv4 the answer is definitely no. Because the > > Linux/FreeBSD > > style setextattr() assumes an atomic replacement of the extended > > attribute, > > it is not semantically compatible (ie. cannot be accurately > > emulated) by > > resource forks. > > > > I do not know of any work for ZFS on FreeBSD w.r.t. this, but I'm > > not a > > ZFS guy. > > Does the NFSv4 protocol support resource forks as a separate entity > from EAs > though? Presumably O_XATTR would turn into a new VOP (VOP_OPENFORK() > or some > such), it wouldn't be shoehorned into the EA APIs. > Yep, that could be done quite easily, from the NFSv4 perspective. NFSv4 calls it Openattr. It isn't completely obvious if NFSv4 supports this recursively (ala Solaris) or only one level deep, but no one seems to care much about the recursive case. Last time this came up for discussion, Jordan Hubbard got quite involved along the lines of ``most of the work is in userland, for archive tools, etc``. I can`t remember what the mailing list thread was called, but it was started by a guy who was a ``resource fork`` advocate (associated with CERN if I recall), where they use Gbyte extended attributes. If others decide to add the VOP calls and make them work for ZFS, I could easily wire in the NFSv4 bits. rick > -- > John Baldwin > From owner-freebsd-hackers@FreeBSD.ORG Mon Sep 8 23:16:28 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3BF4AD2E; Mon, 8 Sep 2014 23:16:28 +0000 (UTC) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id E142E321; Mon, 8 Sep 2014 23:16:27 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ArcEAIUyDlSDaFve/2dsb2JhbABZg2BXBIJ4xnsKhnlTAYEqeIQEAQEEAQEBICsgCxsOCgICDRkCKQEJJgYIBwQBHASIIQ2nYJVmAReBLI1QAQEbNAeCeYFTBZVwg3+EYopEiQmDfSEvB4EIOYEHAQEB X-IronPort-AV: E=Sophos;i="5.04,488,1406606400"; d="scan'208";a="153657440" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-annu.net.uoguelph.ca with ESMTP; 08 Sep 2014 19:16:20 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id A3B8FB3F43; Mon, 8 Sep 2014 19:16:20 -0400 (EDT) Date: Mon, 8 Sep 2014 19:16:20 -0400 (EDT) From: Rick Macklem To: John Baldwin Message-ID: <1152441008.33844915.1410218180664.JavaMail.root@uoguelph.ca> In-Reply-To: <2154953.PmiECqQQIi@ralph.baldwin.cx> Subject: Re: Tool to access ZFS/NFSv4 alternate data streams on FreeBSD? MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.201] X-Mailer: Zimbra 7.2.6_GA_2926 (ZimbraWebClient - FF3.0 (Win)/7.2.6_GA_2926) Cc: freebsd-hackers@freebsd.org, Richard Yao , Lionel Cons , Jan Bramkamp X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Sep 2014 23:16:28 -0000 John Baldwin wrote: > On Friday, September 05, 2014 06:13:10 PM Rick Macklem wrote: > > Lionel Cons wrote: > > > On 5 September 2014 19:26, Richard Yao wrote: > > > > On 09/05/2014 11:35 AM, Jan Bramkamp wrote: > > > >> On 05.09.2014 16:25, Lionel Cons wrote:> Is there any tool > > > >> which > > > >> can be > > > >> used to access ZFS and NFSv4 alternate > > > >> > > > >>> data streams on FreeBSD? > > > >> > > > >> Are you looking for lsextattr(8) and getextattr(8)? > > > >> _______________________________________________ > > > >> freebsd-hackers@freebsd.org mailing list > > > >> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > > > >> To unsubscribe, send any mail to > > > >> "freebsd-hackers-unsubscribe@freebsd.org" > > > > > > > > Do you mean Solaris extended attributes? Those tend to be > > > > called > > > > resource forks on other platforms. Unifying extended attributes > > > > and > > > > resource forks was clever. > > > > > > Yes, they are also called resource forks, or alternate data > > > streams. > > > The attribute files which can be accessed via O_XATTR or cd -@ > > > file/dir on newer ksh/ksh93/bash revisions. > > > > For FreeBSD's NFSv4 the answer is definitely no. Because the > > Linux/FreeBSD > > style setextattr() assumes an atomic replacement of the extended > > attribute, > > it is not semantically compatible (ie. cannot be accurately > > emulated) by > > resource forks. > > > > I do not know of any work for ZFS on FreeBSD w.r.t. this, but I'm > > not a > > ZFS guy. > > Does the NFSv4 protocol support resource forks as a separate entity > from EAs > though? Presumably O_XATTR would turn into a new VOP (VOP_OPENFORK() > or some > such), it wouldn't be shoehorned into the EA APIs. > Oh, just to clarify it, NFSv4 currently only supports resource forks and not EAs. Similar to NFSv4 ACLs, some believed EAs could be emulated via resource forks, but since modifying resource forks isn`t atomic, the ``NFSv4 collective`` has basically given up on that. There is a proposal to add EAs to NFSv4.n (where n is greater than 1), but I have no idea if or when this will exist. rick > -- > John Baldwin > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to > "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@FreeBSD.ORG Tue Sep 9 01:41:49 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 909C8FE; Tue, 9 Sep 2014 01:41:49 +0000 (UTC) Received: from mail-oa0-x22b.google.com (mail-oa0-x22b.google.com [IPv6:2607:f8b0:4003:c02::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E310912A8; Tue, 9 Sep 2014 01:22:22 +0000 (UTC) Received: by mail-oa0-f43.google.com with SMTP id m19so243593oag.16 for ; Mon, 08 Sep 2014 18:22:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=xB+VZYdZV1rusA+oKhoSDx1lw6szFQWBRQy74wM71EA=; b=YG4qVEfrdfRzxkQGV4Dwf6a9eRLa272WqySACKRA+DkR283LC6SaDRfCfkgFuQTpBo ubXxaYfh+ez7EN9T1DEW4BSQjyQle+5UyKFYWJNQr6uF9i72oFEgQuDfIAY/sEqVVuxj ZHvWZxlyn4sFUJgTRMKxy7+Y0JC5v81HhaySdNLoEA2aCyFg9S16JvvXEbjEV1IONj5i KAG4RM0DKuCjU2kOcW+h35ZkONfbAkJxgG/WX4oJiJyu1D93+az3jpGl7+uMkHIQXS2V u9DN+fr04CHjoLXUruE2wExMViOPZooTJcyeSc2GkQKoOIQkgYDr1abfpsQnDuX7CatM ziEA== MIME-Version: 1.0 X-Received: by 10.60.161.71 with SMTP id xq7mr17729308oeb.1.1410225742156; Mon, 08 Sep 2014 18:22:22 -0700 (PDT) Received: by 10.182.87.33 with HTTP; Mon, 8 Sep 2014 18:22:22 -0700 (PDT) Date: Mon, 8 Sep 2014 21:22:22 -0400 Message-ID: Subject: Any recent spam that states it is from me is not. From: Joe Nosay To: FreeBSD PowerPC ML , FreeBSD Mailing List , FreeBSD Hackers , freebsd-current Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Sep 2014 01:41:49 -0000 Please check the headers and then ask me. This is for those who may assume that I do such things. From owner-freebsd-hackers@FreeBSD.ORG Tue Sep 9 04:47:49 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6781C6AF; Tue, 9 Sep 2014 04:47:49 +0000 (UTC) Received: from mail.iXsystems.com (mail.ixsystems.com [12.229.62.4]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "*.ixsystems.com", Issuer "Go Daddy Secure Certification Authority" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 45E93BDF; Tue, 9 Sep 2014 04:47:49 +0000 (UTC) Received: from localhost (mail.ixsystems.com [10.2.55.1]) by mail.iXsystems.com (Postfix) with ESMTP id 833A57F4B8; Mon, 8 Sep 2014 21:47:45 -0700 (PDT) Received: from mail.iXsystems.com ([10.2.55.1]) by localhost (mail.ixsystems.com [10.2.55.1]) (maiad, port 10024) with ESMTP id 71607-06; Mon, 8 Sep 2014 21:47:45 -0700 (PDT) Received: from [10.8.0.38] (unknown [10.8.0.38]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mail.iXsystems.com (Postfix) with ESMTPSA id 0075B7F4B3; Mon, 8 Sep 2014 21:47:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=ixsystems.com; s=newknight0; t=1410238065; bh=aYFX+zW+scPQfqhryTuE23YEAnuwLVQKqZec+rIGe3I=; h=Subject:From:In-Reply-To:Date:Cc:References:To; b=AhWyRDgJlOKp5SLbiLx2gJEN6RnaW6cdYF/22bLCuymq56lculi/gvmSGUyTUTZcY 4873xi+FB+2VKj6Ve9nFI0P/T9kMx/6xSU1HjEZft5dOPfLdrWFKig7U6XDw2441U0 TKE5IccghkzS1CB0W8E+F7bksVdG51KxxyrjnAQ0= Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: Tool to access ZFS/NFSv4 alternate data streams on FreeBSD? From: Jordan Hubbard In-Reply-To: <755175739.33844219.1410217844431.JavaMail.root@uoguelph.ca> Date: Mon, 8 Sep 2014 21:47:42 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <9F4D2C26-F077-4CA7-A532-BA4CE562C50D@ixsystems.com> References: <755175739.33844219.1410217844431.JavaMail.root@uoguelph.ca> To: Rick Macklem X-Mailer: Apple Mail (2.1878.6) Cc: freebsd-hackers@freebsd.org, Richard Yao , Lionel Cons , Jan Bramkamp X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Sep 2014 04:47:49 -0000 Yep. I was just describing the experience that OS X went through in = implementing extattrs / legacy resource fork support. To recap it very = briefly: Having NFSv4 support extattrs (or even named streams, if you = want to go that far) is the comparatively easy part. It=92s backing = them up / copying them around that gets more involved, and if you can=92t = back up certain attributes then you=92re not likely to get anyone to = want to use them, at which point the whole =93sharing=94 aspect kind of = takes a back seat. On Sep 8, 2014, at 4:10 PM, Rick Macklem wrote: > Last time this came up for discussion, Jordan Hubbard got quite = involved > along the lines of ``most of the work is in userland, for archive = tools, etc``. > I can`t remember what the mailing list thread was called, but it was = started > by a guy who was a ``resource fork`` advocate (associated with CERN if = I recall), > where they use Gbyte extended attributes. From owner-freebsd-hackers@FreeBSD.ORG Tue Sep 9 08:39:02 2014 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 02538D2E; Tue, 9 Sep 2014 08:39:02 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7E1112F2; Tue, 9 Sep 2014 08:39:01 +0000 (UTC) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id s898cmTo074518 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 9 Sep 2014 11:38:48 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua s898cmTo074518 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id s898clCr074516; Tue, 9 Sep 2014 11:38:47 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 9 Sep 2014 11:38:47 +0300 From: Konstantin Belousov To: Alan Cox Subject: Re: mmap MAP_NOSYNC regression in 10.x Message-ID: <20140909083847.GB2737@kib.kiev.ua> References: <540903FF.6010602@degoeje.nl> <20140905080633.GM2737@kib.kiev.ua> <5409A4F8.6020204@degoeje.nl> <20140905123826.GP2737@kib.kiev.ua> <540BA416.7010106@rice.edu> <20140908084126.GX2737@kib.kiev.ua> <540DEE8F.5080005@rice.edu> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="6iAIcqJi9p/aaN8Y" Content-Disposition: inline In-Reply-To: <540DEE8F.5080005@rice.edu> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: alc@freebsd.org, Pieter de Goeje , hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Sep 2014 08:39:02 -0000 --6iAIcqJi9p/aaN8Y Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Sep 08, 2014 at 12:59:43PM -0500, Alan Cox wrote: > On 09/08/2014 03:41, Konstantin Belousov wrote: > > On Sat, Sep 06, 2014 at 07:17:26PM -0500, Alan Cox wrote: > >> On 09/05/2014 07:38, Konstantin Belousov wrote: > >>> On Fri, Sep 05, 2014 at 01:56:40PM +0200, Pieter de Goeje wrote: > >>>> Thanks, works for me! > >>> I realized that the patch contains yet another bug. The oflags page > >>> flags update is protected by the exclusive vm object lock, which is o= nly > >>> held in shared mode on the fast path. Below is the fixed patch, where > >>> I take the page lock around setting VPO_NOSYNC (exclusive lock owners > >>> cannot race with fast path since we own shared lock, and two parallel > >>> fast path execution would be handled by the page lock). > >> > >> Suppose that the page is clean and two threads are executing this code > >> concurrently. One's map entry has MAP_NOSYNC set, and the other's > >> doesn't. Let's call these threads NOSYNC and SYNC, respectively. > >> > >> Suppose that the thread SYNC is slightly ahead. It has already > >> performed "m->oflags &=3D ~VPO_NOSYNC;" and now it's about to perform > >> "vm_page_dirty(fs.m);". However, just before the thread SYNC calls > >> vm_page_dirty(), the thread NOSYNC evaluates "m->dirty =3D=3D 0", whic= h is > >> still true, and it performs "m->oflags |=3D VPO_NOSYNC; " > >> > >> This can't happen on the slow path. That is, a fault by a thread > >> without MAP_NOSYNC set on its map entry will reliably clear VPO_NOSYNC. > > As I understand things, it is indeed not possible on the slow path, due > > to PG_RW only set from pmap_enter(), am I right ? I.e. this is another > > place where the rule 'no PG_RW without PG_M' is important. >=20 >=20 > Yes, it's not possible, but I'm a little confused by the rest of your > question, specifically, the statement "no PG_RW without PG_M". Did you > actually mean "no PG_M without PG_RW"? No, I mean what I did wrote, and I was wrong. >=20 >=20 > > Let me formulate my question another way: what are the guarantees we > > provide to the applications when the same page is mapped with and > > without MAP_NOSYNC simultaneously ? Is it contractually guaranteed that > > any write from !MAP_NOSYNC entry triggers write in the syncer activity > > period ? >=20 >=20 > Yes, that is the intent. However, I can think of at least one case > where the existing code doesn't work as intended. Suppose that the > first fault on a !MAP_NOSYNC entry is triggered by a read access. Then, > vm_fault() won't call vm_page_dirty(), but it will nonetheless install a > mapping in the pmap that allows write access. Now, suppose this same > process writes to the page. Finally, suppose that the second fault > happens on a MAP_NOSYNC entry. That fault will see a clean page, i.e., > m->dirty =3D=3D 0, and set VPO_NOSYNC on the page, even though the first > faulting process that wants the page sync'ed has dirtied the page. Yes, you are right. I convinced myself that this cannot happen, due to the false assumption above, and the fact that page flushing removes write bit from pte. >=20 >=20 >=20 > >> The best course of action may be to fall back to the slow path if you > >> actually need to change VPO_NOSYNC's state. Usually, you won't need t= o. > >> > > Let me first try to improve the original patch to handle > > MAP_ENTRY_NOSYNC on fast path as well. It seems to be one of the cases > > when the parallel faults are actually useful. >=20 >=20 > I think it may be time to take a step back, decide what semantics we > really want, and see if there is a better way of implementing those > semantics. The current approach based on toggling VPO_NOSYNC only > really works for the simplest cases. Still, I believe that the current form of the patch completely repeats the existing semantic of the slow path in the fast path. I may propose to avoid putting vm_page_dirty() in the scope of page lock for vm_fault_dirty(), mostly to simplify the logic, not to provide any useful optimization. >=20 >=20 > > One more note: the previous patch handled m->oflags inconsistency for > > setting VPO_NOSYNC operation, but missed the clear one line later. > > I think that increasing the page lock to cover also the vm_page_dirty() > > would fix the race you described, and the second manipulation with > > oflags. > > > > diff --git a/sys/vm/vm_fault.c b/sys/vm/vm_fault.c > > index 30b0456..944b479 100644 > > --- a/sys/vm/vm_fault.c > > +++ b/sys/vm/vm_fault.c > > @@ -174,6 +174,70 @@ unlock_and_deallocate(struct faultstate *fs) > > } > > } > > =20 > > +static void > > +vm_fault_dirty(vm_map_entry_t entry, vm_page_t m, vm_prot_t prot, > > + vm_prot_t fault_type, int fault_flags, boolean_t set_wd) > > +{ > > + boolean_t need_dirty; > > + > > + if (((prot & VM_PROT_WRITE) =3D=3D 0 && > > + (fault_flags & VM_FAULT_DIRTY) =3D=3D 0) || > > + (m->oflags & VPO_UNMANAGED) !=3D 0) > > + return; > > + > > + VM_OBJECT_ASSERT_LOCKED(m->object); > > + > > + need_dirty =3D ((fault_type & VM_PROT_WRITE) !=3D 0 && > > + (fault_flags & VM_FAULT_CHANGE_WIRING) =3D=3D 0) || > > + (fault_flags & VM_FAULT_DIRTY) !=3D 0; > > + > > + if (set_wd) > > + vm_object_set_writeable_dirty(m->object); > > + else > > + /* > > + * If two callers of vm_fault_dirty() with set_wd =3D=3D > > + * FALSE, one for the map entry with MAP_ENTRY_NOSYNC > > + * flag set, other with flag clear, race, it is > > + * possible for the no-NOSYNC thread to see m->dirty > > + * !=3D 0 and not clear VPO_NOSYNC. Take vm_page lock > > + * around manipulation of VPO_NOSYNC and > > + * vm_page_dirty() call, to avoid the race and keep > > + * m->oflags consistent. > > + */ > > + vm_page_lock(m); > > + > > + /* > > + * If this is a NOSYNC mmap we do not want to set VPO_NOSYNC > > + * if the page is already dirty to prevent data written with > > + * the expectation of being synced from not being synced. > > + * Likewise if this entry does not request NOSYNC then make > > + * sure the page isn't marked NOSYNC. Applications sharing > > + * data should use the same flags to avoid ping ponging. > > + */ > > + if ((entry->eflags & MAP_ENTRY_NOSYNC) !=3D 0) { > > + if (m->dirty =3D=3D 0) { > > + m->oflags |=3D VPO_NOSYNC; > > + } > > + } else { > > + m->oflags &=3D ~VPO_NOSYNC; > > + } > > + > > + /* > > + * If the fault is a write, we know that this page is being > > + * written NOW so dirty it explicitly to save on > > + * pmap_is_modified() calls later. > > + * > > + * Also tell the backing pager, if any, that it should remove > > + * any swap backing since the page is now dirty. > > + */ > > + if (need_dirty) > > + vm_page_dirty(m); > > + if (!set_wd) > > + vm_page_unlock(m); > > + if (need_dirty) > > + vm_pager_page_unswapped(m); > > +} > > + > > /* > > * TRYPAGER - used by vm_fault to calculate whether the pager for the > > * current object *might* contain the page. > > @@ -321,11 +385,8 @@ RetryFault:; > > vm_page_hold(m); > > vm_page_unlock(m); > > } > > - if ((fault_type & VM_PROT_WRITE) !=3D 0 && > > - (m->oflags & VPO_UNMANAGED) =3D=3D 0) { > > - vm_page_dirty(m); > > - vm_pager_page_unswapped(m); > > - } > > + vm_fault_dirty(fs.entry, m, prot, fault_type, fault_flags, > > + FALSE); > > VM_OBJECT_RUNLOCK(fs.first_object); > > if (!wired) > > vm_fault_prefault(&fs, vaddr, 0, 0); > > @@ -898,42 +959,7 @@ vnode_locked: > > if (hardfault) > > fs.entry->next_read =3D fs.pindex + faultcount - reqpage; > > =20 > > - if (((prot & VM_PROT_WRITE) !=3D 0 || > > - (fault_flags & VM_FAULT_DIRTY) !=3D 0) && > > - (fs.m->oflags & VPO_UNMANAGED) =3D=3D 0) { > > - vm_object_set_writeable_dirty(fs.object); > > - > > - /* > > - * If this is a NOSYNC mmap we do not want to set VPO_NOSYNC > > - * if the page is already dirty to prevent data written with > > - * the expectation of being synced from not being synced. > > - * Likewise if this entry does not request NOSYNC then make > > - * sure the page isn't marked NOSYNC. Applications sharing > > - * data should use the same flags to avoid ping ponging. > > - */ > > - if (fs.entry->eflags & MAP_ENTRY_NOSYNC) { > > - if (fs.m->dirty =3D=3D 0) > > - fs.m->oflags |=3D VPO_NOSYNC; > > - } else { > > - fs.m->oflags &=3D ~VPO_NOSYNC; > > - } > > - > > - /* > > - * If the fault is a write, we know that this page is being > > - * written NOW so dirty it explicitly to save on=20 > > - * pmap_is_modified() calls later. > > - * > > - * Also tell the backing pager, if any, that it should remove > > - * any swap backing since the page is now dirty. > > - */ > > - if (((fault_type & VM_PROT_WRITE) !=3D 0 && > > - (fault_flags & VM_FAULT_CHANGE_WIRING) =3D=3D 0) || > > - (fault_flags & VM_FAULT_DIRTY) !=3D 0) { > > - vm_page_dirty(fs.m); > > - vm_pager_page_unswapped(fs.m); > > - } > > - } > > - > > + vm_fault_dirty(fs.entry, fs.m, prot, fault_type, fault_flags, TRUE); > > vm_page_assert_xbusied(fs.m); > > =20 > > /* > > diff --git a/sys/vm/vm_page.h b/sys/vm/vm_page.h > > index f12b76c..a45648d 100644 > > --- a/sys/vm/vm_page.h > > +++ b/sys/vm/vm_page.h > > @@ -147,7 +147,7 @@ struct vm_page { > > uint16_t hold_count; /* page hold count (P) */ > > uint16_t flags; /* page PG_* flags (P) */ > > uint8_t aflags; /* access is atomic */ > > - uint8_t oflags; /* page VPO_* flags (O) */ > > + uint8_t oflags; /* page VPO_* flags (OM) */ > > uint8_t queue; /* page queue index (P,Q) */ > > int8_t psind; /* pagesizes[] index (O) */ > > int8_t segind; > > @@ -163,8 +163,9 @@ struct vm_page { > > /* > > * Page flags stored in oflags: > > * > > - * Access to these page flags is synchronized by the lock on the object > > - * containing the page (O). > > + * Access to these page flags is synchronized by the exclusive lock on > > + * the object containing the page, or combination of shared object > > + * lock and the page lock (OM). > > * > > * Note: VPO_UNMANAGED (used by OBJT_DEVICE, OBJT_PHYS and OBJT_SG) > > * indicates that the page is not under PV management but --6iAIcqJi9p/aaN8Y Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJUDryXAAoJEJDCuSvBvK1BKWMQAIu9EE++i0GUBWHz0jiTx3iz rrD4XdwOvNJq4kicpIpvv4d5Q8uzEAOECIYXXVxoHkcyXrtZImFikWUCXAHsIRSU p7ZkxW6NBrWYtsbBuJSQvYINyC6fn+iQYvk10O0CyXBchFHc4um2QB0FqkXui4xF vXNiVFWSe7RBChInGeJZ8H8t2iq73TUqlHuIoRSm/4Fu9PRgSHDH6GA6gY707nLq D+Cb37N9l9OUaV0SVdA+OqiuBi9jGq7/lSPIvQz3HCUmQrWmxP4pPIJOfNTYMQJs jpSHfyaWLcwcGnX1oycVWoPOreuWXT/cfq3jrbEZ4xM3iNrFv09G+o6EQQZIsTzO WoBxtwWygyI73TC4K9i91McLxi+DsUyRM4N6A5qlQ3dQLbFmiGyFtcl7HpNZvdga AqRilih9CXJFi5StuL7ySsG17+1QnJqkLSAbkBcAaHL8K3+aMUBK02oK4UW7X7gI 3q5CBclctvfSogX0DQAxpUzaKZkK0jJgqhpW/tMGA5q1wmJAITwXpw2ckqD7btHj i2L8mtKeGReqdjop4GikXclL7sMCu/HMmpWDNJb/v6nNXM+imX7lCn4MGLAkkZwD lp8t+mVwIOvF0WaJpsE8DoMlV5AfS6WZ1so5S/byldl7kvDuyLeTNAPiLHNWekaq 0A5CaSSr+yCYp0o/P+yL =NVhN -----END PGP SIGNATURE----- --6iAIcqJi9p/aaN8Y-- From owner-freebsd-hackers@FreeBSD.ORG Tue Sep 9 11:47:49 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 18000673 for ; Tue, 9 Sep 2014 11:47:49 +0000 (UTC) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id D051AD3D for ; Tue, 9 Sep 2014 11:47:48 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AsYEAEjoDlSDaFve/2dsb2JhbABZg2BXBIJ4xyAKhnlTAYEpeIQDAQEBAwEBAQEgKyALBRYOCgICDRkCKQEJJgYIBwQBHASIGQgNpiWVbwEXgSyNUAEBGwEzB4J5gVMFlXGDf4RiikSJC4N9IS8HgQg5gQcBAQE X-IronPort-AV: E=Sophos;i="5.04,491,1406606400"; d="scan'208";a="152640460" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 09 Sep 2014 07:47:47 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 5BDAEB3F2B; Tue, 9 Sep 2014 07:47:47 -0400 (EDT) Date: Tue, 9 Sep 2014 07:47:47 -0400 (EDT) From: Rick Macklem To: Jordan Hubbard Message-ID: <1207532459.33927535.1410263267363.JavaMail.root@uoguelph.ca> In-Reply-To: <9F4D2C26-F077-4CA7-A532-BA4CE562C50D@ixsystems.com> Subject: Re: Tool to access ZFS/NFSv4 alternate data streams on FreeBSD? MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 7.2.6_GA_2926 (ZimbraWebClient - FF3.0 (Win)/7.2.6_GA_2926) Cc: freebsd-hackers@freebsd.org, Richard Yao , Lionel Cons , Jan Bramkamp X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Sep 2014 11:47:49 -0000 Jordan Hubbard wrote: > Yep. I was just describing the experience that OS X went through in > implementing extattrs / legacy resource fork support. To recap it > very briefly: Having NFSv4 support extattrs (or even named streams, > if you want to go that far) is the comparatively easy part. It=E2=80=99s > backing them up / copying them around that gets more involved, and > if you can=E2=80=99t back up certain attributes then you=E2=80=99re not l= ikely to > get anyone to want to use them, at which point the whole =E2=80=9Csharing= =E2=80=9D > aspect kind of takes a back seat. >=20 Yep. I strongly suspect you are correct. The question then becomes: - Do we wait and see if someone chooses to get around to doing all the hard userland work. or - Do the easy part in the kernel and then hope someone does the hard userland work because they need it. or - Just decide that the Linux style extended attributes are adequate and not do resource forks at all? If the "collective" does decide to support resource forks, I don't think there is a need to have these supported by all file system ty= pes in FreeBSD. For example for NFSv4 ACLs, they are implemented on UFS and ZFS and I haven't seen expressions of a need for them to be supported by other file system types. I also think that users could live with "if you need resource forks, you have to use ZFS". Btw, personally, I couldn't care less if these are implemented. However, if seems that the kernel part is easy to do and might be useful for folk looking for an alternative to Solaris. Now I'm going to go out on a big limb (flame suit on as we used to say;-) and suggest that Oracle/Solaris will be a dying beast that many users will be looking for an alternative to. Having FreeBSD be a useful alternati= ve might be a good direction for FreeBSD to go. rick > On Sep 8, 2014, at 4:10 PM, Rick Macklem > wrote: >=20 > > Last time this came up for discussion, Jordan Hubbard got quite > > involved > > along the lines of ``most of the work is in userland, for archive > > tools, etc``. > > I can`t remember what the mailing list thread was called, but it > > was started > > by a guy who was a ``resource fork`` advocate (associated with CERN > > if I recall), > > where they use Gbyte extended attributes. >=20 > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to > "freebsd-hackers-unsubscribe@freebsd.org" >=20 From owner-freebsd-hackers@FreeBSD.ORG Tue Sep 9 15:53:21 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 02F7E30E for ; Tue, 9 Sep 2014 15:53:21 +0000 (UTC) Received: from mail-pa0-x22e.google.com (mail-pa0-x22e.google.com [IPv6:2607:f8b0:400e:c03::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CC686E87 for ; Tue, 9 Sep 2014 15:53:20 +0000 (UTC) Received: by mail-pa0-f46.google.com with SMTP id kq14so2665516pab.33 for ; Tue, 09 Sep 2014 08:53:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=aW/g4nQIRSFZIEU3ejNAdbgou8t7KMU77X5dHBcrb54=; b=gJNeY4KoP00DLXc4L3yuh6SnLPM1U6momM+F23bDwWXuaE9+HjzXOmQlHhsQHjVZV7 EH8+xHAH+HIWi1V9WahzF88rDr/vENtjmMi7y7A9ZVXkddDt+Ilw0Ax8BZEkmLSa+njb wDBLdPFq9HfFqE0a0pZn4EObfpGBVIsALziH1Qvnzy6opUUkMMCUcwywiRxyjGVHLxrM z3m9kOLMwFoEHpWP4uFxQPgygrF3d80TNj9nQqNl4MUXHteI7nTX3ozUMU9UyGq3B7AR wRm2RbSwmAbpObLx4Z9tCy9AVXfyLQujMkFQooKNKPY+MaJRUeGlb68SuE0kW2L9G9wN PsXw== MIME-Version: 1.0 X-Received: by 10.70.133.200 with SMTP id pe8mr13314465pdb.40.1410278000247; Tue, 09 Sep 2014 08:53:20 -0700 (PDT) Received: by 10.66.82.37 with HTTP; Tue, 9 Sep 2014 08:53:20 -0700 (PDT) In-Reply-To: <9F4D2C26-F077-4CA7-A532-BA4CE562C50D@ixsystems.com> References: <755175739.33844219.1410217844431.JavaMail.root@uoguelph.ca> <9F4D2C26-F077-4CA7-A532-BA4CE562C50D@ixsystems.com> Date: Tue, 9 Sep 2014 17:53:20 +0200 Message-ID: Subject: Re: Tool to access ZFS/NFSv4 alternate data streams on FreeBSD? From: Simon Toedt To: Jordan Hubbard Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org, Richard Yao , Rick Macklem , Lionel Cons , Jan Bramkamp X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Sep 2014 15:53:21 -0000 On Tue, Sep 9, 2014 at 6:47 AM, Jordan Hubbard wrote: > Yep. I was just describing the experience that OS X went through in impl= ementing extattrs / legacy resource fork support. To recap it very briefly= : Having NFSv4 support extattrs (or even named streams, if you want to go = that far) is the comparatively easy part. It=E2=80=99s backing them up / c= opying them around that gets more involved, and if you can=E2=80=99t back u= p certain attributes then you=E2=80=99re not likely to get anyone to want t= o use them, at which point the whole =E2=80=9Csharing=E2=80=9D aspect kind = of takes a back seat. The native Solaris tools (tar/pax) and the AT&T AST - written by David Korn himself and used widely within AT&T and customers (i.e. cloud) support resource forks via O_XATTR. CERN also has a large set of applications which rely on O_XATTR, so it seems this is not so uncommon. Simon P.S: Solaris UFS and tmpfs support resource forks via O_XATTR and cd -@ in bash4.3 From owner-freebsd-hackers@FreeBSD.ORG Tue Sep 9 15:56:28 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CB820567; Tue, 9 Sep 2014 15:56:28 +0000 (UTC) Received: from mail-pa0-x22d.google.com (mail-pa0-x22d.google.com [IPv6:2607:f8b0:400e:c03::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9508DECF; Tue, 9 Sep 2014 15:56:28 +0000 (UTC) Received: by mail-pa0-f45.google.com with SMTP id rd3so5059385pab.4 for ; Tue, 09 Sep 2014 08:56:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=qVtU8Ca+xSFDyivJicV3KGFNCqGRa0d/xVJnGEi439E=; b=k81QnHN7Gq4IttilII5/9FTMDemg5RWw5y8A6Ep2e4dEI++U5G321Lzr4OeT0SFB+N o45pCsQoNEhIW1amINKbM9oSq79a7q3PkUbSaqCYejv1srnRWVODHnuaLIyQ8OtYgvCx SXYRWBYbfunLSGU73oVbdEs29/LGivQg37nTQCXixqgAAWGA0Q/jS+OvsFYNWBEQ9EhE ztd94WRRi15IND+LNIl4gebrGtB+rRURA3W+D+nCqwJkFwZ+CVcC3H16PbKHKCmkK61T L7kilGk8b1c+ot5sgUmbNHpQUsY6nKiX5SnL9+greqTTQVq3JBQYOLA9eaTjMggbGokd oftg== MIME-Version: 1.0 X-Received: by 10.69.26.134 with SMTP id iy6mr31834927pbd.115.1410278188241; Tue, 09 Sep 2014 08:56:28 -0700 (PDT) Received: by 10.66.82.37 with HTTP; Tue, 9 Sep 2014 08:56:28 -0700 (PDT) In-Reply-To: <14246348.Ehh2xp5iLB@ralph.baldwin.cx> References: <14246348.Ehh2xp5iLB@ralph.baldwin.cx> Date: Tue, 9 Sep 2014 17:56:28 +0200 Message-ID: Subject: Re: O_XATTR support in FreeBSD? From: Simon Toedt To: John Baldwin Content-Type: text/plain; charset=UTF-8 Cc: Rick Macklem , Cedric Blancher , freebsd-hackers@freebsd.org, Richard Yao , Pedro Giffuni , Lionel Cons , Jordan Hubbard X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Sep 2014 15:56:29 -0000 On Mon, Sep 8, 2014 at 5:52 PM, John Baldwin wrote: > On Friday, September 05, 2014 04:24:02 PM Lionel Cons wrote: >> On 1 December 2013 23:05, Lionel Cons wrote: >> > On 27 November 2013 05:52, Tim Kientzle wrote: >> >> On Nov 26, 2013, at 1:51 AM, Cedric Blancher > wrote: >> >>> 5. Support for tar and pax is already there. Its described in >> >>> Solaris's fsattr man page, they use a extended header with filename >> >>> /dev/null (to prevent older tar versions from tripping over the new >> >>> headers) and then have a named attribute header which describes the >> >>> attributes names and flags. >> >> >> >> There are quite a few alternative approaches for storing >> >> extended attributes in tar and pax files. >> > >> > But this discussion is *not* about extended attributes, this >> > discussion is about Alternate Data Streams. Unfortunately the O_XATTR >> > discussion somehow started to cover the Linux "extended attribute >> > system", which is utterly useless in the intended use cases (as said, >> > no access through normal POSIX read(), write(), mmap(), no unlimited >> > size, no sparse data support (aka SEEK_HOLE, SEEK_DATA) etc etc). >> > >> > Lionel >> >> What is the status of O_XATTR (alias alternate data stream support) in >> FreeBSD? We run more and more into the trouble that these kind of >> streams are in use but cannot be accessed or processed on FreeBSD. > > I don't think anyone is even looking at it. I didn't quite follow the > earlier mail about this and some concrete examples probably would help. > Can you do 'open("/path/to/foo@forkname", O_XATTR)' to open the "forkname" > fork of the "/path/to/foo" file? No. You open a file (use O_NONBLOCK to prevent trouble with fifos) and use openat(filefd, "attrname", O_XATTR, ...) Filename "." returns you a fd to the virtual directory of the attr resources, which you can use to fchdir() or otherwise use as normal directory. mkdir(), mkfifo() and mknod() aren't allowed in such a directory. > > -- > John Baldwin > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" Simon From owner-freebsd-hackers@FreeBSD.ORG Tue Sep 9 16:01:10 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6C3A17E2 for ; Tue, 9 Sep 2014 16:01:10 +0000 (UTC) Received: from mail-pd0-x22e.google.com (mail-pd0-x22e.google.com [IPv6:2607:f8b0:400e:c02::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 41568F94 for ; Tue, 9 Sep 2014 16:01:10 +0000 (UTC) Received: by mail-pd0-f174.google.com with SMTP id v10so9293187pde.19 for ; Tue, 09 Sep 2014 09:01:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=HIA87nbMENTFV5SU0qQlIDcjJ+58I8QeRmMd1X/57Ak=; b=LnygK7S3Fg5U1e3Sxs6sUM7VyWBonuqAb/qW+UIZOS2oR5DgV6VOWMzGIjOF0O79OA 74ovJPRurp00d0rpxVmw2FT7rlxWWwzcevfq6jZYytlJ1EUi616I29EFIeYWHff8hQn2 PDn42SXDdIVi8+wWOmkkJasID89yVTNc4RMPuQOWIR/TdheIF6haYitiHP6JC7+00Ma4 ufPGuWT801pwUEfmv4oQACO7xMuIkcSTeNxTy8zmpClVlm1tBW3BWXvEQeywMJxn71ts 0Xu5M0PWH7C9tH0mm5GGi5dalTHX2L/1JMnf31t7taiHdPkhyR6IGJwI4H1jn4Wwegkg ABfg== MIME-Version: 1.0 X-Received: by 10.70.130.138 with SMTP id oe10mr60336848pdb.115.1410278469098; Tue, 09 Sep 2014 09:01:09 -0700 (PDT) Received: by 10.66.82.37 with HTTP; Tue, 9 Sep 2014 09:01:09 -0700 (PDT) In-Reply-To: <1207532459.33927535.1410263267363.JavaMail.root@uoguelph.ca> References: <9F4D2C26-F077-4CA7-A532-BA4CE562C50D@ixsystems.com> <1207532459.33927535.1410263267363.JavaMail.root@uoguelph.ca> Date: Tue, 9 Sep 2014 18:01:09 +0200 Message-ID: Subject: Re: Tool to access ZFS/NFSv4 alternate data streams on FreeBSD? From: Simon Toedt To: Rick Macklem Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org, Richard Yao , Jordan Hubbard , Jan Bramkamp , Lionel Cons X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Sep 2014 16:01:10 -0000 On Tue, Sep 9, 2014 at 1:47 PM, Rick Macklem wrote: > Jordan Hubbard wrote: >> Yep. I was just describing the experience that OS X went through in >> implementing extattrs / legacy resource fork support. To recap it >> very briefly: Having NFSv4 support extattrs (or even named streams, >> if you want to go that far) is the comparatively easy part. It=E2=80=99= s >> backing them up / copying them around that gets more involved, and >> if you can=E2=80=99t back up certain attributes then you=E2=80=99re not = likely to >> get anyone to want to use them, at which point the whole =E2=80=9Csharin= g=E2=80=9D >> aspect kind of takes a back seat. >> > Yep. I strongly suspect you are correct. > > The question then becomes: > - Do we wait and see if someone chooses to get around to doing all > the hard userland work. Solaris tools already have support for this. Also AT&T AST from David Korn have support for O_XATTR, too. > or > - Do the easy part in the kernel and then hope someone does the > hard userland work because they need it. > or > - Just decide that the Linux style extended attributes are adequate > and not do resource forks at all? -1 for adopting the Linux junk. Basically, as they evolve, they go and evolve into Solaris's O_XATTR in the next ten years, with the pain of constant API changes on the way. Each month someone cries about size limit in the Linux style extended attributes or that listing or tar support doesn't work etc Solaris O_XATTR support has the beauty that it works now and is even supported on NFSv4 and is compatible to both Windows and MacOS resource forks. Simon From owner-freebsd-hackers@FreeBSD.ORG Tue Sep 9 21:29:42 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8298C308 for ; Tue, 9 Sep 2014 21:29:42 +0000 (UTC) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 46C6BB1A for ; Tue, 9 Sep 2014 21:29:41 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AsAEAOtwD1SDaFve/2dsb2JhbABZhDcEgnjOegGBI3iEAwEBBAEjVhsYAgINGQJLAQ0GE4g6CKdWlWYBF4EsjUojATMHgnmBUwWpFokLg30hL4EHQYEHAQEB X-IronPort-AV: E=Sophos;i="5.04,493,1406606400"; d="scan'208";a="153930307" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-annu.net.uoguelph.ca with ESMTP; 09 Sep 2014 17:29:40 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 5B32EB3EE4; Tue, 9 Sep 2014 17:29:40 -0400 (EDT) Date: Tue, 9 Sep 2014 17:29:40 -0400 (EDT) From: Rick Macklem To: Simon Toedt Message-ID: <220565922.34288992.1410298180362.JavaMail.root@uoguelph.ca> In-Reply-To: Subject: Re: Tool to access ZFS/NFSv4 alternate data streams on FreeBSD? MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 7.2.6_GA_2926 (ZimbraWebClient - FF3.0 (Win)/7.2.6_GA_2926) Cc: freebsd-hackers@freebsd.org, Richard Yao , Jordan Hubbard , Jan Bramkamp , Lionel Cons X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Sep 2014 21:29:42 -0000 Simon Toedt wrote: > On Tue, Sep 9, 2014 at 1:47 PM, Rick Macklem > wrote: > > Jordan Hubbard wrote: > >> Yep. I was just describing the experience that OS X went through > >> in > >> implementing extattrs / legacy resource fork support. To recap it > >> very briefly: Having NFSv4 support extattrs (or even named > >> streams, > >> if you want to go that far) is the comparatively easy part. It=E2=80= =99s > >> backing them up / copying them around that gets more involved, and > >> if you can=E2=80=99t back up certain attributes then you=E2=80=99re no= t likely to > >> get anyone to want to use them, at which point the whole =E2=80=9Cshar= ing=E2=80=9D > >> aspect kind of takes a back seat. > >> > > Yep. I strongly suspect you are correct. > > > > The question then becomes: > > - Do we wait and see if someone chooses to get around to doing all > > the hard userland work. >=20 > Solaris tools already have support for this. Also AT&T AST from David > Korn have support for O_XATTR, too. >=20 Hopefully others will correct me if I have this incorrect, but I thought CDDL code could only be used for optional components of FreeBSD? I suspect tar and friends are considered core components and that code for this would have to be written by someone (ie. couldn't use CDDL code?). (I'm assuming that these tools are in OpenSolaris.) Be aware that most of FreeBSD's development is done by volunteers in their spare time, so I have no idea if someone is interested in doing this. rick > > or > > - Do the easy part in the kernel and then hope someone does the > > hard userland work because they need it. > > or > > - Just decide that the Linux style extended attributes are adequate > > and not do resource forks at all? >=20 > -1 for adopting the Linux junk. Basically, as they evolve, they go > and > evolve into Solaris's O_XATTR in the next ten years, with the pain of > constant API changes on the way. Each month someone cries about size > limit in the Linux style extended attributes or that listing or tar > support doesn't work etc >=20 > Solaris O_XATTR support has the beauty that it works now and is even > supported on NFSv4 and is compatible to both Windows and MacOS > resource forks. >=20 > Simon >=20 From owner-freebsd-hackers@FreeBSD.ORG Tue Sep 9 22:45:57 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8A8F4C81 for ; Tue, 9 Sep 2014 22:45:57 +0000 (UTC) Received: from mail.iXsystems.com (mail.ixsystems.com [12.229.62.4]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "*.ixsystems.com", Issuer "Go Daddy Secure Certification Authority" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 669D9342 for ; Tue, 9 Sep 2014 22:45:57 +0000 (UTC) Received: from localhost (mail.ixsystems.com [10.2.55.1]) by mail.iXsystems.com (Postfix) with ESMTP id A42CD80D25; Tue, 9 Sep 2014 15:45:56 -0700 (PDT) Received: from mail.iXsystems.com ([10.2.55.1]) by localhost (mail.ixsystems.com [10.2.55.1]) (maiad, port 10024) with ESMTP id 60073-07; Tue, 9 Sep 2014 15:45:56 -0700 (PDT) Received: from [10.2.0.80] (unknown [10.2.0.80]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.iXsystems.com (Postfix) with ESMTPSA id 196E880D1C; Tue, 9 Sep 2014 15:45:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=ixsystems.com; s=newknight0; t=1410302756; bh=AzjW7Y1im/T8GxxEtaJyF7L6owAdrq8s2OGuNwY1wHs=; h=Subject:From:In-Reply-To:Date:Cc:References:To; b=W5lu+vMiIY1KBM3Wzc2TVEnOHlwS/Yq2qa47UoxH2OXl9nktIDzg54DGM70TnscdP akhUwEJJMJ0/Gw/MXcKDhYMSslmxJP6noWCibOQvF5Til1oIP+YD67+SwC0H1ZLMj8 t7f5i+nw7GBpTw+QQ2ub8UuAegbXzA/ANc0EW6SU= Mime-Version: 1.0 (Mac OS X Mail 8.0 \(1974.6\)) Subject: Re: Tool to access ZFS/NFSv4 alternate data streams on FreeBSD? From: Jordan Hubbard In-Reply-To: <220565922.34288992.1410298180362.JavaMail.root@uoguelph.ca> Date: Tue, 9 Sep 2014 15:47:18 -0700 Message-Id: References: <220565922.34288992.1410298180362.JavaMail.root@uoguelph.ca> To: Rick Macklem X-Mailer: Apple Mail (2.1974.6) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-hackers@freebsd.org, Simon Toedt , Lionel Cons , Richard Yao , Jan Bramkamp X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Sep 2014 22:45:57 -0000 > On Sep 9, 2014, at 2:29 PM, Rick Macklem wrote: >=20 > Hopefully others will correct me if I have this incorrect, but I = thought > CDDL code could only be used for optional components of FreeBSD? > I suspect tar and friends are considered core components and that code > for this would have to be written by someone (ie. couldn't use CDDL = code?). > (I'm assuming that these tools are in OpenSolaris.) Well, if someone is willing to add copyfile(3) to FreeBSD (it does all = the magic) then the OS X versions of tar/pax/cpio/=E2=80=A6 are just = forked versions of the FreeBSD tools, but with copyfile(3) support for = handling (serializing / deserializing) EAs. - Jordan From owner-freebsd-hackers@FreeBSD.ORG Wed Sep 10 02:54:15 2014 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3E626A39 for ; Wed, 10 Sep 2014 02:54:15 +0000 (UTC) Received: from kithrup.com (Kithrup.COM [64.142.31.202]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 09C7B3B3 for ; Wed, 10 Sep 2014 02:54:14 +0000 (UTC) Received: from kithrup.com (localhost [127.0.0.1]) by kithrup.com (8.14.4/8.14.4) with ESMTP id s8A2VVLA046427 for ; Tue, 9 Sep 2014 19:31:31 -0700 (PDT) (envelope-from sef@kithrup.com) Received: (from sef@localhost) by kithrup.com (8.14.4/8.14.4/Submit) id s8A2VVfr046426; Tue, 9 Sep 2014 19:31:31 -0700 (PDT) (envelope-from sef) Date: Tue, 9 Sep 2014 19:31:31 -0700 (PDT) From: Sean Eric Fagan Message-Id: <201409100231.s8A2VVfr046426@kithrup.com> To: hackers@freebsd.org Subject: Re: Tool to access ZFS/NFSv4 alternate data streams on FreeBSD? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Sep 2014 02:54:15 -0000 In article Jordan K. Hubbard writes: >Well, if someone is willing to add copyfile(3) to FreeBSD (it does all >the magic) then the OS X versions of tar/pax/cpio/… are just forked >versions of the FreeBSD tools, but with copyfile(3) support for handling >(serializing / deserializing) EAs. copyfile(3)'s serialization sucks -- it's not streamable, and it's limited in terms of size. copyfile(3) as an API also has some serious limitations -- callbacks were added at exactly the wrong time (just before blocks were announced). On the other hand, it does work. From owner-freebsd-hackers@FreeBSD.ORG Wed Sep 10 06:10:28 2014 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 35880E2E; Wed, 10 Sep 2014 06:10:28 +0000 (UTC) Received: from pp1.rice.edu (proofpoint1.mail.rice.edu [128.42.201.100]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id ECCE7C0E; Wed, 10 Sep 2014 06:10:27 +0000 (UTC) Received: from pps.filterd (pp1.rice.edu [127.0.0.1]) by pp1.rice.edu (8.14.5/8.14.5) with SMTP id s8A688mn022353; Wed, 10 Sep 2014 01:10:12 -0500 Received: from mh3.mail.rice.edu (mh3.mail.rice.edu [128.42.199.10]) by pp1.rice.edu with ESMTP id 1p8w07hm32-1; Wed, 10 Sep 2014 01:10:12 -0500 X-Virus-Scanned: by amavis-2.7.0 at mh3.mail.rice.edu, auth channel Received: from 108-254-203-201.lightspeed.hstntx.sbcglobal.net (108-254-203-201.lightspeed.hstntx.sbcglobal.net [108.254.203.201]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) (Authenticated sender: alc) by mh3.mail.rice.edu (Postfix) with ESMTPSA id A5BDF403E5; Wed, 10 Sep 2014 01:10:11 -0500 (CDT) Message-ID: <540FEB43.5030704@rice.edu> Date: Wed, 10 Sep 2014 01:10:11 -0500 From: Alan Cox User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Konstantin Belousov Subject: Re: mmap MAP_NOSYNC regression in 10.x References: <540903FF.6010602@degoeje.nl> <20140905080633.GM2737@kib.kiev.ua> <5409A4F8.6020204@degoeje.nl> <20140905123826.GP2737@kib.kiev.ua> <540BA416.7010106@rice.edu> <20140908084126.GX2737@kib.kiev.ua> <540DEE8F.5080005@rice.edu> <20140909083847.GB2737@kib.kiev.ua> In-Reply-To: <20140909083847.GB2737@kib.kiev.ua> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 kscore.is_bulkscore=0 kscore.compositescore=0 circleOfTrustscore=0 compositescore=0.248919945447816 urlsuspect_oldscore=0.0298999927260837 suspectscore=3 recipient_domain_to_sender_totalscore=0 phishscore=0 bulkscore=0 kscore.is_spamscore=0 recipient_to_sender_totalscore=0 recipient_domain_to_sender_domain_totalscore=498 rbsscore=0.248919945447816 spamscore=0 recipient_to_sender_domain_totalscore=0 urlsuspectscore=0.9 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=7.0.1-1402240000 definitions=main-1409100071 Cc: alc@freebsd.org, Pieter de Goeje , hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Sep 2014 06:10:28 -0000 On 09/09/2014 03:38, Konstantin Belousov wrote: > On Mon, Sep 08, 2014 at 12:59:43PM -0500, Alan Cox wrote: >> On 09/08/2014 03:41, Konstantin Belousov wrote: >>> On Sat, Sep 06, 2014 at 07:17:26PM -0500, Alan Cox wrote: >>>> On 09/05/2014 07:38, Konstantin Belousov wrote: >>>>> On Fri, Sep 05, 2014 at 01:56:40PM +0200, Pieter de Goeje wrote: >>>>>> Thanks, works for me! >>>>> I realized that the patch contains yet another bug. The oflags page >>>>> flags update is protected by the exclusive vm object lock, which is only >>>>> held in shared mode on the fast path. Below is the fixed patch, where >>>>> I take the page lock around setting VPO_NOSYNC (exclusive lock owners >>>>> cannot race with fast path since we own shared lock, and two parallel >>>>> fast path execution would be handled by the page lock). >>>> Suppose that the page is clean and two threads are executing this code >>>> concurrently. One's map entry has MAP_NOSYNC set, and the other's >>>> doesn't. Let's call these threads NOSYNC and SYNC, respectively. >>>> >>>> Suppose that the thread SYNC is slightly ahead. It has already >>>> performed "m->oflags &= ~VPO_NOSYNC;" and now it's about to perform >>>> "vm_page_dirty(fs.m);". However, just before the thread SYNC calls >>>> vm_page_dirty(), the thread NOSYNC evaluates "m->dirty == 0", which is >>>> still true, and it performs "m->oflags |= VPO_NOSYNC; " >>>> >>>> This can't happen on the slow path. That is, a fault by a thread >>>> without MAP_NOSYNC set on its map entry will reliably clear VPO_NOSYNC. >>> As I understand things, it is indeed not possible on the slow path, due >>> to PG_RW only set from pmap_enter(), am I right ? I.e. this is another >>> place where the rule 'no PG_RW without PG_M' is important. >> >> Yes, it's not possible, but I'm a little confused by the rest of your >> question, specifically, the statement "no PG_RW without PG_M". Did you >> actually mean "no PG_M without PG_RW"? > No, I mean what I did wrote, and I was wrong. > >> >>> Let me formulate my question another way: what are the guarantees we >>> provide to the applications when the same page is mapped with and >>> without MAP_NOSYNC simultaneously ? Is it contractually guaranteed that >>> any write from !MAP_NOSYNC entry triggers write in the syncer activity >>> period ? >> >> Yes, that is the intent. However, I can think of at least one case >> where the existing code doesn't work as intended. Suppose that the >> first fault on a !MAP_NOSYNC entry is triggered by a read access. Then, >> vm_fault() won't call vm_page_dirty(), but it will nonetheless install a >> mapping in the pmap that allows write access. Now, suppose this same >> process writes to the page. Finally, suppose that the second fault >> happens on a MAP_NOSYNC entry. That fault will see a clean page, i.e., >> m->dirty == 0, and set VPO_NOSYNC on the page, even though the first >> faulting process that wants the page sync'ed has dirtied the page. > Yes, you are right. I convinced myself that this cannot happen, due to > the false assumption above, and the fact that page flushing removes > write bit from pte. There is another case that I've wondered about and not had the time to check out. What happens when a page is mapped by an entry with MAP_NOSYNC and a write(2) is performed? Is the page being flushed or not? >> >> >>>> The best course of action may be to fall back to the slow path if you >>>> actually need to change VPO_NOSYNC's state. Usually, you won't need to. >>>> >>> Let me first try to improve the original patch to handle >>> MAP_ENTRY_NOSYNC on fast path as well. It seems to be one of the cases >>> when the parallel faults are actually useful. >> >> I think it may be time to take a step back, decide what semantics we >> really want, and see if there is a better way of implementing those >> semantics. The current approach based on toggling VPO_NOSYNC only >> really works for the simplest cases. > Still, I believe that the current form of the patch completely repeats > the existing semantic of the slow path in the fast path. > > I may propose to avoid putting vm_page_dirty() in the scope of page > lock for vm_fault_dirty(), mostly to simplify the logic, not to > provide any useful optimization. I don't think that you can do that. It leads to the race that I described in my first email. >> >>> One more note: the previous patch handled m->oflags inconsistency for >>> setting VPO_NOSYNC operation, but missed the clear one line later. >>> I think that increasing the page lock to cover also the vm_page_dirty() >>> would fix the race you described, and the second manipulation with >>> oflags. >>> >>> diff --git a/sys/vm/vm_fault.c b/sys/vm/vm_fault.c >>> index 30b0456..944b479 100644 >>> --- a/sys/vm/vm_fault.c >>> +++ b/sys/vm/vm_fault.c >>> @@ -174,6 +174,70 @@ unlock_and_deallocate(struct faultstate *fs) >>> } >>> } >>> >>> +static void >>> +vm_fault_dirty(vm_map_entry_t entry, vm_page_t m, vm_prot_t prot, >>> + vm_prot_t fault_type, int fault_flags, boolean_t set_wd) >>> +{ >>> + boolean_t need_dirty; >>> + >>> + if (((prot & VM_PROT_WRITE) == 0 && >>> + (fault_flags & VM_FAULT_DIRTY) == 0) || >>> + (m->oflags & VPO_UNMANAGED) != 0) >>> + return; >>> + >>> + VM_OBJECT_ASSERT_LOCKED(m->object); >>> + >>> + need_dirty = ((fault_type & VM_PROT_WRITE) != 0 && >>> + (fault_flags & VM_FAULT_CHANGE_WIRING) == 0) || >>> + (fault_flags & VM_FAULT_DIRTY) != 0; >>> + >>> + if (set_wd) >>> + vm_object_set_writeable_dirty(m->object); >>> + else >>> + /* >>> + * If two callers of vm_fault_dirty() with set_wd == >>> + * FALSE, one for the map entry with MAP_ENTRY_NOSYNC >>> + * flag set, other with flag clear, race, it is >>> + * possible for the no-NOSYNC thread to see m->dirty >>> + * != 0 and not clear VPO_NOSYNC. Take vm_page lock >>> + * around manipulation of VPO_NOSYNC and >>> + * vm_page_dirty() call, to avoid the race and keep >>> + * m->oflags consistent. >>> + */ >>> + vm_page_lock(m); >>> + >>> + /* >>> + * If this is a NOSYNC mmap we do not want to set VPO_NOSYNC >>> + * if the page is already dirty to prevent data written with >>> + * the expectation of being synced from not being synced. >>> + * Likewise if this entry does not request NOSYNC then make >>> + * sure the page isn't marked NOSYNC. Applications sharing >>> + * data should use the same flags to avoid ping ponging. >>> + */ >>> + if ((entry->eflags & MAP_ENTRY_NOSYNC) != 0) { >>> + if (m->dirty == 0) { >>> + m->oflags |= VPO_NOSYNC; >>> + } >>> + } else { >>> + m->oflags &= ~VPO_NOSYNC; >>> + } >>> + >>> + /* >>> + * If the fault is a write, we know that this page is being >>> + * written NOW so dirty it explicitly to save on >>> + * pmap_is_modified() calls later. >>> + * >>> + * Also tell the backing pager, if any, that it should remove >>> + * any swap backing since the page is now dirty. >>> + */ >>> + if (need_dirty) >>> + vm_page_dirty(m); >>> + if (!set_wd) >>> + vm_page_unlock(m); >>> + if (need_dirty) >>> + vm_pager_page_unswapped(m); >>> +} >>> + >>> /* >>> * TRYPAGER - used by vm_fault to calculate whether the pager for the >>> * current object *might* contain the page. >>> @@ -321,11 +385,8 @@ RetryFault:; >>> vm_page_hold(m); >>> vm_page_unlock(m); >>> } >>> - if ((fault_type & VM_PROT_WRITE) != 0 && >>> - (m->oflags & VPO_UNMANAGED) == 0) { >>> - vm_page_dirty(m); >>> - vm_pager_page_unswapped(m); >>> - } >>> + vm_fault_dirty(fs.entry, m, prot, fault_type, fault_flags, >>> + FALSE); >>> VM_OBJECT_RUNLOCK(fs.first_object); >>> if (!wired) >>> vm_fault_prefault(&fs, vaddr, 0, 0); >>> @@ -898,42 +959,7 @@ vnode_locked: >>> if (hardfault) >>> fs.entry->next_read = fs.pindex + faultcount - reqpage; >>> >>> - if (((prot & VM_PROT_WRITE) != 0 || >>> - (fault_flags & VM_FAULT_DIRTY) != 0) && >>> - (fs.m->oflags & VPO_UNMANAGED) == 0) { >>> - vm_object_set_writeable_dirty(fs.object); >>> - >>> - /* >>> - * If this is a NOSYNC mmap we do not want to set VPO_NOSYNC >>> - * if the page is already dirty to prevent data written with >>> - * the expectation of being synced from not being synced. >>> - * Likewise if this entry does not request NOSYNC then make >>> - * sure the page isn't marked NOSYNC. Applications sharing >>> - * data should use the same flags to avoid ping ponging. >>> - */ >>> - if (fs.entry->eflags & MAP_ENTRY_NOSYNC) { >>> - if (fs.m->dirty == 0) >>> - fs.m->oflags |= VPO_NOSYNC; >>> - } else { >>> - fs.m->oflags &= ~VPO_NOSYNC; >>> - } >>> - >>> - /* >>> - * If the fault is a write, we know that this page is being >>> - * written NOW so dirty it explicitly to save on >>> - * pmap_is_modified() calls later. >>> - * >>> - * Also tell the backing pager, if any, that it should remove >>> - * any swap backing since the page is now dirty. >>> - */ >>> - if (((fault_type & VM_PROT_WRITE) != 0 && >>> - (fault_flags & VM_FAULT_CHANGE_WIRING) == 0) || >>> - (fault_flags & VM_FAULT_DIRTY) != 0) { >>> - vm_page_dirty(fs.m); >>> - vm_pager_page_unswapped(fs.m); >>> - } >>> - } >>> - >>> + vm_fault_dirty(fs.entry, fs.m, prot, fault_type, fault_flags, TRUE); >>> vm_page_assert_xbusied(fs.m); >>> >>> /* >>> diff --git a/sys/vm/vm_page.h b/sys/vm/vm_page.h >>> index f12b76c..a45648d 100644 >>> --- a/sys/vm/vm_page.h >>> +++ b/sys/vm/vm_page.h >>> @@ -147,7 +147,7 @@ struct vm_page { >>> uint16_t hold_count; /* page hold count (P) */ >>> uint16_t flags; /* page PG_* flags (P) */ >>> uint8_t aflags; /* access is atomic */ >>> - uint8_t oflags; /* page VPO_* flags (O) */ >>> + uint8_t oflags; /* page VPO_* flags (OM) */ >>> uint8_t queue; /* page queue index (P,Q) */ >>> int8_t psind; /* pagesizes[] index (O) */ >>> int8_t segind; >>> @@ -163,8 +163,9 @@ struct vm_page { >>> /* >>> * Page flags stored in oflags: >>> * >>> - * Access to these page flags is synchronized by the lock on the object >>> - * containing the page (O). >>> + * Access to these page flags is synchronized by the exclusive lock on >>> + * the object containing the page, or combination of shared object >>> + * lock and the page lock (OM). >>> * >>> * Note: VPO_UNMANAGED (used by OBJT_DEVICE, OBJT_PHYS and OBJT_SG) >>> * indicates that the page is not under PV management but From owner-freebsd-hackers@FreeBSD.ORG Wed Sep 10 07:46:23 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 35441519 for ; Wed, 10 Sep 2014 07:46:23 +0000 (UTC) Received: from mail-qc0-x236.google.com (mail-qc0-x236.google.com [IPv6:2607:f8b0:400d:c01::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id ECF8691B for ; Wed, 10 Sep 2014 07:46:22 +0000 (UTC) Received: by mail-qc0-f182.google.com with SMTP id x13so5682441qcv.13 for ; Wed, 10 Sep 2014 00:46:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=DYn9ZIbBvOzdcMcbMNzOtgspGCAXE29jnfN0d39nzzA=; b=Cg3ock4nwmP8mYWtUMDyXJmc7+H73blUsQaFJbHtiIXtKsPBrl+hBMFUnrtQwsplOG 8gqMsu6TfcCQHZMZqA8Dbe/1Y6StcCaGDBL/Jc/fQ6H9IU6d6x/sq5ZsPzWJCWacUw1m qJkraz5dUYuBW7D82IPIcvBdnnPj8WrJwKjx3BodjCwIC3mObL8hZ1ZqI/kouxKX1wK9 mWrnXLr1LlDvG5WODs9BoplB9uVjuSlqSD6krYZ37aSZCd9QkDVYlEZuTDo7E358SkTd Kf/zYxKBOsurBgupvF5XjUZv3ReE9Jgqnsi7KXlOIifNT1lDidDDE8Qb2OdWoXopD2Ue nmjQ== MIME-Version: 1.0 X-Received: by 10.140.100.237 with SMTP id s100mr34694155qge.92.1410335182002; Wed, 10 Sep 2014 00:46:22 -0700 (PDT) Received: by 10.140.105.245 with HTTP; Wed, 10 Sep 2014 00:46:21 -0700 (PDT) Date: Wed, 10 Sep 2014 09:46:21 +0200 Message-ID: Subject: subr_witness.c "rm_spinlock" and "smp rendezvous" wrong declaration order From: Philippe Jalaber To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Sep 2014 07:46:23 -0000 Hello there, I have been playing with SMP and witness and looking at subr_witness.c. I suspect there is a bug in the declaration order of locks "rm_spinlock" and "smp rendezvous". It leads witness to fire a wrong LOR in SMP. Details: _rm_wlock in kern/kern_rmlock.c calls smp_rendezvous_cpus who locks smp_ipi_mtx then calls smp_rendezvous_action who calls rm_cleanIPI who locks rm_spinlock. So "smp_rendezvous" should be declared before "rm_spinlock" in subr_witness.c Thanks for reading me, Philippe From owner-freebsd-hackers@FreeBSD.ORG Wed Sep 10 09:47:16 2014 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5349DABE; Wed, 10 Sep 2014 09:47:16 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A25591634; Wed, 10 Sep 2014 09:47:15 +0000 (UTC) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id s8A9l5B8029296 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 10 Sep 2014 12:47:05 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua s8A9l5B8029296 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id s8A9l5SH029295; Wed, 10 Sep 2014 12:47:05 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 10 Sep 2014 12:47:05 +0300 From: Konstantin Belousov To: Alan Cox Subject: Re: mmap MAP_NOSYNC regression in 10.x Message-ID: <20140910094705.GG2737@kib.kiev.ua> References: <540903FF.6010602@degoeje.nl> <20140905080633.GM2737@kib.kiev.ua> <5409A4F8.6020204@degoeje.nl> <20140905123826.GP2737@kib.kiev.ua> <540BA416.7010106@rice.edu> <20140908084126.GX2737@kib.kiev.ua> <540DEE8F.5080005@rice.edu> <20140909083847.GB2737@kib.kiev.ua> <540FEB43.5030704@rice.edu> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="q21uyrmm0NnpxAH4" Content-Disposition: inline In-Reply-To: <540FEB43.5030704@rice.edu> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: alc@freebsd.org, Pieter de Goeje , hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Sep 2014 09:47:16 -0000 --q21uyrmm0NnpxAH4 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Sep 10, 2014 at 01:10:11AM -0500, Alan Cox wrote: > On 09/09/2014 03:38, Konstantin Belousov wrote: > > On Mon, Sep 08, 2014 at 12:59:43PM -0500, Alan Cox wrote: > >> On 09/08/2014 03:41, Konstantin Belousov wrote: > >>> On Sat, Sep 06, 2014 at 07:17:26PM -0500, Alan Cox wrote: > >>>> On 09/05/2014 07:38, Konstantin Belousov wrote: > >>>>> On Fri, Sep 05, 2014 at 01:56:40PM +0200, Pieter de Goeje wrote: > >>>>>> Thanks, works for me! > >>>>> I realized that the patch contains yet another bug. The oflags page > >>>>> flags update is protected by the exclusive vm object lock, which is= only > >>>>> held in shared mode on the fast path. Below is the fixed patch, whe= re > >>>>> I take the page lock around setting VPO_NOSYNC (exclusive lock owne= rs > >>>>> cannot race with fast path since we own shared lock, and two parall= el > >>>>> fast path execution would be handled by the page lock). > >>>> Suppose that the page is clean and two threads are executing this co= de > >>>> concurrently. One's map entry has MAP_NOSYNC set, and the other's > >>>> doesn't. Let's call these threads NOSYNC and SYNC, respectively. > >>>> > >>>> Suppose that the thread SYNC is slightly ahead. It has already > >>>> performed "m->oflags &=3D ~VPO_NOSYNC;" and now it's about to perform > >>>> "vm_page_dirty(fs.m);". However, just before the thread SYNC calls > >>>> vm_page_dirty(), the thread NOSYNC evaluates "m->dirty =3D=3D 0", wh= ich is > >>>> still true, and it performs "m->oflags |=3D VPO_NOSYNC; " > >>>> > >>>> This can't happen on the slow path. That is, a fault by a thread > >>>> without MAP_NOSYNC set on its map entry will reliably clear VPO_NOSY= NC. > >>> As I understand things, it is indeed not possible on the slow path, d= ue > >>> to PG_RW only set from pmap_enter(), am I right ? I.e. this is anoth= er > >>> place where the rule 'no PG_RW without PG_M' is important. > >> > >> Yes, it's not possible, but I'm a little confused by the rest of your > >> question, specifically, the statement "no PG_RW without PG_M". Did you > >> actually mean "no PG_M without PG_RW"? > > No, I mean what I did wrote, and I was wrong. > > > >> > >>> Let me formulate my question another way: what are the guarantees we > >>> provide to the applications when the same page is mapped with and > >>> without MAP_NOSYNC simultaneously ? Is it contractually guaranteed t= hat > >>> any write from !MAP_NOSYNC entry triggers write in the syncer activity > >>> period ? > >> > >> Yes, that is the intent. However, I can think of at least one case > >> where the existing code doesn't work as intended. Suppose that the > >> first fault on a !MAP_NOSYNC entry is triggered by a read access. The= n, > >> vm_fault() won't call vm_page_dirty(), but it will nonetheless install= a > >> mapping in the pmap that allows write access. Now, suppose this same > >> process writes to the page. Finally, suppose that the second fault > >> happens on a MAP_NOSYNC entry. That fault will see a clean page, i.e., > >> m->dirty =3D=3D 0, and set VPO_NOSYNC on the page, even though the fir= st > >> faulting process that wants the page sync'ed has dirtied the page. > > Yes, you are right. I convinced myself that this cannot happen, due to > > the false assumption above, and the fact that page flushing removes > > write bit from pte. >=20 >=20 > There is another case that I've wondered about and not had the time to > check out. What happens when a page is mapped by an entry with > MAP_NOSYNC and a write(2) is performed? Is the page being flushed or not? Do you mean write(2) to the file range represented by the page ? If yes, the page mapping mode (or mapping existence at all) is irrelevant there. (Delayed) write is recorded at the buffer layer as B_DELWRI buffer, for filesystems which use buffer cache. Typically, the page flush operation also converts dirty pages into the same dirty buffers. It is syncer or buffer daemon which write out the buffers. >=20 >=20 > >> > >> > >>>> The best course of action may be to fall back to the slow path if you > >>>> actually need to change VPO_NOSYNC's state. Usually, you won't need= to. > >>>> > >>> Let me first try to improve the original patch to handle > >>> MAP_ENTRY_NOSYNC on fast path as well. It seems to be one of the cases > >>> when the parallel faults are actually useful. > >> > >> I think it may be time to take a step back, decide what semantics we > >> really want, and see if there is a better way of implementing those > >> semantics. The current approach based on toggling VPO_NOSYNC only > >> really works for the simplest cases. > > Still, I believe that the current form of the patch completely repeats > > the existing semantic of the slow path in the fast path. > > > > I may propose to avoid putting vm_page_dirty() in the scope of page > > lock for vm_fault_dirty(), mostly to simplify the logic, not to > > provide any useful optimization. >=20 >=20 > I don't think that you can do that. It leads to the race that I > described in my first email. Yes, but I mean, since there are other ways to get dirtyness from !MAP_NOSYNC write accesses to be ignored, there is no new harm from the supposedly rare race. Anyway, my proposal to move the vm_page_dirty() from under the page lock was to make the vm_fault_dirty() code less convoluted, not to micro-optimize things. >=20 >=20 > >> > >>> One more note: the previous patch handled m->oflags inconsistency for > >>> setting VPO_NOSYNC operation, but missed the clear one line later. > >>> I think that increasing the page lock to cover also the vm_page_dirty= () > >>> would fix the race you described, and the second manipulation with > >>> oflags. > >>> > >>> diff --git a/sys/vm/vm_fault.c b/sys/vm/vm_fault.c > >>> index 30b0456..944b479 100644 > >>> --- a/sys/vm/vm_fault.c > >>> +++ b/sys/vm/vm_fault.c > >>> @@ -174,6 +174,70 @@ unlock_and_deallocate(struct faultstate *fs) > >>> } > >>> } > >>> =20 > >>> +static void > >>> +vm_fault_dirty(vm_map_entry_t entry, vm_page_t m, vm_prot_t prot, > >>> + vm_prot_t fault_type, int fault_flags, boolean_t set_wd) > >>> +{ > >>> + boolean_t need_dirty; > >>> + > >>> + if (((prot & VM_PROT_WRITE) =3D=3D 0 && > >>> + (fault_flags & VM_FAULT_DIRTY) =3D=3D 0) || > >>> + (m->oflags & VPO_UNMANAGED) !=3D 0) > >>> + return; > >>> + > >>> + VM_OBJECT_ASSERT_LOCKED(m->object); > >>> + > >>> + need_dirty =3D ((fault_type & VM_PROT_WRITE) !=3D 0 && > >>> + (fault_flags & VM_FAULT_CHANGE_WIRING) =3D=3D 0) || > >>> + (fault_flags & VM_FAULT_DIRTY) !=3D 0; > >>> + > >>> + if (set_wd) > >>> + vm_object_set_writeable_dirty(m->object); > >>> + else > >>> + /* > >>> + * If two callers of vm_fault_dirty() with set_wd =3D=3D > >>> + * FALSE, one for the map entry with MAP_ENTRY_NOSYNC > >>> + * flag set, other with flag clear, race, it is > >>> + * possible for the no-NOSYNC thread to see m->dirty > >>> + * !=3D 0 and not clear VPO_NOSYNC. Take vm_page lock > >>> + * around manipulation of VPO_NOSYNC and > >>> + * vm_page_dirty() call, to avoid the race and keep > >>> + * m->oflags consistent. > >>> + */ > >>> + vm_page_lock(m); > >>> + > >>> + /* > >>> + * If this is a NOSYNC mmap we do not want to set VPO_NOSYNC > >>> + * if the page is already dirty to prevent data written with > >>> + * the expectation of being synced from not being synced. > >>> + * Likewise if this entry does not request NOSYNC then make > >>> + * sure the page isn't marked NOSYNC. Applications sharing > >>> + * data should use the same flags to avoid ping ponging. > >>> + */ > >>> + if ((entry->eflags & MAP_ENTRY_NOSYNC) !=3D 0) { > >>> + if (m->dirty =3D=3D 0) { > >>> + m->oflags |=3D VPO_NOSYNC; > >>> + } > >>> + } else { > >>> + m->oflags &=3D ~VPO_NOSYNC; > >>> + } > >>> + > >>> + /* > >>> + * If the fault is a write, we know that this page is being > >>> + * written NOW so dirty it explicitly to save on > >>> + * pmap_is_modified() calls later. > >>> + * > >>> + * Also tell the backing pager, if any, that it should remove > >>> + * any swap backing since the page is now dirty. > >>> + */ > >>> + if (need_dirty) > >>> + vm_page_dirty(m); > >>> + if (!set_wd) > >>> + vm_page_unlock(m); > >>> + if (need_dirty) > >>> + vm_pager_page_unswapped(m); > >>> +} > >>> + > >>> /* > >>> * TRYPAGER - used by vm_fault to calculate whether the pager for the > >>> * current object *might* contain the page. > >>> @@ -321,11 +385,8 @@ RetryFault:; > >>> vm_page_hold(m); > >>> vm_page_unlock(m); > >>> } > >>> - if ((fault_type & VM_PROT_WRITE) !=3D 0 && > >>> - (m->oflags & VPO_UNMANAGED) =3D=3D 0) { > >>> - vm_page_dirty(m); > >>> - vm_pager_page_unswapped(m); > >>> - } > >>> + vm_fault_dirty(fs.entry, m, prot, fault_type, fault_flags, > >>> + FALSE); > >>> VM_OBJECT_RUNLOCK(fs.first_object); > >>> if (!wired) > >>> vm_fault_prefault(&fs, vaddr, 0, 0); > >>> @@ -898,42 +959,7 @@ vnode_locked: > >>> if (hardfault) > >>> fs.entry->next_read =3D fs.pindex + faultcount - reqpage; > >>> =20 > >>> - if (((prot & VM_PROT_WRITE) !=3D 0 || > >>> - (fault_flags & VM_FAULT_DIRTY) !=3D 0) && > >>> - (fs.m->oflags & VPO_UNMANAGED) =3D=3D 0) { > >>> - vm_object_set_writeable_dirty(fs.object); > >>> - > >>> - /* > >>> - * If this is a NOSYNC mmap we do not want to set VPO_NOSYNC > >>> - * if the page is already dirty to prevent data written with > >>> - * the expectation of being synced from not being synced. > >>> - * Likewise if this entry does not request NOSYNC then make > >>> - * sure the page isn't marked NOSYNC. Applications sharing > >>> - * data should use the same flags to avoid ping ponging. > >>> - */ > >>> - if (fs.entry->eflags & MAP_ENTRY_NOSYNC) { > >>> - if (fs.m->dirty =3D=3D 0) > >>> - fs.m->oflags |=3D VPO_NOSYNC; > >>> - } else { > >>> - fs.m->oflags &=3D ~VPO_NOSYNC; > >>> - } > >>> - > >>> - /* > >>> - * If the fault is a write, we know that this page is being > >>> - * written NOW so dirty it explicitly to save on=20 > >>> - * pmap_is_modified() calls later. > >>> - * > >>> - * Also tell the backing pager, if any, that it should remove > >>> - * any swap backing since the page is now dirty. > >>> - */ > >>> - if (((fault_type & VM_PROT_WRITE) !=3D 0 && > >>> - (fault_flags & VM_FAULT_CHANGE_WIRING) =3D=3D 0) || > >>> - (fault_flags & VM_FAULT_DIRTY) !=3D 0) { > >>> - vm_page_dirty(fs.m); > >>> - vm_pager_page_unswapped(fs.m); > >>> - } > >>> - } > >>> - > >>> + vm_fault_dirty(fs.entry, fs.m, prot, fault_type, fault_flags, TRUE); > >>> vm_page_assert_xbusied(fs.m); > >>> =20 > >>> /* > >>> diff --git a/sys/vm/vm_page.h b/sys/vm/vm_page.h > >>> index f12b76c..a45648d 100644 > >>> --- a/sys/vm/vm_page.h > >>> +++ b/sys/vm/vm_page.h > >>> @@ -147,7 +147,7 @@ struct vm_page { > >>> uint16_t hold_count; /* page hold count (P) */ > >>> uint16_t flags; /* page PG_* flags (P) */ > >>> uint8_t aflags; /* access is atomic */ > >>> - uint8_t oflags; /* page VPO_* flags (O) */ > >>> + uint8_t oflags; /* page VPO_* flags (OM) */ > >>> uint8_t queue; /* page queue index (P,Q) */ > >>> int8_t psind; /* pagesizes[] index (O) */ > >>> int8_t segind; > >>> @@ -163,8 +163,9 @@ struct vm_page { > >>> /* > >>> * Page flags stored in oflags: > >>> * > >>> - * Access to these page flags is synchronized by the lock on the obj= ect > >>> - * containing the page (O). > >>> + * Access to these page flags is synchronized by the exclusive lock = on > >>> + * the object containing the page, or combination of shared object > >>> + * lock and the page lock (OM). > >>> * > >>> * Note: VPO_UNMANAGED (used by OBJT_DEVICE, OBJT_PHYS and OBJT_SG) > >>> * indicates that the page is not under PV management but --q21uyrmm0NnpxAH4 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJUEB4YAAoJEJDCuSvBvK1BlMAP/1fUuvUeuW5brXL/YAsOIciv px3Hi5ASXhQv7PZEYC/rVzjJu1JphiGruh9a64YjHoPPAQPRnYcKn2s9xWzq6pzf uHPRP11FRsP/68fNTSuuflS5/yTKdNcLnD6jSTLDps0Y4FNcIs3qjS2SJpgOs1f9 8stquask/t/lb6IeQ26hn3XpW1k/IH8Farm6i72k7loZBfBx6mGdv5/Di8Ujre/a TaIfNORgL18YTOh6gsMqVIWTB0txLYHVo6KXn6x+LEROkhbpx4My3xClCgreGpht KgkHr8JnfWux7iZ1+bxs4vKPe0djKRwUs9R6ogFcNHi6KJvCWFYCHfBLHK1HPina QPTB7m2NX6rMx4iikWUK0MKlfwXfGYIR64NNmH1dchjMlQL7Jt5yqdUD99Xl+Q7s XV6s6LZtbE9IElN8UuZvks/z9M+jEvR5RoSnuuHINcRMW04uv919S3+RoF5P6N+9 +UZp7+HtBaaaHKrI5BExSsVFiAXGKX0ainqQZfUOQAIrln2+hJF5aZh2Dfy9pHw1 RldOda2jbn7LSQ4e9ZOt+VGx227dA6BEQki3jzm7jNB9vMflj3XCM3J3TELCWwr9 9LBq03Z5DtMu0lgkvVzLAvgg7qN+QFujOzkpMruF62a24l1QJFc8pjb5fHH3VaHx OMzE9GFccew3sIAtQBRe =W+93 -----END PGP SIGNATURE----- --q21uyrmm0NnpxAH4-- From owner-freebsd-hackers@FreeBSD.ORG Thu Sep 11 07:23:32 2014 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7F775C68 for ; Thu, 11 Sep 2014 07:23:32 +0000 (UTC) Received: from puchar.net (puchar.net [188.252.31.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F2E74AAF for ; Thu, 11 Sep 2014 07:23:31 +0000 (UTC) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.14.9/8.14.9) with ESMTP id s8B7I5lj007629 for ; Thu, 11 Sep 2014 09:18:05 +0200 (CEST) (envelope-from wojtek@puchar.net) X-Authentication-Warning: puchar.net: Host puchar-wojtek.intra [10.0.224.6] claimed to be wojtek.dom Received: from wojtek.dom (localhost [127.0.0.1]) by wojtek.dom (8.14.9/8.14.9) with ESMTP id s8B7I5pX061082 for ; Thu, 11 Sep 2014 09:18:05 +0200 (CEST) (envelope-from wojtek@puchar.net) Received: from localhost (wojtek@localhost) by wojtek.dom (8.14.9/8.14.9/Submit) with ESMTP id s8B7I5nf061079 for ; Thu, 11 Sep 2014 09:18:05 +0200 (CEST) (envelope-from wojtek@puchar.net) X-Authentication-Warning: wojtek.dom: wojtek owned process doing -bs Date: Thu, 11 Sep 2014 09:18:05 +0200 (CEST) From: Wojciech Puchar X-X-Sender: wojtek@wojtek.dom To: hackers@freebsd.org Subject: openssl with aes-in or padlock Message-ID: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (puchar.net [10.0.1.1]); Thu, 11 Sep 2014 09:18:05 +0200 (CEST) X-Mailman-Approved-At: Thu, 11 Sep 2014 15:02:16 +0000 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Sep 2014 07:23:32 -0000 how to check if openssl is actually using these instructions? on machine with padlock: #openssl speed -evp aes-256-cbc Doing aes-256-cbc for 3s on 16 size blocks: 732600 aes-256-cbc's in 2.91s Doing aes-256-cbc for 3s on 64 size blocks: 199833 aes-256-cbc's in 2.92s Doing aes-256-cbc for 3s on 256 size blocks: 50469 aes-256-cbc's in 2.91s Doing aes-256-cbc for 3s on 1024 size blocks: 25060 aes-256-cbc's in 2.92s Doing aes-256-cbc for 3s on 8192 size blocks: 3145 aes-256-cbc's in 2.93s OpenSSL 1.0.1e-freebsd 11 Feb 2013 built on: date not available options:bn(64,32) rc4(8x,mmx) des(ptr,risc1,16,long) aes(partial) idea(int) blowfish(idx) compiler: cc The 'numbers' are in 1000s of bytes per second processed. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes aes-256-cbc 4033.24k 4377.09k 4445.61k 8782.52k 8794.06k #openssl engine (dynamic) Dynamic engine loading support in the same time dd from geli encrypted ramdisk to /dev/null is 66MB/s how to enable padlock or aes-in in openssl? From owner-freebsd-hackers@FreeBSD.ORG Thu Sep 11 16:58:58 2014 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F242C3EE for ; Thu, 11 Sep 2014 16:58:58 +0000 (UTC) Received: from puchar.net (puchar.net [188.252.31.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 79C8780C for ; Thu, 11 Sep 2014 16:58:58 +0000 (UTC) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.14.9/8.14.9) with ESMTP id s8BGwuax016109 for ; Thu, 11 Sep 2014 18:58:56 +0200 (CEST) (envelope-from wojtek@puchar.net) X-Authentication-Warning: puchar.net: Host puchar-wojtek.intra [10.0.224.6] claimed to be wojtek.dom Received: from wojtek.dom (localhost [127.0.0.1]) by wojtek.dom (8.14.9/8.14.9) with ESMTP id s8BGwu7j001220 for ; Thu, 11 Sep 2014 18:58:56 +0200 (CEST) (envelope-from wojtek@puchar.net) Received: from localhost (wojtek@localhost) by wojtek.dom (8.14.9/8.14.9/Submit) with ESMTP id s8BGwuKq001217 for ; Thu, 11 Sep 2014 18:58:56 +0200 (CEST) (envelope-from wojtek@puchar.net) X-Authentication-Warning: wojtek.dom: wojtek owned process doing -bs Date: Thu, 11 Sep 2014 18:58:56 +0200 (CEST) From: Wojciech Puchar X-X-Sender: wojtek@wojtek.dom To: hackers@freebsd.org Subject: openssl with aes-in or padlock Message-ID: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (puchar.net [10.0.1.1]); Thu, 11 Sep 2014 18:58:56 +0200 (CEST) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Sep 2014 16:58:59 -0000 how to check if openssl is actually using these instructions? on machine with padlock: #openssl speed -evp aes-256-cbc Doing aes-256-cbc for 3s on 16 size blocks: 732600 aes-256-cbc's in 2.91s Doing aes-256-cbc for 3s on 64 size blocks: 199833 aes-256-cbc's in 2.92s Doing aes-256-cbc for 3s on 256 size blocks: 50469 aes-256-cbc's in 2.91s Doing aes-256-cbc for 3s on 1024 size blocks: 25060 aes-256-cbc's in 2.92s Doing aes-256-cbc for 3s on 8192 size blocks: 3145 aes-256-cbc's in 2.93s OpenSSL 1.0.1e-freebsd 11 Feb 2013 built on: date not available options:bn(64,32) rc4(8x,mmx) des(ptr,risc1,16,long) aes(partial) idea(int) blowfish(idx) compiler: cc The 'numbers' are in 1000s of bytes per second processed. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes aes-256-cbc 4033.24k 4377.09k 4445.61k 8782.52k 8794.06k #openssl engine (dynamic) Dynamic engine loading support in the same time dd from geli encrypted ramdisk to /dev/null is 66MB/s how to enable padlock or aes-in in openssl? _______________________________________________ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" From owner-freebsd-hackers@FreeBSD.ORG Thu Sep 11 18:03:00 2014 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D979AEAA for ; Thu, 11 Sep 2014 18:03:00 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "funkthat.com", Issuer "funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 95989ED1 for ; Thu, 11 Sep 2014 18:03:00 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id s8BI2wcO057870 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 11 Sep 2014 11:02:59 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id s8BI2wml057867; Thu, 11 Sep 2014 11:02:58 -0700 (PDT) (envelope-from jmg) Date: Thu, 11 Sep 2014 11:02:58 -0700 From: John-Mark Gurney To: Wojciech Puchar Subject: Re: openssl with aes-in or padlock Message-ID: <20140911180258.GN82175@funkthat.com> Mail-Followup-To: Wojciech Puchar , hackers@freebsd.org References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Thu, 11 Sep 2014 11:02:59 -0700 (PDT) Cc: hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Sep 2014 18:03:00 -0000 Wojciech Puchar wrote this message on Thu, Sep 11, 2014 at 18:58 +0200: > how to check if openssl is actually using these instructions? > > on machine with padlock: > > #openssl speed -evp aes-256-cbc First off, you won't get much speed up w/ CBC encrypt... Try testing using aes-256-ctr instead... CBC can't process multiple blocks in parallel like CTR can... if you measure the cbc _decrypt_ speed, you should see a big improvement as CBC decrypt can be parallelized... > in the same time dd from geli encrypted ramdisk to /dev/null is 66MB/s geli uses a different framework for it's crypto processing.. for geli, make sure you have the aesni kernel module loaded before you attach to a geli disk... You should get kernel messages like the following: GEOM_ELI: Device gpt/werner.eli created. GEOM_ELI: Encryption: AES-XTS 256 GEOM_ELI: Crypto: hardware notice the Crypto: hardware line.. Also, make sure that your geli sector size is 4k instead of 512... This reduces the loop overhead, and as modern disks usually use 4k anyways, there isn't much of a lose going to the larger sector size... Also, if you want to do pure testing, you can use geom_zero, and turn off clearing w/ kern.geom.zero.clear so that gzero won't bother zeroing the buffer, then you'll be able to better measure geli's overhead/performance.. > how to enable padlock or aes-in in openssl? For OpenSSL, you need at least 1.0 for AES-NI to be detected and used... I'm not familar w/ padlock if it can be used from userland, w/o the kernel, but I don't believe it is... If you have a machine w/o AES-NI, but has padlock, you can load the padlock and cryptodev kernel modules, and then OpenSSL should detect that /dev/crypto is present and use that... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-hackers@FreeBSD.ORG Thu Sep 11 21:33:34 2014 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4BA23847 for ; Thu, 11 Sep 2014 21:33:34 +0000 (UTC) Received: from puchar.net (puchar.net [188.252.31.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C7583931 for ; Thu, 11 Sep 2014 21:33:33 +0000 (UTC) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.14.9/8.14.9) with ESMTP id s8BLXSiW022578; Thu, 11 Sep 2014 23:33:29 +0200 (CEST) (envelope-from wojtek@puchar.net) X-Authentication-Warning: puchar.net: Host puchar-wojtek.intra [10.0.224.6] claimed to be wojtek.dom Received: from wojtek.dom (localhost [127.0.0.1]) by wojtek.dom (8.14.9/8.14.9) with ESMTP id s8BLXSEG002156; Thu, 11 Sep 2014 23:33:28 +0200 (CEST) (envelope-from wojtek@puchar.net) Received: from localhost (wojtek@localhost) by wojtek.dom (8.14.9/8.14.9/Submit) with ESMTP id s8BLXRRG002153; Thu, 11 Sep 2014 23:33:28 +0200 (CEST) (envelope-from wojtek@puchar.net) X-Authentication-Warning: wojtek.dom: wojtek owned process doing -bs Date: Thu, 11 Sep 2014 23:33:27 +0200 (CEST) From: Wojciech Puchar X-X-Sender: wojtek@wojtek.dom To: John-Mark Gurney Subject: Re: openssl with aes-in or padlock In-Reply-To: <20140911180258.GN82175@funkthat.com> Message-ID: References: <20140911180258.GN82175@funkthat.com> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (puchar.net [10.0.1.1]); Thu, 11 Sep 2014 23:33:29 +0200 (CEST) Cc: hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Sep 2014 21:33:34 -0000 >> #openssl speed -evp aes-256-cbc > > First off, you won't get much speed up w/ CBC encrypt... Try testing > using aes-256-ctr instead... CBC can't process multiple blocks in > parallel like CTR can... if you measure the cbc _decrypt_ speed, you > should see a big improvement as CBC decrypt can be parallelized... > >> in the same time dd from geli encrypted ramdisk to /dev/null is 66MB/s > > geli uses a different framework for it's crypto processing.. for geli, > make sure you have the aesni kernel module loaded before you attach > to a geli disk... You should get kernel messages like the following: > GEOM_ELI: Device gpt/werner.eli created. > GEOM_ELI: Encryption: AES-XTS 256 > GEOM_ELI: Crypto: hardware yes i have this. contrary to what you say - both AES-XTC and AES-CBC gets MUCH faster with AES-NI. > notice the Crypto: hardware line.. Also, make sure that your geli > sector size is 4k instead of 512... This reduces the loop overhead, as i already said - geli works fast and make use of AES-NI or padlock openssl does not From owner-freebsd-hackers@FreeBSD.ORG Thu Sep 11 21:37:09 2014 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AB4B1A58 for ; Thu, 11 Sep 2014 21:37:09 +0000 (UTC) Received: from dmz-mailsec-scanner-3.mit.edu (dmz-mailsec-scanner-3.mit.edu [18.9.25.14]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48E7D967 for ; Thu, 11 Sep 2014 21:37:09 +0000 (UTC) X-AuditID: 1209190e-f79d46d000003643-53-541215fd6068 Received: from mailhub-auth-2.mit.edu ( [18.7.62.36]) (using TLS with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by dmz-mailsec-scanner-3.mit.edu (Symantec Messaging Gateway) with SMTP id 0A.08.13891.DF512145; Thu, 11 Sep 2014 17:37:01 -0400 (EDT) Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) by mailhub-auth-2.mit.edu (8.13.8/8.9.2) with ESMTP id s8BLb0M0002096; Thu, 11 Sep 2014 17:37:01 -0400 Received: from multics.mit.edu (system-low-sipb.mit.edu [18.187.2.37]) (authenticated bits=56) (User authenticated as kaduk@ATHENA.MIT.EDU) by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id s8BLawqK000466 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Thu, 11 Sep 2014 17:37:00 -0400 Received: (from kaduk@localhost) by multics.mit.edu (8.12.9.20060308) id s8BLavxI014351; Thu, 11 Sep 2014 17:36:57 -0400 (EDT) Date: Thu, 11 Sep 2014 17:36:57 -0400 (EDT) From: Benjamin Kaduk To: Wojciech Puchar Subject: Re: openssl with aes-in or padlock In-Reply-To: Message-ID: References: <20140911180258.GN82175@funkthat.com> User-Agent: Alpine 1.10 (GSO 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFupgleLIzCtJLcpLzFFi42IRYrdT0f0rKhRisOKIusWGBYUW97ueMFnc fH6MzYHZY8an+Swel09tZvZo+PWHPYA5issmJTUnsyy1SN8ugStjY99ttoLTTBVHlt9iamD8 z9jFyMkhIWAi8e32TFYIW0ziwr31bCC2kMBsJolZ76y7GLmA7I2MEpvatjJDOIeYJKadmMsC 4TQwSkycfQOsnUVAW2Jew1Iwm01ARWLmm41go0QE1CWunp4KZjML2Ej8en2NHcQWBqp/de4W C4jNCRR/2t0BFOfg4BVwlPi5KRdi/kRGiZmTJ4KdKiqgI7F6/xSwel4BQYmTM5+wQMzUklg+ fRvLBEbBWUhSs5CkFjAyrWKUTcmt0s1NzMwpTk3WLU5OzMtLLdI11svNLNFLTSndxAgKXk5J vh2MXw8qHWIU4GBU4uGtYBEMEWJNLCuuzD3EKMnBpCTKqy8iFCLEl5SfUpmRWJwRX1Sak1p8 iFGCg1lJhNdCCCjHm5JYWZValA+TkuZgURLn3fSDL0RIID2xJDU7NbUgtQgmK8PBoSTBqwOM UiHBotT01Iq0zJwShDQTByfIcB6g4b4gNbzFBYm5xZnpEPlTjMYcLU1ve5k41nV+62cSYsnL z0uVEueVAykVACnNKM2DmwZLQK8YxYGeE+YVB6niASYvuHmvgFYxAa06aMwPsqokESEl1cDI XfZuud58yZyHS/9M1pry6n1VlxGnfqlY3v/ZbPPOrvx5bHP2iQO/XL/W7Z8eLWZgtODpzuSw 3dw1D+Iqe0/1n623mJEWxL3bw0Zb5FN6jJ/pPFbT2XYfnkdrfzjN+VWA5eaUfRITtF0tvnTN 2+LuXtVQH76H6465/lKjqLIIP45wDTarQ/uVWIozEg21mIuKEwHXhVnBGwMAAA== Cc: John-Mark Gurney , hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Sep 2014 21:37:09 -0000 On Thu, 11 Sep 2014, Wojciech Puchar wrote: > as i already said - geli works fast and make use of AES-NI or padlock > > openssl does not The openssl from ports appears to have an option for padlock, which is disabled by default. I didn't look enough to see whether its 'ASM' option secretly meant aesni or not. -Ben From owner-freebsd-hackers@FreeBSD.ORG Thu Sep 11 21:48:05 2014 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AE09EE6F for ; Thu, 11 Sep 2014 21:48:05 +0000 (UTC) Received: from mail-pa0-f43.google.com (mail-pa0-f43.google.com [209.85.220.43]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 81BE3A49 for ; Thu, 11 Sep 2014 21:48:05 +0000 (UTC) Received: by mail-pa0-f43.google.com with SMTP id fa1so10151797pad.2 for ; Thu, 11 Sep 2014 14:48:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:references:mime-version:in-reply-to:content-type :content-transfer-encoding:message-id:cc:from:subject:date:to; bh=5Mg8pfaCYivDu5wB68yAeBqEc4bJDr5YRVXswp3+HR4=; b=Le32I5tMAR3cvqFNMYBiEVUvXQQ9iCHnaNLD+J03NFWUIlRPhIn/AZUkB3hDn7LDdL Vy7KfGLL/8ajk3/hwICyDytK0gCb4gGA4j/8h7i9Rq0LZR4ru0YSz89dBzqsm78C7GX3 hvOIWROgHDD7wwtfAszHJitQnVy4bLj1LJ9/g7zsW/UnCZZ0E121lVZIEsCvgPZd7a8l XkF6i0/7MG94mrXFxMLRtVE2Oihajct6cn9n/JbN0RrCjOvk+IhDYSzuirbGIlE+nqMe fTniFjhNU+jL551/jc7XfLET14AteY69ORMZlf6yXXP79jB+bU+JVl45lvOuw/qem+2v RUpQ== X-Gm-Message-State: ALoCoQm49ym8PKRthsJhFWWdacJBHrLoodTIP+Edi+ZXBqvFK++L7CaA+jSJPBHq8eVRfsjr6LBu X-Received: by 10.68.220.71 with SMTP id pu7mr5745360pbc.22.1410472084703; Thu, 11 Sep 2014 14:48:04 -0700 (PDT) Received: from [51.218.197.21] ([66.87.119.21]) by mx.google.com with ESMTPSA id cu3sm1954767pbb.48.2014.09.11.14.47.58 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 11 Sep 2014 14:48:03 -0700 (PDT) References: <20140911180258.GN82175@funkthat.com> Mime-Version: 1.0 (1.0) In-Reply-To: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Message-Id: <62E8AD7E-346F-4F77-9628-6D5121D7AD6D@netgate.com> X-Mailer: iPhone Mail (11D257) From: Jim Thompson Subject: Re: openssl with aes-in or padlock Date: Thu, 11 Sep 2014 14:47:54 -0700 To: Wojciech Puchar Cc: John-Mark Gurney , "hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Sep 2014 21:48:05 -0000 We just fixed IPSEC to use AES-GCM (with support for AES-NI on hardware that= supports it.) OpenSSL / OpenVPN is probably next.=20 -- Jim On Sep 11, 2014, at 14:33, Wojciech Puchar wrote: >>> #openssl speed -evp aes-256-cbc >>=20 >> First off, you won't get much speed up w/ CBC encrypt... Try testing >> using aes-256-ctr instead... CBC can't process multiple blocks in >> parallel like CTR can... if you measure the cbc _decrypt_ speed, you >> should see a big improvement as CBC decrypt can be parallelized... >>=20 >>> in the same time dd from geli encrypted ramdisk to /dev/null is 66MB/s >>=20 >> geli uses a different framework for it's crypto processing.. for geli, >> make sure you have the aesni kernel module loaded before you attach >> to a geli disk... You should get kernel messages like the following: >> GEOM_ELI: Device gpt/werner.eli created. >> GEOM_ELI: Encryption: AES-XTS 256 >> GEOM_ELI: Crypto: hardware >=20 > yes i have this. contrary to what you say - both AES-XTC and AES-CBC gets M= UCH faster with AES-NI. >=20 >> notice the Crypto: hardware line.. Also, make sure that your geli >> sector size is 4k instead of 512... This reduces the loop overhead, >=20 > as i already said - geli works fast and make use of AES-NI or padlock >=20 > openssl does not > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"= From owner-freebsd-hackers@FreeBSD.ORG Thu Sep 11 23:47:38 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B2D6AB44 for ; Thu, 11 Sep 2014 23:47:38 +0000 (UTC) Received: from nskntqsrv01p.mx.bigpond.com (nskntqsrv01p.mx.bigpond.com [61.9.168.231]) by mx1.freebsd.org (Postfix) with ESMTP id 4B8A27E1 for ; Thu, 11 Sep 2014 23:47:37 +0000 (UTC) Received: from nskntcmgw07p ([61.9.169.167]) by nskntmtas03p.mx.bigpond.com with ESMTP id <20140911231516.SGTB7575.nskntmtas03p.mx.bigpond.com@nskntcmgw07p> for ; Thu, 11 Sep 2014 23:15:16 +0000 Received: from hermes.heuristicsystems.com.au ([121.210.107.100]) by nskntcmgw07p with BigPond Outbound id pzFG1o00U29zwdD01zFGdt; Thu, 11 Sep 2014 23:15:16 +0000 X-Authority-Analysis: v=2.0 cv=JN65Qr2b c=1 sm=1 a=SEJ2iDwVkb98DYvesvueMw==:17 a=0aGaYNJWb7cA:10 a=esKmG0L1hUUA:10 a=k80-YZaA8ycA:10 a=N659UExz7-8A:10 a=GHIR_BbyAAAA:8 a=6I5d2MoRAAAA:8 a=enLiN83gAY6DEwUNq44A:9 a=pILNOxqGKmIA:10 a=SV7veod9ZcQA:10 a=SEJ2iDwVkb98DYvesvueMw==:117 Received: from [10.0.5.3] (ewsw01.hs [10.0.5.3]) (authenticated bits=0) by hermes.heuristicsystems.com.au (8.14.5/8.13.6) with ESMTP id s8BNFBhX014416 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT) for ; Fri, 12 Sep 2014 09:15:13 +1000 (EST) (envelope-from dewayne.geraghty@heuristicsystems.com.au) Message-ID: <54122CFD.3070702@heuristicsystems.com.au> Date: Fri, 12 Sep 2014 09:15:09 +1000 From: Dewayne Geraghty User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Thunderbird/31.1.0 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: Re: openssl with aes-in or padlock References: In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Sep 2014 23:47:38 -0000 On 12/09/2014 2:58 AM, Wojciech Puchar wrote: > how to check if openssl is actually using these instructions? > > on machine with padlock: > > #openssl speed -evp aes-256-cbc > Doing aes-256-cbc for 3s on 16 size blocks: 732600 aes-256-cbc's in 2.91s > Doing aes-256-cbc for 3s on 64 size blocks: 199833 aes-256-cbc's in 2.92s > Doing aes-256-cbc for 3s on 256 size blocks: 50469 aes-256-cbc's in 2.91s > Doing aes-256-cbc for 3s on 1024 size blocks: 25060 aes-256-cbc's in > 2.92s > Doing aes-256-cbc for 3s on 8192 size blocks: 3145 aes-256-cbc's in 2.93s > OpenSSL 1.0.1e-freebsd 11 Feb 2013 > built on: date not available > options:bn(64,32) rc4(8x,mmx) des(ptr,risc1,16,long) aes(partial) > idea(int) blowfish(idx) > compiler: cc > The 'numbers' are in 1000s of bytes per second processed. > type 16 bytes 64 bytes 256 bytes 1024 bytes > 8192 bytes > aes-256-cbc 4033.24k 4377.09k 4445.61k 8782.52k > 8794.06k > > > #openssl engine > (dynamic) Dynamic engine loading support > > > > in the same time dd from geli encrypted ramdisk to /dev/null is 66MB/s > > how to enable padlock or aes-in in openssl? > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to > "freebsd-hackers-unsubscribe@freebsd.org" > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to > "freebsd-hackers-unsubscribe@freebsd.org" > > Wojciech, I have a very old single core VIA-15000 (1.5MHz) padlock server in use, so the numbers may be adversely affected: # openssl speed -evp aes-256-cbc To get the most accurate results, try to run this program when this computer is idle. Doing aes-256-cbc for 3s on 16 size blocks: 14239761 aes-256-cbc's in 2.97s Doing aes-256-cbc for 3s on 64 size blocks: 10999641 aes-256-cbc's in 2.96s Doing aes-256-cbc for 3s on 256 size blocks: 5845504 aes-256-cbc's in 2.98s Doing aes-256-cbc for 3s on 1024 size blocks: 2023702 aes-256-cbc's in 2.98s Doing aes-256-cbc for 3s on 8192 size blocks: 283165 aes-256-cbc's in 2.96s OpenSSL 0.9.7e-p1 25 Oct 2004 built on: Thu Sep 27 11:13:38 EST 2007 options:bn(64,32) md2(int) rc4(idx,int) des(ptr,risc1,16,long) aes(partial) blowfish(idx) compiler: gcc -DOPENSSL_THREADS -pthread -D_REENTRANT -D_THREAD_SAFE -D_THREADSAFE -DDSO_DLFCN -DHAVE_DLFCN_H -DOPENSSL_NO_KRB5 -DTERMIOS -DL_ENDIAN -fomit-frame-pointer -O3 -m486 -Wall -DSHA1_ASM -DMD5_ASM -DRMD160_ASM available timing options: USE_TOD HZ=128 [sysconf value] timing function used: getrusage The 'numbers' are in 1000s of bytes per second processed. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes aes-256-cbc 76594.07k 237795.53k 501951.53k 694914.86k 782587.93k On a single core VIA C7 Processor 1000MHz (FreeBSD 8.2 firewall) # openssl speed -evp aes-256-cbc Doing aes-256-cbc for 3s on 16 size blocks: 8270982 aes-256-cbc's in 2.91s Doing aes-256-cbc for 3s on 64 size blocks: 6672866 aes-256-cbc's in 2.96s Doing aes-256-cbc for 3s on 256 size blocks: 3652460 aes-256-cbc's in 2.95s Doing aes-256-cbc for 3s on 1024 size blocks: 1313482 aes-256-cbc's in 2.97s Doing aes-256-cbc for 3s on 8192 size blocks: 188472 aes-256-cbc's in 2.98s OpenSSL 1.0.0d 8 Feb 2011 built on: Mon Mar 7 14:18:26 EST 2011 options:bn(64,32) md2(int) rc4(4x,int) des(ptr,risc1,16,long) aes(partial) idea(int) blowfish(idx) compiler: cc -fPIC -DOPENSSL_PIC -DZLIB_SHARED -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -DTERMIOS -O3 -fomit-frame-pointer -Wall -O2 -pipe -pipe -O2 -g0 -ggdb0 -DSTRIP_FBSDID -UDEBUGGING -UEBUGGING -march=prescott -mtune=prescott -march=prescott -O3 -fno-strict-aliasing -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DRMD160_ASM -DAES_ASM -DWHIRLPOOL_ASM The 'numbers' are in 1000s of bytes per second processed. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes aes-256-cbc 45412.79k 144232.50k 317463.69k 453054.51k 518706.60k These are the kind of figures that you should expect on a padlock device. We turn on the padlock option during the build, and add these to our openssl.cnf (though it may no longer be necessary with the 8.x or later). openssl_conf = openssl_init [openssl_init] engines = engine_section [engine_section] padlock = padlock_section [padlock_section] default_algorithms = ALL Please note - the only reliable measure is to actually encrypt and decrypt files, we've found that the openssl speed test really isn't a comparative good measure. I'd suggest something like: dd if=/dev/zero bs=1m count=100 | openssl enc -e -aes-256-cbc -pass pass:obscure | openssl enc -d -aes-256-cbc -pass pass:obscure > /dev/null So for reference: the VIA/padlock 1.5MHz server transfers in 1.4 seconds (around 74MB/s), the 1MHz firewall transfers in 1.98s. Regards, Dewayne. From owner-freebsd-hackers@FreeBSD.ORG Fri Sep 12 00:24:13 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 443F6782 for ; Fri, 12 Sep 2014 00:24:13 +0000 (UTC) Received: from mail-pd0-x22b.google.com (mail-pd0-x22b.google.com [IPv6:2607:f8b0:400e:c02::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 19652B25 for ; Fri, 12 Sep 2014 00:24:13 +0000 (UTC) Received: by mail-pd0-f171.google.com with SMTP id p10so11459864pdj.2 for ; Thu, 11 Sep 2014 17:24:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=nNhIveQ0OEksg0ii7XxqO1h/ZxR8sOPmik64qZmUV0Q=; b=YMCg403/60FX2Vn0VyKveKtjViqLIMiQ9hUn9zhEu+/dGv9O7C3YVh9KLimXurO4pW gniclKwUzUYYeikBVI1Pj6JLO58k8OAknlQUy3Re6Th+REyaWrPquIqbNKyYXtFcQOif zjU65h3tnquAWaA+DaxZJmKNHs8qqqRIIJW1TV0OXh5aNNEh68bT4eEzdbPpBr1kkD+/ RkNAJtIjRlXJQX2rDI8ILQJZiBdTl9mxA1sa6n/TKDQ+qboHtr84FMEv6jJPMsXNeHsp qfU0pgRwGJKHr74ZL4HSTaXXw2h8sseZsKijtcL1OmIukWMBa1SU/NNZqfnqeEYrSpfx C1EA== MIME-Version: 1.0 X-Received: by 10.68.203.5 with SMTP id km5mr6143522pbc.91.1410481452651; Thu, 11 Sep 2014 17:24:12 -0700 (PDT) Received: by 10.70.132.2 with HTTP; Thu, 11 Sep 2014 17:24:12 -0700 (PDT) In-Reply-To: <220565922.34288992.1410298180362.JavaMail.root@uoguelph.ca> References: <220565922.34288992.1410298180362.JavaMail.root@uoguelph.ca> Date: Fri, 12 Sep 2014 02:24:12 +0200 Message-ID: Subject: Re: Tool to access ZFS/NFSv4 alternate data streams on FreeBSD? From: Lionel Cons To: Rick Macklem Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Freebsd hackers list , Simon Toedt , Jordan Hubbard , Richard Yao , Jan Bramkamp X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Sep 2014 00:24:13 -0000 On 9 September 2014 23:29, Rick Macklem wrote: > Simon Toedt wrote: >> On Tue, Sep 9, 2014 at 1:47 PM, Rick Macklem >> wrote: >> > Jordan Hubbard wrote: >> >> Yep. I was just describing the experience that OS X went through >> >> in >> >> implementing extattrs / legacy resource fork support. To recap it >> >> very briefly: Having NFSv4 support extattrs (or even named >> >> streams, >> >> if you want to go that far) is the comparatively easy part. It=E2=80= =99s >> >> backing them up / copying them around that gets more involved, and >> >> if you can=E2=80=99t back up certain attributes then you=E2=80=99re n= ot likely to >> >> get anyone to want to use them, at which point the whole =E2=80=9Csha= ring=E2=80=9D >> >> aspect kind of takes a back seat. >> >> >> > Yep. I strongly suspect you are correct. >> > >> > The question then becomes: >> > - Do we wait and see if someone chooses to get around to doing all >> > the hard userland work. >> >> Solaris tools already have support for this. Also AT&T AST from David >> Korn have support for O_XATTR, too. >> > Hopefully others will correct me if I have this incorrect, but I thought > CDDL code could only be used for optional components of FreeBSD? > I suspect tar and friends are considered core components and that code > for this would have to be written by someone (ie. couldn't use CDDL code?= ). > (I'm assuming that these tools are in OpenSolaris.) I don't think you FreeBSD should *copy* the code. But it can be used for reference how the extended tar headers for filesystem forks should look like. That's all. > > Be aware that most of FreeBSD's development is done by volunteers in thei= r > spare time, so I have no idea if someone is interested in doing this. If anyone can get the kernel parts I think we can sponsor someone to do the userland work. Lionel From owner-freebsd-hackers@FreeBSD.ORG Fri Sep 12 00:45:44 2014 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D6C60CE4 for ; Fri, 12 Sep 2014 00:45:44 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "funkthat.com", Issuer "funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id B4575CE9 for ; Fri, 12 Sep 2014 00:45:44 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id s8C0jgPH062909 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 11 Sep 2014 17:45:43 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id s8C0jfHd062908; Thu, 11 Sep 2014 17:45:41 -0700 (PDT) (envelope-from jmg) Date: Thu, 11 Sep 2014 17:45:41 -0700 From: John-Mark Gurney To: Wojciech Puchar Subject: Re: openssl with aes-in or padlock Message-ID: <20140912004541.GQ82175@funkthat.com> Mail-Followup-To: Wojciech Puchar , hackers@freebsd.org References: <20140911180258.GN82175@funkthat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Thu, 11 Sep 2014 17:45:43 -0700 (PDT) Cc: hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Sep 2014 00:45:45 -0000 Wojciech Puchar wrote this message on Thu, Sep 11, 2014 at 23:33 +0200: > >>#openssl speed -evp aes-256-cbc > > > >First off, you won't get much speed up w/ CBC encrypt... Try testing > >using aes-256-ctr instead... CBC can't process multiple blocks in > >parallel like CTR can... if you measure the cbc _decrypt_ speed, you > >should see a big improvement as CBC decrypt can be parallelized... > > > >>in the same time dd from geli encrypted ramdisk to /dev/null is 66MB/s > > > >geli uses a different framework for it's crypto processing.. for geli, > >make sure you have the aesni kernel module loaded before you attach > >to a geli disk... You should get kernel messages like the following: > >GEOM_ELI: Device gpt/werner.eli created. > >GEOM_ELI: Encryption: AES-XTS 256 > >GEOM_ELI: Crypto: hardware > > yes i have this. contrary to what you say - both AES-XTC and AES-CBC gets > MUCH faster with AES-NI. Well, AES-NI CBC may be faster w/ AES-NI, but it's not as fast as using another mode... AES-XTS should be many times faster than CBC... Also, above you compared two different modes... on CBC encrypt (the OpenSSL test) and CBC decrypt (the geli test) so of course you're going to get very different performances... You didn't tell me if you're using a new enough version of OpenSSL or not.. What release are you using? iirc, 10.0-R was the first release that the included OpenSSL suppoed AES-NI... You can always install OpenSSL from ports to get a version that supports AES-NI... > >notice the Crypto: hardware line.. Also, make sure that your geli > >sector size is 4k instead of 512... This reduces the loop overhead, > > as i already said - geli works fast and make use of AES-NI or padlock > > openssl does not For comparision: $ openssl speed -evp aes-256-cbc [...] type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes aes-256-cbc 348326.67k 408978.79k 431361.86k 456647.06k 460708.89k $ openssl speed -decrypt -evp aes-256-cbc [...] type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes aes-256-cbc 318017.94k 1263192.77k 2938389.38k 3288584.50k 3400531.97k notice, encrypt is 460MB/sec vs 3.4GB/sec decrypt... This is HEAD: $ openssl version OpenSSL 1.0.1h-freebsd 5 Jun 2014 Also, 66MB/sec seems VERY slow to me to be using AES-NI, but it just could be you're on 9.x which didn't have very good AES-NI kernel support... We need more information about which version of FreeBSD, and what processor you have before we can be more help... The first part of dmesg would be useful... and also openssl version too.. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-hackers@FreeBSD.ORG Fri Sep 12 15:45:52 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 75AFDEB8 for ; Fri, 12 Sep 2014 15:45:52 +0000 (UTC) Received: from mail-pd0-x233.google.com (mail-pd0-x233.google.com [IPv6:2607:f8b0:400e:c02::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B716FDA for ; Fri, 12 Sep 2014 15:45:52 +0000 (UTC) Received: by mail-pd0-f179.google.com with SMTP id g10so1472116pdj.38 for ; Fri, 12 Sep 2014 08:45:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=UEFHQ+xcpKJIeab72VsE1brNAogWl+eUvmov0JiHu38=; b=M7hLFaMRFmJXvLEQc7kIYI6mxnSzFfvhTjXUZg0id438AuKDGjUjpe7/UORbMeU95w m835ITfROvgeL5BHcFcmK7UZ5+4RYYVs4PpOb0mtQQ6YZ57ZSIPMuSMM110b2QHDeKiX qrAoaWOkMFz1gjhE4oObqK217jtGVYL+1AwT1ZykosPocvdBHGV7euakvdridqp1xE7d bt+4dgB4rmP3BYWNReBr0efEF0GfqQ2U2AIBKbUpYcpvagChFCyiQJcPT0GdytpcOmFX CLd9Ge+5Xy4fbikI7er2hsAC28DblVLFGXLHaWANG/WeFSylWT1ZHATMSMKrq38zpRhD kpAQ== MIME-Version: 1.0 X-Received: by 10.68.109.5 with SMTP id ho5mr13743180pbb.13.1410536751733; Fri, 12 Sep 2014 08:45:51 -0700 (PDT) Received: by 10.66.82.37 with HTTP; Fri, 12 Sep 2014 08:45:51 -0700 (PDT) In-Reply-To: References: <220565922.34288992.1410298180362.JavaMail.root@uoguelph.ca> Date: Fri, 12 Sep 2014 17:45:51 +0200 Message-ID: Subject: Re: Tool to access ZFS/NFSv4 alternate data streams on FreeBSD? From: Simon Toedt To: Jordan Hubbard Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org, Richard Yao , Rick Macklem , Lionel Cons , Jan Bramkamp X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Sep 2014 15:45:52 -0000 On Wed, Sep 10, 2014 at 12:47 AM, Jordan Hubbard wrote: > > On Sep 9, 2014, at 2:29 PM, Rick Macklem wrote: > > Hopefully others will correct me if I have this incorrect, but I thought > CDDL code could only be used for optional components of FreeBSD? > I suspect tar and friends are considered core components and that code > for this would have to be written by someone (ie. couldn't use CDDL code?= ). > (I'm assuming that these tools are in OpenSolaris.) > > > Well, if someone is willing to add copyfile(3) to FreeBSD (it does all th= e > magic) then the OS X versions of tar/pax/cpio/=E2=80=A6 are just forked v= ersions of > the FreeBSD tools, but with copyfile(3) support for handling (serializing= / > deserializing) EAs. > > - Jordan Does FreeBSD partake in Google Summer Of Code? Adding O_XATTR support sounds like an excellent project for it which could benefit all of the *BSD, from FreeBSD over NetBSD up to Dragonfly... Simon From owner-freebsd-hackers@FreeBSD.ORG Fri Sep 12 15:47:28 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 77270FB6 for ; Fri, 12 Sep 2014 15:47:28 +0000 (UTC) Received: from mail-pd0-x22b.google.com (mail-pd0-x22b.google.com [IPv6:2607:f8b0:400e:c02::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B7D72F for ; Fri, 12 Sep 2014 15:47:28 +0000 (UTC) Received: by mail-pd0-f171.google.com with SMTP id p10so1491379pdj.2 for ; Fri, 12 Sep 2014 08:47:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=BBaHArMEsVhoHd34qM8IEnmAn4/Z7hzdpXdFDWhuuqU=; b=Vygkg+q17KbczSBVzaJtaXHcu9GfhbnpOXK5edahS8Blx3ydvo2e3NtiaTlKvLn/RE MShjgR/nBVTFoO7PanwLlNkYqKFRhAyMQdklG2KkoXIXIEInMKpzedgzN1z2GfYgb21+ j14u1zWaL33U3MP7jSpPEXBLLY2er1kjd5gVsF9B4QfiEvcNjfCaEio8j/jS3PcJNMSE wgZtq2QcTwj+w5PHWidQO6fBCGpKUNQGoCwPz5OuxXNN49faXMf5f+0ay9pOTHOjUimH gu6p2CKoUMNHaF2mMgUaV9/pKy42zWUJ1jG0GTkgGh4ZWkf/J7Fde9YMyjrK9ORoj8Xu zn9w== MIME-Version: 1.0 X-Received: by 10.68.109.5 with SMTP id ho5mr13758192pbb.13.1410536847930; Fri, 12 Sep 2014 08:47:27 -0700 (PDT) Received: by 10.66.82.37 with HTTP; Fri, 12 Sep 2014 08:47:27 -0700 (PDT) In-Reply-To: References: <220565922.34288992.1410298180362.JavaMail.root@uoguelph.ca> Date: Fri, 12 Sep 2014 17:47:27 +0200 Message-ID: Subject: Re: Tool to access ZFS/NFSv4 alternate data streams on FreeBSD? From: Simon Toedt To: Lionel Cons Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Freebsd hackers list , Richard Yao , Rick Macklem , Jordan Hubbard , Jan Bramkamp X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Sep 2014 15:47:28 -0000 On Fri, Sep 12, 2014 at 2:24 AM, Lionel Cons wro= te: > On 9 September 2014 23:29, Rick Macklem wrote: >> Simon Toedt wrote: >>> On Tue, Sep 9, 2014 at 1:47 PM, Rick Macklem >>> wrote: >>> > Jordan Hubbard wrote: >>> >> Yep. I was just describing the experience that OS X went through >>> >> in >>> >> implementing extattrs / legacy resource fork support. To recap it >>> >> very briefly: Having NFSv4 support extattrs (or even named >>> >> streams, >>> >> if you want to go that far) is the comparatively easy part. It=E2= =80=99s >>> >> backing them up / copying them around that gets more involved, and >>> >> if you can=E2=80=99t back up certain attributes then you=E2=80=99re = not likely to >>> >> get anyone to want to use them, at which point the whole =E2=80=9Csh= aring=E2=80=9D >>> >> aspect kind of takes a back seat. >>> >> >>> > Yep. I strongly suspect you are correct. >>> > >>> > The question then becomes: >>> > - Do we wait and see if someone chooses to get around to doing all >>> > the hard userland work. >>> >>> Solaris tools already have support for this. Also AT&T AST from David >>> Korn have support for O_XATTR, too. >>> >> Hopefully others will correct me if I have this incorrect, but I thought >> CDDL code could only be used for optional components of FreeBSD? >> I suspect tar and friends are considered core components and that code >> for this would have to be written by someone (ie. couldn't use CDDL code= ?). >> (I'm assuming that these tools are in OpenSolaris.) > > I don't think you FreeBSD should *copy* the code. But it can be used > for reference how the extended tar headers for filesystem forks should > look like. That's all. > >> >> Be aware that most of FreeBSD's development is done by volunteers in the= ir >> spare time, so I have no idea if someone is interested in doing this. > > If anyone can get the kernel parts I think we can sponsor someone to > do the userland work. How much money would CERN offer? :) Simon From owner-freebsd-hackers@FreeBSD.ORG Fri Sep 12 16:25:43 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9153F7CE; Fri, 12 Sep 2014 16:25:43 +0000 (UTC) Received: from mail-oi0-x229.google.com (mail-oi0-x229.google.com [IPv6:2607:f8b0:4003:c06::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4D50C7D1; Fri, 12 Sep 2014 16:25:43 +0000 (UTC) Received: by mail-oi0-f41.google.com with SMTP id u20so678512oif.0 for ; Fri, 12 Sep 2014 09:25:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=3eher2zxi6GwYbzSGDDFB94zerltCdZY/YeRTCrbM+0=; b=Nd+8eQMCAigCxRMrLEZ7Wdo4VMNsXvoH175FMkeqtmPge6+yKhbzwP+uJQrDafHvoq 85IVODxn/ECKAFyXPpzJT+yGqfEy28pkrNxVc246d+PXn2ivE1QzOxobzX23VWQYSwGn 9JZcTix0NYTdJr4O4DtI21CgM3714ad3RqOlWnyFt0yi0agMf5PFsx3DeLL67qMJzxWW MxhMeWpvHlEl9oAY8V7FFsS9WYRx5LTrcw/LcUQkB8weYpo94XyLPsJ/e1EHdBzV1n2o PBWO5sRvsE23eHx68kjW2vj2sSbwfRkrLph46TMyNl8EX6CfNTOY+OZU0x7ze9Hbi9Lb Hx7g== MIME-Version: 1.0 X-Received: by 10.182.128.34 with SMTP id nl2mr9701837obb.44.1410539142452; Fri, 12 Sep 2014 09:25:42 -0700 (PDT) Sender: tomek.cedro@gmail.com Received: by 10.202.48.15 with HTTP; Fri, 12 Sep 2014 09:25:42 -0700 (PDT) In-Reply-To: References: Date: Fri, 12 Sep 2014 18:25:42 +0200 X-Google-Sender-Auth: ArOkDXn3zi2vHGBbWJ1XcwOc5co Message-ID: Subject: Re: FreeBSD and WiDi / Miracast / WiFi Direct HDMI streaming From: CeDeROM To: Waitman Gobble Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-hackers@freebsd.org" , FreeBSD Questions Mailing List X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Sep 2014 16:25:43 -0000 PTV3000 does not seems to work even with the stock Android/Nexus (Streamcast) nor Windows 7 (Intel WiDi). I have ordered a Streamcast dongle from Google, lets give it a try :-) Then lets try to port OpenWFD to FreeBSD with any (or both) hadrware dongles support :-) -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info From owner-freebsd-hackers@FreeBSD.ORG Fri Sep 12 16:32:53 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9D734A86; Fri, 12 Sep 2014 16:32:53 +0000 (UTC) Received: from mail-oi0-x22f.google.com (mail-oi0-x22f.google.com [IPv6:2607:f8b0:4003:c06::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5797C8BB; Fri, 12 Sep 2014 16:32:53 +0000 (UTC) Received: by mail-oi0-f47.google.com with SMTP id a141so677980oig.20 for ; Fri, 12 Sep 2014 09:32:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=GZHRBfoUes/Rn8WOIDj5yCVk7ts3RQBOEsWB/gW45fs=; b=Q6QjftSjXLh6s76XJWn5cK8piOF2JPcvW/IlNBPYu/Pt4TdDfBSfFL0yqGHM07zSnJ 7bUYqMrR4nui+Bpj6Xt6Law61SRytO664tmZTpi/DOXGz2ZMeUmLJdJtoJlHoRVm1N/h EzPtf4sm+MS/Rz61b9qnVURGtSZfrdkA7GeRTnYkravLAv8mzRQdKCi5XH+3354qDNzp qHirzqgGYcgJjP1MoEb4/u5jQ10SmFKXb/pqKeRMC9dvCm+BnwdNpX1TxF8fOwNuJdAa leeJyTFBjNn4cflC7NzcrJHG7k36+asiuJQIYRgCnTPq7/VCJdUuqrheTEwWqRsg4QjK NTcA== MIME-Version: 1.0 X-Received: by 10.60.78.132 with SMTP id b4mr9587982oex.21.1410539572628; Fri, 12 Sep 2014 09:32:52 -0700 (PDT) Sender: tomek.cedro@gmail.com Received: by 10.202.48.15 with HTTP; Fri, 12 Sep 2014 09:32:52 -0700 (PDT) In-Reply-To: References: Date: Fri, 12 Sep 2014 18:32:52 +0200 X-Google-Sender-Auth: ruTUXlZRxH3MWfD8YevtTkgh0u8 Message-ID: Subject: Re: FreeBSD and WiDi / Miracast / WiFi Direct HDMI streaming From: CeDeROM To: Waitman Gobble Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-hackers@freebsd.org" , FreeBSD Questions Mailing List X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Sep 2014 16:32:53 -0000 On Fri, Sep 12, 2014 at 6:29 PM, Waitman Gobble wrote: > I bought some google dongle thingy that I've never tried, maybe it's a > streamcast, i'll check. > I fiddled around with the OpenWFD the other day to see if it would compile.. > There are a couple of 'linuxy' files that need looked at. Cool :-) It would be nice to have wireless HDMI on our beloved FreeBSD =) >From a first glance OpenWFD is a userspace daemon so maybe it wont be sooo hard to port :-) I just wonder how it would work with Xorg :-) Maybe next week I will have some time to look :-) -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info From owner-freebsd-hackers@FreeBSD.ORG Fri Sep 12 16:29:48 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 04531978; Fri, 12 Sep 2014 16:29:48 +0000 (UTC) Received: from mail-lb0-x22b.google.com (mail-lb0-x22b.google.com [IPv6:2a00:1450:4010:c04::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 529C2807; Fri, 12 Sep 2014 16:29:47 +0000 (UTC) Received: by mail-lb0-f171.google.com with SMTP id 10so1233693lbg.16 for ; Fri, 12 Sep 2014 09:29:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=WQ7NVMI0asg394LKzeuMxTMBfTzMjjR94NG25HbPgOE=; b=YMrPelrxw00XcDuXHkKQJ9B4qz5y9yjmOCQ+EIKvzvsIlTFnV95txjhFOs5NmY1o15 dkl+V1AuRrUbRdGuKEYpFZJYfXWZrtThSKb6zUtjuTTAW9JQpGtfX33+SZityMyrfHE/ iCMgsJ101zyWpdLs+UMyVZpn/2jWJdqRDcoJxmB8YF7ebZqZpZ2Y4vxx+n3dYFspmWcm wtOqOZH7+HKhW5hgFlfV+3Ft+yPOHiydRbitD5kFNtNro0cCBwFzJtl+SUCCo9Jsmi4a QiPcpbwpUb8SXUqFdTIT3e/vmPd9do9eLEV3EYA0eGJovmO/qxw9NhQ7bYdPIMDPCf5P aS2g== MIME-Version: 1.0 X-Received: by 10.152.23.42 with SMTP id j10mr10405098laf.82.1410539385149; Fri, 12 Sep 2014 09:29:45 -0700 (PDT) Received: by 10.25.136.132 with HTTP; Fri, 12 Sep 2014 09:29:45 -0700 (PDT) In-Reply-To: References: Date: Fri, 12 Sep 2014 09:29:45 -0700 Message-ID: Subject: Re: FreeBSD and WiDi / Miracast / WiFi Direct HDMI streaming From: Waitman Gobble To: CeDeROM X-Mailman-Approved-At: Fri, 12 Sep 2014 16:38:30 +0000 Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-hackers@freebsd.org" , FreeBSD Questions Mailing List X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Sep 2014 16:29:48 -0000 On Fri, Sep 12, 2014 at 9:25 AM, CeDeROM wrote: > PTV3000 does not seems to work even with the stock Android/Nexus > (Streamcast) nor Windows 7 (Intel WiDi). > > I have ordered a Streamcast dongle from Google, lets give it a try :-) > > Then lets try to port OpenWFD to FreeBSD with any (or both) hadrware > dongles support :-) > > -- > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > I bought some google dongle thingy that I've never tried, maybe it's a streamcast, i'll check. I fiddled around with the OpenWFD the other day to see if it would compile.. There are a couple of 'linuxy' files that need looked at. -- Waitman Gobble San Jose California USA 510-830-7975 From owner-freebsd-hackers@FreeBSD.ORG Fri Sep 12 16:39:46 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F171C14D; Fri, 12 Sep 2014 16:39:46 +0000 (UTC) Received: from mail-oi0-x22a.google.com (mail-oi0-x22a.google.com [IPv6:2607:f8b0:4003:c06::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AC4C4945; Fri, 12 Sep 2014 16:39:46 +0000 (UTC) Received: by mail-oi0-f42.google.com with SMTP id e131so624834oig.1 for ; Fri, 12 Sep 2014 09:39:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=/NIP16+2obgriaCpWTZIgf6qWAw7jogNkSGxXNKmn8w=; b=er3pnN1lRGgSJMDYCmPINub8KUyk0VGo/dOMwlchKI+fPd6LwXeg+VcVDQlSVA3Nsc LCsm1bVz3jvAIA3S4cHZIHCGOYlf4e7MxjUW9XEnqGbiy71Ea+/YK8vj+b06r3bvdux9 cySxOQgGw14/7th7sM6Lnszzx3QyJf0E3UprBo9zlcRPp2jabPRnklFpF3gcWBV6HRes sS/qkA++rSvauNM5Ud7/OhaeQn1RJCBtm2UNitx4zOXXCI41jwlBD74hzCih/3Fs8PcR TsMUtXN5kMhMNM7RDXdBFFg7EO0T2/LWx3A/qH5YepHH1v+RpGXpyUO2VDpXISNsEhIr a/pA== MIME-Version: 1.0 X-Received: by 10.182.60.232 with SMTP id k8mr9409833obr.54.1410539985871; Fri, 12 Sep 2014 09:39:45 -0700 (PDT) Sender: tomek.cedro@gmail.com Received: by 10.202.48.15 with HTTP; Fri, 12 Sep 2014 09:39:45 -0700 (PDT) In-Reply-To: References: Date: Fri, 12 Sep 2014 18:39:45 +0200 X-Google-Sender-Auth: 8I8vD6iS_xv3nmKdkD5LBkUTero Message-ID: Subject: Re: FreeBSD and WiDi / Miracast / WiFi Direct HDMI streaming From: CeDeROM To: Waitman Gobble Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-hackers@freebsd.org" , FreeBSD Questions Mailing List X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Sep 2014 16:39:47 -0000 On Fri, Sep 12, 2014 at 6:25 PM, CeDeROM wrote: > I have ordered a Streamcast dongle from Google, lets give it a try :-) Sorry, I have refernced a CHROMECAST device :-) http://www.google.com/chrome/devices/chromecast/ -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info From owner-freebsd-hackers@FreeBSD.ORG Fri Sep 12 17:04:57 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 12863CBD for ; Fri, 12 Sep 2014 17:04:57 +0000 (UTC) Received: from mail-ie0-x231.google.com (mail-ie0-x231.google.com [IPv6:2607:f8b0:4001:c03::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D238EC2F for ; Fri, 12 Sep 2014 17:04:56 +0000 (UTC) Received: by mail-ie0-f177.google.com with SMTP id rd18so1229528iec.8 for ; Fri, 12 Sep 2014 10:04:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc:content-type; bh=KB8Vb8U5ZQAJx2WeMK2DFUov5Ft8lqOWVU4Qkketcog=; b=a83wGuyRwFqiudDzw+oL1gW2zj+4Ajf6KrJtUh3jybIBI2x2L/dw2Ckx9R+bIPF45e PoSdWUBhbMd5OBZtBF+XA4Q8Ppf6mLaPkhFc0qxZFYRn5y+MvtCar7OgF04mUo+5OhBm rDfwL6vMpX0IDR9sFtAByc2wyCBTh35GsShJCwvF2NOkkQLuhRK9KaR/Lo5T5L4GvMJ6 YI38ejbz98VGJR3ZzNI5WlaUhPiFZ8U00pcqFAQfU7G+kqWui8uqTEp8VzgAb9EEMfzk 2Pqxse6quTbQv1l0jam6deA5vJTczrUBdL61Z7nsaxO8DG7cdYnVuY9O7qGzBdgofO43 7XWA== X-Received: by 10.42.233.75 with SMTP id jx11mr11058678icb.22.1410541496305; Fri, 12 Sep 2014 10:04:56 -0700 (PDT) MIME-Version: 1.0 Sender: carpeddiem@gmail.com Received: by 10.107.44.1 with HTTP; Fri, 12 Sep 2014 10:04:36 -0700 (PDT) In-Reply-To: <20140607184126.GQ3991@kib.kiev.ua> References: <538A3432.5010303@gmx.us> <53930E19.8090603@gmx.us> <53931963.4040604@selasky.org> <53932314.6010108@gmx.us> <1402153691.709851721.u6k6kkkk@frv35.fwdcdn.com> <53933110.8060300@gmx.us> <1402157083.156846225.m95e69ke@frv35.fwdcdn.com> <53934250.1090403@gmx.us> <20140607175752.GP3991@kib.kiev.ua> <20140607184126.GQ3991@kib.kiev.ua> From: Ed Maste Date: Fri, 12 Sep 2014 13:04:36 -0400 X-Google-Sender-Auth: IKG-KGxbR56nCwfxgt2b0HYoJko Message-ID: Subject: Re: Fwd: Interrupt Overload To: Konstantin Belousov Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Sep 2014 17:04:57 -0000 On 7 June 2014 14:41, Konstantin Belousov wrote: > Try to set the tunable hw.drm.msi to 0 before i915 driver is loaded. > I.e. the easiest is to set it at loader prompt. If you load driver > by starting Xorg, then kenv hw.drm.msi=0 would be enough. A similar issue is reported in PR 193500, but unlike this case it's reported that hw.drm.msi=0 does not solve it. I've added a request to the PR for vmstat -i during the storm. From owner-freebsd-hackers@FreeBSD.ORG Fri Sep 12 21:12:59 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C7208ABB for ; Fri, 12 Sep 2014 21:12:59 +0000 (UTC) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 8A7FDA61 for ; Fri, 12 Sep 2014 21:12:59 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ArEEAABhE1SDaFve/2dsb2JhbABfg2BXBIJ4xXUKhnpUAYEneIQDAQEBAwEBAQEgKyALBRYYAgINGQIpAQkmBggHBAEcBIgZCA2nQ5VyAReBLI0/EAIBGwEzB4I4QRKBQQWWBIQBhGKKTYkTg30hLweBQYEHAQEB X-IronPort-AV: E=Sophos;i="5.04,515,1406606400"; d="scan'208";a="153778336" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 12 Sep 2014 17:12:58 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 91793B403E; Fri, 12 Sep 2014 17:12:58 -0400 (EDT) Date: Fri, 12 Sep 2014 17:12:58 -0400 (EDT) From: Rick Macklem To: Lionel Cons Message-ID: <1757754054.35708274.1410556378584.JavaMail.root@uoguelph.ca> In-Reply-To: Subject: Re: Tool to access ZFS/NFSv4 alternate data streams on FreeBSD? MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 7.2.6_GA_2926 (ZimbraWebClient - FF3.0 (Win)/7.2.6_GA_2926) Cc: Freebsd hackers list , Simon Toedt , Jordan Hubbard , Richard Yao , Jan Bramkamp X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Sep 2014 21:13:00 -0000 Lionel Cons wrote: > On 9 September 2014 23:29, Rick Macklem wrote: > > Simon Toedt wrote: > >> On Tue, Sep 9, 2014 at 1:47 PM, Rick Macklem > >> > >> wrote: > >> > Jordan Hubbard wrote: > >> >> Yep. I was just describing the experience that OS X went > >> >> through > >> >> in > >> >> implementing extattrs / legacy resource fork support. To recap > >> >> it > >> >> very briefly: Having NFSv4 support extattrs (or even named > >> >> streams, > >> >> if you want to go that far) is the comparatively easy part. > >> >> It=E2=80=99s > >> >> backing them up / copying them around that gets more involved, > >> >> and > >> >> if you can=E2=80=99t back up certain attributes then you=E2=80=99re= not likely > >> >> to > >> >> get anyone to want to use them, at which point the whole > >> >> =E2=80=9Csharing=E2=80=9D > >> >> aspect kind of takes a back seat. > >> >> > >> > Yep. I strongly suspect you are correct. > >> > > >> > The question then becomes: > >> > - Do we wait and see if someone chooses to get around to doing > >> > all > >> > the hard userland work. > >> > >> Solaris tools already have support for this. Also AT&T AST from > >> David > >> Korn have support for O_XATTR, too. > >> > > Hopefully others will correct me if I have this incorrect, but I > > thought > > CDDL code could only be used for optional components of FreeBSD? > > I suspect tar and friends are considered core components and that > > code > > for this would have to be written by someone (ie. couldn't use CDDL > > code?). > > (I'm assuming that these tools are in OpenSolaris.) >=20 > I don't think you FreeBSD should *copy* the code. But it can be used > for reference how the extended tar headers for filesystem forks > should > look like. That's all. >=20 > > > > Be aware that most of FreeBSD's development is done by volunteers > > in their > > spare time, so I have no idea if someone is interested in doing > > this. >=20 > If anyone can get the kernel parts I think we can sponsor someone to > do the userland work. >=20 Hmm, well others that are more familiar with the traditions of the FreeBSD will need to comment. I have never figured out how the "collective" makes technical direction decisions. I think that you might want to email on freebsd-arch@ proposing that suppor= t for openat()/resource forks be added to FreeBSD. Then, based on what respon= ses occur to this "big picture" proposal, whether or not to do it may become apparent? If the collective thinks it is appropriate, I can probably find time to do the kernel bits for NFSv4/ZFS. (I have no interest in trying to add support to UFS or doing the userland bits.) rick > Lionel > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to > "freebsd-hackers-unsubscribe@freebsd.org" From owner-freebsd-hackers@FreeBSD.ORG Sat Sep 13 07:34:48 2014 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4F0D18E1 for ; Sat, 13 Sep 2014 07:34:48 +0000 (UTC) Received: from puchar.net (puchar.net [188.252.31.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C763A8E2 for ; Sat, 13 Sep 2014 07:34:47 +0000 (UTC) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.14.9/8.14.9) with ESMTP id s8D7YaxC092461; Sat, 13 Sep 2014 09:34:36 +0200 (CEST) (envelope-from wojtek@puchar.net) X-Authentication-Warning: puchar.net: Host puchar-wojtek.intra [10.0.224.6] claimed to be wojtek.dom Received: from wojtek.dom (localhost [127.0.0.1]) by wojtek.dom (8.14.9/8.14.9) with ESMTP id s8D7YabT018155; Sat, 13 Sep 2014 09:34:36 +0200 (CEST) (envelope-from wojtek@puchar.net) Received: from localhost (wojtek@localhost) by wojtek.dom (8.14.9/8.14.9/Submit) with ESMTP id s8D7YZsg018152; Sat, 13 Sep 2014 09:34:35 +0200 (CEST) (envelope-from wojtek@puchar.net) X-Authentication-Warning: wojtek.dom: wojtek owned process doing -bs Date: Sat, 13 Sep 2014 09:34:35 +0200 (CEST) From: Wojciech Puchar X-X-Sender: wojtek@wojtek.dom To: John-Mark Gurney Subject: Re: openssl with aes-in or padlock In-Reply-To: <20140912004541.GQ82175@funkthat.com> Message-ID: References: <20140911180258.GN82175@funkthat.com> <20140912004541.GQ82175@funkthat.com> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (puchar.net [10.0.1.1]); Sat, 13 Sep 2014 09:34:36 +0200 (CEST) Cc: hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Sep 2014 07:34:48 -0000 >> MUCH faster with AES-NI. > > Well, AES-NI CBC may be faster w/ AES-NI, but it's not as fast as using > another mode... AES-XTS should be many times faster than CBC... Also, 30% maybe with geli. > above you compared two different modes... on CBC encrypt (the OpenSSL > test) and CBC decrypt (the geli test) so of course you're going to get > very different performances... > > You didn't tell me if you're using a new enough version of OpenSSL or > not.. What release are you using? iirc, 10.0-R was the first release OpenSSL 1.0.1e-freebsd 11 Feb 2013 From owner-freebsd-hackers@FreeBSD.ORG Sat Sep 13 07:35:22 2014 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 82CC89C4 for ; Sat, 13 Sep 2014 07:35:22 +0000 (UTC) Received: from puchar.net (puchar.net [188.252.31.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0695F8EB for ; Sat, 13 Sep 2014 07:35:21 +0000 (UTC) Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.14.9/8.14.9) with ESMTP id s8D7ZJnd092480; Sat, 13 Sep 2014 09:35:20 +0200 (CEST) (envelope-from wojtek@puchar.net) X-Authentication-Warning: puchar.net: Host puchar-wojtek.intra [10.0.224.6] claimed to be wojtek.dom Received: from wojtek.dom (localhost [127.0.0.1]) by wojtek.dom (8.14.9/8.14.9) with ESMTP id s8D7ZJa7018167; Sat, 13 Sep 2014 09:35:19 +0200 (CEST) (envelope-from wojtek@puchar.net) Received: from localhost (wojtek@localhost) by wojtek.dom (8.14.9/8.14.9/Submit) with ESMTP id s8D7ZJw8018164; Sat, 13 Sep 2014 09:35:19 +0200 (CEST) (envelope-from wojtek@puchar.net) X-Authentication-Warning: wojtek.dom: wojtek owned process doing -bs Date: Sat, 13 Sep 2014 09:35:19 +0200 (CEST) From: Wojciech Puchar X-X-Sender: wojtek@wojtek.dom To: Jim Thompson Subject: Re: openssl with aes-in or padlock In-Reply-To: <62E8AD7E-346F-4F77-9628-6D5121D7AD6D@netgate.com> Message-ID: References: <20140911180258.GN82175@funkthat.com> <62E8AD7E-346F-4F77-9628-6D5121D7AD6D@netgate.com> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (puchar.net [10.0.1.1]); Sat, 13 Sep 2014 09:35:20 +0200 (CEST) Cc: John-Mark Gurney , "hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Sep 2014 07:35:22 -0000 will it be available on FreeBSD 10 ? On Thu, 11 Sep 2014, Jim Thompson wrote: > We just fixed IPSEC to use AES-GCM (with support for AES-NI on hardware that supports it.) > > OpenSSL / OpenVPN is probably next. > > -- Jim > > On Sep 11, 2014, at 14:33, Wojciech Puchar wrote: > >>>> #openssl speed -evp aes-256-cbc >>> >>> First off, you won't get much speed up w/ CBC encrypt... Try testing >>> using aes-256-ctr instead... CBC can't process multiple blocks in >>> parallel like CTR can... if you measure the cbc _decrypt_ speed, you >>> should see a big improvement as CBC decrypt can be parallelized... >>> >>>> in the same time dd from geli encrypted ramdisk to /dev/null is 66MB/s >>> >>> geli uses a different framework for it's crypto processing.. for geli, >>> make sure you have the aesni kernel module loaded before you attach >>> to a geli disk... You should get kernel messages like the following: >>> GEOM_ELI: Device gpt/werner.eli created. >>> GEOM_ELI: Encryption: AES-XTS 256 >>> GEOM_ELI: Crypto: hardware >> >> yes i have this. contrary to what you say - both AES-XTC and AES-CBC gets MUCH faster with AES-NI. >> >>> notice the Crypto: hardware line.. Also, make sure that your geli >>> sector size is 4k instead of 512... This reduces the loop overhead, >> >> as i already said - geli works fast and make use of AES-NI or padlock >> >> openssl does not >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > >