From owner-freebsd-hackers@freebsd.org Thu Aug 16 16:28:36 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CAF77106CF24 for ; Thu, 16 Aug 2018 16:28:35 +0000 (UTC) (envelope-from doubleble@outlook.com) Received: from NAM02-BL2-obe.outbound.protection.outlook.com (mail-oln040092003066.outbound.protection.outlook.com [40.92.3.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT TLS CA 4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 11DC18B7D2 for ; Thu, 16 Aug 2018 16:28:34 +0000 (UTC) (envelope-from doubleble@outlook.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0aKi9+JF8PrAP2g3D4sxamQTB/c9IHbobJWoLn+zud8=; b=hOGHDtCGipX4jNdv/bxPnJJGO5YrkoyVBQ/lxHsfNFPY9ZOwDDF56luOzUBEvSkERo2GB4CE49S4vpyWVVDRQBUHiyIk5IUDxfBtFBdzE9594E0O94nWoSeTm5IV3LQwOAHl+RfG3hIGdfa7tSrawK4T8L2vFpEjXQDexCoJ8iGf5BEhC4t3FSqRdSedk0wFIFXLmQKmLgriCliwoRGozL01i9ztJToXvtIXpgK1dUIsKyhGC14z9CF81v3pi+rnoZEz4NyDLwJww+l97i/GjdmmiWSUp1EzPnmHOt5niiE5ug6mOkofmzqGNphsMdPlSrI7CpdSfdb6ABuLAYx2Pw== Received: from SN1NAM02FT008.eop-nam02.prod.protection.outlook.com (10.152.72.58) by SN1NAM02HT139.eop-nam02.prod.protection.outlook.com (10.152.73.223) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.1059.14; Thu, 16 Aug 2018 16:28:33 +0000 Received: from BN6PR2201MB1523.namprd22.prod.outlook.com (10.152.72.58) by SN1NAM02FT008.mail.protection.outlook.com (10.152.72.119) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.20.1059.14 via Frontend Transport; Thu, 16 Aug 2018 16:28:33 +0000 Received: from BN6PR2201MB1523.namprd22.prod.outlook.com ([fe80::4c9a:37fe:d25e:45e]) by BN6PR2201MB1523.namprd22.prod.outlook.com ([fe80::4c9a:37fe:d25e:45e%2]) with mapi id 15.20.1059.017; Thu, 16 Aug 2018 16:28:33 +0000 From: Double Tong To: "freebsd-hackers@freebsd.org" Subject: Create a thread with a separate file descriptor table (set RFFDG flag) Thread-Topic: Create a thread with a separate file descriptor table (set RFFDG flag) Thread-Index: AQHUNXm4sY9W87Z5LU6r9yREhH5aIQ== Date: Thu, 16 Aug 2018 16:28:33 +0000 Message-ID: Accept-Language: en-CA, zh-CN, en-US Content-Language: en-CA X-MS-Has-Attach: X-MS-TNEF-Correlator: x-incomingtopheadermarker: OriginalChecksum:E87CE54C87FFBFDEA1FA71A6EADE8416FF56B4B5374CF5DCBF905C76F5E1AD22; UpperCasedChecksum:9BCB135E09A2376DE7422D7F08610F4D0D542F241BE8BBAC41E14EF32FBC5FF9; SizeAsReceived:6995; Count:44 x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [DyUPIF8F0BKBqOuIIKId5nD15iAbKzJXZkduKaEbtUQ=] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; SN1NAM02HT139; 6:ouRRlHpei4nfc+gLR5bRcTpkHlkCFm8BKVQcqc5gvv3dsH9Vc1rHAh/ova1O4q3GoVGPqRUPvIta0oA55xLbYvvmCDxmALR0uBVTiTE0ILdqOUupiKN8yK31mpn5Vmj9fZrE7sNKYS7ouj0045X95qLNTsqbYgqFnG2e+70wFZ14Gi8QKtXXNi7gXkQBp9m/9o6hyDa/JSq3bKQfY658jf1wgyCOVhUtGXO6X5y0RcsksotW/FBYICz96uB/+RqQVlj5YYxgpoC++Im/SGQ0q1GMqXEh8Z2LdGaylbvs15rHQAb/8/a9qoQ9vocs/esYHrTVGUEJhhB9p4Z3Hn8AYz6XJX4nZFGAAiyymYtysca2q5/Dql+Fx8bM68lJh3bg7PoLWjBgbP0/hWSVgNHmRE2+u0fcXgZbzhJeeTbtu76vtyaUBkieaT5WadpmOmL3P38k5fPhjVR7l4OuT3npeA==; 5:T7O4qwmQ25rdN+admoe85hWWB+oH9XNNZvvsJXeIXVFWcq2YxgSn9T2S9Vbh7F4PngGydRhqfG3DWNtqrUEbRE4ISKs88+F6iOn4xbopqUjaP4ziWwZxlYp0ZIs0P8/bz/p6vIkd4WZinrn04cWmasPq9czorV1eyBzp9muV074=; 7:ScSQc3yhb6OiyXyUI/vDsvQaAcvyRnA5QhQPWZJ9m5XCwz4NPkNC6Q/mr+xPGy0J/76U60xACGSQptOyfubb2OawdlFXECppmhhuiE59OyoM6qFfFKXK6ZmunHqUgNYfmvd4pa70IkMziV+ECvcXkRwVId6YVL2zpoznN57lFLqgGxHVvGn7mu778SQSiqSbrR2rZrhNCpSCx2jCXMj4brj7QMeYo35YE8GRUPjJQwx9oqX2ozXtrP2XuOaeS39a x-incomingheadercount: 44 x-eopattributedmessage: 0 x-microsoft-antispam: BCL:0; PCL:0; RULEID:(7020095)(201702061078)(5061506573)(5061507331)(1603103135)(2017031320274)(2017031324274)(2017031323274)(2017031322404)(1603101475)(1601125500)(1701031045); SRVR:SN1NAM02HT139; x-ms-traffictypediagnostic: SN1NAM02HT139: x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(4566010)(82015058); SRVR:SN1NAM02HT139; BCL:0; PCL:0; RULEID:; SRVR:SN1NAM02HT139; x-forefront-prvs: 07665BE9D1 x-forefront-antispam-report: SFV:NSPM; SFS:(7070007)(189003)(501624003)(199004)(106356001)(25786009)(6306002)(6436002)(8676002)(105586002)(8936002)(2351001)(2501003)(5250100002)(26005)(6346003)(55016002)(6916009)(54896002)(5660300001)(5640700003)(20460500001)(102836004)(104016004)(82202002)(87572001)(105004)(56003)(476003)(345774005)(68736007)(426003)(5024004)(14444005)(14454004)(256004)(86362001)(2900100001)(99286004)(7696005)(486006)(74316002)(81156014)(97736004)(19627405001)(33656002); DIR:OUT; SFP:1901; SCL:1; SRVR:SN1NAM02HT139; H:BN6PR2201MB1523.namprd22.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: outlook.com does not designate permitted sender hosts) authentication-results: spf=none (sender IP is ) smtp.mailfrom=doubleble@outlook.com; x-microsoft-antispam-message-info: DthhwvYVugjka6AQLQ0kGUGnv2kejaRHW6zTh/U2UNI5wtg6ZzHji3rpNIgxhUbUXWecBJiEEGU3wqg5bNVcMuzSrDnOGcf2SHibf+9PrcNOm++cPH4iRH9iGwO1KaQPvubhMmdDFgXQq2/k5yHcmCNZt9/lY6YN2JFxAjvJHTnEoi5o0/QUtLSAV79Ao8o/0qgEjGzifp02nbG1oz+g3sl+1HxRn7njlFW8C/S8XI4= MIME-Version: 1.0 X-OriginatorOrg: outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 9a4e3081-9524-43cf-bfc3-dcaef82d5da1 X-MS-Exchange-CrossTenant-Network-Message-Id: 86707581-2bde-4ff2-84b2-08d603954f1e X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 9a4e3081-9524-43cf-bfc3-dcaef82d5da1 X-MS-Exchange-CrossTenant-originalarrivaltime: 16 Aug 2018 16:28:33.1289 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Internet X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN1NAM02HT139 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Aug 2018 16:28:36 -0000 Hello, I want to create a thread with a separate file descriptor table to have bet= ter performance with kevent(2). On Linux, I was using unshare(2) syscall to= achieve this, which as far as I know there is no equivalent or similar sys= call in FreeBSD. I have posted on freebsd forums (https://forums.freebsd.org/threads/create-= a-thread-with-a-separate-file-descriptor-table-set-rffdg-flag.67143/), and = now I understood the following: rfork_thread(3) is deprecated in favor of = pthread_create(3). rfork_thread(3) is written in assembly language to perfo= rm stack swapping, which means if rfork_thread(3) no longer exists in the b= uild, it can damage our program's portability if it relies on rfork_thread. With the above consideration, I still have the following questions: 1. Is there an elegant way to create a thread with a separate file descr= iptor table? 2. If you are thinking about using rfork_thread(3) to do this, I am work= ing on this direction. I am using waitpid to join these "threads", and the = thread exits in the middle of execution with status 0x8b collected by waitp= id. I guess this status means invalid page access. I wrote a tiny program (= attached below) to reflect the code I am using in my program, I appreciate = if you would like to take a look at it to see if there is anything I was no= t doing correctly. 3. As I was reading the code of pthread_create, it allocates a pthread s= truct on the top of thread, and then calls clone, which freebsd implemented= its version of clone that actually calls rfork (I did not find the source = of freebsd's clone, can someone provides a link?). So I believe theoretical= ly there should be a way to achieve this in the user space. And if I am not= using pthread related APIs, then missing pthread struct should be fine as = well? 4. On Linux, after calling unshare(CLONE_FLIES), I got performance incre= ase around 10% with 1000 concurrent TCP connections. I am instructed to imp= lement this by my supervisor, and I do not have much details about why the = performance would increase. Would this also works for freebsd as well (keve= nt calls)? Thank you for any help, comments in advance! my_rfork_test.cc: -------------------------- // This program runs well, except status code is non zero. In my bigger pro= gram, it terminates in the middle of the routine with status 0x8b #include #include #include #include #include const size_t VEC_SIZE =3D 1000; static int thread_routine(void* arg) { std::cout<<"init thread "<** vectors =3D new std::vector*[VEC_SIZE]; for(int i =3D 0; i < VEC_SIZE; i++) { vectors[i] =3D new std::vector(10000); std::cout<<"vec "<(2)); int status =3D 0; waitpid(child, &status, 0x0); std::cout<<"return status 0x"<