From owner-freebsd-arm@FreeBSD.ORG Sat Nov 2 13:33:45 2013 Return-Path: Delivered-To: freebsd-arm@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id CF6E3DEC for ; Sat, 2 Nov 2013 13:33:45 +0000 (UTC) (envelope-from ian@FreeBSD.org) Received: from mho-02-ewr.mailhop.org (mho-02-ewr.mailhop.org [204.13.248.72]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A4DF625A4 for ; Sat, 2 Nov 2013 13:33:45 +0000 (UTC) Received: from c-24-8-230-52.hsd1.co.comcast.net ([24.8.230.52] helo=damnhippie.dyndns.org) by mho-02-ewr.mailhop.org with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.72) (envelope-from ) id 1VcbKi-0003qE-ES; Sat, 02 Nov 2013 13:33:44 +0000 Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id rA2DXfAD058421; Sat, 2 Nov 2013 07:33:41 -0600 (MDT) (envelope-from ian@FreeBSD.org) X-Mail-Handler: Dyn Standard SMTP by Dyn X-Originating-IP: 24.8.230.52 X-Report-Abuse-To: abuse@dyndns.com (see http://www.dyndns.com/services/sendlabs/outbound_abuse.html for abuse reporting information) X-MHO-User: U2FsdGVkX1+626rbGikDehmoZ5CFnVsh Subject: Re: sshd crash From: Ian Lepore To: Tim Kientzle In-Reply-To: References: <1383313834.31172.65.camel@revolution.hippie.lan> <1383328423.31172.92.camel@revolution.hippie.lan> <1383343354.31172.102.camel@revolution.hippie.lan> Content-Type: text/plain; charset="us-ascii" Date: Sat, 02 Nov 2013 07:33:40 -0600 Message-ID: <1383399220.31172.116.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: freebsd-arm@FreeBSD.org, Howard Su X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Nov 2013 13:33:46 -0000 On Fri, 2013-11-01 at 22:35 -0700, Tim Kientzle wrote: > On Nov 1, 2013, at 3:02 PM, Ian Lepore wrote: > > > On Sat, 2013-11-02 at 02:40 +0800, Jia-Shiun Li wrote: > >> On Sat, Nov 2, 2013 at 1:53 AM, Ian Lepore wrote: > >>> On Sat, 2013-11-02 at 01:44 +0800, Jia-Shiun Li wrote: > >>>> may I add: putty causes this to happen. mine 0.62. But ssh from another > >>>> FreeBSD host has no problem. > >>>> > >>>> I suspect it to be some issues related to memory or malloc issues > >>>> specific to bbb. 'tmux a -d' without existing detached sessions > >>>> causes tmux client to core dump. But sshd and it are both fine on rpi. > >>>> > >>>> -Jia-Shiun. > >>> > >>> This is the first I've heard of being able to ssh to an arm platform > >>> that doesn't have PrivSep disabled, since about July or so. I've never > >>> heard a report yet that anything on the client side could make a > >>> difference. > >>> > >>> It's definitely not a beaglebone thing, it happens on every arm board > >>> I've got... dreamplug, rpi, bbw, imx53, wandboard. > >> > >> > >> Ok let me make sure I did not mix things up. ;) > >> > >> IIRC I once saw similar issue on rpi shortly. But after another > >> weekly update it was gone. I did not pay too much attention on rpi, > >> and thought it was bbb specific. > >> > >> I did not change sshd_config, UsePrivilegeSeparation supposed > >> remaining on as default is. > > I started looking into it a couple of months ago but didn't get > very far; Diane Bruce got a lot further than I did. > > If I recall correctly, it started up when the malloc libc symbols > were changed. That may have altered what malloc implementation > sshd used. > > So it could be a long-standing stray write that jemalloc just > happens to detect. > > It could also be related to locking (there's some multi-threaded > crypto code in sshd that may be involved). There's lots of stuff with lock in the name, but I don't think there are actually any threads involved in sshd, just forking. ldd says sshd doesn't link to libthr. I'm not sure it's a mundane stray-write either. The routine that's asserting is checking to see if the contents of a page are all-zero because a jemalloc internal flag is set that says it should be. I had the routine print the non-zero data it found, and it looks like this: not-zero at 0 0x20c99000 = 0x20800a00 not-zero at 1 0x20c99004 = 0x00000001 not-zero at 2 0x20c99008 = 0x0000002f not-zero at 3 0x20c9900c = 0xffffffff not-zero at 4 0x20c99010 = 0x00007fff not-zero at 5 0x20c99014 = 0x00000003 not-zero at 96 0x20c99180 = 0x5a5a5a5a not-zero at 97 0x20c99184 = 0x5a5a5a5a not-zero at 98 0x20c99188 = 0x5a5a5a5a The 0x5a continues to the end of the page. So jemalloc has metadata that says it thinks the page is all-zeroes, and the page is a mix of data and some zeroes and the 5a junk-fill byte. It seems more like the metadata is in error somehow. (Maybe a stray write hit the metadata.) -- Ian