Date: Sat, 18 Jul 2009 18:09:54 +0100 From: Andrew Brampton <brampton+freebsd@gmail.com> To: "Joe R. Jah" <jjah@cloud.ccsf.cc.ca.us> Cc: freebsd-questions@freebsd.org Subject: Re: OT: wget bug Message-ID: <d41814900907181009q1d4f4b62k9818607a65421419@mail.gmail.com> In-Reply-To: <20090718093237.Y19472@cloud.ccsf.cc.ca.us> References: <20090717144049.C35992@cloud.ccsf.cc.ca.us> <d41814900907180452p29244911nd2570909e7274791@mail.gmail.com> <20090718093237.Y19472@cloud.ccsf.cc.ca.us>
next in thread | previous in thread | raw e-mail | index | archive | help
2009/7/18 Joe R. Jah <jjah@cloud.ccsf.cc.ca.us>: > Thank you Andrew. =C2=A0Yes the server is truly returning 401. =C2=A0I ha= ve already > reconfigured wget to download everything regardless of their timestamp, > but it's a waste of bandwidth, because most of the site is unchanged. > > Do you know of any workaround in wget, or an alternative tool to ONLY > download newer files by http? > Joe, There are two ways to check if the file has been changed. One, read the time the file was last changed, or two, read the file and compare it to a old copy. Wget was obviously trying to do option 1 but this is denied by the remote server. You most likely could get it to do option 2, however by doing so you are wasting bandwidth downloading unchanged files just to check if they had been changed. If you have control over the remote webserver, then the simplest way to solve this problem is to configure the webserver not to return 401 when wget sends the If-Modified-Since HTTP header. A better solution, again assuming you have control of the remote server, is to use "rsync" as it is designed for this kind of task. If you don't have control over the remote server, then you are stuck with your current solution. Andrew
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?d41814900907181009q1d4f4b62k9818607a65421419>