Date: Mon, 27 Oct 2014 18:55:57 +0200 From: Konstantin Belousov <kostikbel@gmail.com> To: John Baldwin <jhb@freebsd.org> Cc: freebsd-arch@freebsd.org Subject: Re: RfC: fueword(9) and casueword(9) Message-ID: <20141027165557.GC1877@kib.kiev.ua> In-Reply-To: <2048849.GkvWliFbyg@ralph.baldwin.cx> References: <20141021094539.GA1877@kib.kiev.ua> <20141022002825.H2080@besplex.bde.org> <20141021162306.GE1877@kib.kiev.ua> <2048849.GkvWliFbyg@ralph.baldwin.cx>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Oct 27, 2014 at 11:17:51AM -0400, John Baldwin wrote:
> On Tuesday, October 21, 2014 07:23:06 PM Konstantin Belousov wrote:
> > On Wed, Oct 22, 2014 at 01:41:12AM +1100, Bruce Evans wrote:
> > > A new API should try to fix these __DEVOLATILE() abominations. I think it
> > > is safe, and even correct, to declare the pointers as volatile const void
> > > *, since the functions really can handle volatile data, unlike copyin().
> > >
> > > Atomic op functions are declared as taking pointers to volatile for
> > > similar reasons. Often they are applied to non-volatile data, but
> > > adding a qualifier is type-safe and doesn't cost efficiency since the
> > > pointer access is is not known to the compiler. (The last point is not
> > > so clear -- the compiler can see things in the functions since they are
> > > inline asm. fueword() isn't inline so its (in)efficiency is not changed.)
> > >
> > > The atomic read functions are not declared as taking pointers to const.
> > > The __DECONST() abomination might be used to work around this bug.
> >
> > I prefer to not complicate the fetch(9) KPI due to the mistakes in the
> > umtx structures definitions. I think that it is bug to mark the lock
> > words with volatile. I want the fueword(9) interface to be as much
> > similar to fuword(9), in particular, volatile seems to be not needed.
>
> I agree with Bruce here. casuword() already accepts volatile. I also
> think umtx is correct in marking the field as volatile. They are subject
> to change without the compiler's knowledge albeit by other threads
> rather than signal handlers. Having them marked volatile doesn't really
> matter for the kernel, but the header is also used in userland and is
> relevant in sem_new.c, etc.
You agree with making fueword() accept volatile const void * as the
address ? Or do you agree with the existence of the volatile type
qualifier for the lock field of umtx structures ?
I definitely do not want to make fueword() different from fuword() in
this aspect. If changing both fueword() and fuword() to take volatile
const * address, this should be different patch. At least because
that existing changes to kern_umtx.c are really complicated due to
changing very delicate logic, and I do not want to add unrelated and
splittable modifications to something which I expect to require
more debugging in the wild.
Below is the current version, which passed Peter' stress2 load on x86.
I also did smoke-testing on powerpc64. After make tinderbox finishes
successfully for the patch, I consider the change ready.
diff --git a/share/man/man9/Makefile b/share/man/man9/Makefile
index bc21dc6..fb63e78 100644
--- a/share/man/man9/Makefile
+++ b/share/man/man9/Makefile
@@ -581,6 +581,9 @@ MLINKS+=condvar.9 cv_broadcast.9 \
MLINKS+=config_intrhook.9 config_intrhook_disestablish.9 \
config_intrhook.9 config_intrhook_establish.9
MLINKS+=contigmalloc.9 contigfree.9
+MLINKS+=casuword.9 casueword.9 \
+ casuword.9 casueword32.9 \
+ casuword.9 casuword32.9
MLINKS+=copy.9 copyin.9 \
copy.9 copyin_nofault.9 \
copy.9 copyinstr.9 \
@@ -688,7 +691,10 @@ MLINKS+=fetch.9 fubyte.9 \
fetch.9 fuword.9 \
fetch.9 fuword16.9 \
fetch.9 fuword32.9 \
- fetch.9 fuword64.9
+ fetch.9 fuword64.9 \
+ fetch.9 fueword.9 \
+ fetch.9 fueword32.9 \
+ fetch.9 fueword64.9
MLINKS+=firmware.9 firmware_get.9 \
firmware.9 firmware_put.9 \
firmware.9 firmware_register.9 \
diff --git a/share/man/man9/casuword.9 b/share/man/man9/casuword.9
new file mode 100644
index 0000000..34a0f1d
--- /dev/null
+++ b/share/man/man9/casuword.9
@@ -0,0 +1,95 @@
+.\" Copyright (c) 2014 The FreeBSD Foundation
+.\" All rights reserved.
+.\"
+.\" Part of this documentation was written by
+.\" Konstantin Belousov <kib@FreeBSD.org> under sponsorship
+.\" from the FreeBSD Foundation.
+.\"
+.\" Redistribution and use in source and binary forms, with or without
+.\" modification, are permitted provided that the following conditions
+.\" are met:
+.\" 1. Redistributions of source code must retain the above copyright
+.\" notice, this list of conditions and the following disclaimer.
+.\" 2. Redistributions in binary form must reproduce the above copyright
+.\" notice, this list of conditions and the following disclaimer in the
+.\" documentation and/or other materials provided with the distribution.
+.\"
+.\" THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND
+.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE
+.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+.\" SUCH DAMAGE.
+.\"
+.\" $FreeBSD$
+.\"
+.Dd October 21, 2014
+.Dt CASU 9
+.Os
+.Sh NAME
+.Nm casueword ,
+.Nm casueword32 ,
+.Nm casuword ,
+.Nm casuword32
+.Nd fetch, compare and store data from user-space
+.Sh SYNOPSIS
+.In sys/types.h
+.In sys/systm.h
+.Ft int
+.Fn casueword "volatile u_long *base" "u_long oldval" "u_long *oldvalp" "u_long newval"
+.Ft int
+.Fn casueword32 "volatile uint32_t *base" "uint32_t oldval" "uint32_t *oldvalp" "uint32_t newval"
+.Ft u_long
+.Fn casuword "volatile u_long *base" "u_long oldval" "u_long newval"
+.Ft uint32_t
+.Fn casuword32 "volatile uint32_t *base" "uint32_t oldval" "uint32_t newval"
+.Sh DESCRIPTION
+The
+.Nm
+functions are designed to perform atomic compare-and-swap operation on
+the value in the usermode memory of the current process.
+.Pp
+The
+.Nm
+routines reads the value from user memory with address
+.Pa base ,
+and compare the value read with
+.Pa oldval .
+If the values are equal,
+.Pa newval
+is written to the
+.Pa *base .
+In case of
+.Fn casueword32
+and
+.Fn casueword ,
+old value is stored into the (kernel-mode) variable pointed by
+.Pa *oldvalp .
+The userspace value must be naturally aligned.
+.Pp
+The callers of
+.Fn casuword
+and
+.Fn casuword32
+functions cannot distinguish between -1 read from
+userspace and function failure.
+.Sh RETURN VALUES
+The
+.Fn casuword
+and
+.Fn casuword32
+functions return the data fetched or -1 on failure.
+The
+.Fn casueword
+and
+.Fn casueword32
+functions return 0 on success and -1 on failure.
+.Sh SEE ALSO
+.Xr atomic 9 ,
+.Xr fetch 9 ,
+.Xr store 9
diff --git a/share/man/man9/fetch.9 b/share/man/man9/fetch.9
index ccf6866..7e13cbc 100644
--- a/share/man/man9/fetch.9
+++ b/share/man/man9/fetch.9
@@ -34,7 +34,7 @@
.\"
.\" $FreeBSD$
.\"
-.Dd October 5, 2009
+.Dd October 21, 2014
.Dt FETCH 9
.Os
.Sh NAME
@@ -44,11 +44,13 @@
.Nm fuword ,
.Nm fuword16 ,
.Nm fuword32 ,
-.Nm fuword64
+.Nm fuword64 ,
+.Nm fueword ,
+.Nm fueword32 ,
+.Nm fueword64
.Nd fetch data from user-space
.Sh SYNOPSIS
.In sys/types.h
-.In sys/time.h
.In sys/systm.h
.Ft int
.Fn fubyte "const void *base"
@@ -60,27 +62,38 @@
.Fn fuword32 "const void *base"
.Ft int64_t
.Fn fuword64 "const void *base"
+.Ft long
+.Fn fueword "const void *base" "long *val"
+.Ft int32_t
+.Fn fueword32 "const void *base" "int32_t *val"
+.Ft int64_t
+.Fn fueword64 "const void *base" "int64_t *val"
.In sys/resourcevar.h
.Ft int
.Fn fuswintr "void *base"
.Sh DESCRIPTION
The
.Nm
-functions are designed to copy small amounts of data from user-space.
+functions are designed to copy small amounts of data from user-space
+of the current process.
+If read is successful, it is performed atomically.
+The data read must be naturally aligned.
.Pp
The
.Nm
routines provide the following functionality:
-.Bl -tag -width "fuswintr()"
+.Bl -tag -width "fueword32()"
.It Fn fubyte
Fetches a byte of data from the user-space address
.Pa base .
+The byte read is zero-extended into the results variable.
.It Fn fuword
-Fetches a word of data from the user-space address
+Fetches a word of data (long) from the user-space address
.Pa base .
.It Fn fuword16
Fetches 16 bits of data from the user-space address
.Pa base .
+The half-word read is zero-extended into the results variable.
.It Fn fuword32
Fetches 32 bits of data from the user-space address
.Pa base .
@@ -91,11 +104,46 @@ Fetches 64 bits of data from the user-space address
Fetches a short word of data from the user-space address
.Pa base .
This function is safe to call during an interrupt context.
+.It Fn fueword
+Fetches a word of data (long) from the user-space address
+.Pa base
+and stores the result in the variable pointed by
+.Pa val .
+.It Fn fueword32
+Fetches 32 bits of data from the user-space address
+.Pa base
+and stores the result in the variable pointed by
+.Pa val .
+.It Fn fueword64
+Fetches 64 bits of data from the user-space address
+.Pa base
+and stores the result in the variable pointed by
+.Pa val .
.El
+.Pp
+The callers of
+.Fn fuword ,
+.Fn fuword32
+and
+.Fn fuword64
+functions cannot distinguish between -1 read from
+userspace and function failure.
.Sh RETURN VALUES
The
-.Nm
+.Fn fubyte ,
+.Fn fuword ,
+.Fn fuword16 ,
+.Fn fuword32 ,
+.Fn fuword64 ,
+and
+.Fn fuswintr
functions return the data fetched or -1 on failure.
+The
+.Fn fueword ,
+.Fn fueword32
+and
+.Fn fueword64
+functions return 0 on success and -1 on failure.
.Sh SEE ALSO
.Xr copy 9 ,
.Xr store 9
diff --git a/sys/amd64/amd64/support.S b/sys/amd64/amd64/support.S
index 4897367..50e653d 100644
--- a/sys/amd64/amd64/support.S
+++ b/sys/amd64/amd64/support.S
@@ -312,12 +312,13 @@ copyin_fault:
END(copyin)
/*
- * casuword32. Compare and set user integer. Returns -1 or the current value.
- * dst = %rdi, old = %rsi, new = %rdx
+ * casueword32. Compare and set user integer. Returns -1 on fault,
+ * 0 if access was successful. Old value is written to *oldp.
+ * dst = %rdi, old = %esi, oldp = %rdx, new = %ecx
*/
-ENTRY(casuword32)
- movq PCPU(CURPCB),%rcx
- movq $fusufault,PCB_ONFAULT(%rcx)
+ENTRY(casueword32)
+ movq PCPU(CURPCB),%r8
+ movq $fusufault,PCB_ONFAULT(%r8)
movq $VM_MAXUSER_ADDRESS-4,%rax
cmpq %rax,%rdi /* verify address is valid */
@@ -327,26 +328,34 @@ ENTRY(casuword32)
#ifdef SMP
lock
#endif
- cmpxchgl %edx,(%rdi) /* new = %edx */
+ cmpxchgl %ecx,(%rdi) /* new = %ecx */
/*
* The old value is in %eax. If the store succeeded it will be the
* value we expected (old) from before the store, otherwise it will
- * be the current value.
+ * be the current value. Save %eax into %esi to prepare the return
+ * value.
*/
+ movl %eax,%esi
+ xorl %eax,%eax
+ movq %rax,PCB_ONFAULT(%r8)
- movq PCPU(CURPCB),%rcx
- movq $0,PCB_ONFAULT(%rcx)
+ /*
+ * Access the oldp after the pcb_onfault is cleared, to correctly
+ * catch corrupted pointer.
+ */
+ movl %esi,(%rdx) /* oldp = %rdx */
ret
-END(casuword32)
+END(casueword32)
/*
- * casuword. Compare and set user word. Returns -1 or the current value.
- * dst = %rdi, old = %rsi, new = %rdx
+ * casueword. Compare and set user long. Returns -1 on fault,
+ * 0 if access was successful. Old value is written to *oldp.
+ * dst = %rdi, old = %rsi, oldp = %rdx, new = %rcx
*/
-ENTRY(casuword)
- movq PCPU(CURPCB),%rcx
- movq $fusufault,PCB_ONFAULT(%rcx)
+ENTRY(casueword)
+ movq PCPU(CURPCB),%r8
+ movq $fusufault,PCB_ONFAULT(%r8)
movq $VM_MAXUSER_ADDRESS-4,%rax
cmpq %rax,%rdi /* verify address is valid */
@@ -356,28 +365,28 @@ ENTRY(casuword)
#ifdef SMP
lock
#endif
- cmpxchgq %rdx,(%rdi) /* new = %rdx */
+ cmpxchgq %rcx,(%rdi) /* new = %rcx */
/*
- * The old value is in %eax. If the store succeeded it will be the
+ * The old value is in %rax. If the store succeeded it will be the
* value we expected (old) from before the store, otherwise it will
* be the current value.
*/
-
- movq PCPU(CURPCB),%rcx
- movq $fusufault,PCB_ONFAULT(%rcx)
- movq $0,PCB_ONFAULT(%rcx)
+ movq %rax,%rsi
+ xorl %eax,%eax
+ movq %rax,PCB_ONFAULT(%r8)
+ movq %rsi,(%rdx)
ret
-END(casuword)
+END(casueword)
/*
* Fetch (load) a 64-bit word, a 32-bit word, a 16-bit word, or an 8-bit
- * byte from user memory. All these functions are MPSAFE.
- * addr = %rdi
+ * byte from user memory.
+ * addr = %rdi, valp = %rsi
*/
-ALTENTRY(fuword64)
-ENTRY(fuword)
+ALTENTRY(fueword64)
+ENTRY(fueword)
movq PCPU(CURPCB),%rcx
movq $fusufault,PCB_ONFAULT(%rcx)
@@ -385,13 +394,15 @@ ENTRY(fuword)
cmpq %rax,%rdi /* verify address is valid */
ja fusufault
- movq (%rdi),%rax
- movq $0,PCB_ONFAULT(%rcx)
+ xorl %eax,%eax
+ movq (%rdi),%r11
+ movq %rax,PCB_ONFAULT(%rcx)
+ movq %r11,(%rsi)
ret
END(fuword64)
END(fuword)
-ENTRY(fuword32)
+ENTRY(fueword32)
movq PCPU(CURPCB),%rcx
movq $fusufault,PCB_ONFAULT(%rcx)
@@ -399,10 +410,12 @@ ENTRY(fuword32)
cmpq %rax,%rdi /* verify address is valid */
ja fusufault
- movl (%rdi),%eax
- movq $0,PCB_ONFAULT(%rcx)
+ xorl %eax,%eax
+ movl (%rdi),%r11d
+ movq %rax,PCB_ONFAULT(%rcx)
+ movl %r11d,(%rsi)
ret
-END(fuword32)
+END(fueword32)
/*
* fuswintr() and suswintr() are specialized variants of fuword16() and
diff --git a/sys/amd64/ia32/ia32_syscall.c b/sys/amd64/ia32/ia32_syscall.c
index 0cdec6f..92249f9 100644
--- a/sys/amd64/ia32/ia32_syscall.c
+++ b/sys/amd64/ia32/ia32_syscall.c
@@ -110,7 +110,7 @@ ia32_fetch_syscall_args(struct thread *td, struct syscall_args *sa)
struct proc *p;
struct trapframe *frame;
caddr_t params;
- u_int32_t args[8];
+ u_int32_t args[8], tmp;
int error, i;
p = td->td_proc;
@@ -126,7 +126,10 @@ ia32_fetch_syscall_args(struct thread *td, struct syscall_args *sa)
/*
* Code is first argument, followed by actual args.
*/
- sa->code = fuword32(params);
+ error = fueword32(params, &tmp);
+ if (error == -1)
+ return (EFAULT);
+ sa->code = tmp;
params += sizeof(int);
} else if (sa->code == SYS___syscall) {
/*
@@ -135,7 +138,10 @@ ia32_fetch_syscall_args(struct thread *td, struct syscall_args *sa)
* We use a 32-bit fetch in case params is not
* aligned.
*/
- sa->code = fuword32(params);
+ error = fueword32(params, &tmp);
+ if (error == -1)
+ return (EFAULT);
+ sa->code = tmp;
params += sizeof(quad_t);
}
if (p->p_sysent->sv_mask)
diff --git a/sys/arm/include/param.h b/sys/arm/include/param.h
index 4a64607..6267154 100644
--- a/sys/arm/include/param.h
+++ b/sys/arm/include/param.h
@@ -149,4 +149,8 @@
#define pgtok(x) ((x) * (PAGE_SIZE / 1024))
+#ifdef _KERNEL
+#define NO_FUEWORD 1
+#endif
+
#endif /* !_ARM_INCLUDE_PARAM_H_ */
diff --git a/sys/compat/freebsd32/freebsd32_misc.c b/sys/compat/freebsd32/freebsd32_misc.c
index 8ec949f..5ea062e 100644
--- a/sys/compat/freebsd32/freebsd32_misc.c
+++ b/sys/compat/freebsd32/freebsd32_misc.c
@@ -1832,16 +1832,21 @@ freebsd32_sysctl(struct thread *td, struct freebsd32_sysctl_args *uap)
{
int error, name[CTL_MAXNAME];
size_t j, oldlen;
+ uint32_t tmp;
if (uap->namelen > CTL_MAXNAME || uap->namelen < 2)
return (EINVAL);
error = copyin(uap->name, name, uap->namelen * sizeof(int));
if (error)
return (error);
- if (uap->oldlenp)
- oldlen = fuword32(uap->oldlenp);
- else
+ if (uap->oldlenp) {
+ error = fueword32(uap->oldlenp, &tmp);
+ oldlen = tmp;
+ } else {
oldlen = 0;
+ }
+ if (error != 0)
+ return (EFAULT);
error = userland_sysctl(td, name, uap->namelen,
uap->old, &oldlen, 1,
uap->new, uap->newlen, &j, SCTL_MASK32);
diff --git a/sys/i386/i386/support.s b/sys/i386/i386/support.s
index c126f78..0a08012 100644
--- a/sys/i386/i386/support.s
+++ b/sys/i386/i386/support.s
@@ -389,16 +389,16 @@ copyin_fault:
ret
/*
- * casuword. Compare and set user word. Returns -1 or the current value.
+ * casueword. Compare and set user word. Returns -1 on fault,
+ * 0 on non-faulting access. The current value is in *oldp.
*/
-
-ALTENTRY(casuword32)
-ENTRY(casuword)
+ALTENTRY(casueword32)
+ENTRY(casueword)
movl PCPU(CURPCB),%ecx
movl $fusufault,PCB_ONFAULT(%ecx)
movl 4(%esp),%edx /* dst */
movl 8(%esp),%eax /* old */
- movl 12(%esp),%ecx /* new */
+ movl 16(%esp),%ecx /* new */
cmpl $VM_MAXUSER_ADDRESS-4,%edx /* verify address is valid */
ja fusufault
@@ -416,17 +416,20 @@ ENTRY(casuword)
movl PCPU(CURPCB),%ecx
movl $0,PCB_ONFAULT(%ecx)
+ movl 12(%esp),%edx /* oldp */
+ movl %eax,(%edx)
+ xorl %eax,%eax
ret
-END(casuword32)
-END(casuword)
+END(casueword32)
+END(casueword)
/*
* Fetch (load) a 32-bit word, a 16-bit word, or an 8-bit byte from user
- * memory. All these functions are MPSAFE.
+ * memory.
*/
-ALTENTRY(fuword32)
-ENTRY(fuword)
+ALTENTRY(fueword32)
+ENTRY(fueword)
movl PCPU(CURPCB),%ecx
movl $fusufault,PCB_ONFAULT(%ecx)
movl 4(%esp),%edx /* from */
@@ -436,9 +439,12 @@ ENTRY(fuword)
movl (%edx),%eax
movl $0,PCB_ONFAULT(%ecx)
+ movl 8(%esp),%edx
+ movl %eax,(%edx)
+ xorl %eax,%eax
ret
-END(fuword32)
-END(fuword)
+END(fueword32)
+END(fueword)
/*
* fuswintr() and suswintr() are specialized variants of fuword16() and
diff --git a/sys/i386/i386/trap.c b/sys/i386/i386/trap.c
index 1d0d104..84d6ec3 100644
--- a/sys/i386/i386/trap.c
+++ b/sys/i386/i386/trap.c
@@ -1059,6 +1059,7 @@ cpu_fetch_syscall_args(struct thread *td, struct syscall_args *sa)
struct proc *p;
struct trapframe *frame;
caddr_t params;
+ long tmp;
int error;
p = td->td_proc;
@@ -1074,14 +1075,20 @@ cpu_fetch_syscall_args(struct thread *td, struct syscall_args *sa)
/*
* Code is first argument, followed by actual args.
*/
- sa->code = fuword(params);
+ error = fueword(params, &tmp);
+ if (error == -1)
+ return (EFAULT);
+ sa->code = tmp;
params += sizeof(int);
} else if (sa->code == SYS___syscall) {
/*
* Like syscall, but code is a quad, so as to maintain
* quad alignment for the rest of the arguments.
*/
- sa->code = fuword(params);
+ error = fueword(params, &tmp);
+ if (error == -1)
+ return (EFAULT);
+ sa->code = tmp;
params += sizeof(quad_t);
}
diff --git a/sys/kern/kern_exec.c b/sys/kern/kern_exec.c
index 09212c8..45d4c6f 100644
--- a/sys/kern/kern_exec.c
+++ b/sys/kern/kern_exec.c
@@ -1091,7 +1091,7 @@ int
exec_copyin_args(struct image_args *args, char *fname,
enum uio_seg segflg, char **argv, char **envv)
{
- char *argp, *envp;
+ u_long argp, envp;
int error;
size_t length;
@@ -1127,13 +1127,17 @@ exec_copyin_args(struct image_args *args, char *fname,
/*
* extract arguments first
*/
- while ((argp = (caddr_t) (intptr_t) fuword(argv++))) {
- if (argp == (caddr_t) -1) {
+ for (;;) {
+ error = fueword(argv++, &argp);
+ if (error == -1) {
error = EFAULT;
goto err_exit;
}
- if ((error = copyinstr(argp, args->endp,
- args->stringspace, &length))) {
+ if (argp == 0)
+ break;
+ error = copyinstr((void *)(uintptr_t)argp, args->endp,
+ args->stringspace, &length);
+ if (error != 0) {
if (error == ENAMETOOLONG)
error = E2BIG;
goto err_exit;
@@ -1149,13 +1153,17 @@ exec_copyin_args(struct image_args *args, char *fname,
* extract environment strings
*/
if (envv) {
- while ((envp = (caddr_t)(intptr_t)fuword(envv++))) {
- if (envp == (caddr_t)-1) {
+ for (;;) {
+ error = fueword(envv++, &envp);
+ if (error == -1) {
error = EFAULT;
goto err_exit;
}
- if ((error = copyinstr(envp, args->endp,
- args->stringspace, &length))) {
+ if (envp == 0)
+ break;
+ error = copyinstr((void *)(uintptr_t)envp,
+ args->endp, args->stringspace, &length);
+ if (error != 0) {
if (error == ENAMETOOLONG)
error = E2BIG;
goto err_exit;
diff --git a/sys/kern/kern_umtx.c b/sys/kern/kern_umtx.c
index c815e36..58e76bc 100644
--- a/sys/kern/kern_umtx.c
+++ b/sys/kern/kern_umtx.c
@@ -510,6 +510,15 @@ umtxq_unbusy(struct umtx_key *key)
wakeup_one(uc);
}
+static inline void
+umtxq_unbusy_unlocked(struct umtx_key *key)
+{
+
+ umtxq_lock(key);
+ umtxq_unbusy(key);
+ umtxq_unlock(key);
+}
+
static struct umtxq_queue *
umtxq_queue_lookup(struct umtx_key *key, int q)
{
@@ -847,6 +856,7 @@ do_wait(struct thread *td, void *addr, u_long id,
struct abs_timeout timo;
struct umtx_q *uq;
u_long tmp;
+ uint32_t tmp32;
int error = 0;
uq = td->td_umtxq;
@@ -860,18 +870,29 @@ do_wait(struct thread *td, void *addr, u_long id,
umtxq_lock(&uq->uq_key);
umtxq_insert(uq);
umtxq_unlock(&uq->uq_key);
- if (compat32 == 0)
- tmp = fuword(addr);
- else
- tmp = (unsigned int)fuword32(addr);
+ if (compat32 == 0) {
+ error = fueword(addr, &tmp);
+ if (error != 0)
+ error = EFAULT;
+ } else {
+ error = fueword32(addr, &tmp32);
+ if (error == 0)
+ tmp = tmp32;
+ else
+ error = EFAULT;
+ }
umtxq_lock(&uq->uq_key);
- if (tmp == id)
- error = umtxq_sleep(uq, "uwait", timeout == NULL ?
- NULL : &timo);
- if ((uq->uq_flags & UQF_UMTXQ) == 0)
- error = 0;
- else
+ if (error == 0) {
+ if (tmp == id)
+ error = umtxq_sleep(uq, "uwait", timeout == NULL ?
+ NULL : &timo);
+ if ((uq->uq_flags & UQF_UMTXQ) == 0)
+ error = 0;
+ else
+ umtxq_remove(uq);
+ } else if ((uq->uq_flags & UQF_UMTXQ) != 0) {
umtxq_remove(uq);
+ }
umtxq_unlock(&uq->uq_key);
umtx_key_release(&uq->uq_key);
if (error == ERESTART)
@@ -908,11 +929,11 @@ do_lock_normal(struct thread *td, struct umutex *m, uint32_t flags,
struct abs_timeout timo;
struct umtx_q *uq;
uint32_t owner, old, id;
- int error = 0;
+ int error, rv;
id = td->td_tid;
uq = td->td_umtxq;
-
+ error = 0;
if (timeout != NULL)
abs_timeout_init2(&timo, timeout);
@@ -921,7 +942,9 @@ do_lock_normal(struct thread *td, struct umutex *m, uint32_t flags,
* can fault on any access.
*/
for (;;) {
- owner = fuword32(__DEVOLATILE(void *, &m->m_owner));
+ rv = fueword32(__DEVOLATILE(void *, &m->m_owner), &owner);
+ if (rv == -1)
+ return (EFAULT);
if (mode == _UMUTEX_WAIT) {
if (owner == UMUTEX_UNOWNED || owner == UMUTEX_CONTESTED)
return (0);
@@ -929,31 +952,31 @@ do_lock_normal(struct thread *td, struct umutex *m, uint32_t flags,
/*
* Try the uncontested case. This should be done in userland.
*/
- owner = casuword32(&m->m_owner, UMUTEX_UNOWNED, id);
+ rv = casueword32(&m->m_owner, UMUTEX_UNOWNED,
+ &owner, id);
+ /* The address was invalid. */
+ if (rv == -1)
+ return (EFAULT);
/* The acquire succeeded. */
if (owner == UMUTEX_UNOWNED)
return (0);
- /* The address was invalid. */
- if (owner == -1)
- return (EFAULT);
-
/* If no one owns it but it is contested try to acquire it. */
if (owner == UMUTEX_CONTESTED) {
- owner = casuword32(&m->m_owner,
- UMUTEX_CONTESTED, id | UMUTEX_CONTESTED);
+ rv = casueword32(&m->m_owner,
+ UMUTEX_CONTESTED, &owner,
+ id | UMUTEX_CONTESTED);
+ /* The address was invalid. */
+ if (rv == -1)
+ return (EFAULT);
if (owner == UMUTEX_CONTESTED)
return (0);
- /* The address was invalid. */
- if (owner == -1)
- return (EFAULT);
-
- error = umtxq_check_susp(td);
- if (error != 0)
- return (error);
+ rv = umtxq_check_susp(td);
+ if (rv != 0)
+ return (rv);
/* If this failed the lock has changed, restart. */
continue;
@@ -985,10 +1008,11 @@ do_lock_normal(struct thread *td, struct umutex *m, uint32_t flags,
* either some one else has acquired the lock or it has been
* released.
*/
- old = casuword32(&m->m_owner, owner, owner | UMUTEX_CONTESTED);
+ rv = casueword32(&m->m_owner, owner, &old,
+ owner | UMUTEX_CONTESTED);
/* The address was invalid. */
- if (old == -1) {
+ if (rv == -1) {
umtxq_lock(&uq->uq_key);
umtxq_remove(uq);
umtxq_unbusy(&uq->uq_key);
@@ -1033,16 +1057,16 @@ do_unlock_normal(struct thread *td, struct umutex *m, uint32_t flags)
/*
* Make sure we own this mtx.
*/
- owner = fuword32(__DEVOLATILE(uint32_t *, &m->m_owner));
- if (owner == -1)
+ error = fueword32(__DEVOLATILE(uint32_t *, &m->m_owner), &owner);
+ if (error == -1)
return (EFAULT);
if ((owner & ~UMUTEX_CONTESTED) != id)
return (EPERM);
if ((owner & UMUTEX_CONTESTED) == 0) {
- old = casuword32(&m->m_owner, owner, UMUTEX_UNOWNED);
- if (old == -1)
+ error = casueword32(&m->m_owner, owner, &old, UMUTEX_UNOWNED);
+ if (error == -1)
return (EFAULT);
if (old == owner)
return (0);
@@ -1064,14 +1088,14 @@ do_unlock_normal(struct thread *td, struct umutex *m, uint32_t flags)
* there is zero or one thread only waiting for it.
* Otherwise, it must be marked as contested.
*/
- old = casuword32(&m->m_owner, owner,
- count <= 1 ? UMUTEX_UNOWNED : UMUTEX_CONTESTED);
+ error = casueword32(&m->m_owner, owner, &old,
+ count <= 1 ? UMUTEX_UNOWNED : UMUTEX_CONTESTED);
umtxq_lock(&key);
umtxq_signal(&key,1);
umtxq_unbusy(&key);
umtxq_unlock(&key);
umtx_key_release(&key);
- if (old == -1)
+ if (error == -1)
return (EFAULT);
if (old != owner)
return (EINVAL);
@@ -1091,14 +1115,16 @@ do_wake_umutex(struct thread *td, struct umutex *m)
int error;
int count;
- owner = fuword32(__DEVOLATILE(uint32_t *, &m->m_owner));
- if (owner == -1)
+ error = fueword32(__DEVOLATILE(uint32_t *, &m->m_owner), &owner);
+ if (error == -1)
return (EFAULT);
if ((owner & ~UMUTEX_CONTESTED) != 0)
return (0);
- flags = fuword32(&m->m_flags);
+ error = fueword32(&m->m_flags, &flags);
+ if (error == -1)
+ return (EFAULT);
/* We should only ever be in here for contested locks */
if ((error = umtx_key_get(m, TYPE_NORMAL_UMUTEX, GET_SHARE(flags),
@@ -1110,16 +1136,20 @@ do_wake_umutex(struct thread *td, struct umutex *m)
count = umtxq_count(&key);
umtxq_unlock(&key);
- if (count <= 1)
- owner = casuword32(&m->m_owner, UMUTEX_CONTESTED, UMUTEX_UNOWNED);
+ if (count <= 1) {
+ error = casueword32(&m->m_owner, UMUTEX_CONTESTED, &owner,
+ UMUTEX_UNOWNED);
+ if (error == -1)
+ error = EFAULT;
+ }
umtxq_lock(&key);
- if (count != 0 && (owner & ~UMUTEX_CONTESTED) == 0)
+ if (error == 0 && count != 0 && (owner & ~UMUTEX_CONTESTED) == 0)
umtxq_signal(&key, 1);
umtxq_unbusy(&key);
umtxq_unlock(&key);
umtx_key_release(&key);
- return (0);
+ return (error);
}
/*
@@ -1162,41 +1192,49 @@ do_wake2_umutex(struct thread *td, struct umutex *m, uint32_t flags)
* any memory.
*/
if (count > 1) {
- owner = fuword32(__DEVOLATILE(uint32_t *, &m->m_owner));
- while ((owner & UMUTEX_CONTESTED) ==0) {
- old = casuword32(&m->m_owner, owner,
- owner|UMUTEX_CONTESTED);
+ error = fueword32(__DEVOLATILE(uint32_t *, &m->m_owner),
+ &owner);
+ if (error == -1)
+ error = EFAULT;
+ while (error == 0 && (owner & UMUTEX_CONTESTED) == 0) {
+ error = casueword32(&m->m_owner, owner, &old,
+ owner | UMUTEX_CONTESTED);
+ if (error == -1) {
+ error = EFAULT;
+ break;
+ }
if (old == owner)
break;
owner = old;
- if (old == -1)
- break;
error = umtxq_check_susp(td);
if (error != 0)
break;
}
} else if (count == 1) {
- owner = fuword32(__DEVOLATILE(uint32_t *, &m->m_owner));
- while ((owner & ~UMUTEX_CONTESTED) != 0 &&
+ error = fueword32(__DEVOLATILE(uint32_t *, &m->m_owner),
+ &owner);
+ if (error == -1)
+ error = EFAULT;
+ while (error == 0 && (owner & ~UMUTEX_CONTESTED) != 0 &&
(owner & UMUTEX_CONTESTED) == 0) {
- old = casuword32(&m->m_owner, owner,
- owner|UMUTEX_CONTESTED);
+ error = casueword32(&m->m_owner, owner, &old,
+ owner | UMUTEX_CONTESTED);
+ if (error == -1) {
+ error = EFAULT;
+ break;
+ }
if (old == owner)
break;
owner = old;
- if (old == -1)
- break;
error = umtxq_check_susp(td);
if (error != 0)
break;
}
}
umtxq_lock(&key);
- if (owner == -1) {
- error = EFAULT;
+ if (error == EFAULT) {
umtxq_signal(&key, INT_MAX);
- }
- else if (count != 0 && (owner & ~UMUTEX_CONTESTED) == 0)
+ } else if (count != 0 && (owner & ~UMUTEX_CONTESTED) == 0)
umtxq_signal(&key, 1);
umtxq_unbusy(&key);
umtxq_unlock(&key);
@@ -1576,7 +1614,7 @@ do_lock_pi(struct thread *td, struct umutex *m, uint32_t flags,
struct umtx_q *uq;
struct umtx_pi *pi, *new_pi;
uint32_t id, owner, old;
- int error;
+ int error, rv;
id = td->td_tid;
uq = td->td_umtxq;
@@ -1619,7 +1657,12 @@ do_lock_pi(struct thread *td, struct umutex *m, uint32_t flags,
/*
* Try the uncontested case. This should be done in userland.
*/
- owner = casuword32(&m->m_owner, UMUTEX_UNOWNED, id);
+ rv = casueword32(&m->m_owner, UMUTEX_UNOWNED, &owner, id);
+ /* The address was invalid. */
+ if (rv == -1) {
+ error = EFAULT;
+ break;
+ }
/* The acquire succeeded. */
if (owner == UMUTEX_UNOWNED) {
@@ -1627,16 +1670,15 @@ do_lock_pi(struct thread *td, struct umutex *m, uint32_t flags,
break;
}
- /* The address was invalid. */
- if (owner == -1) {
- error = EFAULT;
- break;
- }
-
/* If no one owns it but it is contested try to acquire it. */
if (owner == UMUTEX_CONTESTED) {
- owner = casuword32(&m->m_owner,
- UMUTEX_CONTESTED, id | UMUTEX_CONTESTED);
+ rv = casueword32(&m->m_owner,
+ UMUTEX_CONTESTED, &owner, id | UMUTEX_CONTESTED);
+ /* The address was invalid. */
+ if (rv == -1) {
+ error = EFAULT;
+ break;
+ }
if (owner == UMUTEX_CONTESTED) {
umtxq_lock(&uq->uq_key);
@@ -1647,12 +1689,6 @@ do_lock_pi(struct thread *td, struct umutex *m, uint32_t flags,
break;
}
- /* The address was invalid. */
- if (owner == -1) {
- error = EFAULT;
- break;
- }
-
error = umtxq_check_susp(td);
if (error != 0)
break;
@@ -1683,13 +1719,12 @@ do_lock_pi(struct thread *td, struct umutex *m, uint32_t flags,
* either some one else has acquired the lock or it has been
* released.
*/
- old = casuword32(&m->m_owner, owner, owner | UMUTEX_CONTESTED);
+ rv = casueword32(&m->m_owner, owner, &old,
+ owner | UMUTEX_CONTESTED);
/* The address was invalid. */
- if (old == -1) {
- umtxq_lock(&uq->uq_key);
- umtxq_unbusy(&uq->uq_key);
- umtxq_unlock(&uq->uq_key);
+ if (rv == -1) {
+ umtxq_unbusy_unlocked(&uq->uq_key);
error = EFAULT;
break;
}
@@ -1741,8 +1776,8 @@ do_unlock_pi(struct thread *td, struct umutex *m, uint32_t flags)
/*
* Make sure we own this mtx.
*/
- owner = fuword32(__DEVOLATILE(uint32_t *, &m->m_owner));
- if (owner == -1)
+ error = fueword32(__DEVOLATILE(uint32_t *, &m->m_owner), &owner);
+ if (error == -1)
return (EFAULT);
if ((owner & ~UMUTEX_CONTESTED) != id)
@@ -1750,8 +1785,8 @@ do_unlock_pi(struct thread *td, struct umutex *m, uint32_t flags)
/* This should be done in userland */
if ((owner & UMUTEX_CONTESTED) == 0) {
- old = casuword32(&m->m_owner, owner, UMUTEX_UNOWNED);
- if (old == -1)
+ error = casueword32(&m->m_owner, owner, &old, UMUTEX_UNOWNED);
+ if (error == -1)
return (EFAULT);
if (old == owner)
return (0);
@@ -1809,14 +1844,12 @@ do_unlock_pi(struct thread *td, struct umutex *m, uint32_t flags)
* there is zero or one thread only waiting for it.
* Otherwise, it must be marked as contested.
*/
- old = casuword32(&m->m_owner, owner,
- count <= 1 ? UMUTEX_UNOWNED : UMUTEX_CONTESTED);
+ error = casueword32(&m->m_owner, owner, &old,
+ count <= 1 ? UMUTEX_UNOWNED : UMUTEX_CONTESTED);
- umtxq_lock(&key);
- umtxq_unbusy(&key);
- umtxq_unlock(&key);
+ umtxq_unbusy_unlocked(&key);
umtx_key_release(&key);
- if (old == -1)
+ if (error == -1)
return (EFAULT);
if (old != owner)
return (EINVAL);
@@ -1835,7 +1868,7 @@ do_lock_pp(struct thread *td, struct umutex *m, uint32_t flags,
struct umtx_pi *pi;
uint32_t ceiling;
uint32_t owner, id;
- int error, pri, old_inherited_pri, su;
+ int error, pri, old_inherited_pri, su, rv;
id = td->td_tid;
uq = td->td_umtxq;
@@ -1853,7 +1886,12 @@ do_lock_pp(struct thread *td, struct umutex *m, uint32_t flags,
umtxq_busy(&uq->uq_key);
umtxq_unlock(&uq->uq_key);
- ceiling = RTP_PRIO_MAX - fuword32(&m->m_ceilings[0]);
+ rv = fueword32(&m->m_ceilings[0], &ceiling);
+ if (rv == -1) {
+ error = EFAULT;
+ goto out;
+ }
+ ceiling = RTP_PRIO_MAX - ceiling;
if (ceiling > RTP_PRIO_MAX) {
error = EINVAL;
goto out;
@@ -1874,17 +1912,16 @@ do_lock_pp(struct thread *td, struct umutex *m, uint32_t flags,
}
mtx_unlock_spin(&umtx_lock);
- owner = casuword32(&m->m_owner,
- UMUTEX_CONTESTED, id | UMUTEX_CONTESTED);
-
- if (owner == UMUTEX_CONTESTED) {
- error = 0;
+ rv = casueword32(&m->m_owner,
+ UMUTEX_CONTESTED, &owner, id | UMUTEX_CONTESTED);
+ /* The address was invalid. */
+ if (rv == -1) {
+ error = EFAULT;
break;
}
- /* The address was invalid. */
- if (owner == -1) {
- error = EFAULT;
+ if (owner == UMUTEX_CONTESTED) {
+ error = 0;
break;
}
@@ -1946,9 +1983,7 @@ do_lock_pp(struct thread *td, struct umutex *m, uint32_t flags,
}
out:
- umtxq_lock(&uq->uq_key);
- umtxq_unbusy(&uq->uq_key);
- umtxq_unlock(&uq->uq_key);
+ umtxq_unbusy_unlocked(&uq->uq_key);
umtx_key_release(&uq->uq_key);
return (error);
}
@@ -1973,8 +2008,8 @@ do_unlock_pp(struct thread *td, struct umutex *m, uint32_t flags)
/*
* Make sure we own this mtx.
*/
- owner = fuword32(__DEVOLATILE(uint32_t *, &m->m_owner));
- if (owner == -1)
+ error = fueword32(__DEVOLATILE(uint32_t *, &m->m_owner), &owner);
+ if (error == -1)
return (EFAULT);
if ((owner & ~UMUTEX_CONTESTED) != id)
@@ -2047,9 +2082,11 @@ do_set_ceiling(struct thread *td, struct umutex *m, uint32_t ceiling,
uint32_t save_ceiling;
uint32_t owner, id;
uint32_t flags;
- int error;
+ int error, rv;
- flags = fuword32(&m->m_flags);
+ error = fueword32(&m->m_flags, &flags);
+ if (error == -1)
+ return (EFAULT);
if ((flags & UMUTEX_PRIO_PROTECT) == 0)
return (EINVAL);
if (ceiling > RTP_PRIO_MAX)
@@ -2064,10 +2101,18 @@ do_set_ceiling(struct thread *td, struct umutex *m, uint32_t ceiling,
umtxq_busy(&uq->uq_key);
umtxq_unlock(&uq->uq_key);
- save_ceiling = fuword32(&m->m_ceilings[0]);
+ rv = fueword32(&m->m_ceilings[0], &save_ceiling);
+ if (rv == -1) {
+ error = EFAULT;
+ break;
+ }
- owner = casuword32(&m->m_owner,
- UMUTEX_CONTESTED, id | UMUTEX_CONTESTED);
+ rv = casueword32(&m->m_owner,
+ UMUTEX_CONTESTED, &owner, id | UMUTEX_CONTESTED);
+ if (rv == -1) {
+ error = EFAULT;
+ break;
+ }
if (owner == UMUTEX_CONTESTED) {
suword32(&m->m_ceilings[0], ceiling);
@@ -2077,12 +2122,6 @@ do_set_ceiling(struct thread *td, struct umutex *m, uint32_t ceiling,
break;
}
- /* The address was invalid. */
- if (owner == -1) {
- error = EFAULT;
- break;
- }
-
if ((owner & ~UMUTEX_CONTESTED) == id) {
suword32(&m->m_ceilings[0], ceiling);
error = 0;
@@ -2129,8 +2168,8 @@ do_lock_umutex(struct thread *td, struct umutex *m,
uint32_t flags;
int error;
- flags = fuword32(&m->m_flags);
- if (flags == -1)
+ error = fueword32(&m->m_flags, &flags);
+ if (error == -1)
return (EFAULT);
switch(flags & (UMUTEX_PRIO_INHERIT | UMUTEX_PRIO_PROTECT)) {
@@ -2164,9 +2203,10 @@ static int
do_unlock_umutex(struct thread *td, struct umutex *m)
{
uint32_t flags;
+ int error;
- flags = fuword32(&m->m_flags);
- if (flags == -1)
+ error = fueword32(&m->m_flags, &flags);
+ if (error == -1)
return (EFAULT);
switch(flags & (UMUTEX_PRIO_INHERIT | UMUTEX_PRIO_PROTECT)) {
@@ -2187,21 +2227,27 @@ do_cv_wait(struct thread *td, struct ucond *cv, struct umutex *m,
{
struct abs_timeout timo;
struct umtx_q *uq;
- uint32_t flags;
- uint32_t clockid;
+ uint32_t flags, clockid, hasw;
int error;
uq = td->td_umtxq;
- flags = fuword32(&cv->c_flags);
+ error = fueword32(&cv->c_flags, &flags);
+ if (error == -1)
+ return (EFAULT);
error = umtx_key_get(cv, TYPE_CV, GET_SHARE(flags), &uq->uq_key);
if (error != 0)
return (error);
if ((wflags & CVWAIT_CLOCKID) != 0) {
- clockid = fuword32(&cv->c_clockid);
+ error = fueword32(&cv->c_clockid, &clockid);
+ if (error == -1) {
+ umtx_key_release(&uq->uq_key);
+ return (EFAULT);
+ }
if (clockid < CLOCK_REALTIME ||
clockid >= CLOCK_THREAD_CPUTIME_ID) {
/* hmm, only HW clock id will work. */
+ umtx_key_release(&uq->uq_key);
return (EINVAL);
}
} else {
@@ -2217,12 +2263,12 @@ do_cv_wait(struct thread *td, struct ucond *cv, struct umutex *m,
* Set c_has_waiters to 1 before releasing user mutex, also
* don't modify cache line when unnecessary.
*/
- if (fuword32(__DEVOLATILE(uint32_t *, &cv->c_has_waiters)) == 0)
+ error = fueword32(__DEVOLATILE(uint32_t *, &cv->c_has_waiters),
+ &hasw);
+ if (error == 0 && hasw == 0)
suword32(__DEVOLATILE(uint32_t *, &cv->c_has_waiters), 1);
- umtxq_lock(&uq->uq_key);
- umtxq_unbusy(&uq->uq_key);
- umtxq_unlock(&uq->uq_key);
+ umtxq_unbusy_unlocked(&uq->uq_key);
error = do_unlock_umutex(td, m);
@@ -2276,7 +2322,9 @@ do_cv_signal(struct thread *td, struct ucond *cv)
int error, cnt, nwake;
uint32_t flags;
- flags = fuword32(&cv->c_flags);
+ error = fueword32(&cv->c_flags, &flags);
+ if (error == -1)
+ return (EFAULT);
if ((error = umtx_key_get(cv, TYPE_CV, GET_SHARE(flags), &key)) != 0)
return (error);
umtxq_lock(&key);
@@ -2287,6 +2335,8 @@ do_cv_signal(struct thread *td, struct ucond *cv)
umtxq_unlock(&key);
error = suword32(
__DEVOLATILE(uint32_t *, &cv->c_has_waiters), 0);
+ if (error == -1)
+ error = EFAULT;
umtxq_lock(&key);
}
umtxq_unbusy(&key);
@@ -2302,7 +2352,9 @@ do_cv_broadcast(struct thread *td, struct ucond *cv)
int error;
uint32_t flags;
- flags = fuword32(&cv->c_flags);
+ error = fueword32(&cv->c_flags, &flags);
+ if (error == -1)
+ return (EFAULT);
if ((error = umtx_key_get(cv, TYPE_CV, GET_SHARE(flags), &key)) != 0)
return (error);
@@ -2312,10 +2364,10 @@ do_cv_broadcast(struct thread *td, struct ucond *cv)
umtxq_unlock(&key);
error = suword32(__DEVOLATILE(uint32_t *, &cv->c_has_waiters), 0);
+ if (error == -1)
+ error = EFAULT;
- umtxq_lock(&key);
- umtxq_unbusy(&key);
- umtxq_unlock(&key);
+ umtxq_unbusy_unlocked(&key);
umtx_key_release(&key);
return (error);
@@ -2329,10 +2381,12 @@ do_rw_rdlock(struct thread *td, struct urwlock *rwlock, long fflag, struct _umtx
uint32_t flags, wrflags;
int32_t state, oldstate;
int32_t blocked_readers;
- int error;
+ int error, rv;
uq = td->td_umtxq;
- flags = fuword32(&rwlock->rw_flags);
+ error = fueword32(&rwlock->rw_flags, &flags);
+ if (error == -1)
+ return (EFAULT);
error = umtx_key_get(rwlock, TYPE_RWLOCK, GET_SHARE(flags), &uq->uq_key);
if (error != 0)
return (error);
@@ -2345,15 +2399,22 @@ do_rw_rdlock(struct thread *td, struct urwlock *rwlock, long fflag, struct _umtx
wrflags |= URWLOCK_WRITE_WAITERS;
for (;;) {
- state = fuword32(__DEVOLATILE(int32_t *, &rwlock->rw_state));
+ rv = fueword32(__DEVOLATILE(int32_t *, &rwlock->rw_state),
+ &state);
+ if (rv == -1) {
+ umtx_key_release(&uq->uq_key);
+ return (EFAULT);
+ }
+
/* try to lock it */
while (!(state & wrflags)) {
if (__predict_false(URWLOCK_READER_COUNT(state) == URWLOCK_MAX_READERS)) {
umtx_key_release(&uq->uq_key);
return (EAGAIN);
}
- oldstate = casuword32(&rwlock->rw_state, state, state + 1);
- if (oldstate == -1) {
+ rv = casueword32(&rwlock->rw_state, state,
+ &oldstate, state + 1);
+ if (rv == -1) {
umtx_key_release(&uq->uq_key);
return (EFAULT);
}
@@ -2379,12 +2440,17 @@ do_rw_rdlock(struct thread *td, struct urwlock *rwlock, long fflag, struct _umtx
* re-read the state, in case it changed between the try-lock above
* and the check below
*/
- state = fuword32(__DEVOLATILE(int32_t *, &rwlock->rw_state));
+ rv = fueword32(__DEVOLATILE(int32_t *, &rwlock->rw_state),
+ &state);
+ if (rv == -1)
+ error = EFAULT;
/* set read contention bit */
- while ((state & wrflags) && !(state & URWLOCK_READ_WAITERS)) {
- oldstate = casuword32(&rwlock->rw_state, state, state | URWLOCK_READ_WAITERS);
- if (oldstate == -1) {
+ while (error == 0 && (state & wrflags) &&
+ !(state & URWLOCK_READ_WAITERS)) {
+ rv = casueword32(&rwlock->rw_state, state,
+ &oldstate, state | URWLOCK_READ_WAITERS);
+ if (rv == -1) {
error = EFAULT;
break;
}
@@ -2396,17 +2462,13 @@ do_rw_rdlock(struct thread *td, struct urwlock *rwlock, long fflag, struct _umtx
break;
}
if (error != 0) {
- umtxq_lock(&uq->uq_key);
- umtxq_unbusy(&uq->uq_key);
- umtxq_unlock(&uq->uq_key);
+ umtxq_unbusy_unlocked(&uq->uq_key);
break;
}
/* state is changed while setting flags, restart */
if (!(state & wrflags)) {
- umtxq_lock(&uq->uq_key);
- umtxq_unbusy(&uq->uq_key);
- umtxq_unlock(&uq->uq_key);
+ umtxq_unbusy_unlocked(&uq->uq_key);
error = umtxq_check_susp(td);
if (error != 0)
break;
@@ -2415,7 +2477,13 @@ do_rw_rdlock(struct thread *td, struct urwlock *rwlock, long fflag, struct _umtx
sleep:
/* contention bit is set, before sleeping, increase read waiter count */
- blocked_readers = fuword32(&rwlock->rw_blocked_readers);
+ rv = fueword32(&rwlock->rw_blocked_readers,
+ &blocked_readers);
+ if (rv == -1) {
+ umtxq_unbusy_unlocked(&uq->uq_key);
+ error = EFAULT;
+ break;
+ }
suword32(&rwlock->rw_blocked_readers, blocked_readers+1);
while (state & wrflags) {
@@ -2431,18 +2499,32 @@ sleep:
umtxq_unlock(&uq->uq_key);
if (error)
break;
- state = fuword32(__DEVOLATILE(int32_t *, &rwlock->rw_state));
+ rv = fueword32(__DEVOLATILE(int32_t *,
+ &rwlock->rw_state), &state);
+ if (rv == -1) {
+ error = EFAULT;
+ break;
+ }
}
/* decrease read waiter count, and may clear read contention bit */
- blocked_readers = fuword32(&rwlock->rw_blocked_readers);
+ rv = fueword32(&rwlock->rw_blocked_readers,
+ &blocked_readers);
+ if (rv == -1) {
+ umtxq_unbusy_unlocked(&uq->uq_key);
+ error = EFAULT;
+ break;
+ }
suword32(&rwlock->rw_blocked_readers, blocked_readers-1);
if (blocked_readers == 1) {
- state = fuword32(__DEVOLATILE(int32_t *, &rwlock->rw_state));
- for (;;) {
- oldstate = casuword32(&rwlock->rw_state, state,
- state & ~URWLOCK_READ_WAITERS);
- if (oldstate == -1) {
+ rv = fueword32(__DEVOLATILE(int32_t *,
+ &rwlock->rw_state), &state);
+ if (rv == -1)
+ error = EFAULT;
+ while (error == 0) {
+ rv = casueword32(&rwlock->rw_state, state,
+ &oldstate, state & ~URWLOCK_READ_WAITERS);
+ if (rv == -1) {
error = EFAULT;
break;
}
@@ -2450,14 +2532,10 @@ sleep:
break;
state = oldstate;
error = umtxq_check_susp(td);
- if (error != 0)
- break;
}
}
- umtxq_lock(&uq->uq_key);
- umtxq_unbusy(&uq->uq_key);
- umtxq_unlock(&uq->uq_key);
+ umtxq_unbusy_unlocked(&uq->uq_key);
if (error != 0)
break;
}
@@ -2476,10 +2554,12 @@ do_rw_wrlock(struct thread *td, struct urwlock *rwlock, struct _umtx_time *timeo
int32_t state, oldstate;
int32_t blocked_writers;
int32_t blocked_readers;
- int error;
+ int error, rv;
uq = td->td_umtxq;
- flags = fuword32(&rwlock->rw_flags);
+ error = fueword32(&rwlock->rw_flags, &flags);
+ if (error == -1)
+ return (EFAULT);
error = umtx_key_get(rwlock, TYPE_RWLOCK, GET_SHARE(flags), &uq->uq_key);
if (error != 0)
return (error);
@@ -2489,10 +2569,16 @@ do_rw_wrlock(struct thread *td, struct urwlock *rwlock, struct _umtx_time *timeo
blocked_readers = 0;
for (;;) {
- state = fuword32(__DEVOLATILE(int32_t *, &rwlock->rw_state));
+ rv = fueword32(__DEVOLATILE(int32_t *, &rwlock->rw_state),
+ &state);
+ if (rv == -1) {
+ umtx_key_release(&uq->uq_key);
+ return (EFAULT);
+ }
while (!(state & URWLOCK_WRITE_OWNER) && URWLOCK_READER_COUNT(state) == 0) {
- oldstate = casuword32(&rwlock->rw_state, state, state | URWLOCK_WRITE_OWNER);
- if (oldstate == -1) {
+ rv = casueword32(&rwlock->rw_state, state,
+ &oldstate, state | URWLOCK_WRITE_OWNER);
+ if (rv == -1) {
umtx_key_release(&uq->uq_key);
return (EFAULT);
}
@@ -2528,12 +2614,17 @@ do_rw_wrlock(struct thread *td, struct urwlock *rwlock, struct _umtx_time *timeo
* re-read the state, in case it changed between the try-lock above
* and the check below
*/
- state = fuword32(__DEVOLATILE(int32_t *, &rwlock->rw_state));
+ rv = fueword32(__DEVOLATILE(int32_t *, &rwlock->rw_state),
+ &state);
+ if (rv == -1)
+ error = EFAULT;
- while (((state & URWLOCK_WRITE_OWNER) || URWLOCK_READER_COUNT(state) != 0) &&
- (state & URWLOCK_WRITE_WAITERS) == 0) {
- oldstate = casuword32(&rwlock->rw_state, state, state | URWLOCK_WRITE_WAITERS);
- if (oldstate == -1) {
+ while (error == 0 && ((state & URWLOCK_WRITE_OWNER) ||
+ URWLOCK_READER_COUNT(state) != 0) &&
+ (state & URWLOCK_WRITE_WAITERS) == 0) {
+ rv = casueword32(&rwlock->rw_state, state,
+ &oldstate, state | URWLOCK_WRITE_WAITERS);
+ if (rv == -1) {
error = EFAULT;
break;
}
@@ -2545,23 +2636,25 @@ do_rw_wrlock(struct thread *td, struct urwlock *rwlock, struct _umtx_time *timeo
break;
}
if (error != 0) {
- umtxq_lock(&uq->uq_key);
- umtxq_unbusy(&uq->uq_key);
- umtxq_unlock(&uq->uq_key);
+ umtxq_unbusy_unlocked(&uq->uq_key);
break;
}
if (!(state & URWLOCK_WRITE_OWNER) && URWLOCK_READER_COUNT(state) == 0) {
- umtxq_lock(&uq->uq_key);
- umtxq_unbusy(&uq->uq_key);
- umtxq_unlock(&uq->uq_key);
+ umtxq_unbusy_unlocked(&uq->uq_key);
error = umtxq_check_susp(td);
if (error != 0)
break;
continue;
}
sleep:
- blocked_writers = fuword32(&rwlock->rw_blocked_writers);
+ rv = fueword32(&rwlock->rw_blocked_writers,
+ &blocked_writers);
+ if (rv == -1) {
+ umtxq_unbusy_unlocked(&uq->uq_key);
+ error = EFAULT;
+ break;
+ }
suword32(&rwlock->rw_blocked_writers, blocked_writers+1);
while ((state & URWLOCK_WRITE_OWNER) || URWLOCK_READER_COUNT(state) != 0) {
@@ -2577,17 +2670,34 @@ sleep:
umtxq_unlock(&uq->uq_key);
if (error)
break;
- state = fuword32(__DEVOLATILE(int32_t *, &rwlock->rw_state));
+ rv = fueword32(__DEVOLATILE(int32_t *,
+ &rwlock->rw_state), &state);
+ if (rv == -1) {
+ error = EFAULT;
+ break;
+ }
}
- blocked_writers = fuword32(&rwlock->rw_blocked_writers);
+ rv = fueword32(&rwlock->rw_blocked_writers,
+ &blocked_writers);
+ if (rv == -1) {
+ umtxq_unbusy_unlocked(&uq->uq_key);
+ error = EFAULT;
+ break;
+ }
suword32(&rwlock->rw_blocked_writers, blocked_writers-1);
if (blocked_writers == 1) {
- state = fuword32(__DEVOLATILE(int32_t *, &rwlock->rw_state));
+ rv = fueword32(__DEVOLATILE(int32_t *,
+ &rwlock->rw_state), &state);
+ if (rv == -1) {
+ umtxq_unbusy_unlocked(&uq->uq_key);
+ error = EFAULT;
+ break;
+ }
for (;;) {
- oldstate = casuword32(&rwlock->rw_state, state,
- state & ~URWLOCK_WRITE_WAITERS);
- if (oldstate == -1) {
+ rv = casueword32(&rwlock->rw_state, state,
+ &oldstate, state & ~URWLOCK_WRITE_WAITERS);
+ if (rv == -1) {
error = EFAULT;
break;
}
@@ -2603,13 +2713,17 @@ sleep:
if (error != 0)
break;
}
- blocked_readers = fuword32(&rwlock->rw_blocked_readers);
+ rv = fueword32(&rwlock->rw_blocked_readers,
+ &blocked_readers);
+ if (rv == -1) {
+ umtxq_unbusy_unlocked(&uq->uq_key);
+ error = EFAULT;
+ break;
+ }
} else
blocked_readers = 0;
- umtxq_lock(&uq->uq_key);
- umtxq_unbusy(&uq->uq_key);
- umtxq_unlock(&uq->uq_key);
+ umtxq_unbusy_unlocked(&uq->uq_key);
}
umtx_key_release(&uq->uq_key);
@@ -2624,20 +2738,26 @@ do_rw_unlock(struct thread *td, struct urwlock *rwlock)
struct umtx_q *uq;
uint32_t flags;
int32_t state, oldstate;
- int error, q, count;
+ int error, rv, q, count;
uq = td->td_umtxq;
- flags = fuword32(&rwlock->rw_flags);
+ error = fueword32(&rwlock->rw_flags, &flags);
+ if (error == -1)
+ return (EFAULT);
error = umtx_key_get(rwlock, TYPE_RWLOCK, GET_SHARE(flags), &uq->uq_key);
if (error != 0)
return (error);
- state = fuword32(__DEVOLATILE(int32_t *, &rwlock->rw_state));
+ error = fueword32(__DEVOLATILE(int32_t *, &rwlock->rw_state), &state);
+ if (error == -1) {
+ error = EFAULT;
+ goto out;
+ }
if (state & URWLOCK_WRITE_OWNER) {
for (;;) {
- oldstate = casuword32(&rwlock->rw_state, state,
- state & ~URWLOCK_WRITE_OWNER);
- if (oldstate == -1) {
+ rv = casueword32(&rwlock->rw_state, state,
+ &oldstate, state & ~URWLOCK_WRITE_OWNER);
+ if (rv == -1) {
error = EFAULT;
goto out;
}
@@ -2655,9 +2775,9 @@ do_rw_unlock(struct thread *td, struct urwlock *rwlock)
}
} else if (URWLOCK_READER_COUNT(state) != 0) {
for (;;) {
- oldstate = casuword32(&rwlock->rw_state, state,
- state - 1);
- if (oldstate == -1) {
+ rv = casueword32(&rwlock->rw_state, state,
+ &oldstate, state - 1);
+ if (rv == -1) {
error = EFAULT;
goto out;
}
@@ -2716,11 +2836,13 @@ do_sem_wait(struct thread *td, struct _usem *sem, struct _umtx_time *timeout)
{
struct abs_timeout timo;
struct umtx_q *uq;
- uint32_t flags, count;
- int error;
+ uint32_t flags, count, count1;
+ int error, rv;
uq = td->td_umtxq;
- flags = fuword32(&sem->_flags);
+ error = fueword32(&sem->_flags, &flags);
+ if (error == -1)
+ return (EFAULT);
error = umtx_key_get(sem, TYPE_SEM, GET_SHARE(flags), &uq->uq_key);
if (error != 0)
return (error);
@@ -2732,15 +2854,16 @@ do_sem_wait(struct thread *td, struct _usem *sem, struct _umtx_time *timeout)
umtxq_busy(&uq->uq_key);
umtxq_insert(uq);
umtxq_unlock(&uq->uq_key);
- casuword32(&sem->_has_waiters, 0, 1);
- count = fuword32(__DEVOLATILE(uint32_t *, &sem->_count));
- if (count != 0) {
+ rv = casueword32(&sem->_has_waiters, 0, &count1, 1);
+ if (rv == 0)
+ rv = fueword32(__DEVOLATILE(uint32_t *, &sem->_count), &count);
+ if (rv == -1 || count != 0) {
umtxq_lock(&uq->uq_key);
umtxq_unbusy(&uq->uq_key);
umtxq_remove(uq);
umtxq_unlock(&uq->uq_key);
umtx_key_release(&uq->uq_key);
- return (0);
+ return (rv == -1 ? EFAULT : 0);
}
umtxq_lock(&uq->uq_key);
umtxq_unbusy(&uq->uq_key);
@@ -2771,7 +2894,9 @@ do_sem_wake(struct thread *td, struct _usem *sem)
int error, cnt;
uint32_t flags;
- flags = fuword32(&sem->_flags);
+ error = fueword32(&sem->_flags, &flags);
+ if (error == -1)
+ return (EFAULT);
if ((error = umtx_key_get(sem, TYPE_SEM, GET_SHARE(flags), &key)) != 0)
return (error);
umtxq_lock(&key);
@@ -2789,6 +2914,8 @@ do_sem_wake(struct thread *td, struct _usem *sem)
error = suword32(
__DEVOLATILE(uint32_t *, &sem->_has_waiters), 0);
umtxq_lock(&key);
+ if (error == -1)
+ error = EFAULT;
}
}
umtxq_unbusy(&key);
@@ -2804,7 +2931,7 @@ do_sem2_wait(struct thread *td, struct _usem2 *sem, struct _umtx_time *timeout)
struct abs_timeout timo;
struct umtx_q *uq;
uint32_t count, flags;
- int error;
+ int error, rv;
uq = td->td_umtxq;
flags = fuword32(&sem->_flags);
@@ -2819,8 +2946,8 @@ do_sem2_wait(struct thread *td, struct _usem2 *sem, struct _umtx_time *timeout)
umtxq_busy(&uq->uq_key);
umtxq_insert(uq);
umtxq_unlock(&uq->uq_key);
- count = fuword32(__DEVOLATILE(uint32_t *, &sem->_count));
- if (count == -1) {
+ rv = fueword32(__DEVOLATILE(uint32_t *, &sem->_count), &count);
+ if (rv == -1) {
umtxq_lock(&uq->uq_key);
umtxq_unbusy(&uq->uq_key);
umtxq_remove(uq);
@@ -2839,8 +2966,8 @@ do_sem2_wait(struct thread *td, struct _usem2 *sem, struct _umtx_time *timeout)
}
if (count == USEM_HAS_WAITERS)
break;
- count = casuword32(&sem->_count, 0, USEM_HAS_WAITERS);
- if (count == -1) {
+ rv = casueword32(&sem->_count, 0, &count, USEM_HAS_WAITERS);
+ if (rv == -1) {
umtxq_lock(&uq->uq_key);
umtxq_unbusy(&uq->uq_key);
umtxq_remove(uq);
@@ -2877,10 +3004,12 @@ static int
do_sem2_wake(struct thread *td, struct _usem2 *sem)
{
struct umtx_key key;
- int error, cnt;
+ int error, cnt, rv;
uint32_t count, flags;
- flags = fuword32(&sem->_flags);
+ rv = fueword32(&sem->_flags, &flags);
+ if (rv == -1)
+ return (EFAULT);
if ((error = umtx_key_get(sem, TYPE_SEM, GET_SHARE(flags), &key)) != 0)
return (error);
umtxq_lock(&key);
@@ -2895,12 +3024,12 @@ do_sem2_wake(struct thread *td, struct _usem2 *sem)
*/
if (cnt == 1) {
umtxq_unlock(&key);
- count = fuword32(__DEVOLATILE(uint32_t *,
- &sem->_count));
- while (count != -1 && count & USEM_HAS_WAITERS)
- count = casuword32(&sem->_count, count,
+ rv = fueword32(__DEVOLATILE(uint32_t *, &sem->_count),
+ &count);
+ while (rv != -1 && count & USEM_HAS_WAITERS)
+ rv = casueword32(&sem->_count, count, &count,
count & ~USEM_HAS_WAITERS);
- if (count == -1)
+ if (rv == -1)
error = EFAULT;
umtxq_lock(&key);
}
diff --git a/sys/kern/subr_uio.c b/sys/kern/subr_uio.c
index f2e6e32..f2bbb0c 100644
--- a/sys/kern/subr_uio.c
+++ b/sys/kern/subr_uio.c
@@ -7,6 +7,11 @@
* Co. or Unix System Laboratories, Inc. and are reproduced herein with
* the permission of UNIX System Laboratories, Inc.
*
+ * Copyright (c) 2014 The FreeBSD Foundation
+ *
+ * Portions of this software were developed by Konstantin Belousov
+ * under sponsorship from the FreeBSD Foundation.
+ *
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
@@ -438,3 +443,128 @@ copyout_unmap(struct thread *td, vm_offset_t addr, size_t sz)
return (0);
}
+
+#ifdef NO_FUEWORD
+/*
+ * XXXKIB The temporal implementation of fue*() functions which do not
+ * handle usermode -1 properly, mixing it with the fault code. Keep
+ * this until MD code is written. Currently sparc64, mips and arm do
+ * not have proper implementation.
+ */
+
+int
+fueword(const void *base, long *val)
+{
+ long res;
+
+ res = fuword(base);
+ if (res == -1)
+ return (-1);
+ *val = res;
+ return (0);
+}
+
+int
+fueword32(const void *base, int32_t *val)
+{
+ int32_t res;
+
+ res = fuword32(base);
+ if (res == -1)
+ return (-1);
+ *val = res;
+ return (0);
+}
+
+#ifdef _LP64
+int
+fueword64(const void *base, int64_t *val)
+{
+ int32_t res;
+
+ res = fuword64(base);
+ if (res == -1)
+ return (-1);
+ *val = res;
+ return (0);
+}
+#endif
+
+int
+casueword32(volatile uint32_t *base, uint32_t oldval, uint32_t *oldvalp,
+ uint32_t newval)
+{
+ int32_t ov;
+
+ ov = casuword32(base, oldval, newval);
+ if (ov == -1)
+ return (-1);
+ *oldvalp = ov;
+ return (0);
+}
+
+int
+casueword(volatile u_long *p, u_long oldval, u_long *oldvalp, u_long newval)
+{
+ u_long ov;
+
+ ov = casuword(p, oldval, newval);
+ if (ov == -1)
+ return (-1);
+ *oldvalp = ov;
+ return (0);
+}
+#else /* NO_FUEWORD */
+int32_t
+fuword32(const void *addr)
+{
+ int rv;
+ int32_t val;
+
+ rv = fueword32(addr, &val);
+ return (rv == -1 ? -1 : val);
+}
+
+#ifdef _LP64
+int64_t
+fuword64(const void *addr)
+{
+ int rv;
+ int64_t val;
+
+ rv = fueword64(addr, &val);
+ return (rv == -1 ? -1 : val);
+}
+#endif /* _LP64 */
+
+long
+fuword(const void *addr)
+{
+ long val;
+ int rv;
+
+ rv = fueword(addr, &val);
+ return (rv == -1 ? -1 : val);
+}
+
+uint32_t
+casuword32(volatile uint32_t *addr, uint32_t old, uint32_t new)
+{
+ int rv;
+ uint32_t val;
+
+ rv = casueword32(addr, old, &val, new);
+ return (rv == -1 ? -1 : val);
+}
+
+u_long
+casuword(volatile u_long *addr, u_long old, u_long new)
+{
+ int rv;
+ u_long val;
+
+ rv = casueword(addr, old, &val, new);
+ return (rv == -1 ? -1 : val);
+}
+
+#endif /* NO_FUEWORD */
diff --git a/sys/kern/vfs_acl.c b/sys/kern/vfs_acl.c
index 93626fb..e9361e5 100644
--- a/sys/kern/vfs_acl.c
+++ b/sys/kern/vfs_acl.c
@@ -148,6 +148,7 @@ acl_copyin(void *user_acl, struct acl *kernel_acl, acl_type_t type)
static int
acl_copyout(struct acl *kernel_acl, void *user_acl, acl_type_t type)
{
+ uint32_t am;
int error;
struct oldacl old;
@@ -162,8 +163,11 @@ acl_copyout(struct acl *kernel_acl, void *user_acl, acl_type_t type)
break;
default:
- if (fuword32((char *)user_acl +
- offsetof(struct acl, acl_maxcnt)) != ACL_MAX_ENTRIES)
+ error = fueword32((char *)user_acl +
+ offsetof(struct acl, acl_maxcnt), &am);
+ if (error == -1)
+ return (EFAULT);
+ if (am != ACL_MAX_ENTRIES)
return (EINVAL);
error = copyout(kernel_acl, user_acl, sizeof(*kernel_acl));
diff --git a/sys/mips/include/param.h b/sys/mips/include/param.h
index 2d1d7f1..90f3e6f 100644
--- a/sys/mips/include/param.h
+++ b/sys/mips/include/param.h
@@ -178,4 +178,8 @@
#define pgtok(x) ((x) * (PAGE_SIZE / 1024))
+#ifdef _KERNEL
+#define NO_FUEWORD 1
+#endif
+
#endif /* !_MIPS_INCLUDE_PARAM_H_ */
diff --git a/sys/net/if_spppsubr.c b/sys/net/if_spppsubr.c
index 9dc55c5..c0f8e39 100644
--- a/sys/net/if_spppsubr.c
+++ b/sys/net/if_spppsubr.c
@@ -5060,7 +5060,8 @@ sppp_params(struct sppp *sp, u_long cmd, void *data)
* Check the cmd word first before attempting to fetch all the
* data.
*/
- if ((subcmd = fuword(ifr->ifr_data)) == -1) {
+ rv = fueword(ifr->ifr_data, &subcmd);
+ if (rv == -1) {
rv = EFAULT;
goto quit;
}
diff --git a/sys/powerpc/powerpc/copyinout.c b/sys/powerpc/powerpc/copyinout.c
index dcfab80..a337c8b 100644
--- a/sys/powerpc/powerpc/copyinout.c
+++ b/sys/powerpc/powerpc/copyinout.c
@@ -405,14 +405,13 @@ fubyte(const void *addr)
return (val);
}
-#ifdef __powerpc64__
-int32_t
-fuword32(const void *addr)
+int
+fuword16(const void *addr)
{
struct thread *td;
pmap_t pm;
faultbuf env;
- int32_t *p, val;
+ uint16_t *p, val;
td = curthread;
pm = &td->td_proc->p_vmspace->vm_pmap;
@@ -432,15 +431,14 @@ fuword32(const void *addr)
td->td_pcb->pcb_onfault = NULL;
return (val);
}
-#endif
-long
-fuword(const void *addr)
+int
+fueword32(const void *addr, int32_t *val)
{
struct thread *td;
pmap_t pm;
faultbuf env;
- long *p, val;
+ int32_t *p;
td = curthread;
pm = &td->td_proc->p_vmspace->vm_pmap;
@@ -455,22 +453,71 @@ fuword(const void *addr)
return (-1);
}
- val = *p;
+ *val = *p;
td->td_pcb->pcb_onfault = NULL;
- return (val);
+ return (0);
}
-#ifndef __powerpc64__
-int32_t
-fuword32(const void *addr)
+#ifdef __powerpc64__
+int
+fueword64(const void *addr, int64_t *val)
{
- return ((int32_t)fuword(addr));
+ struct thread *td;
+ pmap_t pm;
+ faultbuf env;
+ int64_t *p;
+
+ td = curthread;
+ pm = &td->td_proc->p_vmspace->vm_pmap;
+
+ if (setfault(env)) {
+ td->td_pcb->pcb_onfault = NULL;
+ return (-1);
+ }
+
+ if (map_user_ptr(pm, addr, (void **)&p, sizeof(*p), NULL)) {
+ td->td_pcb->pcb_onfault = NULL;
+ return (-1);
+ }
+
+ *val = *p;
+
+ td->td_pcb->pcb_onfault = NULL;
+ return (0);
}
#endif
-uint32_t
-casuword32(volatile uint32_t *addr, uint32_t old, uint32_t new)
+int
+fueword(const void *addr, long *val)
+{
+ struct thread *td;
+ pmap_t pm;
+ faultbuf env;
+ long *p;
+
+ td = curthread;
+ pm = &td->td_proc->p_vmspace->vm_pmap;
+
+ if (setfault(env)) {
+ td->td_pcb->pcb_onfault = NULL;
+ return (-1);
+ }
+
+ if (map_user_ptr(pm, addr, (void **)&p, sizeof(*p), NULL)) {
+ td->td_pcb->pcb_onfault = NULL;
+ return (-1);
+ }
+
+ *val = *p;
+
+ td->td_pcb->pcb_onfault = NULL;
+ return (0);
+}
+
+int
+casueword32(volatile uint32_t *addr, uint32_t old, uint32_t *oldvalp,
+ uint32_t new)
{
struct thread *td;
pmap_t pm;
@@ -507,18 +554,21 @@ casuword32(volatile uint32_t *addr, uint32_t old, uint32_t new)
td->td_pcb->pcb_onfault = NULL;
- return (val);
+ *oldvalp = val;
+ return (0);
}
#ifndef __powerpc64__
-u_long
-casuword(volatile u_long *addr, u_long old, u_long new)
+int
+casueword(volatile u_long *addr, u_long old, u_long *oldvalp, u_long new)
{
- return (casuword32((volatile uint32_t *)addr, old, new));
+
+ return (casueword32((volatile uint32_t *)addr, old,
+ (uint32_t *)oldvalp, new));
}
#else
-u_long
-casuword(volatile u_long *addr, u_long old, u_long new)
+int
+casueword(volatile u_long *addr, u_long old, u_long *oldvalp, u_long new)
{
struct thread *td;
pmap_t pm;
@@ -555,7 +605,7 @@ casuword(volatile u_long *addr, u_long old, u_long new)
td->td_pcb->pcb_onfault = NULL;
- return (val);
+ *oldvalp = val;
+ return (0);
}
#endif
-
diff --git a/sys/sparc64/include/param.h b/sys/sparc64/include/param.h
index e59f2c4..46bacae 100644
--- a/sys/sparc64/include/param.h
+++ b/sys/sparc64/include/param.h
@@ -146,4 +146,8 @@
#define pgtok(x) ((unsigned long)(x) * (PAGE_SIZE / 1024))
+#ifdef _KERNEL
+#define NO_FUEWORD 1
+#endif
+
#endif /* !_SPARC64_INCLUDE_PARAM_H_ */
diff --git a/sys/sys/systm.h b/sys/sys/systm.h
index f4eae57..6e5ee61 100644
--- a/sys/sys/systm.h
+++ b/sys/sys/systm.h
@@ -254,16 +254,23 @@ int copyout_nofault(const void * __restrict kaddr, void * __restrict udaddr,
int fubyte(const void *base);
long fuword(const void *base);
-int fuword16(void *base);
+int fuword16(const void *base);
int32_t fuword32(const void *base);
int64_t fuword64(const void *base);
+int fueword(const void *base, long *val);
+int fueword32(const void *base, int32_t *val);
+int fueword64(const void *base, int64_t *val);
int subyte(void *base, int byte);
int suword(void *base, long word);
int suword16(void *base, int word);
int suword32(void *base, int32_t word);
int suword64(void *base, int64_t word);
uint32_t casuword32(volatile uint32_t *base, uint32_t oldval, uint32_t newval);
-u_long casuword(volatile u_long *p, u_long oldval, u_long newval);
+u_long casuword(volatile u_long *p, u_long oldval, u_long newval);
+int casueword32(volatile uint32_t *base, uint32_t oldval, uint32_t *oldvalp,
+ uint32_t newval);
+int casueword(volatile u_long *p, u_long oldval, u_long *oldvalp,
+ u_long newval);
void realitexpire(void *);
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20141027165557.GC1877>