Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 14 Jan 2014 21:01:07 +0000
From:      Julien Grall <julien.grall@linaro.org>
To:        freebsd-xen@freebsd.org, xen-devel@lists.xen.org, gibbs@freebsd.org, freebsd-arm@FreeBSD.org
Cc:        stefano.stabellini@eu.citrix.com, ian.campbell@citrix.com, roger.pau@citrix.com
Subject:   [RFC] Add support for Xen ARM guest on FreeBSD
Message-ID:  <1389733267-20822-1-git-send-email-julien.grall@linaro.org>

next in thread | raw e-mail | index | archive | help
Hello,

During the last couple of months I have been working on porting FreeBSD on
Xen on ARM.

It's my first big project on FreeBSD and I'm not sure if I have cc the right
MLs and people. If not, could somebody point me to the right email address?

With this patch series I'm able to boot FreeBSD with a single VCPU on every
32 bits platform supported by Xen.

The patch series is divided in 5 parts:
	* #1-#12: Clean up and bug fix with some options
	* #13-#17: Update the Xen interface to Xen 4.4 and fix compilation
	* #18-#32: Update Xen code to support ARM guest
	* #33-#38: Update ARM code to support Xen guest
	* #39-#40: Add platform code to support ARM guest

I have pushed a branch on my git based on Royger's pvh work version 10:
	git clone git://xenbits.xen.org/people/julieng/freebsd.git -b xen-arm
	http://xenbits.xen.org/gitweb/?p=people/julieng/freebsd.git;a=shortlog;h=refs/heads/xen-arm

If needed, I can send the patch one by one the different mailing list.

This new support brings 2 open questions (for both Xen and FreeBSD community).
When a new guest is created, the toolstack will generated a device tree which
will contains:
	- The amount of memory
	- The description of the platform (gic, timer, hypervisor node)
	- PSCI node for SMP bringup

Until now, Xen on ARM supported only Linux-based OS. When the support of
device tree was added in Xen for guest, we chose to use Linux device tree
bindings (gic, timer,...). It seems that FreeBSD choose a different way to
implement device tree:
	- strictly respect ePAR (for interrupt-parent property)
	- gic bindings different (only 1 interrupt cell)

I would like to come with a common device tree specification (bindings, ...)
across every operating system. I know the Linux community is working on having
a device tree bindings out of the kernel tree. Does FreeBSD community plan to
work with Linux community for this purpose?

As specified early, the device tree can vary accross Xen version and user input
(eg the memory). I have noticed few places (mainly the timer) where the IRQs
number are harcoded.

In the long-term, it would be nice to see FreeBSD booting out-of-box (eg the
device tree is directly provided by the board firmware). I plan to add support
for Device Tree loading via Linux Boot ABI, it the way that Xen use to boot a
new guest.

The second question is related to memory attribute for page table. The early
page table setup by FreeBSD are using Write-Through memory attribute which
result to a likely random (not every time the same place) crash before the
real page table are initialized.
Replacing Write-Through by Write-Back made FreeBSD boot correctly. Even today,
I have no idea if it's a requirement from Xen or a bug (either in Xen or
FreeBSD).

The code is taking its first faltering steps, therefore the TODO is quite big:
	- Try the code on x86 architecture. I did lots of rework for the event
	channels code and didn't even try to compile
	- Clean up event channel code
	- Clean up xen control code
	- Add support for Device Tree loading via Linux Boot ABI
	- Fixing crashing on userspace. Could be related to the patch
	series "arm SMP on Cortex-A15"
	- Add guest SMP support
	- DOM0 support?

Any help, comments, questions are welcomed.

Sincerely yours,

============= Instruction to test FreeBSD on Xen on ARM ===========

Xen upstream tree doesn't fully support ELF loading for ARM. You will need to
recompile the tools with the following 2 patches:
	- https://patches.linaro.org/22228/
	- https://patches.linaro.org/22227/

To compile and boot Xen on your board, you can refer to the wiki page:
http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions

The instruction to compile FreeBSD for Xen on ARM:
$ truncate -s 512 xenvm.img
$ sudo mdconfig -f xenvm.img -u0
$ sudo newfs /dev/md0
$ sudo mount /dev/md0 /mnt

$ sudo make TARGET_ARCH=armv6 kernel-toolchain
$ sudo make TARGET_ARCH=armv6 KERNCONF=XENHVM buildkernel
$ sudo make TARGET_ARCH=armv6 buildworld
$ sudo make TARGET_ARCH=armv6 DESTDIR=/mnt installworld distribution

$ echo "/dev/xbd0	/	ufs	rw	1	1" > /mnt/etc/fstab
$ vi /mnt/etc/ttys (add the line 'xc0 "/usr/libexec/getty Pc" xterm on secure")

$ sudo umount /mnt
$ sudo mdconfig -d u0

Then you can copy the rootfs and the kernel to DOM 0 on your board.

To boot the a FreeBSD your will required the following configuration file
$ cat freebsd.xl
kernel="kernel"
memory=64
name="freebsd"
vcpus=1
autoballon="off"
disk=[ 'phy:/dev/loop0,xvda,w' ]
$ losetup /dev/loop0 xenvm.img
$ xl create freebsd.xl
$ xl console freebsd

Julien Grall (40):
  xen/netfront: Don't need to include machine/intr_machdep.h
  xen/blkfront: Don't need to include machine/intr_machdep.h
  xen/control: Remove include machine/intr_machdep.h
  xen: Define xen_intr_handle_upcall in common headers
  xen: Remove duplicate features.h header in i386 arch
  xen/hypercall: Allow HYPERVISOR_console_write to take a const string
  xen/timer: Make xen timer optional
  xen/console: Fix build when DDB is enabled
  xen/console: clean up identify callback
  xen/xenstore: xs_probe should return BUS_PROBE_NOWILDCARD
  xen/control: xctlr_probe shoud return BUS_PROBE_NOWILDCARD
  xen/hypervisor: Be sure to set __XEN_INTERFACE_VERSION__
  xen/interface: Update interface to Xen 4.4 headers
  xen: Bump __XEN_INTERFACE_VERSION__ to 4.3
  xen/baloon: Fix compilation with Xen 4.4 headers
  xen/netfront: Fix compilation with Xen 4.4 headers
  xen/gnttab: Fix compilation with Xen 4.4 headers
  xen/ballon: Use correct type for frame list
  xen/netback: Fix printf format for xen_pfn_t
  xen/netfront: Use the correct type for rx_pfn_array
  xen/gnttab: Use the right type for the frames
  xen/gnttab: Export resume_frames
  xen/gnttab: Add a guard for xenpci functions
  xen/netfront: Define PTE flags per architecture
  xen/control: Implement suspend has panic for ARM
  xen: Introduce xen_pmap
  xen: move x86/xen/xenpv.c in dev/xen/xenpv.c
  xen/xenpv: Only add isa for x86 architecture
  xen/xenpv: Protect xenpci code by DEV_XENPCI
  xen: move xen_intr.c to common code
  xen/evtchn: Rework the event channels handler to make it generic
  xen/console: handle console for HVM
  arm/timer: Handle timer with no clock-frequency property
  arm/timer: WORKAROUND The timer should support DT...
  arm: Detect new revision for cortex A15
  arm: add ELFNOTE macro
  arm: Implement disable_intr
  arm: Use Write-Back as memory attribute for early page table
  dts: Add xenvm.dts
  arm: Add xenhvm platform

 sys/amd64/conf/GENERIC                  |    4 +-
 sys/amd64/include/apicvar.h             |    1 -
 sys/amd64/include/xen/hypercall.h       |    2 +-
 sys/amd64/include/xen/xen-os.h          |    2 +
 sys/amd64/include/xen/xenpmap.h         |    2 +-
 sys/arm/arm/cpufunc.c                   |    2 +
 sys/arm/arm/generic_timer.c             |   12 +-
 sys/arm/arm/locore.S                    |    4 +-
 sys/arm/conf/XENHVM                     |   97 +++
 sys/arm/include/armreg.h                |    2 +
 sys/arm/include/asmacros.h              |   26 +
 sys/arm/include/cpufunc.h               |    6 +
 sys/arm/include/xen/hypercall.h         |   50 ++
 sys/arm/include/xen/synch_bitops.h      |   52 ++
 sys/arm/include/xen/xen-os.h            |   29 +
 sys/arm/include/xen/xenfunc.h           |    4 +
 sys/arm/include/xen/xenvar.h            |   14 +
 sys/arm/xenhvm/elf.S                    |   12 +
 sys/arm/xenhvm/files.xenhvm             |   18 +
 sys/arm/xenhvm/hypercall.S              |  107 +++
 sys/arm/xenhvm/xen-dt.c                 |  148 ++++
 sys/arm/xenhvm/xenhvm_machdep.c         |  132 ++++
 sys/boot/fdt/dts/xenvm.dts              |   77 ++
 sys/conf/files                          |    4 +-
 sys/conf/options                        |    3 +
 sys/conf/options.arm                    |    1 +
 sys/dev/xen/balloon/balloon.c           |    6 +-
 sys/dev/xen/blkfront/blkfront.c         |    1 -
 sys/dev/xen/console/console.c           |   28 +-
 sys/dev/xen/console/xencons_ring.c      |   41 +-
 sys/dev/xen/control/control.c           |   19 +-
 sys/dev/xen/netback/netback.c           |    6 +-
 sys/dev/xen/netfront/netfront.c         |   15 +-
 sys/dev/xen/xenpci/xenpci.c             |    2 +-
 sys/dev/xen/xenpv.c                     |  133 ++++
 sys/i386/conf/GENERIC                   |    4 +-
 sys/i386/conf/XEN                       |    1 +
 sys/i386/include/apicvar.h              |    1 -
 sys/i386/include/xen/features.h         |   22 -
 sys/i386/include/xen/hypercall.h        |    2 +-
 sys/i386/include/xen/xen-os.h           |    2 +
 sys/i386/include/xen/xenvar.h           |    2 +-
 sys/i386/xen/xen_machdep.c              |    2 +-
 sys/x86/xen/xen_intr.c                  | 1241 -------------------------------
 sys/x86/xen/xenpv.c                     |  128 ----
 sys/xen/gnttab.c                        |   23 +-
 sys/xen/hypervisor.h                    |    3 +-
 sys/xen/interface/arch-arm.h            |  259 +++++--
 sys/xen/interface/arch-arm/hvm/save.h   |    2 +-
 sys/xen/interface/arch-x86/hvm/save.h   |   25 +-
 sys/xen/interface/arch-x86/xen-mca.h    |    2 +-
 sys/xen/interface/arch-x86/xen-x86_32.h |    2 +-
 sys/xen/interface/arch-x86/xen-x86_64.h |    2 +-
 sys/xen/interface/arch-x86/xen.h        |   31 +-
 sys/xen/interface/callback.h            |    4 +-
 sys/xen/interface/dom0_ops.h            |    2 +-
 sys/xen/interface/domctl.h              |   65 +-
 sys/xen/interface/elfnote.h             |   10 +-
 sys/xen/interface/event_channel.h       |    5 +-
 sys/xen/interface/features.h            |   16 +-
 sys/xen/interface/gcov.h                |  115 +++
 sys/xen/interface/grant_table.h         |    8 +-
 sys/xen/interface/hvm/hvm_xs_strings.h  |   80 ++
 sys/xen/interface/hvm/ioreq.h           |   20 +-
 sys/xen/interface/hvm/params.h          |   14 +-
 sys/xen/interface/hvm/pvdrivers.h       |   47 ++
 sys/xen/interface/hvm/save.h            |    4 +-
 sys/xen/interface/io/blkif.h            |  104 ++-
 sys/xen/interface/io/console.h          |    2 +-
 sys/xen/interface/io/fbif.h             |    2 +-
 sys/xen/interface/io/kbdif.h            |    2 +-
 sys/xen/interface/io/netif.h            |   47 +-
 sys/xen/interface/io/pciif.h            |    3 +-
 sys/xen/interface/io/protocols.h        |    5 +-
 sys/xen/interface/io/tpmif.h            |   68 +-
 sys/xen/interface/io/usbif.h            |    1 -
 sys/xen/interface/io/vscsiif.h          |   42 +-
 sys/xen/interface/io/xenbus.h           |   20 +-
 sys/xen/interface/io/xs_wire.h          |    7 +-
 sys/xen/interface/kexec.h               |    7 +-
 sys/xen/interface/mem_event.h           |    4 +-
 sys/xen/interface/memory.h              |   80 +-
 sys/xen/interface/nmi.h                 |    7 +-
 sys/xen/interface/physdev.h             |   33 +-
 sys/xen/interface/platform.h            |   27 +-
 sys/xen/interface/sched.h               |   87 ++-
 sys/xen/interface/sysctl.h              |   47 +-
 sys/xen/interface/tmem.h                |    2 +-
 sys/xen/interface/trace.h               |   93 ++-
 sys/xen/interface/vcpu.h                |    2 +-
 sys/xen/interface/version.h             |    6 +-
 sys/xen/interface/xen-compat.h          |    2 +-
 sys/xen/interface/xen.h                 |  128 +++-
 sys/xen/interface/xenoprof.h            |    2 +-
 sys/xen/interface/xsm/flask_op.h        |    8 +
 sys/xen/xen-os.h                        |    2 +-
 sys/xen/xen_intr.c                      | 1072 ++++++++++++++++++++++++++
 sys/xen/xen_intr.h                      |    8 +-
 sys/xen/xenstore/xenstore.c             |    4 +-
 99 files changed, 3339 insertions(+), 1791 deletions(-)
 create mode 100644 sys/arm/conf/XENHVM
 create mode 100644 sys/arm/include/xen/hypercall.h
 create mode 100644 sys/arm/include/xen/synch_bitops.h
 create mode 100644 sys/arm/include/xen/xen-os.h
 create mode 100644 sys/arm/include/xen/xenfunc.h
 create mode 100644 sys/arm/include/xen/xenvar.h
 create mode 100644 sys/arm/xenhvm/elf.S
 create mode 100644 sys/arm/xenhvm/files.xenhvm
 create mode 100644 sys/arm/xenhvm/hypercall.S
 create mode 100644 sys/arm/xenhvm/xen-dt.c
 create mode 100644 sys/arm/xenhvm/xenhvm_machdep.c
 create mode 100644 sys/boot/fdt/dts/xenvm.dts
 create mode 100644 sys/dev/xen/xenpv.c
 delete mode 100644 sys/i386/include/xen/features.h
 delete mode 100644 sys/x86/xen/xen_intr.c
 delete mode 100644 sys/x86/xen/xenpv.c
 create mode 100644 sys/xen/interface/gcov.h
 create mode 100644 sys/xen/interface/hvm/hvm_xs_strings.h
 create mode 100644 sys/xen/interface/hvm/pvdrivers.h
 create mode 100644 sys/xen/xen_intr.c

-- 
1.8.0




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1389733267-20822-1-git-send-email-julien.grall>