Ceph on FreeBSD

From owner-svn-doc-all@freebsd.org Wed Apr 13 16:51:57 2016 Return-Path: Delivered-To: svn-doc-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 645D9B0EF7F; Wed, 13 Apr 2016 16:51:57 +0000 (UTC) (envelope-from wblock@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3D4A61E70; Wed, 13 Apr 2016 16:51:57 +0000 (UTC) (envelope-from wblock@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id u3DGpu2B017006; Wed, 13 Apr 2016 16:51:56 GMT (envelope-from wblock@FreeBSD.org) Received: (from wblock@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id u3DGpuqO017005; Wed, 13 Apr 2016 16:51:56 GMT (envelope-from wblock@FreeBSD.org) Message-Id: <201604131651.u3DGpuqO017005@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: wblock set sender to wblock@FreeBSD.org using -f From: Warren Block Date: Wed, 13 Apr 2016 16:51:56 +0000 (UTC) To: doc-committers@freebsd.org, svn-doc-all@freebsd.org, svn-doc-head@freebsd.org Subject: svn commit: r48612 - head/en_US.ISO8859-1/htdocs/news/status X-SVN-Group: doc-head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-doc-all@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "SVN commit messages for the entire doc trees \(except for " user" , " projects" , and " translations" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Apr 2016 16:51:57 -0000 Author: wblock Date: Wed Apr 13 16:51:56 2016 New Revision: 48612 URL: https://svnweb.freebsd.org/changeset/doc/48612 Log: Add Ceph report from Willem Jan Withagen . Modified: head/en_US.ISO8859-1/htdocs/news/status/report-2016-01-2016-03.xml Modified: head/en_US.ISO8859-1/htdocs/news/status/report-2016-01-2016-03.xml ============================================================================== --- head/en_US.ISO8859-1/htdocs/news/status/report-2016-01-2016-03.xml Wed Apr 13 16:15:00 2016 (r48611) +++ head/en_US.ISO8859-1/htdocs/news/status/report-2016-01-2016-03.xml Wed Apr 13 16:51:56 2016 (r48612) @@ -2013,4 +2013,202 @@ + + + Ceph on FreeBSD + + + + + Willem Jan + Withagen + + wjw@digiware.nl + + + + + Ceph main site + Main repository + My Fork + The git PULL with all changes + + + +

Ceph is a distributed object store and file system designed + to provide excellent performance, reliability and + scalability.

+ +

+
Object Storage
+ +
Ceph provides seamless access to objects using native + language bindings or radosgw, a REST interface that is + compatible with applications written for S3 and Swift.
+
+
Block Storage
+ +
Ceph's RADOS Block Device (RBD) provides access to block + device images that are striped and replicated across the + entire storage cluster.
+
+
File System
+ +
Ceph provides a POSIX-compliant network file system that + aims for high performance, large data storage, and maximum + compatibility with legacy applications.
+

+ +

I started looking into Ceph, because the HAST solution with + CARP and ggate did not really do what I wanted. But + I am aiming for running a Ceph storage cluster of storage + nodes that are running ZFS. The end station would be running + bhyve on RBD disk that are stored in Ceph.

+ +

The FreeBSD build will build most of the tools in Ceph. Note + that the RBD-dependent items will not work since FreeBSD does + not have RBD yet.

+ +

Build Prerequisites

+ +

Compiling and building Ceph is tested on 11-CURRENT. It uses + the CLANG toolset taht is available, which needs to be at + least 3.7. Clang 3.4 (on 10.2-STABLE) does not have all the + required capabilites to compile everything.

+ +

This setup will get things running for FreeBSD:

+ +

+
Install bash and link it in /bin (requires root + privileges):
+ +
sudo pkg install bash
+ +
sudo ln -s /usr/local/bin/bash /bin/bash
+
+
Building Ceph
+ +
./do_freebsd.sh
+

+ +

Parts Not Yet Included

+ +

+
RBD
+ +
Rados Block Devices is implemented in the Linux kernel. + It seems that there used to be a userspace implementation + first. And perhaps ggated could be used as a + template since it does some of the same functions, other + than just between two disks. And it has a userspace + counterpart.
+
+
BlueStore
+ +
FreeBSD and Linux have a different AIO API, and that + needs to be made compatible. Next to that is the + discussion in FreeBSD about aio_cancel not + working for all device types.
+
+
CephFS
+ +
Cython tries to access an internal field in dirent which + does not compile.
+

+ +

Tests that verify the correct working of the above are also + excluded from the test set.

+ +

Tests Not Yet Included

+ +

+
ceph-detect-init/run-tox.sh
+ +
Because the current implementation does not know anything + about FreeBSD rc-init.
+
+
Tests that make use of nosestests
+ +
Calling these does not really work since + nosetests is not in /usr/bin, and + calling through /usr/bin/env/nosetests does not + work on FreeBSD.
+
+
test/pybind/test_ceph_argparse.py
+
+
test/pybind/test_ceph_daemon.py
+

+ +

Things To Investigate

+ +

+
ceph-{osd,mon} need two signals before they + actually terminate.
+
+
ceph_erasure_code --debug-osd 20 --plugin_exists jerasure + crashes due to SIGSEGV. This is a pointer reference that + gets modified outside the regular programflow. Probably + due to a programming error but perhaps wrong mixing and + matching of many libraries.
+

+ + + + +

Current and foremost task is to get the test set to + complete without errors. This includes fixing several + coredumps.

+ +

Run integration tests to see if the FreeBSD daemons will + work with a Linux Ceph platform.

+ + + +

Get the Python tests that are currently excluded to work, + and test OKE.

+ + + +

Compile and test the user space RBD (Rados Block + Device).

+ +

Investigate and see if an in-kernel RBD device could be + developed a la ggate.

+ + + +

Integrate the FreeBSD /etc/rc.d init scripts in + the Ceph stack for testing and running Ceph on production + machines.

+ + +