Date: Thu, 13 Mar 2014 13:48:46 -0400 From: Richard Yao <ryao@gentoo.org> To: CeDeROM <cederom@tlen.pl>, freebsd-hackers@freebsd.org Subject: Re: GSoC proposition: multiplatform UFS2 driver Message-ID: <5321EF7E.7000500@gentoo.org> In-Reply-To: <CAFYkXjm1fCQDSvHTHSpmAbLPqNz0kwaWf%2BajLdaoBD3bLnMDAA@mail.gmail.com> References: <CAFYkXjm1fCQDSvHTHSpmAbLPqNz0kwaWf%2BajLdaoBD3bLnMDAA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --8B3H5GWiSqHQdrWk2e6ktg97efq3Gmblt Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 03/12/2014 12:54 PM, CeDeROM wrote: > My proposition is to create universal UFS2 driver that would work on > Windows, Linux, and maybe other OS, so we can use native UFS2 as data > partition among various OS. >=20 I like this idea. In particular, I would like to have UFS2 SU+J available for use in certain Linux VMs and was thinking about this possibility earlier in the week. However, I no longer qualify to be a GSoC student and I do not have time to do it myself. Speaking as someone who might be a GSoC mentor (in Gentoo, not FreeBSD) this year, I can say that this proposal is rather terse and you would need to add detail in order to make a formal proposal. However, you did the right thing in emailing the FreeBSD Hackers mailing list for feedback on this idea. Speaking as a kernel filesystem hacker (with most of my experience in Linux), I think this could be too ambitious depending on how you add detail. In particular, Linux has a moving target of a VFS (which causes much pain) and Windows' IFS is very different from the SunOS VFS that has become the basis for implementing multiple filesystems on different UNIX implementations and copy-cats. However, lets ignore that and go over the specifics of what you can expect and what decisions I suggest you make. The very first thing you will need to decide is whether this would be a port to native kernel interfaces, a port to FUSE/Dokan or some combination of the two. I believe that selecting FUSE/Dokan would make your project much easier, but if selected, I believe that it will limit the appeal of the result. This in part because people have a general distaste for FUSE that is probably more than what is warranted. It is also in part because FUSE would be unsuitable for the VM environment use case that a native port to Linux could achieve. Second, you will need to do is identify the *BSD interfaces in use so you know precisely what needs to be wrapped on each platform when writing wrapper code. I suggest keeping as much wrapper code as possible separate so that it is easy to see what needs to be reimplemented for a given platform. It would be best to rely on the C Pre-Processor to make this wrapper code support multiple platforms instead of writing an entirely separate wrapper for each platform. Adopting autotools to generate the C Pre-Processor definitions will make support for multiple platforms easier, especially in the case of Linux's VFS. Third, you will need to think about platforms. You mentioned Windows and Linux, but I strongly suggest that you refocus on Linux, Mac OS X and Illumos for reasons I stated above. All three of them can be considered open source for your purposes, they have similar development tools and they are basically everyone but Windows. If you follow my advice and find yourself with extra time, you could try tackling Windows as what I will call a bonus goal. Fourth, you will need to setup development environments for each platform, learn how to use the development tools to write filesystem kernel modules or FUSE/Dokan programs and learn the APIs. If this involves VMs, you will want to make sure that the storage IO is quick so you do not waste time waiting for slow VFS operations during compilation and testing. You will also want to figure out who to contact when you have questions about things. If you follow my advice about refocusing on Linux, Mac OS X and Illumos, you will likely find people in the broader Open ZFS community willing to answer general questions. Lets consider each platform individually. On Illumos, you will find the tools, kernel design and VFS API to be very similar to those on FreeBSD. In particular, they both have strong UNIX heritage and the VFS implementations in both are direct descendants of the original SunOS VFS, which predated all others. You should find the Illumos mailing lists a good place to ask questions should you have a= ny. On Mac OS X, you will first need to get your hands on it. You can either use an actual Mac OS X system or Darwin, which is the OSS core of Mac OS X that includes its kernel and base userland. Apple stopped releasing Darwin ISOs at version 8 while the current Mac OS X is at version 13, but interface stability should be good enough that the version 8 ISOs should be sufficient. However, you could also get version 9 ISOs through Pure Darwin. Pure Darwin is an attempt to continue Apple's releases that suffered from a chronic lack of the requisite man power to achieve its goals, but it should be sufficient for yours. As far as technical aspects of a Mac OS X port are concerned, you will find similarities with FreeBSD in its tools and VFS API. Mac OS X's XNU kernel is a descendant of Carnegie Mellon University's Mach microkernel. While XNU itself is not a microkernel and incorporates significant amounts of BSD code, you will encounter significant differences with FreeBSD that are greater than those you will encounter with Illumos. In particular, low level kernel functions such as memory allocation and thread management have been replaced with their Mach counterparts and unless you are familiar with XNU, you will find the design of loadable kernel modules to be unlike anything you have ever seen. Given that the majority of XNU kernel developers operate behind closed doors, I cannot recommend a list full of developers for you to use to ask questions, but I think you could try asking the Mac ZFS developers. The present Mac ZFS project lead is Jorgen Lundman. You can find him under the handle lundman in #mac-zfs on freenode. You will also find myself there under the handle ryao. If this project goes forward as a native port and you ask when I am present, I am willing to ask him to consider such questions as coming from me. While I cannot speak for him, I can say that I have answered enough of his questions in the past that I imagine that he would consider answering yours as returning the favor. On Linux, you will find that the development tools and kernel design are similar to FreeBSD, but the VFS API is substantially different from the VFS in other UNIX implementations/clones/copy-cats. You will also find that different kernel releases lack API compatibility, such that you can expect 1 or 2 things to break every release. In addition, distribution specific patches mean that API compatibility within what appears to be the same kernel release can break. The implication being that kernel version checks do not work universally. I strongly recommend using autotools to handle this. If you use autotools, I suggest adopting the autotools code from ZFSOnLinux, which should provide you with a solid framework to handle API compatibility problems. It would also make handling many kernels rather easy to do. I suspect licensing might be a problem, but I believe its authors (of which I am one) are open to releasing it under a BSD-style license should someone request it. I discussed licensing of the autotools code with the ZFSOnLinux project lead during my review of how ZFSOnLinux's licensing affected Gentoo's ZFS packaging. My recollection is that he was willing to change the license if the current one prevented use, before I found that the licensing was fine for Gentoo's use. I believe he, myself and the few others who contributed would be willing to release the code under a different license should you want to use the code in a project that requires a different F/OSS license. As for questions and the distribution, I am willing to volunteer myself to answer questions involving a native Linux UFS2 SU+J port and I think Gentoo would be an excellent choice of distribution to use for this. You would need to learn how to build your own kernels (which would be true for any distribution), but I am willing to answer questions there too. As for Windows, I know that it is dramatically different in its use of IFS, but I cannot say much else. If you have questions, you could probably contact the ReactOS developers with your questions. Keep in mind that ReactOS' kernel is not capable of handling alternative filesystems through IFS modules at this time, so their ability to answer your questions could be limited. They will also likely try to get you to volunteer to write the requisite code necessary to do that. That has been my past experience whenever I ask them technical questions about Windows filesystems. That being said, happy coding. :) --8B3H5GWiSqHQdrWk2e6ktg97efq3Gmblt Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBAgAGBQJTIe+AAAoJECDuEZm+6ExkkNkP/0bjl9cN57E+MF5GiGpzKO6r fbL5k2IR5z9MNzsmRqujwmKqqykrsLcD7qHOs2IbXZpivKBTS3f5L1/NRO+vPycD A3V3joyUeGkCcEYwCB+GcsF3Z7ufaHuPQIFVoDcmfMzbmDKrlYpctzaZ8Tb8xU1v cCXwEFSap6JFg3cDW2rN6ibAR49jd4SMx9FHamUNBm3V5YKCrEn9BA1IDWaJ2fXV aCuAc+C+MOUm/brhc7Tl9QzbKXh7OaAM2qVRvqIzj8A/xJly3RdFrrafJJc0fAMx P0QZ/BhLHkbp0dfG5+STmzUrPOtNJTClqimTtFf+5ZpTKXuylCinifmWk9qUGHsj DoBvbtBeslwrMU2nL+DWRqfJ+8FMsaEWhGvdMpIyabDypDW46nnx1/aUEBH1kVgF wrDwQp/hyVqnTi6oaLVADD3qTREJeERqUeEY7LrqCs2r4HRiWHERulQ+j6AKgewW RLJx7URf9iuuJ/7wEF7j2vPUlNDNICScYl417c1xdHg9EDLDX7VhNuzO7iuiB3pF IaqHl3xlEGirErmeYcXtT57yigklRdIEt2LPfKoFrT7CuIOSVpEmspdt0O+5mSur w2+5zOb4Yv7eoZGOZ4T4E6ZjlbLsPEP+4bzqT/kfWU4G+92jaTOuxD8lMC6i0167 iRl4u2ffZJIDDsl3Ek+O =OpT1 -----END PGP SIGNATURE----- --8B3H5GWiSqHQdrWk2e6ktg97efq3Gmblt--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5321EF7E.7000500>