Date: Sun, 13 Oct 2019 11:01:06 +0200 From: "Dave Cottlehuber" <dch@skunkwerks.at> To: freebsd-questions <freebsd-questions@freebsd.org> Subject: Re: Ansible for FreeBSD - use cases? Message-ID: <b9b6d85d-d59c-4b31-a396-24d8c806c7a6@www.fastmail.com> In-Reply-To: <20191005141507.GA1223@admin.sibptus.ru> References: <20191005141507.GA1223@admin.sibptus.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, 5 Oct 2019, at 16:15, Victor Sudakov wrote: > Dear Colleagues, > > Is anyone using ansible on FreeBSD for anything useful in production? I look after a number of customers with reasonably complex requirements, bare metal, network provisioning, jails & custom apps. Typically we use bare metal systems to boot up FreeBSD, and then ansible-ify everything else from that. > I've experimented with the pkgng module, it does install packages :-) > Most modules are meant for Linux however. > > What FreeBSD tasks do you automate with ansible, and with what modules? I highly recommend sysutils/py-mitogen (for massive speed up) & https://github.com/austinhyde/ansible-sshjail (for remote jail access, works like you expect it to. The core functionality driven by ansible is roughly: - node bootstrap - sshd, users, ntp - sudo, sysctls, unbound - tarsnaps, pf, zfs - freebsd-update & IDS - set up zerotier mesh network to link jails across servers - jails (currently with iocell but soon "simple jails") - haproxy, pf, bgp networking, spiped, TLS certs - collectd, rsyslog, smtp - everything else lives in jails - graylog, riemann, vault - clustered rabbitmq & couchdb - a few bespoke apps in erlang & elixir - build server - setup poudriere - kick off builds from a custom ports repo Updating apps is as simple as: - use BGP to offline a node from incoming data - shut down jails - zfs destroy zroot/jails - re-run ansible to deploy all the things - let haproxy figure out things are up again We have datasets like zroot/jailed/app which contains all the permanent data, and the app & config lives in zroot/jails/app which is both transient and deployed by ansible. Some of the stuff never changes, so I've started using pkg-create(8) to wrap up a lot of the stuff that I used to have as configs, and simply deploy that as a package via ansible. > I use net/rdist6 to update some configs on remote hosts, so I don't > think I'll benefit much from file copying modules of ansible. Ansible shines for the case where you need more variable injection than rdist or simple shell scripts can provide, but there are a few things that I *don't* enjoy about ansible. 1. ansible suffers from its own success. GitHub is a terrible solution for managing 1000s of ever-changing PRs for modules, and I don't think they've yet found an appropriate approach for engaging with the community to keep PRs flowing. This is a generic issue for any FLOSS project of this level of activity. 2. hiera - if you've used puppet, you'll miss this. While ansible provides a similar hash table to merge inventory data, none of the community provided roles can use this so everything is "namespaced" via snake_case: postgres_config_thing instead of some postgres.config.thing (where each level is an actual hash). Where this matters is when you want a group of nodes to be able to override a particular setting, but inherit elsewhere. 3. DAG - if you're doing DevOps with tools that have no concept of dependencies, you're doing it wrong. For example, if 90% of the play is already set up, there is no need to re-run it all from scratch every time. I work around this with keeping plays in small task-focused sets, and using conditional clauses to skip large chunks of plays if the final condition of the play is already set. 4. versions, the core is changing continuously. You basically need to keep up with the leading edge to remain in support. This means updating your plays every quarter, and patching up anything that's out of whack. There's always something I use that's being deprecated, but only once has this been problematic. That aside, generally I'm pretty happy with ansible, especially for FreeBSD. Most roles comprise the same steps: - make a new jail - install some packages - setup a few directories & zfs datasets - ensure daemon is running - ensure data is synced across cluster nodes/jails - tweak settings for jails & proxies for a home grown "service mesh" I guess it takes around 1-2 hours to make an ansible role for any given application we wish to run in this way, across multiple jails. It would be great if FreeBSD were more programmable in some ways - more libxo for plumbing orchestration tools into the system, things like iflib or a libfw to programmatically manage networks instead of wrapping ansible line-in-file and block functions, but right now it's good enough. Jail and zfs support makes isolating things really easy and the performance is more than sufficient for my needs. The zfs books & the jails book are excellent companions. Happy to expand on this if something above is of interest. A+ Dave
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?b9b6d85d-d59c-4b31-a396-24d8c806c7a6>