# etcdadm - automation for etcd clusters

## Table of Contents

<!-- toc -->
- [Summary](#summary)
- [Motivation](#motivation)
  - [Goals](#goals)
  - [Non-Goals](#non-goals)
- [Proposal](#proposal)
  - [User Stories](#user-stories)
    - [Manual Cluster Creation](#manual-cluster-creation)
    - [Automatic Cluster Creation](#automatic-cluster-creation)
    - [Automatic Cluster Creation with EBS volumes](#automatic-cluster-creation-with-ebs-volumes)
  - [Implementation Details/Notes/Constraints](#implementation-detailsnotesconstraints)
  - [Risks and Mitigations](#risks-and-mitigations)
- [Graduation Criteria](#graduation-criteria)
- [Implementation History](#implementation-history)
- [Infrastructure Needed](#infrastructure-needed)
<!-- /toc -->

## Summary

etcdadm makes operation of etcd for the Kubernetes control plane easy, on clouds
and on bare-metal, including both single-node and HA configurations.

It is able to perform cluster reconfigurations, upgrades / downgrades, and
backups / restores.

## Motivation

Today each installation tool must reimplement etcd operation, and this is
difficult.  It also leads to ecosystem fragmentation - e.g. etcd backups from
one tool are not necessarily compatible with the backups from other tools.  The
failure modes are subtle and rare, and thus the kubernetes project benefits from
having more collaboration.


### Goals

The following key tasks are in scope:

* Cluster creation
* Cluster teardown
* Cluster resizing / membership changes
* Cluster backups
* Disaster recovery or restore from backup
* Cluster upgrades
* Cluster downgrades
* PKI management

We will implement this functionality both as a base layer of imperative (manual
CLI) operation, and a self-management layer which should enable automated
in "safe" scenarios (with fallback to manual operation).

We'll also optionally support limited interaction with cloud infrastructure, for
example for mounting volumes and peer-discovery.  This is primarily for the
self-management layer, but we'll expose it via etcdadm for consistency and for
power-users.  The tasks are limited today to listing & mounting a persistent
volume, and listing instances to find peers.  A full solution for management of
machines or networks (for example) is out of scope, though we might share some
example configurations for exposition.  We expect kubernetes installation
tooling to configure the majority of the cloud infrastructure here, because both
the configurations and the configuration tooling varies widely.

The big reason that volume mounting is in scope is that volume mounting acts as
a simple mutex on most clouds - it is a cheap way to boost the safety of our
leader/gossip algorithms, because we have an external source of truth.

We'll also support reading & writing backups to S3 / GCS etc.

### Non-Goals

* The project is not targeted at operation of an etcd cluster for use other than
  by Kubernetes apiserver.  We are not building a general-purpose etcd operation
  toolkit.  Likely it will work well for other use-cases, but other tools may be
  more suitable.
* As described above, we aren't building a full "turn up an etcd cluster on a
  cloud solution"; we expect this to be a building block for use by kubernetes
  installation tooling (e.g. cluster API solutions).

## Proposal

We will combine the [etcdadm](https://github.com/platform9/etcdadm) from
Platform9 with the [etcd-manager](https://github.com/kopeio/etcd-manager)
project from kopeio / @justinsb.

etcdadm gives us easy to use CLI commands, which will form the base layer of
operation.  Automation should ideally describe what it is doing in terms of
etcdadm commands, though we will also expose etcdadm as a go-library for easier
consumption, following the kubectl pattern of a `cmd/` layer calling into a
`pkg/` layer.  This means the end-user can understand the operation of the
tooling, and advanced users can feel confident that they can use the CLI tooling
for advanced operations.

etcd-manager provides automation of the common scenarios, particularly when
running on a cloud.  It will be rebased to work in terms of etcdadm CLI
operations (which will likely require some functionality to be added to etcdadm
itself).  Where automation is not known to be safe, etcd-manager can stop and
allow for manual intervention using the CLI.

kops is currently using etcd-manager, and we aim to switch to the (new) etcadm asap.

We expect other tooling (e.g. cluster-api implementations) to adopt this project
for etcd management going forwards, and do a first integration or two if it
hasn't happened already.

### User Stories

#### Manual Cluster Creation

A cluster operator setting up a cluster manually will be able to do so using etcdadm and kubeadm.

The basic flow looks like:

* On a master machine, run `etcdadm init`, making note of the `etcdadm join
  <endpoint>` command
* On each other master machine, copy the CA certificate and key from one of the
  other masters, then run the `etcdadm join <endpoint>` command.
* Run kubeadm following the [external etcd procedure](https://kubernetes.io/docs/setup/independent/high-availability/#external-etcd)

This results in an multi-node ("HA") etcd cluster.

#### Automatic Cluster Creation

etcd-manager works by coordinating via a shared filesystem-like store (e.g. S3
or GCS) and/or via cloud APIs (e.g. EC2 or GCE).  In doing so it is able to
automate the manual commands, which is very handy for running in a cloud
environment like AWS or GCE.

The basic flow would look like:

* The user writes a configuration file to GCS using `etcdadm seed
  gs://mybucket/cluster1/etcd1 version=3.2.12 nodes=3`
* On each master machine, run `etcdadm auto gs://mybucket/cluster1/etcd1`.
  (Likely the user will have to run that persistently, either as a systemd
  service or a static pod.)

`etcdadm auto` downloads the target configuration from GCS, discovers other
peers also running etcdadm, gossips with them to do basic leader election.  When
sufficient nodes are available to form a quorum, it starts etcd.

#### Automatic Cluster Creation with EBS volumes

etcdadm can also automatically mount EBS volumes.  The workflow looks like this:

* As before, write a configuration file using `etcadm seed ...`, but this time
  passing additional arguments "--volume-tag cluster=mycluster"
* Create EBS volumes with the matching tags
* On each master machine, run `etcdadm auto ...` as before.  Now etcdadm will
  try to mount a volume with the correct tags before acting as a member of the
  cluster.

### Implementation Details/Notes/Constraints

* There will be some changes needed to both platform9/etcdadm (e.g. etcd2
  support) and kopeio/etcd-manager (to rebase on top of etcdadm).
* It is unlikely that e.g. GKE / EKS will use etcdadm (at least initially),
  which limits the pool of contributors.

### Risks and Mitigations

* Automatic mode may make incorrect decisions and break a cluster.  Mitigation:
  automated backups, and a willingness to stop and wait for a fix / operator
  intervention (CLI mode).
* Automatic mode relies on peer-to-peer discovery and gossiping, which is less
  reliable than Raft.  Mitigation: rely on Raft as much as possible, be very
  conservative in automated operations (favor correctness over availability or
  speed).  etcd non-voting members will make this much more reliable.

## Graduation Criteria

etcdadm will be considered successful when it is used by the majority of OSS
cluster installations.

## Implementation History

* Much SIG discussion
* Initial proposal to SIG 2018-10-09
* Initial KEP draft 2018-10-22
* Added clarification of cloud interaction 2018-10-23

## Infrastructure Needed

* etcdadm will be a subproject under sig-cluster-lifecycle
