diff --git a/content/post/rancher-k3s-with-galera.md b/content/post/rancher-k3s-with-galera.md new file mode 100644 index 0000000..c9e9056 --- /dev/null +++ b/content/post/rancher-k3s-with-galera.md @@ -0,0 +1,401 @@ +--- +date: 2016-04-05T00:17:00+01:00 +description: Installing Rancher k3s with MariaDB Galera +tags: +- HOWTO +- Kubernetes +- Galera +title: Howto Install Rancher k3s with MariaDB Galera +--- + +In this blog Post we gonna install a HA Rancher Kubernetes Cluster with a MariaDB Galera Cluster as Datastore. + + + +#### Outline + +- [Prepare Ubuntu Bionic Server]() +- [Install MariaDB Galera]() +- [Deploy K3S]() +- [Install kubectl and helm]() +- [Install MetalLB]() +- [Install cert-manager for Let's Encrypt]() +- [Install Rancher]() +- [Forward HTTP/HTTPS to the Rancher Load Balancer IP]() + +#### Prepare Ubuntu Bionic Server + +You need 3 Nodes, 4 CPU, >8GiB RAM, 100GiB Disk, I have 3 Nodes 4 CPU, 24 GiB RAM, 250GiB Disk. + +Install Ubuntu on one Server, remove snapd, ufw, cloud-init. +Then clone it and edit /etc/hosts /etc/hostname /etc/netplan/50-cloud-init.yaml and `rm -f /etc/ssh/ssh_host_*` - reboot. + +#### Install MariaDB Galera + +Install the [MariaDB repo](https://downloads.mariadb.org/mariadb/repositories/#distro=Ubuntu&distro_release=bionic--ubuntu_bionic&mirror=host-europe&version=10.4) on each server. + +```bash +sudo apt-get install software-properties-common +sudo apt-key adv --fetch-keys 'https://mariadb.org/mariadb_release_signing_key.asc' +sudo add-apt-repository 'deb [arch=amd64,arm64,ppc64el] http://ftp.hosteurope.de/mirror/mariadb.org/repo/10.4/ubuntu bionic main' + +sudo apt update +sudo apt install mariadb-server mariadb-client +``` + +##### Secure MariaDB on each Node + +Set Password with mysql_secure_installation: + +``` +pcdummy@rancher01:~$ sudo mysql_secure_installation + +NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB + SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY! + +In order to log into MariaDB to secure it, we'll need the current +password for the root user. If you've just installed MariaDB, and +haven't set the root password yet, you should just press enter here. + +Enter current password for root (enter for none): +OK, successfully used password, moving on... + +Setting the root password or using the unix_socket ensures that nobody +can log into the MariaDB root user without the proper authorisation. + +You already have your root account protected, so you can safely answer 'n'. + +Switch to unix_socket authentication [Y/n] n + ... skipping. + +You already have your root account protected, so you can safely answer 'n'. + +Change the root password? [Y/n] +New password: +Re-enter new password: +Password updated successfully! +Reloading privilege tables.. + ... Success! + + +By default, a MariaDB installation has an anonymous user, allowing anyone +to log into MariaDB without having to have a user account created for +them. This is intended only for testing, and to make the installation +go a bit smoother. You should remove them before moving into a +production environment. + +Remove anonymous users? [Y/n] Y + ... Success! + +Normally, root should only be allowed to connect from 'localhost'. This +ensures that someone cannot guess at the root password from the network. + +Disallow root login remotely? [Y/n] n + ... skipping. + +By default, MariaDB comes with a database named 'test' that anyone can +access. This is also intended only for testing, and should be removed +before moving into a production environment. + +Remove test database and access to it? [Y/n] Y + - Dropping test database... + ... Success! + - Removing privileges on test database... + ... Success! + +Reloading the privilege tables will ensure that all changes made so far +will take effect immediately. + +Reload privilege tables now? [Y/n] Y + ... Success! + +Cleaning up... + +All done! If you've completed all of the above steps, your MariaDB +installation should now be secure. + +Thanks for using MariaDB! +``` + +##### Configure Galera on each Node + +Stop Mariadb + +``` +sudo systemctl stop mariadb +``` + +Liste on all Interfaces (if you want configure it to listen only on a specific address): + +```bash +sudo sed -i 's/bind-address\t\t= 127.0.0.1/#bind-address\t\t= 127.0.0.1/g' /etc/mysql/my.cnf +``` + +Enable Galera, Paste the following into /etc/mysql/mariadb.conf.d/99-cluster.cnf + +``` +[galera] + +wsrep_on = on +wsrep_provider = /usr/lib/galera/libgalera_smm.so +wsrep_cluster_address = gcomm://10.128.1.17,10.128.1.18,10.128.1.19 +wsrep_cluster_name = k3s_cluster_0 + +default_storage_engine = InnoDB +innodb_autoinc_lock_mode = 2 +innodb_doublewrite = 1 + +binlog_format = ROW +``` + +And change the ip addresse for `wsrep_cluster_address` + +Some tuning if you use this Galera cluster for other purposes + +``` +sudo nano /etc/mysql/mariadb.conf.d/98-tuning.cnf +``` + +``` +[mysqld] +key_buffer_size=256M +thread_stack=192K +thread_cache_size=8 +max_connections=1000 +innodb_buffer_pool_size=2G +query_cache_limit=2M +query_cache_size=0 +query_cache_type=0 +table_open_cache=128 +join_buffer_size=512k +table_definition_cache=-1 +performance_schema=ON +innodb_log_file_size=256M +innodb_buffer_pool_instances=2 +tmp_table_size=32M +max_heap_table_size=32M +``` + +##### Bootstrap the cluster + +One **one** node run `sudo galera_new_cluster` + +One the other 2 nodes run: `sudo systemctl start mariadb.service` + +##### Create the k3s Database + +One one node run: + +```bash +mysql -u root -p +``` + +```sql +CREATE DATABASE `k3s`; +GRANT ALL PRIVILEGES ON `k3s`.* TO 'k3s'@'%' IDENTIFIED BY ''; +``` + + +#### Deploy k3s + +Install k3s one each nodes, one after another: + +```bash +curl -sfL https://get.k3s.io | sh -s - server --datastore-endpoint="mysql://k3s:@tcp(localhost:3306)/k3s" --no-deploy servicelb +``` + +Check the nodes after. + +```bash +sudo k3s kubectl get nodes +``` + +One one node copy the config (I choose node1 for that): + +```bash +mkdir ~/.kube +sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config +sudo chown -R $(whoami): ~/.kube +``` + +#### Install kubectl and helm + +Install kubectl + +```bash +sudo apt-get update && sudo apt-get install -y apt-transport-https +curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - +echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list +sudo apt-get update +sudo apt-get install -y kubectl +``` + +Install helm to ~/bin + +```bash +wget https://get.helm.sh/helm-v3.2.1-linux-amd64.tar.gz +tar xfz helm-v3.2.1-linux-amd64.tar.gz +mkdir ~/bin +mv linux-amd64/helm linux-amd64/tiller ~/bin +rm -rf linux-amd64 +``` + +Init helm on the cluster + +```bash +kubectl --namespace kube-system create serviceaccount tiller; +kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller; +helm init --service-account tiller; +``` + +#### Install MetalLB + +Install MetalLB (change the address range!) + +```bash +helm install stable/metallb \ + --name metallb \ + --namespace kube-system \ + --set configInline.address-pools[0].name=default \ + --set configInline.address-pools[0].protocol=layer2 \ + --set configInline.address-pools[0].addresses[0]=10.128.3.1-10.128.3.254 +``` + +Check the deployment + +```bash +kubectl get pods -n kube-system -l app=metallb -o wide +``` + +#### Install cert-manager for Let's Encrypt + +```bash +kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.12/deploy/manifests/00-crds.yaml +kubectl create namespace cert-manager +helm repo add jetstack https://charts.jetstack.io +helm repo update +helm install jetstack/cert-manager \ + --name cert-manager \ + --namespace cert-manager \ + --version v0.12.0 +``` + +```bash +$ kubectl get pods --namespace cert-manager +NAME READY STATUS RESTARTS AGE +cert-manager-6bcdf8c5cc-5bcrg 1/1 Running 0 54s +cert-manager-cainjector-6659d6844d-zrr5h 1/1 Running 0 54s +cert-manager-webhook-547567b88f-ptrlg 1/1 Running 0 54s +``` + +#### Install Rancher + +```bash +helm repo add rancher-latest https://releases.rancher.com/server-charts/latest +kubectl create namespace cattle-system +helm install rancher-latest/rancher \ + --name rancher \ + --namespace cattle-system \ + --set hostname=rancher.example.org \ + --set ingress.tls.source=letsEncrypt \ + --set letsEncrypt.email=support@example.org +``` + +```bash +$ kubectl -n cattle-system rollout status deploy/rancher +Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available... + +Waiting for deployment "rancher" rollout to finish: 1 of 3 updated replicas are available... +Waiting for deployment "rancher" rollout to finish: 2 of 3 updated replicas are available... +deployment "rancher" successfully rolled out +``` + +#### Forward HTTP/HTTPS to the Rancher Load Balancer IP + +```bash +$ kubectl -n kube-system describe service/traefik +Name: traefik +Namespace: kube-system +Labels: app=traefik + chart=traefik-1.81.0 + heritage=Helm + release=traefik +Annotations: field.cattle.io/publicEndpoints: + [{"addresses":["10.128.3.1"],"port":80,"protocol":"TCP","serviceName":"kube-system:traefik","allNodes":false},{"addresses":["10.128.3.1"],... +Selector: app=traefik,release=traefik +Type: LoadBalancer +IP: 10.43.47.63 +LoadBalancer Ingress: 10.128.3.1 +Port: http 80/TCP +TargetPort: http/TCP +NodePort: http 32316/TCP +Endpoints: 10.42.0.6:80 +Port: https 443/TCP +TargetPort: https/TCP +NodePort: https 30752/TCP +Endpoints: 10.42.0.6:443 +Session Affinity: None +External Traffic Policy: Cluster +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal IPAllocated 16m metallb-controller Assigned IP "10.128.3.1" + Normal nodeAssigned 3m10s (x5 over 16m) metallb-speaker announcing from node "rancher01" +``` + +Here the IP is 10.128.3.1 i forward HTTP (80) and HTTPS (443) to it. + + +Wait for the Let's Encrypt Cert +``` +$ kubectl -n cattle-system describe certificate +Name: tls-rancher-ingress +Namespace: cattle-system +Labels: app=rancher + chart=rancher-2.4.3 + heritage=Tiller + release=rancher +Annotations: +API Version: cert-manager.io/v1alpha2 +Kind: Certificate +Metadata: + Creation Timestamp: 2020-05-07T23:07:41Z + Generation: 1 + Owner References: + API Version: extensions/v1beta1 + Block Owner Deletion: true + Controller: true + Kind: Ingress + Name: rancher + UID: 625bd78c-819a-4ba5-8ed0-4e8cf0497860 + Resource Version: 23050 + Self Link: /apis/cert-manager.io/v1alpha2/namespaces/cattle-system/certificates/tls-rancher-ingress + UID: 6e0886ec-8b2d-4459-8c60-f994f269a146 +Spec: + Dns Names: + rancher.example.org + Issuer Ref: + Group: cert-manager.io + Kind: Issuer + Name: rancher + Secret Name: tls-rancher-ingress +Status: + Conditions: + Last Transition Time: 2020-05-07T23:07:41Z + Message: Waiting for CertificateRequest "tls-rancher-ingress-2753661366" to complete + Reason: InProgress + Status: False + Type: Ready +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal GeneratedKey 18m cert-manager Generated a new private key + Normal Requested 18m cert-manager Created new CertificateRequest resource "tls-rancher-ingress-2753661366" +``` + +I had some troubles with cert-manager where it wasn't able to access http:// without that it is not able to generate the certificate. + +#### Links + +[kauri.io](https://kauri.io/38-install-and-configure-a-kubernetes-cluster-with/418b3bc1e0544fbc955a4bbba6fff8a9/a) - Some Informations from there +[howtoforge Galera](https://www.howtoforge.com/how-to-setup-mariadb-galera-multi-master-synchronous-replication-using-debian-10/) - Install Galera \ No newline at end of file