--- date: 2016-04-05T00:17:00+01:00 description: Installing Rancher k3s with MariaDB Galera tags: - HOWTO - Kubernetes - Galera title: Howto Install Rancher k3s with MariaDB Galera --- In this blog Post we gonna install a HA Rancher Kubernetes Cluster with a MariaDB Galera Cluster as Datastore. #### Outline - [Prepare Ubuntu Bionic Server]() - [Install MariaDB Galera]() - [Deploy K3S]() - [Install kubectl and helm]() - [Install MetalLB]() - [Install cert-manager for Let's Encrypt]() - [Install Rancher]() - [Forward HTTP/HTTPS to the Rancher Load Balancer IP]() #### Prepare Ubuntu Bionic Server You need 3 Nodes, 4 CPU, >8GiB RAM, 100GiB Disk, I have 3 Nodes 4 CPU, 24 GiB RAM, 250GiB Disk. Install Ubuntu on one Server, remove snapd, ufw, cloud-init. Then clone it and edit /etc/hosts /etc/hostname /etc/netplan/50-cloud-init.yaml and `rm -f /etc/ssh/ssh_host_*` - reboot. #### Install MariaDB Galera Install the [MariaDB repo](https://downloads.mariadb.org/mariadb/repositories/#distro=Ubuntu&distro_release=bionic--ubuntu_bionic&mirror=host-europe&version=10.4) on each server. ```bash sudo apt-get install software-properties-common sudo apt-key adv --fetch-keys 'https://mariadb.org/mariadb_release_signing_key.asc' sudo add-apt-repository 'deb [arch=amd64,arm64,ppc64el] http://ftp.hosteurope.de/mirror/mariadb.org/repo/10.4/ubuntu bionic main' sudo apt update sudo apt install mariadb-server mariadb-client ``` ##### Secure MariaDB on each Node Set Password with mysql_secure_installation: ``` pcdummy@rancher01:~$ sudo mysql_secure_installation NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY! In order to log into MariaDB to secure it, we'll need the current password for the root user. If you've just installed MariaDB, and haven't set the root password yet, you should just press enter here. Enter current password for root (enter for none): OK, successfully used password, moving on... Setting the root password or using the unix_socket ensures that nobody can log into the MariaDB root user without the proper authorisation. You already have your root account protected, so you can safely answer 'n'. Switch to unix_socket authentication [Y/n] n ... skipping. You already have your root account protected, so you can safely answer 'n'. Change the root password? [Y/n] New password: Re-enter new password: Password updated successfully! Reloading privilege tables.. ... Success! By default, a MariaDB installation has an anonymous user, allowing anyone to log into MariaDB without having to have a user account created for them. This is intended only for testing, and to make the installation go a bit smoother. You should remove them before moving into a production environment. Remove anonymous users? [Y/n] Y ... Success! Normally, root should only be allowed to connect from 'localhost'. This ensures that someone cannot guess at the root password from the network. Disallow root login remotely? [Y/n] n ... skipping. By default, MariaDB comes with a database named 'test' that anyone can access. This is also intended only for testing, and should be removed before moving into a production environment. Remove test database and access to it? [Y/n] Y - Dropping test database... ... Success! - Removing privileges on test database... ... Success! Reloading the privilege tables will ensure that all changes made so far will take effect immediately. Reload privilege tables now? [Y/n] Y ... Success! Cleaning up... All done! If you've completed all of the above steps, your MariaDB installation should now be secure. Thanks for using MariaDB! ``` ##### Configure Galera on each Node Stop Mariadb ``` sudo systemctl stop mariadb ``` Liste on all Interfaces (if you want configure it to listen only on a specific address): ```bash sudo sed -i 's/bind-address\t\t= 127.0.0.1/#bind-address\t\t= 127.0.0.1/g' /etc/mysql/my.cnf ``` Enable Galera, Paste the following into /etc/mysql/mariadb.conf.d/99-cluster.cnf ``` [galera] wsrep_on = on wsrep_provider = /usr/lib/galera/libgalera_smm.so wsrep_cluster_address = gcomm://10.128.1.17,10.128.1.18,10.128.1.19 wsrep_cluster_name = k3s_cluster_0 default_storage_engine = InnoDB innodb_autoinc_lock_mode = 2 innodb_doublewrite = 1 binlog_format = ROW ``` And change the ip addresse for `wsrep_cluster_address` Some tuning if you use this Galera cluster for other purposes ``` sudo nano /etc/mysql/mariadb.conf.d/98-tuning.cnf ``` ``` [mysqld] key_buffer_size=256M thread_stack=192K thread_cache_size=8 max_connections=1000 innodb_buffer_pool_size=2G query_cache_limit=2M query_cache_size=0 query_cache_type=0 table_open_cache=128 join_buffer_size=512k table_definition_cache=-1 performance_schema=ON innodb_log_file_size=256M innodb_buffer_pool_instances=2 tmp_table_size=32M max_heap_table_size=32M ``` ##### Bootstrap the cluster One **one** node run `sudo galera_new_cluster` One the other 2 nodes run: `sudo systemctl start mariadb.service` ##### Create the k3s Database One one node run: ```bash mysql -u root -p ``` ```sql CREATE DATABASE `k3s`; GRANT ALL PRIVILEGES ON `k3s`.* TO 'k3s'@'%' IDENTIFIED BY ''; ``` #### Deploy k3s Install k3s one each nodes, one after another: ```bash curl -sfL https://get.k3s.io | sh -s - server --datastore-endpoint="mysql://k3s:@tcp(localhost:3306)/k3s" --no-deploy servicelb ``` Check the nodes after. ```bash sudo k3s kubectl get nodes ``` One one node copy the config (I choose node1 for that): ```bash mkdir ~/.kube sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config sudo chown -R $(whoami): ~/.kube ``` #### Install kubectl and helm Install kubectl ```bash sudo apt-get update && sudo apt-get install -y apt-transport-https curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list sudo apt-get update sudo apt-get install -y kubectl ``` Install helm to ~/bin ```bash wget https://get.helm.sh/helm-v3.2.1-linux-amd64.tar.gz tar xfz helm-v3.2.1-linux-amd64.tar.gz mkdir ~/bin mv linux-amd64/helm linux-amd64/tiller ~/bin rm -rf linux-amd64 ``` Init helm on the cluster ```bash kubectl --namespace kube-system create serviceaccount tiller; kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller; helm init --service-account tiller; ``` #### Install MetalLB Install MetalLB (change the address range!) ```bash helm install stable/metallb \ --name metallb \ --namespace kube-system \ --set configInline.address-pools[0].name=default \ --set configInline.address-pools[0].protocol=layer2 \ --set configInline.address-pools[0].addresses[0]=10.128.3.1-10.128.3.254 ``` Check the deployment ```bash kubectl get pods -n kube-system -l app=metallb -o wide ``` #### Install cert-manager for Let's Encrypt ```bash kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.12/deploy/manifests/00-crds.yaml kubectl create namespace cert-manager helm repo add jetstack https://charts.jetstack.io helm repo update helm install jetstack/cert-manager \ --name cert-manager \ --namespace cert-manager \ --version v0.12.0 ``` ```bash $ kubectl get pods --namespace cert-manager NAME READY STATUS RESTARTS AGE cert-manager-6bcdf8c5cc-5bcrg 1/1 Running 0 54s cert-manager-cainjector-6659d6844d-zrr5h 1/1 Running 0 54s cert-manager-webhook-547567b88f-ptrlg 1/1 Running 0 54s ``` #### Install Rancher ```bash helm repo add rancher-latest https://releases.rancher.com/server-charts/latest kubectl create namespace cattle-system helm install rancher-latest/rancher \ --name rancher \ --namespace cattle-system \ --set hostname=rancher.example.org \ --set ingress.tls.source=letsEncrypt \ --set letsEncrypt.email=support@example.org ``` ```bash $ kubectl -n cattle-system rollout status deploy/rancher Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available... Waiting for deployment "rancher" rollout to finish: 1 of 3 updated replicas are available... Waiting for deployment "rancher" rollout to finish: 2 of 3 updated replicas are available... deployment "rancher" successfully rolled out ``` #### Forward HTTP/HTTPS to the Rancher Load Balancer IP ```bash $ kubectl -n kube-system describe service/traefik Name: traefik Namespace: kube-system Labels: app=traefik chart=traefik-1.81.0 heritage=Helm release=traefik Annotations: field.cattle.io/publicEndpoints: [{"addresses":["10.128.3.1"],"port":80,"protocol":"TCP","serviceName":"kube-system:traefik","allNodes":false},{"addresses":["10.128.3.1"],... Selector: app=traefik,release=traefik Type: LoadBalancer IP: 10.43.47.63 LoadBalancer Ingress: 10.128.3.1 Port: http 80/TCP TargetPort: http/TCP NodePort: http 32316/TCP Endpoints: 10.42.0.6:80 Port: https 443/TCP TargetPort: https/TCP NodePort: https 30752/TCP Endpoints: 10.42.0.6:443 Session Affinity: None External Traffic Policy: Cluster Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal IPAllocated 16m metallb-controller Assigned IP "10.128.3.1" Normal nodeAssigned 3m10s (x5 over 16m) metallb-speaker announcing from node "rancher01" ``` Here the IP is 10.128.3.1 i forward HTTP (80) and HTTPS (443) to it. Wait for the Let's Encrypt Cert ``` $ kubectl -n cattle-system describe certificate Name: tls-rancher-ingress Namespace: cattle-system Labels: app=rancher chart=rancher-2.4.3 heritage=Tiller release=rancher Annotations: API Version: cert-manager.io/v1alpha2 Kind: Certificate Metadata: Creation Timestamp: 2020-05-07T23:07:41Z Generation: 1 Owner References: API Version: extensions/v1beta1 Block Owner Deletion: true Controller: true Kind: Ingress Name: rancher UID: 625bd78c-819a-4ba5-8ed0-4e8cf0497860 Resource Version: 23050 Self Link: /apis/cert-manager.io/v1alpha2/namespaces/cattle-system/certificates/tls-rancher-ingress UID: 6e0886ec-8b2d-4459-8c60-f994f269a146 Spec: Dns Names: rancher.example.org Issuer Ref: Group: cert-manager.io Kind: Issuer Name: rancher Secret Name: tls-rancher-ingress Status: Conditions: Last Transition Time: 2020-05-07T23:07:41Z Message: Waiting for CertificateRequest "tls-rancher-ingress-2753661366" to complete Reason: InProgress Status: False Type: Ready Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal GeneratedKey 18m cert-manager Generated a new private key Normal Requested 18m cert-manager Created new CertificateRequest resource "tls-rancher-ingress-2753661366" ``` I had some troubles with cert-manager where it wasn't able to access http:// without that it is not able to generate the certificate. #### Links [kauri.io](https://kauri.io/38-install-and-configure-a-kubernetes-cluster-with/418b3bc1e0544fbc955a4bbba6fff8a9/a) - Some Informations from there [howtoforge Galera](https://www.howtoforge.com/how-to-setup-mariadb-galera-multi-master-synchronous-replication-using-debian-10/) - Install Galera