--- date: 2020-05-08T02:21:00+02:00 description: Installing Rancher k3s with MariaDB Galera tags: - HOWTO - Kubernetes - Galera title: Howto Install Rancher k3s with MariaDB Galera --- In this blog Post we gonna install a HA Rancher Kubernetes Cluster with a MariaDB Galera Cluster as Datastore. #### Outline - [Prepare Ubuntu Bionic Server]() - [Install MariaDB Galera]() - [Deploy K3S]() - [Install kubectl and helm]() - [Install MetalLB]() - [Install cert-manager for Let's Encrypt]() - [Install Rancher]() - [Forward HTTP/HTTPS to the Rancher Load Balancer IP]() #### Prepare Ubuntu Bionic Server You need 3 Nodes, 4 CPU, >8GiB RAM, 100GiB Disk, I have 3 Nodes 4 CPU, 24 GiB RAM, 250GiB Disk. Install Ubuntu on one Server, remove snapd, ufw, cloud-init. Then clone it and edit /etc/hosts /etc/hostname /etc/netplan/50-cloud-init.yaml and `rm -f /etc/ssh/ssh_host_*` - reboot. #### Install MariaDB Galera Install the [MariaDB repo](https://downloads.mariadb.org/mariadb/repositories/#distro=Ubuntu&distro_release=bionic--ubuntu_bionic&mirror=host-europe&version=10.4) on each server. ```bash sudo apt-get install software-properties-common sudo apt-key adv --fetch-keys 'https://mariadb.org/mariadb_release_signing_key.asc' sudo add-apt-repository 'deb [arch=amd64] http://ftp.hosteurope.de/mirror/mariadb.org/repo/10.4/ubuntu bionic main' sudo apt update sudo apt install mariadb-server mariadb-client mariadb-backup ``` ##### Secure MariaDB on each Node Set Password with mysql_secure_installation: ```bash pcdummy@rancher01:~$ sudo mysql_secure_installation NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY! In order to log into MariaDB to secure it, we'll need the current password for the root user. If you've just installed MariaDB, and haven't set the root password yet, you should just press enter here. Enter current password for root (enter for none): OK, successfully used password, moving on... Setting the root password or using the unix_socket ensures that nobody can log into the MariaDB root user without the proper authorisation. You already have your root account protected, so you can safely answer 'n'. Switch to unix_socket authentication [Y/n] n ... skipping. You already have your root account protected, so you can safely answer 'n'. Change the root password? [Y/n] New password: Re-enter new password: Password updated successfully! Reloading privilege tables.. ... Success! By default, a MariaDB installation has an anonymous user, allowing anyone to log into MariaDB without having to have a user account created for them. This is intended only for testing, and to make the installation go a bit smoother. You should remove them before moving into a production environment. Remove anonymous users? [Y/n] Y ... Success! Normally, root should only be allowed to connect from 'localhost'. This ensures that someone cannot guess at the root password from the network. Disallow root login remotely? [Y/n] n ... skipping. By default, MariaDB comes with a database named 'test' that anyone can access. This is also intended only for testing, and should be removed before moving into a production environment. Remove test database and access to it? [Y/n] Y - Dropping test database... ... Success! - Removing privileges on test database... ... Success! Reloading the privilege tables will ensure that all changes made so far will take effect immediately. Reload privilege tables now? [Y/n] Y ... Success! Cleaning up... All done! If you've completed all of the above steps, your MariaDB installation should now be secure. Thanks for using MariaDB! ``` ##### Configure Galera on each Node Stop Mariadb ```bash sudo systemctl stop mariadb ``` Liste on all Interfaces (if you want configure it to listen only on a specific address): ```bash sudo sed -i 's/max_connections\t\t= 100/max_connections\t\t= 1000/g' /etc/mysql/my.cnf sudo sed -i 's/bind-address\t\t= 127.0.0.1/#bind-address\t\t= 127.0.0.1/g' /etc/mysql/my.cnf ``` Enable Galera, Paste the following into /etc/mysql/mariadb.conf.d/99-cluster.cnf ```INI [galera] wsrep_on = on wsrep_provider = /usr/lib/galera/libgalera_smm.so wsrep_cluster_address = gcomm://10.128.1.17,10.128.1.18,10.128.1.19 wsrep_cluster_name = k3s_cluster_0 wsrep_provider_options="gcache.size=512M" wsrep_sst_method = mariabackup wsrep_sst_auth = root:SuperSecretRootPassword default_storage_engine = InnoDB innodb_autoinc_lock_mode = 2 innodb_doublewrite = 1 binlog_format = ROW ``` And change the ip addresse for `wsrep_cluster_address` Some tuning if you use this Galera cluster for other purposes ```bash sudo nano /etc/mysql/mariadb.conf.d/98-tuning.cnf ``` ```INI [mysqld] key_buffer_size=256M thread_stack=192K thread_cache_size=8 max_connections=1000 innodb_buffer_pool_size=2G query_cache_limit=2M query_cache_size=0 query_cache_type=0 table_open_cache=128 join_buffer_size=512k table_definition_cache=-1 performance_schema=ON innodb_log_file_size=256M innodb_buffer_pool_instances=2 tmp_table_size=32M max_heap_table_size=32M ``` ##### Bootstrap the cluster One **one** node run `sudo galera_new_cluster` One the other 2 nodes run: `sudo systemctl start mariadb.service` ##### Check the MariaDB Galera Cluster ```bash mysql -u root -p -e "SELECT * FROM information_schema.global_status WHERE variable_name IN ('WSREP_CLUSTER_STATUS','WSREP_LOCAL_STATE_COMMENT','WSREP_CLUSTER_SIZE','WSREP_EVS_REPL_LATENCY','WSREP_EVS_DELAYED','WSREP_READY');" ``` ##### Install and configure MaxScale ```bash sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 135659E928C12247 sudo add-apt-repository 'deb [arch=amd64] http://downloads.mariadb.com/MaxScale/2.2/ubuntu bionic main' sudo apt install maxscale ``` ```sql mysql -u root -p CREATE USER 'maxscale'@'%' IDENTIFIED BY 'SuperSecretPassword'; GRANT SELECT ON mysql.user TO 'maxscale'@'%'; GRANT SELECT ON mysql.db TO 'maxscale'@'%'; GRANT SELECT ON mysql.tables_priv TO 'maxscale'@'%'; GRANT SELECT ON mysql.roles_mapping TO 'maxscale'@'%'; GRANT SHOW DATABASES ON *.* TO 'maxscale'@'%'; GRANT REPLICATION CLIENT ON *.* TO 'maxscale'@'%'; GRANT SUPER ON *.* TO maxscale@'%'; FLUSH PRIVILEGES; exit ``` Generate MaxScale Keys **note the encrypted password** and write replace it in /etc/maxscale.cnf ```bash sudo systemctl stop maxscale sudo maxkeys /var/lib/maxscale/ sudo maxpasswd 'SuperSecretPassword' sudo chown maxscale: /var/lib/maxscale/.secrets ``` /etc/maxscale.cnf ```INI # MaxScale documentation: # https://mariadb.com/kb/en/mariadb-enterprise/mariadb-maxscale-22/ # Global parameters # # Complete list of configuration options: # https://mariadb.com/kb/en/mariadb-enterprise/mariadb-maxscale-22-mariadb-maxscale-configuration-usage-scenarios/ [maxscale] threads=auto # Server definitions # # Set the address of the server to the network # address of a MariaDB server. # [server1] type=server address=10.248.8.1 port=3306 protocol=MariaDBBackend [server2] type=server address=10.248.8.2 port=3306 protocol=MariaDBBackend [server3] type=server address=10.248.8.3 port=3306 protocol=MariaDBBackend # Monitor for the servers # # This will keep MaxScale aware of the state of the servers. # MariaDB Monitor documentation: # https://mariadb.com/kb/en/mariadb-enterprise/mariadb-maxscale-22-mariadb-monitor/ [MariaDB-Monitor] type=monitor module=galeramon servers=server1,server2,server3 user=maxscale passwd=D83ED4E84351BD822950FDE5C2991889 monitor_interval=2000 # Service definitions # # Service Definition for a read-only service and # a read/write splitting service. # # ReadWriteSplit documentation: # https://mariadb.com/kb/en/mariadb-enterprise/mariadb-maxscale-22-readwritesplit/ [Read-Write-Service] type=service router=readwritesplit servers=server1,server2,server3 user=maxscale passwd=D83ED4E84351BD822950FDE5C2991889 # This service enables the use of the MaxAdmin interface # MaxScale administration guide: # https://mariadb.com/kb/en/mariadb-enterprise/mariadb-maxscale-22-maxadmin-admin-interface/ [MaxAdmin-Service] type=service router=cli # Listener definitions for the services # # These listeners represent the ports the # services will listen on. # [Read-Write-Listener] type=listener service=Read-Write-Service protocol=MariaDBClient port=4006 [MaxAdmin-Listener] type=listener service=MaxAdmin-Service protocol=maxscaled socket=default ``` ```bash sudo systemctl start maxscale sudo systemctl enable maxscale ``` ##### Create the k3s Database One one node run: ```bash mysql -u root -p ``` ```sql CREATE DATABASE `k3s`; GRANT ALL PRIVILEGES ON `k3s`.* TO 'k3s'@'%' IDENTIFIED BY ''; ``` #### Deploy k3s Install k3s one each nodes, one after another: ```bash curl -sfL https://get.k3s.io | sh -s - server --datastore-endpoint="mysql://k3s:SuperSecretPassword@tcp(localhost:4006)/k3s" --no-deploy servicelb --no-deploy traefik ``` Check the nodes after. ```bash sudo k3s kubectl get nodes ``` One one node copy the config (I choose node1 for that): ```bash mkdir ~/.kube sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config sudo chown -R $(whoami): ~/.kube ``` #### Install kubectl and helm Install kubectl ```bash sudo apt-get update && sudo apt-get install -y apt-transport-https curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list sudo apt-get update sudo apt-get install -y kubectl ``` Install helm to ~/bin ```bash curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 chmod 700 get_helm.sh ./get_helm.sh ``` #### Install MetalLB Install MetalLB (change the address range!) See: [metallb install](https://metallb.universe.tf/installation/) ```bash kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml # On first install only kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)" ``` metallb.yaml -> kubectl apply -f metallb.yaml ```yaml apiVersion: v1 kind: ConfigMap metadata: namespace: metallb-system name: config data: config: | address-pools: - name: default protocol: layer2 addresses: - 10.248.11.1-10.248.11.253 ``` Check the deployment ```bash kubectl get pods -n metallb-system -l app=metallb -o wide ``` #### Install cert-manager for Let's Encrypt ```bash kubectl create namespace cert-manager kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.15.0/cert-manager.crds.yaml helm repo add jetstack https://charts.jetstack.io helm repo update helm install \ cert-manager jetstack/cert-manager \ --namespace cert-manager \ --version v0.15.0 ``` ```bash $ kubectl get pods --namespace cert-manager NAME READY STATUS RESTARTS AGE cert-manager-6bcdf8c5cc-5bcrg 1/1 Running 0 54s cert-manager-cainjector-6659d6844d-zrr5h 1/1 Running 0 54s cert-manager-webhook-547567b88f-ptrlg 1/1 Running 0 54s ``` #### Install Nginx ```bash helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx helm repo update helm install ingress-nginx ingress-nginx/ingress-nginx \ --namespace kube-system \ --set controller.image.runAsUser=101 \ --set defaultBackend.enabled=false ``` #### Install Rancher ```bash helm repo add rancher-latest https://releases.rancher.com/server-charts/latest helm repo update kubectl create namespace cattle-system helm install rancher rancher-latest/rancher \ --namespace cattle-system \ --set hostname=rancher.example.org \ --set ingress.tls.source=letsEncrypt \ --set letsEncrypt.email=support@example.org ``` ```bash $ kubectl -n cattle-system rollout status deploy/rancher Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available... Waiting for deployment "rancher" rollout to finish: 1 of 3 updated replicas are available... Waiting for deployment "rancher" rollout to finish: 2 of 3 updated replicas are available... deployment "rancher" successfully rolled out ``` #### Forward HTTP/HTTPS to the Rancher Load Balancer IP ```bash $ kubectl -n kube-system describe service/traefik Name: traefik Namespace: kube-system Labels: app=traefik chart=traefik-1.81.0 heritage=Helm release=traefik Annotations: field.cattle.io/publicEndpoints: [{"addresses":["10.128.3.1"],"port":80,"protocol":"TCP","serviceName":"kube-system:traefik","allNodes":false},{"addresses":["10.128.3.1"],... Selector: app=traefik,release=traefik Type: LoadBalancer IP: 10.43.47.63 LoadBalancer Ingress: 10.128.3.1 Port: http 80/TCP TargetPort: http/TCP NodePort: http 32316/TCP Endpoints: 10.42.0.6:80 Port: https 443/TCP TargetPort: https/TCP NodePort: https 30752/TCP Endpoints: 10.42.0.6:443 Session Affinity: None External Traffic Policy: Cluster Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal IPAllocated 16m metallb-controller Assigned IP "10.128.3.1" Normal nodeAssigned 3m10s (x5 over 16m) metallb-speaker announcing from node "rancher01" ``` Here the IP is 10.128.3.1 i forward HTTP (80) and HTTPS (443) to it. Wait for the Let's Encrypt Cert ```bash $ kubectl -n cattle-system describe certificate Name: tls-rancher-ingress Namespace: cattle-system Labels: app=rancher chart=rancher-2.4.3 heritage=Tiller release=rancher Annotations: API Version: cert-manager.io/v1alpha2 Kind: Certificate Metadata: Creation Timestamp: 2020-05-07T23:07:41Z Generation: 1 Owner References: API Version: extensions/v1beta1 Block Owner Deletion: true Controller: true Kind: Ingress Name: rancher UID: 625bd78c-819a-4ba5-8ed0-4e8cf0497860 Resource Version: 23050 Self Link: /apis/cert-manager.io/v1alpha2/namespaces/cattle-system/certificates/tls-rancher-ingress UID: 6e0886ec-8b2d-4459-8c60-f994f269a146 Spec: Dns Names: rancher.example.org Issuer Ref: Group: cert-manager.io Kind: Issuer Name: rancher Secret Name: tls-rancher-ingress Status: Conditions: Last Transition Time: 2020-05-07T23:07:41Z Message: Waiting for CertificateRequest "tls-rancher-ingress-2753661366" to complete Reason: InProgress Status: False Type: Ready Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal GeneratedKey 18m cert-manager Generated a new private key Normal Requested 18m cert-manager Created new CertificateRequest resource "tls-rancher-ingress-2753661366" ``` I had some troubles with cert-manager where it wasn't able to access http:// without that it is not able to generate the certificate. #### Links [kauri.io](https://kauri.io/38-install-and-configure-a-kubernetes-cluster-with/418b3bc1e0544fbc955a4bbba6fff8a9/a) - Some Informations from there [howtoforge Galera](https://www.howtoforge.com/how-to-setup-mariadb-galera-multi-master-synchronous-replication-using-debian-10/) - Install Galera