You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
562 lines
16 KiB
Markdown
562 lines
16 KiB
Markdown
---
|
|
date: 2020-05-08T02:21:00+02:00
|
|
description: Installing Rancher k3s with MariaDB Galera
|
|
tags:
|
|
- HOWTO
|
|
- Kubernetes
|
|
- Galera
|
|
title: Howto Install Rancher k3s with MariaDB Galera
|
|
---
|
|
|
|
In this blog Post we gonna install a HA Rancher Kubernetes Cluster with a MariaDB Galera Cluster as Datastore.
|
|
|
|
<!--more-->
|
|
|
|
#### Outline
|
|
|
|
- [Prepare Ubuntu Bionic Server]()
|
|
- [Install MariaDB Galera]()
|
|
- [Deploy K3S]()
|
|
- [Install kubectl and helm]()
|
|
- [Install MetalLB]()
|
|
- [Install cert-manager for Let's Encrypt]()
|
|
- [Install Rancher]()
|
|
- [Forward HTTP/HTTPS to the Rancher Load Balancer IP]()
|
|
|
|
#### Prepare Ubuntu Bionic Server
|
|
|
|
You need 3 Nodes, 4 CPU, >8GiB RAM, 100GiB Disk, I have 3 Nodes 4 CPU, 24 GiB RAM, 250GiB Disk.
|
|
|
|
Install Ubuntu on one Server, remove snapd, ufw, cloud-init.
|
|
Then clone it and edit /etc/hosts /etc/hostname /etc/netplan/50-cloud-init.yaml and `rm -f /etc/ssh/ssh_host_*` - reboot.
|
|
|
|
#### Install MariaDB Galera
|
|
|
|
Install the [MariaDB repo](https://downloads.mariadb.org/mariadb/repositories/#distro=Ubuntu&distro_release=bionic--ubuntu_bionic&mirror=host-europe&version=10.4) on each server.
|
|
|
|
```bash
|
|
sudo apt-get install software-properties-common
|
|
sudo apt-key adv --fetch-keys 'https://mariadb.org/mariadb_release_signing_key.asc'
|
|
sudo add-apt-repository 'deb [arch=amd64] http://ftp.hosteurope.de/mirror/mariadb.org/repo/10.4/ubuntu bionic main'
|
|
|
|
sudo apt update
|
|
sudo apt install mariadb-server mariadb-client mariadb-backup
|
|
```
|
|
|
|
##### Secure MariaDB on each Node
|
|
|
|
Set Password with mysql_secure_installation:
|
|
|
|
```bash
|
|
pcdummy@rancher01:~$ sudo mysql_secure_installation
|
|
|
|
NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
|
|
SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!
|
|
|
|
In order to log into MariaDB to secure it, we'll need the current
|
|
password for the root user. If you've just installed MariaDB, and
|
|
haven't set the root password yet, you should just press enter here.
|
|
|
|
Enter current password for root (enter for none):
|
|
OK, successfully used password, moving on...
|
|
|
|
Setting the root password or using the unix_socket ensures that nobody
|
|
can log into the MariaDB root user without the proper authorisation.
|
|
|
|
You already have your root account protected, so you can safely answer 'n'.
|
|
|
|
Switch to unix_socket authentication [Y/n] n
|
|
... skipping.
|
|
|
|
You already have your root account protected, so you can safely answer 'n'.
|
|
|
|
Change the root password? [Y/n]
|
|
New password:
|
|
Re-enter new password:
|
|
Password updated successfully!
|
|
Reloading privilege tables..
|
|
... Success!
|
|
|
|
|
|
By default, a MariaDB installation has an anonymous user, allowing anyone
|
|
to log into MariaDB without having to have a user account created for
|
|
them. This is intended only for testing, and to make the installation
|
|
go a bit smoother. You should remove them before moving into a
|
|
production environment.
|
|
|
|
Remove anonymous users? [Y/n] Y
|
|
... Success!
|
|
|
|
Normally, root should only be allowed to connect from 'localhost'. This
|
|
ensures that someone cannot guess at the root password from the network.
|
|
|
|
Disallow root login remotely? [Y/n] n
|
|
... skipping.
|
|
|
|
By default, MariaDB comes with a database named 'test' that anyone can
|
|
access. This is also intended only for testing, and should be removed
|
|
before moving into a production environment.
|
|
|
|
Remove test database and access to it? [Y/n] Y
|
|
- Dropping test database...
|
|
... Success!
|
|
- Removing privileges on test database...
|
|
... Success!
|
|
|
|
Reloading the privilege tables will ensure that all changes made so far
|
|
will take effect immediately.
|
|
|
|
Reload privilege tables now? [Y/n] Y
|
|
... Success!
|
|
|
|
Cleaning up...
|
|
|
|
All done! If you've completed all of the above steps, your MariaDB
|
|
installation should now be secure.
|
|
|
|
Thanks for using MariaDB!
|
|
```
|
|
|
|
##### Configure Galera on each Node
|
|
|
|
Stop Mariadb
|
|
|
|
```bash
|
|
sudo systemctl stop mariadb
|
|
```
|
|
|
|
Liste on all Interfaces (if you want configure it to listen only on a specific address):
|
|
|
|
```bash
|
|
sudo sed -i 's/max_connections\t\t= 100/max_connections\t\t= 1000/g' /etc/mysql/my.cnf
|
|
sudo sed -i 's/bind-address\t\t= 127.0.0.1/#bind-address\t\t= 127.0.0.1/g' /etc/mysql/my.cnf
|
|
```
|
|
|
|
Enable Galera, Paste the following into /etc/mysql/mariadb.conf.d/99-cluster.cnf
|
|
|
|
```INI
|
|
[galera]
|
|
|
|
wsrep_on = on
|
|
wsrep_provider = /usr/lib/galera/libgalera_smm.so
|
|
wsrep_cluster_address = gcomm://10.128.1.17,10.128.1.18,10.128.1.19
|
|
wsrep_cluster_name = k3s_cluster_0
|
|
wsrep_provider_options="gcache.size=512M"
|
|
wsrep_sst_method = mariabackup
|
|
wsrep_sst_auth = root:SuperSecretRootPassword
|
|
|
|
default_storage_engine = InnoDB
|
|
innodb_autoinc_lock_mode = 2
|
|
innodb_doublewrite = 1
|
|
|
|
binlog_format = ROW
|
|
```
|
|
|
|
And change the ip addresse for `wsrep_cluster_address`
|
|
|
|
Some tuning if you use this Galera cluster for other purposes
|
|
|
|
```bash
|
|
sudo nano /etc/mysql/mariadb.conf.d/98-tuning.cnf
|
|
```
|
|
|
|
```INI
|
|
[mysqld]
|
|
key_buffer_size=256M
|
|
thread_stack=192K
|
|
thread_cache_size=8
|
|
max_connections=1000
|
|
innodb_buffer_pool_size=2G
|
|
query_cache_limit=2M
|
|
query_cache_size=0
|
|
query_cache_type=0
|
|
table_open_cache=128
|
|
join_buffer_size=512k
|
|
table_definition_cache=-1
|
|
performance_schema=ON
|
|
innodb_log_file_size=256M
|
|
innodb_buffer_pool_instances=2
|
|
tmp_table_size=32M
|
|
max_heap_table_size=32M
|
|
```
|
|
|
|
##### Bootstrap the cluster
|
|
|
|
One **one** node run `sudo galera_new_cluster`
|
|
|
|
One the other 2 nodes run: `sudo systemctl start mariadb.service`
|
|
|
|
##### Check the MariaDB Galera Cluster
|
|
|
|
```bash
|
|
mysql -u root -p -e "SELECT * FROM information_schema.global_status WHERE variable_name IN ('WSREP_CLUSTER_STATUS','WSREP_LOCAL_STATE_COMMENT','WSREP_CLUSTER_SIZE','WSREP_EVS_REPL_LATENCY','WSREP_EVS_DELAYED','WSREP_READY');"
|
|
```
|
|
|
|
|
|
##### Install and configure MaxScale
|
|
|
|
```bash
|
|
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 135659E928C12247
|
|
sudo add-apt-repository 'deb [arch=amd64] http://downloads.mariadb.com/MaxScale/2.2/ubuntu bionic main'
|
|
sudo apt install maxscale
|
|
```
|
|
|
|
```sql
|
|
mysql -u root -p
|
|
CREATE USER 'maxscale'@'%' IDENTIFIED BY 'SuperSecretPassword';
|
|
GRANT SELECT ON mysql.user TO 'maxscale'@'%';
|
|
GRANT SELECT ON mysql.db TO 'maxscale'@'%';
|
|
GRANT SELECT ON mysql.tables_priv TO 'maxscale'@'%';
|
|
GRANT SELECT ON mysql.roles_mapping TO 'maxscale'@'%';
|
|
GRANT SHOW DATABASES ON *.* TO 'maxscale'@'%';
|
|
GRANT REPLICATION CLIENT ON *.* TO 'maxscale'@'%';
|
|
GRANT SUPER ON *.* TO maxscale@'%';
|
|
FLUSH PRIVILEGES;
|
|
exit
|
|
```
|
|
|
|
Generate MaxScale Keys **note the encrypted password** and write replace it in /etc/maxscale.cnf
|
|
|
|
```bash
|
|
sudo systemctl stop maxscale
|
|
sudo maxkeys /var/lib/maxscale/
|
|
sudo maxpasswd 'SuperSecretPassword'
|
|
sudo chown maxscale: /var/lib/maxscale/.secrets
|
|
```
|
|
|
|
/etc/maxscale.cnf
|
|
|
|
```INI
|
|
# MaxScale documentation:
|
|
# https://mariadb.com/kb/en/mariadb-enterprise/mariadb-maxscale-22/
|
|
|
|
# Global parameters
|
|
#
|
|
# Complete list of configuration options:
|
|
# https://mariadb.com/kb/en/mariadb-enterprise/mariadb-maxscale-22-mariadb-maxscale-configuration-usage-scenarios/
|
|
|
|
[maxscale]
|
|
threads=auto
|
|
|
|
# Server definitions
|
|
#
|
|
# Set the address of the server to the network
|
|
# address of a MariaDB server.
|
|
#
|
|
|
|
[server1]
|
|
type=server
|
|
address=10.248.8.1
|
|
port=3306
|
|
protocol=MariaDBBackend
|
|
|
|
[server2]
|
|
type=server
|
|
address=10.248.8.2
|
|
port=3306
|
|
protocol=MariaDBBackend
|
|
|
|
[server3]
|
|
type=server
|
|
address=10.248.8.3
|
|
port=3306
|
|
protocol=MariaDBBackend
|
|
|
|
# Monitor for the servers
|
|
#
|
|
# This will keep MaxScale aware of the state of the servers.
|
|
# MariaDB Monitor documentation:
|
|
# https://mariadb.com/kb/en/mariadb-enterprise/mariadb-maxscale-22-mariadb-monitor/
|
|
|
|
[MariaDB-Monitor]
|
|
type=monitor
|
|
module=galeramon
|
|
servers=server1,server2,server3
|
|
user=maxscale
|
|
passwd=D83ED4E84351BD822950FDE5C2991889
|
|
monitor_interval=2000
|
|
|
|
# Service definitions
|
|
#
|
|
# Service Definition for a read-only service and
|
|
# a read/write splitting service.
|
|
#
|
|
|
|
# ReadWriteSplit documentation:
|
|
# https://mariadb.com/kb/en/mariadb-enterprise/mariadb-maxscale-22-readwritesplit/
|
|
|
|
[Read-Write-Service]
|
|
type=service
|
|
router=readwritesplit
|
|
servers=server1,server2,server3
|
|
user=maxscale
|
|
passwd=D83ED4E84351BD822950FDE5C2991889
|
|
|
|
# This service enables the use of the MaxAdmin interface
|
|
# MaxScale administration guide:
|
|
# https://mariadb.com/kb/en/mariadb-enterprise/mariadb-maxscale-22-maxadmin-admin-interface/
|
|
|
|
[MaxAdmin-Service]
|
|
type=service
|
|
router=cli
|
|
|
|
# Listener definitions for the services
|
|
#
|
|
# These listeners represent the ports the
|
|
# services will listen on.
|
|
#
|
|
|
|
[Read-Write-Listener]
|
|
type=listener
|
|
service=Read-Write-Service
|
|
protocol=MariaDBClient
|
|
port=4006
|
|
|
|
[MaxAdmin-Listener]
|
|
type=listener
|
|
service=MaxAdmin-Service
|
|
protocol=maxscaled
|
|
socket=default
|
|
```
|
|
|
|
```bash
|
|
sudo systemctl start maxscale
|
|
sudo systemctl enable maxscale
|
|
```
|
|
|
|
##### Create the k3s Database
|
|
|
|
One one node run:
|
|
|
|
```bash
|
|
mysql -u root -p
|
|
```
|
|
|
|
```sql
|
|
CREATE DATABASE `k3s`;
|
|
GRANT ALL PRIVILEGES ON `k3s`.* TO 'k3s'@'%' IDENTIFIED BY '<superSecret>';
|
|
```
|
|
|
|
|
|
#### Deploy k3s
|
|
|
|
Install k3s one each nodes, one after another:
|
|
|
|
```bash
|
|
curl -sfL https://get.k3s.io | sh -s - server --datastore-endpoint="mysql://k3s:SuperSecretPassword@tcp(localhost:4006)/k3s" --no-deploy servicelb --no-deploy traefik
|
|
```
|
|
|
|
Check the nodes after.
|
|
|
|
```bash
|
|
sudo k3s kubectl get nodes
|
|
```
|
|
|
|
One one node copy the config (I choose node1 for that):
|
|
|
|
```bash
|
|
mkdir ~/.kube
|
|
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
|
|
sudo chown -R $(whoami): ~/.kube
|
|
```
|
|
|
|
#### Install kubectl and helm
|
|
|
|
Install kubectl
|
|
|
|
```bash
|
|
sudo apt-get update && sudo apt-get install -y apt-transport-https
|
|
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
|
|
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
|
|
sudo apt-get update
|
|
sudo apt-get install -y kubectl
|
|
```
|
|
|
|
Install helm to ~/bin
|
|
|
|
```bash
|
|
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
|
|
chmod 700 get_helm.sh
|
|
./get_helm.sh
|
|
```
|
|
|
|
#### Install MetalLB
|
|
|
|
Install MetalLB (change the address range!)
|
|
|
|
See: [metallb install](https://metallb.universe.tf/installation/)
|
|
|
|
```bash
|
|
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml
|
|
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml
|
|
# On first install only
|
|
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
|
|
```
|
|
|
|
metallb.yaml -> kubectl apply -f metallb.yaml
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
namespace: metallb-system
|
|
name: config
|
|
data:
|
|
config: |
|
|
address-pools:
|
|
- name: default
|
|
protocol: layer2
|
|
addresses:
|
|
- 10.248.11.1-10.248.11.253
|
|
```
|
|
|
|
Check the deployment
|
|
|
|
```bash
|
|
kubectl get pods -n metallb-system -l app=metallb -o wide
|
|
```
|
|
|
|
#### Install cert-manager for Let's Encrypt
|
|
|
|
```bash
|
|
kubectl create namespace cert-manager
|
|
kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.15.0/cert-manager.crds.yaml
|
|
helm repo add jetstack https://charts.jetstack.io
|
|
helm repo update
|
|
helm install \
|
|
cert-manager jetstack/cert-manager \
|
|
--namespace cert-manager \
|
|
--version v0.15.0
|
|
```
|
|
|
|
```bash
|
|
$ kubectl get pods --namespace cert-manager
|
|
NAME READY STATUS RESTARTS AGE
|
|
cert-manager-6bcdf8c5cc-5bcrg 1/1 Running 0 54s
|
|
cert-manager-cainjector-6659d6844d-zrr5h 1/1 Running 0 54s
|
|
cert-manager-webhook-547567b88f-ptrlg 1/1 Running 0 54s
|
|
```
|
|
|
|
#### Install Nginx
|
|
|
|
```bash
|
|
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
|
|
helm repo update
|
|
helm install ingress-nginx ingress-nginx/ingress-nginx \
|
|
--namespace kube-system \
|
|
--set controller.image.runAsUser=101 \
|
|
--set defaultBackend.enabled=false
|
|
```
|
|
|
|
#### Install Rancher
|
|
|
|
```bash
|
|
helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
|
|
helm repo update
|
|
kubectl create namespace cattle-system
|
|
helm install rancher rancher-latest/rancher \
|
|
--namespace cattle-system \
|
|
--set hostname=rancher.example.org \
|
|
--set ingress.tls.source=letsEncrypt \
|
|
--set letsEncrypt.email=support@example.org
|
|
```
|
|
|
|
```bash
|
|
$ kubectl -n cattle-system rollout status deploy/rancher
|
|
Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available...
|
|
|
|
Waiting for deployment "rancher" rollout to finish: 1 of 3 updated replicas are available...
|
|
Waiting for deployment "rancher" rollout to finish: 2 of 3 updated replicas are available...
|
|
deployment "rancher" successfully rolled out
|
|
```
|
|
|
|
#### Forward HTTP/HTTPS to the Rancher Load Balancer IP
|
|
|
|
```bash
|
|
$ kubectl -n kube-system describe service/traefik
|
|
Name: traefik
|
|
Namespace: kube-system
|
|
Labels: app=traefik
|
|
chart=traefik-1.81.0
|
|
heritage=Helm
|
|
release=traefik
|
|
Annotations: field.cattle.io/publicEndpoints:
|
|
[{"addresses":["10.128.3.1"],"port":80,"protocol":"TCP","serviceName":"kube-system:traefik","allNodes":false},{"addresses":["10.128.3.1"],...
|
|
Selector: app=traefik,release=traefik
|
|
Type: LoadBalancer
|
|
IP: 10.43.47.63
|
|
LoadBalancer Ingress: 10.128.3.1
|
|
Port: http 80/TCP
|
|
TargetPort: http/TCP
|
|
NodePort: http 32316/TCP
|
|
Endpoints: 10.42.0.6:80
|
|
Port: https 443/TCP
|
|
TargetPort: https/TCP
|
|
NodePort: https 30752/TCP
|
|
Endpoints: 10.42.0.6:443
|
|
Session Affinity: None
|
|
External Traffic Policy: Cluster
|
|
Events:
|
|
Type Reason Age From Message
|
|
---- ------ ---- ---- -------
|
|
Normal IPAllocated 16m metallb-controller Assigned IP "10.128.3.1"
|
|
Normal nodeAssigned 3m10s (x5 over 16m) metallb-speaker announcing from node "rancher01"
|
|
```
|
|
|
|
Here the IP is 10.128.3.1 i forward HTTP (80) and HTTPS (443) to it.
|
|
|
|
|
|
Wait for the Let's Encrypt Cert
|
|
|
|
```bash
|
|
$ kubectl -n cattle-system describe certificate
|
|
Name: tls-rancher-ingress
|
|
Namespace: cattle-system
|
|
Labels: app=rancher
|
|
chart=rancher-2.4.3
|
|
heritage=Tiller
|
|
release=rancher
|
|
Annotations: <none>
|
|
API Version: cert-manager.io/v1alpha2
|
|
Kind: Certificate
|
|
Metadata:
|
|
Creation Timestamp: 2020-05-07T23:07:41Z
|
|
Generation: 1
|
|
Owner References:
|
|
API Version: extensions/v1beta1
|
|
Block Owner Deletion: true
|
|
Controller: true
|
|
Kind: Ingress
|
|
Name: rancher
|
|
UID: 625bd78c-819a-4ba5-8ed0-4e8cf0497860
|
|
Resource Version: 23050
|
|
Self Link: /apis/cert-manager.io/v1alpha2/namespaces/cattle-system/certificates/tls-rancher-ingress
|
|
UID: 6e0886ec-8b2d-4459-8c60-f994f269a146
|
|
Spec:
|
|
Dns Names:
|
|
rancher.example.org
|
|
Issuer Ref:
|
|
Group: cert-manager.io
|
|
Kind: Issuer
|
|
Name: rancher
|
|
Secret Name: tls-rancher-ingress
|
|
Status:
|
|
Conditions:
|
|
Last Transition Time: 2020-05-07T23:07:41Z
|
|
Message: Waiting for CertificateRequest "tls-rancher-ingress-2753661366" to complete
|
|
Reason: InProgress
|
|
Status: False
|
|
Type: Ready
|
|
Events:
|
|
Type Reason Age From Message
|
|
---- ------ ---- ---- -------
|
|
Normal GeneratedKey 18m cert-manager Generated a new private key
|
|
Normal Requested 18m cert-manager Created new CertificateRequest resource "tls-rancher-ingress-2753661366"
|
|
```
|
|
|
|
I had some troubles with cert-manager where it wasn't able to access http://<rancher-FQDN> without that it is not able to generate the certificate.
|
|
|
|
#### Links
|
|
|
|
[kauri.io](https://kauri.io/38-install-and-configure-a-kubernetes-cluster-with/418b3bc1e0544fbc955a4bbba6fff8a9/a) - Some Informations from there
|
|
[howtoforge Galera](https://www.howtoforge.com/how-to-setup-mariadb-galera-multi-master-synchronous-replication-using-debian-10/) - Install Galera |