You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
401 lines
12 KiB
Markdown
401 lines
12 KiB
Markdown
4 years ago
|
---
|
||
|
date: 2016-04-05T00:17:00+01:00
|
||
|
description: Installing Rancher k3s with MariaDB Galera
|
||
|
tags:
|
||
|
- HOWTO
|
||
|
- Kubernetes
|
||
|
- Galera
|
||
|
title: Howto Install Rancher k3s with MariaDB Galera
|
||
|
---
|
||
|
|
||
|
In this blog Post we gonna install a HA Rancher Kubernetes Cluster with a MariaDB Galera Cluster as Datastore.
|
||
|
|
||
|
<!--more-->
|
||
|
|
||
|
#### Outline
|
||
|
|
||
|
- [Prepare Ubuntu Bionic Server]()
|
||
|
- [Install MariaDB Galera]()
|
||
|
- [Deploy K3S]()
|
||
|
- [Install kubectl and helm]()
|
||
|
- [Install MetalLB]()
|
||
|
- [Install cert-manager for Let's Encrypt]()
|
||
|
- [Install Rancher]()
|
||
|
- [Forward HTTP/HTTPS to the Rancher Load Balancer IP]()
|
||
|
|
||
|
#### Prepare Ubuntu Bionic Server
|
||
|
|
||
|
You need 3 Nodes, 4 CPU, >8GiB RAM, 100GiB Disk, I have 3 Nodes 4 CPU, 24 GiB RAM, 250GiB Disk.
|
||
|
|
||
|
Install Ubuntu on one Server, remove snapd, ufw, cloud-init.
|
||
|
Then clone it and edit /etc/hosts /etc/hostname /etc/netplan/50-cloud-init.yaml and `rm -f /etc/ssh/ssh_host_*` - reboot.
|
||
|
|
||
|
#### Install MariaDB Galera
|
||
|
|
||
|
Install the [MariaDB repo](https://downloads.mariadb.org/mariadb/repositories/#distro=Ubuntu&distro_release=bionic--ubuntu_bionic&mirror=host-europe&version=10.4) on each server.
|
||
|
|
||
|
```bash
|
||
|
sudo apt-get install software-properties-common
|
||
|
sudo apt-key adv --fetch-keys 'https://mariadb.org/mariadb_release_signing_key.asc'
|
||
|
sudo add-apt-repository 'deb [arch=amd64,arm64,ppc64el] http://ftp.hosteurope.de/mirror/mariadb.org/repo/10.4/ubuntu bionic main'
|
||
|
|
||
|
sudo apt update
|
||
|
sudo apt install mariadb-server mariadb-client
|
||
|
```
|
||
|
|
||
|
##### Secure MariaDB on each Node
|
||
|
|
||
|
Set Password with mysql_secure_installation:
|
||
|
|
||
|
```
|
||
|
pcdummy@rancher01:~$ sudo mysql_secure_installation
|
||
|
|
||
|
NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
|
||
|
SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!
|
||
|
|
||
|
In order to log into MariaDB to secure it, we'll need the current
|
||
|
password for the root user. If you've just installed MariaDB, and
|
||
|
haven't set the root password yet, you should just press enter here.
|
||
|
|
||
|
Enter current password for root (enter for none):
|
||
|
OK, successfully used password, moving on...
|
||
|
|
||
|
Setting the root password or using the unix_socket ensures that nobody
|
||
|
can log into the MariaDB root user without the proper authorisation.
|
||
|
|
||
|
You already have your root account protected, so you can safely answer 'n'.
|
||
|
|
||
|
Switch to unix_socket authentication [Y/n] n
|
||
|
... skipping.
|
||
|
|
||
|
You already have your root account protected, so you can safely answer 'n'.
|
||
|
|
||
|
Change the root password? [Y/n]
|
||
|
New password:
|
||
|
Re-enter new password:
|
||
|
Password updated successfully!
|
||
|
Reloading privilege tables..
|
||
|
... Success!
|
||
|
|
||
|
|
||
|
By default, a MariaDB installation has an anonymous user, allowing anyone
|
||
|
to log into MariaDB without having to have a user account created for
|
||
|
them. This is intended only for testing, and to make the installation
|
||
|
go a bit smoother. You should remove them before moving into a
|
||
|
production environment.
|
||
|
|
||
|
Remove anonymous users? [Y/n] Y
|
||
|
... Success!
|
||
|
|
||
|
Normally, root should only be allowed to connect from 'localhost'. This
|
||
|
ensures that someone cannot guess at the root password from the network.
|
||
|
|
||
|
Disallow root login remotely? [Y/n] n
|
||
|
... skipping.
|
||
|
|
||
|
By default, MariaDB comes with a database named 'test' that anyone can
|
||
|
access. This is also intended only for testing, and should be removed
|
||
|
before moving into a production environment.
|
||
|
|
||
|
Remove test database and access to it? [Y/n] Y
|
||
|
- Dropping test database...
|
||
|
... Success!
|
||
|
- Removing privileges on test database...
|
||
|
... Success!
|
||
|
|
||
|
Reloading the privilege tables will ensure that all changes made so far
|
||
|
will take effect immediately.
|
||
|
|
||
|
Reload privilege tables now? [Y/n] Y
|
||
|
... Success!
|
||
|
|
||
|
Cleaning up...
|
||
|
|
||
|
All done! If you've completed all of the above steps, your MariaDB
|
||
|
installation should now be secure.
|
||
|
|
||
|
Thanks for using MariaDB!
|
||
|
```
|
||
|
|
||
|
##### Configure Galera on each Node
|
||
|
|
||
|
Stop Mariadb
|
||
|
|
||
|
```
|
||
|
sudo systemctl stop mariadb
|
||
|
```
|
||
|
|
||
|
Liste on all Interfaces (if you want configure it to listen only on a specific address):
|
||
|
|
||
|
```bash
|
||
|
sudo sed -i 's/bind-address\t\t= 127.0.0.1/#bind-address\t\t= 127.0.0.1/g' /etc/mysql/my.cnf
|
||
|
```
|
||
|
|
||
|
Enable Galera, Paste the following into /etc/mysql/mariadb.conf.d/99-cluster.cnf
|
||
|
|
||
|
```
|
||
|
[galera]
|
||
|
|
||
|
wsrep_on = on
|
||
|
wsrep_provider = /usr/lib/galera/libgalera_smm.so
|
||
|
wsrep_cluster_address = gcomm://10.128.1.17,10.128.1.18,10.128.1.19
|
||
|
wsrep_cluster_name = k3s_cluster_0
|
||
|
|
||
|
default_storage_engine = InnoDB
|
||
|
innodb_autoinc_lock_mode = 2
|
||
|
innodb_doublewrite = 1
|
||
|
|
||
|
binlog_format = ROW
|
||
|
```
|
||
|
|
||
|
And change the ip addresse for `wsrep_cluster_address`
|
||
|
|
||
|
Some tuning if you use this Galera cluster for other purposes
|
||
|
|
||
|
```
|
||
|
sudo nano /etc/mysql/mariadb.conf.d/98-tuning.cnf
|
||
|
```
|
||
|
|
||
|
```
|
||
|
[mysqld]
|
||
|
key_buffer_size=256M
|
||
|
thread_stack=192K
|
||
|
thread_cache_size=8
|
||
|
max_connections=1000
|
||
|
innodb_buffer_pool_size=2G
|
||
|
query_cache_limit=2M
|
||
|
query_cache_size=0
|
||
|
query_cache_type=0
|
||
|
table_open_cache=128
|
||
|
join_buffer_size=512k
|
||
|
table_definition_cache=-1
|
||
|
performance_schema=ON
|
||
|
innodb_log_file_size=256M
|
||
|
innodb_buffer_pool_instances=2
|
||
|
tmp_table_size=32M
|
||
|
max_heap_table_size=32M
|
||
|
```
|
||
|
|
||
|
##### Bootstrap the cluster
|
||
|
|
||
|
One **one** node run `sudo galera_new_cluster`
|
||
|
|
||
|
One the other 2 nodes run: `sudo systemctl start mariadb.service`
|
||
|
|
||
|
##### Create the k3s Database
|
||
|
|
||
|
One one node run:
|
||
|
|
||
|
```bash
|
||
|
mysql -u root -p
|
||
|
```
|
||
|
|
||
|
```sql
|
||
|
CREATE DATABASE `k3s`;
|
||
|
GRANT ALL PRIVILEGES ON `k3s`.* TO 'k3s'@'%' IDENTIFIED BY '<superSecret>';
|
||
|
```
|
||
|
|
||
|
|
||
|
#### Deploy k3s
|
||
|
|
||
|
Install k3s one each nodes, one after another:
|
||
|
|
||
|
```bash
|
||
|
curl -sfL https://get.k3s.io | sh -s - server --datastore-endpoint="mysql://k3s:<superSecret>@tcp(localhost:3306)/k3s" --no-deploy servicelb
|
||
|
```
|
||
|
|
||
|
Check the nodes after.
|
||
|
|
||
|
```bash
|
||
|
sudo k3s kubectl get nodes
|
||
|
```
|
||
|
|
||
|
One one node copy the config (I choose node1 for that):
|
||
|
|
||
|
```bash
|
||
|
mkdir ~/.kube
|
||
|
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
|
||
|
sudo chown -R $(whoami): ~/.kube
|
||
|
```
|
||
|
|
||
|
#### Install kubectl and helm
|
||
|
|
||
|
Install kubectl
|
||
|
|
||
|
```bash
|
||
|
sudo apt-get update && sudo apt-get install -y apt-transport-https
|
||
|
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
|
||
|
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
|
||
|
sudo apt-get update
|
||
|
sudo apt-get install -y kubectl
|
||
|
```
|
||
|
|
||
|
Install helm to ~/bin
|
||
|
|
||
|
```bash
|
||
|
wget https://get.helm.sh/helm-v3.2.1-linux-amd64.tar.gz
|
||
|
tar xfz helm-v3.2.1-linux-amd64.tar.gz
|
||
|
mkdir ~/bin
|
||
|
mv linux-amd64/helm linux-amd64/tiller ~/bin
|
||
|
rm -rf linux-amd64
|
||
|
```
|
||
|
|
||
|
Init helm on the cluster
|
||
|
|
||
|
```bash
|
||
|
kubectl --namespace kube-system create serviceaccount tiller;
|
||
|
kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller;
|
||
|
helm init --service-account tiller;
|
||
|
```
|
||
|
|
||
|
#### Install MetalLB
|
||
|
|
||
|
Install MetalLB (change the address range!)
|
||
|
|
||
|
```bash
|
||
|
helm install stable/metallb \
|
||
|
--name metallb \
|
||
|
--namespace kube-system \
|
||
|
--set configInline.address-pools[0].name=default \
|
||
|
--set configInline.address-pools[0].protocol=layer2 \
|
||
|
--set configInline.address-pools[0].addresses[0]=10.128.3.1-10.128.3.254
|
||
|
```
|
||
|
|
||
|
Check the deployment
|
||
|
|
||
|
```bash
|
||
|
kubectl get pods -n kube-system -l app=metallb -o wide
|
||
|
```
|
||
|
|
||
|
#### Install cert-manager for Let's Encrypt
|
||
|
|
||
|
```bash
|
||
|
kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.12/deploy/manifests/00-crds.yaml
|
||
|
kubectl create namespace cert-manager
|
||
|
helm repo add jetstack https://charts.jetstack.io
|
||
|
helm repo update
|
||
|
helm install jetstack/cert-manager \
|
||
|
--name cert-manager \
|
||
|
--namespace cert-manager \
|
||
|
--version v0.12.0
|
||
|
```
|
||
|
|
||
|
```bash
|
||
|
$ kubectl get pods --namespace cert-manager
|
||
|
NAME READY STATUS RESTARTS AGE
|
||
|
cert-manager-6bcdf8c5cc-5bcrg 1/1 Running 0 54s
|
||
|
cert-manager-cainjector-6659d6844d-zrr5h 1/1 Running 0 54s
|
||
|
cert-manager-webhook-547567b88f-ptrlg 1/1 Running 0 54s
|
||
|
```
|
||
|
|
||
|
#### Install Rancher
|
||
|
|
||
|
```bash
|
||
|
helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
|
||
|
kubectl create namespace cattle-system
|
||
|
helm install rancher-latest/rancher \
|
||
|
--name rancher \
|
||
|
--namespace cattle-system \
|
||
|
--set hostname=rancher.example.org \
|
||
|
--set ingress.tls.source=letsEncrypt \
|
||
|
--set letsEncrypt.email=support@example.org
|
||
|
```
|
||
|
|
||
|
```bash
|
||
|
$ kubectl -n cattle-system rollout status deploy/rancher
|
||
|
Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available...
|
||
|
|
||
|
Waiting for deployment "rancher" rollout to finish: 1 of 3 updated replicas are available...
|
||
|
Waiting for deployment "rancher" rollout to finish: 2 of 3 updated replicas are available...
|
||
|
deployment "rancher" successfully rolled out
|
||
|
```
|
||
|
|
||
|
#### Forward HTTP/HTTPS to the Rancher Load Balancer IP
|
||
|
|
||
|
```bash
|
||
|
$ kubectl -n kube-system describe service/traefik
|
||
|
Name: traefik
|
||
|
Namespace: kube-system
|
||
|
Labels: app=traefik
|
||
|
chart=traefik-1.81.0
|
||
|
heritage=Helm
|
||
|
release=traefik
|
||
|
Annotations: field.cattle.io/publicEndpoints:
|
||
|
[{"addresses":["10.128.3.1"],"port":80,"protocol":"TCP","serviceName":"kube-system:traefik","allNodes":false},{"addresses":["10.128.3.1"],...
|
||
|
Selector: app=traefik,release=traefik
|
||
|
Type: LoadBalancer
|
||
|
IP: 10.43.47.63
|
||
|
LoadBalancer Ingress: 10.128.3.1
|
||
|
Port: http 80/TCP
|
||
|
TargetPort: http/TCP
|
||
|
NodePort: http 32316/TCP
|
||
|
Endpoints: 10.42.0.6:80
|
||
|
Port: https 443/TCP
|
||
|
TargetPort: https/TCP
|
||
|
NodePort: https 30752/TCP
|
||
|
Endpoints: 10.42.0.6:443
|
||
|
Session Affinity: None
|
||
|
External Traffic Policy: Cluster
|
||
|
Events:
|
||
|
Type Reason Age From Message
|
||
|
---- ------ ---- ---- -------
|
||
|
Normal IPAllocated 16m metallb-controller Assigned IP "10.128.3.1"
|
||
|
Normal nodeAssigned 3m10s (x5 over 16m) metallb-speaker announcing from node "rancher01"
|
||
|
```
|
||
|
|
||
|
Here the IP is 10.128.3.1 i forward HTTP (80) and HTTPS (443) to it.
|
||
|
|
||
|
|
||
|
Wait for the Let's Encrypt Cert
|
||
|
```
|
||
|
$ kubectl -n cattle-system describe certificate
|
||
|
Name: tls-rancher-ingress
|
||
|
Namespace: cattle-system
|
||
|
Labels: app=rancher
|
||
|
chart=rancher-2.4.3
|
||
|
heritage=Tiller
|
||
|
release=rancher
|
||
|
Annotations: <none>
|
||
|
API Version: cert-manager.io/v1alpha2
|
||
|
Kind: Certificate
|
||
|
Metadata:
|
||
|
Creation Timestamp: 2020-05-07T23:07:41Z
|
||
|
Generation: 1
|
||
|
Owner References:
|
||
|
API Version: extensions/v1beta1
|
||
|
Block Owner Deletion: true
|
||
|
Controller: true
|
||
|
Kind: Ingress
|
||
|
Name: rancher
|
||
|
UID: 625bd78c-819a-4ba5-8ed0-4e8cf0497860
|
||
|
Resource Version: 23050
|
||
|
Self Link: /apis/cert-manager.io/v1alpha2/namespaces/cattle-system/certificates/tls-rancher-ingress
|
||
|
UID: 6e0886ec-8b2d-4459-8c60-f994f269a146
|
||
|
Spec:
|
||
|
Dns Names:
|
||
|
rancher.example.org
|
||
|
Issuer Ref:
|
||
|
Group: cert-manager.io
|
||
|
Kind: Issuer
|
||
|
Name: rancher
|
||
|
Secret Name: tls-rancher-ingress
|
||
|
Status:
|
||
|
Conditions:
|
||
|
Last Transition Time: 2020-05-07T23:07:41Z
|
||
|
Message: Waiting for CertificateRequest "tls-rancher-ingress-2753661366" to complete
|
||
|
Reason: InProgress
|
||
|
Status: False
|
||
|
Type: Ready
|
||
|
Events:
|
||
|
Type Reason Age From Message
|
||
|
---- ------ ---- ---- -------
|
||
|
Normal GeneratedKey 18m cert-manager Generated a new private key
|
||
|
Normal Requested 18m cert-manager Created new CertificateRequest resource "tls-rancher-ingress-2753661366"
|
||
|
```
|
||
|
|
||
|
I had some troubles with cert-manager where it wasn't able to access http://<rancher-FQDN> without that it is not able to generate the certificate.
|
||
|
|
||
|
#### Links
|
||
|
|
||
|
[kauri.io](https://kauri.io/38-install-and-configure-a-kubernetes-cluster-with/418b3bc1e0544fbc955a4bbba6fff8a9/a) - Some Informations from there
|
||
|
[howtoforge Galera](https://www.howtoforge.com/how-to-setup-mariadb-galera-multi-master-synchronous-replication-using-debian-10/) - Install Galera
|