parent
7d63d40028
commit
3e90fd1d04
@ -0,0 +1,401 @@
|
||||
---
|
||||
date: 2016-04-05T00:17:00+01:00
|
||||
description: Installing Rancher k3s with MariaDB Galera
|
||||
tags:
|
||||
- HOWTO
|
||||
- Kubernetes
|
||||
- Galera
|
||||
title: Howto Install Rancher k3s with MariaDB Galera
|
||||
---
|
||||
|
||||
In this blog Post we gonna install a HA Rancher Kubernetes Cluster with a MariaDB Galera Cluster as Datastore.
|
||||
|
||||
<!--more-->
|
||||
|
||||
#### Outline
|
||||
|
||||
- [Prepare Ubuntu Bionic Server]()
|
||||
- [Install MariaDB Galera]()
|
||||
- [Deploy K3S]()
|
||||
- [Install kubectl and helm]()
|
||||
- [Install MetalLB]()
|
||||
- [Install cert-manager for Let's Encrypt]()
|
||||
- [Install Rancher]()
|
||||
- [Forward HTTP/HTTPS to the Rancher Load Balancer IP]()
|
||||
|
||||
#### Prepare Ubuntu Bionic Server
|
||||
|
||||
You need 3 Nodes, 4 CPU, >8GiB RAM, 100GiB Disk, I have 3 Nodes 4 CPU, 24 GiB RAM, 250GiB Disk.
|
||||
|
||||
Install Ubuntu on one Server, remove snapd, ufw, cloud-init.
|
||||
Then clone it and edit /etc/hosts /etc/hostname /etc/netplan/50-cloud-init.yaml and `rm -f /etc/ssh/ssh_host_*` - reboot.
|
||||
|
||||
#### Install MariaDB Galera
|
||||
|
||||
Install the [MariaDB repo](https://downloads.mariadb.org/mariadb/repositories/#distro=Ubuntu&distro_release=bionic--ubuntu_bionic&mirror=host-europe&version=10.4) on each server.
|
||||
|
||||
```bash
|
||||
sudo apt-get install software-properties-common
|
||||
sudo apt-key adv --fetch-keys 'https://mariadb.org/mariadb_release_signing_key.asc'
|
||||
sudo add-apt-repository 'deb [arch=amd64,arm64,ppc64el] http://ftp.hosteurope.de/mirror/mariadb.org/repo/10.4/ubuntu bionic main'
|
||||
|
||||
sudo apt update
|
||||
sudo apt install mariadb-server mariadb-client
|
||||
```
|
||||
|
||||
##### Secure MariaDB on each Node
|
||||
|
||||
Set Password with mysql_secure_installation:
|
||||
|
||||
```
|
||||
pcdummy@rancher01:~$ sudo mysql_secure_installation
|
||||
|
||||
NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
|
||||
SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!
|
||||
|
||||
In order to log into MariaDB to secure it, we'll need the current
|
||||
password for the root user. If you've just installed MariaDB, and
|
||||
haven't set the root password yet, you should just press enter here.
|
||||
|
||||
Enter current password for root (enter for none):
|
||||
OK, successfully used password, moving on...
|
||||
|
||||
Setting the root password or using the unix_socket ensures that nobody
|
||||
can log into the MariaDB root user without the proper authorisation.
|
||||
|
||||
You already have your root account protected, so you can safely answer 'n'.
|
||||
|
||||
Switch to unix_socket authentication [Y/n] n
|
||||
... skipping.
|
||||
|
||||
You already have your root account protected, so you can safely answer 'n'.
|
||||
|
||||
Change the root password? [Y/n]
|
||||
New password:
|
||||
Re-enter new password:
|
||||
Password updated successfully!
|
||||
Reloading privilege tables..
|
||||
... Success!
|
||||
|
||||
|
||||
By default, a MariaDB installation has an anonymous user, allowing anyone
|
||||
to log into MariaDB without having to have a user account created for
|
||||
them. This is intended only for testing, and to make the installation
|
||||
go a bit smoother. You should remove them before moving into a
|
||||
production environment.
|
||||
|
||||
Remove anonymous users? [Y/n] Y
|
||||
... Success!
|
||||
|
||||
Normally, root should only be allowed to connect from 'localhost'. This
|
||||
ensures that someone cannot guess at the root password from the network.
|
||||
|
||||
Disallow root login remotely? [Y/n] n
|
||||
... skipping.
|
||||
|
||||
By default, MariaDB comes with a database named 'test' that anyone can
|
||||
access. This is also intended only for testing, and should be removed
|
||||
before moving into a production environment.
|
||||
|
||||
Remove test database and access to it? [Y/n] Y
|
||||
- Dropping test database...
|
||||
... Success!
|
||||
- Removing privileges on test database...
|
||||
... Success!
|
||||
|
||||
Reloading the privilege tables will ensure that all changes made so far
|
||||
will take effect immediately.
|
||||
|
||||
Reload privilege tables now? [Y/n] Y
|
||||
... Success!
|
||||
|
||||
Cleaning up...
|
||||
|
||||
All done! If you've completed all of the above steps, your MariaDB
|
||||
installation should now be secure.
|
||||
|
||||
Thanks for using MariaDB!
|
||||
```
|
||||
|
||||
##### Configure Galera on each Node
|
||||
|
||||
Stop Mariadb
|
||||
|
||||
```
|
||||
sudo systemctl stop mariadb
|
||||
```
|
||||
|
||||
Liste on all Interfaces (if you want configure it to listen only on a specific address):
|
||||
|
||||
```bash
|
||||
sudo sed -i 's/bind-address\t\t= 127.0.0.1/#bind-address\t\t= 127.0.0.1/g' /etc/mysql/my.cnf
|
||||
```
|
||||
|
||||
Enable Galera, Paste the following into /etc/mysql/mariadb.conf.d/99-cluster.cnf
|
||||
|
||||
```
|
||||
[galera]
|
||||
|
||||
wsrep_on = on
|
||||
wsrep_provider = /usr/lib/galera/libgalera_smm.so
|
||||
wsrep_cluster_address = gcomm://10.128.1.17,10.128.1.18,10.128.1.19
|
||||
wsrep_cluster_name = k3s_cluster_0
|
||||
|
||||
default_storage_engine = InnoDB
|
||||
innodb_autoinc_lock_mode = 2
|
||||
innodb_doublewrite = 1
|
||||
|
||||
binlog_format = ROW
|
||||
```
|
||||
|
||||
And change the ip addresse for `wsrep_cluster_address`
|
||||
|
||||
Some tuning if you use this Galera cluster for other purposes
|
||||
|
||||
```
|
||||
sudo nano /etc/mysql/mariadb.conf.d/98-tuning.cnf
|
||||
```
|
||||
|
||||
```
|
||||
[mysqld]
|
||||
key_buffer_size=256M
|
||||
thread_stack=192K
|
||||
thread_cache_size=8
|
||||
max_connections=1000
|
||||
innodb_buffer_pool_size=2G
|
||||
query_cache_limit=2M
|
||||
query_cache_size=0
|
||||
query_cache_type=0
|
||||
table_open_cache=128
|
||||
join_buffer_size=512k
|
||||
table_definition_cache=-1
|
||||
performance_schema=ON
|
||||
innodb_log_file_size=256M
|
||||
innodb_buffer_pool_instances=2
|
||||
tmp_table_size=32M
|
||||
max_heap_table_size=32M
|
||||
```
|
||||
|
||||
##### Bootstrap the cluster
|
||||
|
||||
One **one** node run `sudo galera_new_cluster`
|
||||
|
||||
One the other 2 nodes run: `sudo systemctl start mariadb.service`
|
||||
|
||||
##### Create the k3s Database
|
||||
|
||||
One one node run:
|
||||
|
||||
```bash
|
||||
mysql -u root -p
|
||||
```
|
||||
|
||||
```sql
|
||||
CREATE DATABASE `k3s`;
|
||||
GRANT ALL PRIVILEGES ON `k3s`.* TO 'k3s'@'%' IDENTIFIED BY '<superSecret>';
|
||||
```
|
||||
|
||||
|
||||
#### Deploy k3s
|
||||
|
||||
Install k3s one each nodes, one after another:
|
||||
|
||||
```bash
|
||||
curl -sfL https://get.k3s.io | sh -s - server --datastore-endpoint="mysql://k3s:<superSecret>@tcp(localhost:3306)/k3s" --no-deploy servicelb
|
||||
```
|
||||
|
||||
Check the nodes after.
|
||||
|
||||
```bash
|
||||
sudo k3s kubectl get nodes
|
||||
```
|
||||
|
||||
One one node copy the config (I choose node1 for that):
|
||||
|
||||
```bash
|
||||
mkdir ~/.kube
|
||||
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
|
||||
sudo chown -R $(whoami): ~/.kube
|
||||
```
|
||||
|
||||
#### Install kubectl and helm
|
||||
|
||||
Install kubectl
|
||||
|
||||
```bash
|
||||
sudo apt-get update && sudo apt-get install -y apt-transport-https
|
||||
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
|
||||
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y kubectl
|
||||
```
|
||||
|
||||
Install helm to ~/bin
|
||||
|
||||
```bash
|
||||
wget https://get.helm.sh/helm-v3.2.1-linux-amd64.tar.gz
|
||||
tar xfz helm-v3.2.1-linux-amd64.tar.gz
|
||||
mkdir ~/bin
|
||||
mv linux-amd64/helm linux-amd64/tiller ~/bin
|
||||
rm -rf linux-amd64
|
||||
```
|
||||
|
||||
Init helm on the cluster
|
||||
|
||||
```bash
|
||||
kubectl --namespace kube-system create serviceaccount tiller;
|
||||
kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller;
|
||||
helm init --service-account tiller;
|
||||
```
|
||||
|
||||
#### Install MetalLB
|
||||
|
||||
Install MetalLB (change the address range!)
|
||||
|
||||
```bash
|
||||
helm install stable/metallb \
|
||||
--name metallb \
|
||||
--namespace kube-system \
|
||||
--set configInline.address-pools[0].name=default \
|
||||
--set configInline.address-pools[0].protocol=layer2 \
|
||||
--set configInline.address-pools[0].addresses[0]=10.128.3.1-10.128.3.254
|
||||
```
|
||||
|
||||
Check the deployment
|
||||
|
||||
```bash
|
||||
kubectl get pods -n kube-system -l app=metallb -o wide
|
||||
```
|
||||
|
||||
#### Install cert-manager for Let's Encrypt
|
||||
|
||||
```bash
|
||||
kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.12/deploy/manifests/00-crds.yaml
|
||||
kubectl create namespace cert-manager
|
||||
helm repo add jetstack https://charts.jetstack.io
|
||||
helm repo update
|
||||
helm install jetstack/cert-manager \
|
||||
--name cert-manager \
|
||||
--namespace cert-manager \
|
||||
--version v0.12.0
|
||||
```
|
||||
|
||||
```bash
|
||||
$ kubectl get pods --namespace cert-manager
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
cert-manager-6bcdf8c5cc-5bcrg 1/1 Running 0 54s
|
||||
cert-manager-cainjector-6659d6844d-zrr5h 1/1 Running 0 54s
|
||||
cert-manager-webhook-547567b88f-ptrlg 1/1 Running 0 54s
|
||||
```
|
||||
|
||||
#### Install Rancher
|
||||
|
||||
```bash
|
||||
helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
|
||||
kubectl create namespace cattle-system
|
||||
helm install rancher-latest/rancher \
|
||||
--name rancher \
|
||||
--namespace cattle-system \
|
||||
--set hostname=rancher.example.org \
|
||||
--set ingress.tls.source=letsEncrypt \
|
||||
--set letsEncrypt.email=support@example.org
|
||||
```
|
||||
|
||||
```bash
|
||||
$ kubectl -n cattle-system rollout status deploy/rancher
|
||||
Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available...
|
||||
|
||||
Waiting for deployment "rancher" rollout to finish: 1 of 3 updated replicas are available...
|
||||
Waiting for deployment "rancher" rollout to finish: 2 of 3 updated replicas are available...
|
||||
deployment "rancher" successfully rolled out
|
||||
```
|
||||
|
||||
#### Forward HTTP/HTTPS to the Rancher Load Balancer IP
|
||||
|
||||
```bash
|
||||
$ kubectl -n kube-system describe service/traefik
|
||||
Name: traefik
|
||||
Namespace: kube-system
|
||||
Labels: app=traefik
|
||||
chart=traefik-1.81.0
|
||||
heritage=Helm
|
||||
release=traefik
|
||||
Annotations: field.cattle.io/publicEndpoints:
|
||||
[{"addresses":["10.128.3.1"],"port":80,"protocol":"TCP","serviceName":"kube-system:traefik","allNodes":false},{"addresses":["10.128.3.1"],...
|
||||
Selector: app=traefik,release=traefik
|
||||
Type: LoadBalancer
|
||||
IP: 10.43.47.63
|
||||
LoadBalancer Ingress: 10.128.3.1
|
||||
Port: http 80/TCP
|
||||
TargetPort: http/TCP
|
||||
NodePort: http 32316/TCP
|
||||
Endpoints: 10.42.0.6:80
|
||||
Port: https 443/TCP
|
||||
TargetPort: https/TCP
|
||||
NodePort: https 30752/TCP
|
||||
Endpoints: 10.42.0.6:443
|
||||
Session Affinity: None
|
||||
External Traffic Policy: Cluster
|
||||
Events:
|
||||
Type Reason Age From Message
|
||||
---- ------ ---- ---- -------
|
||||
Normal IPAllocated 16m metallb-controller Assigned IP "10.128.3.1"
|
||||
Normal nodeAssigned 3m10s (x5 over 16m) metallb-speaker announcing from node "rancher01"
|
||||
```
|
||||
|
||||
Here the IP is 10.128.3.1 i forward HTTP (80) and HTTPS (443) to it.
|
||||
|
||||
|
||||
Wait for the Let's Encrypt Cert
|
||||
```
|
||||
$ kubectl -n cattle-system describe certificate
|
||||
Name: tls-rancher-ingress
|
||||
Namespace: cattle-system
|
||||
Labels: app=rancher
|
||||
chart=rancher-2.4.3
|
||||
heritage=Tiller
|
||||
release=rancher
|
||||
Annotations: <none>
|
||||
API Version: cert-manager.io/v1alpha2
|
||||
Kind: Certificate
|
||||
Metadata:
|
||||
Creation Timestamp: 2020-05-07T23:07:41Z
|
||||
Generation: 1
|
||||
Owner References:
|
||||
API Version: extensions/v1beta1
|
||||
Block Owner Deletion: true
|
||||
Controller: true
|
||||
Kind: Ingress
|
||||
Name: rancher
|
||||
UID: 625bd78c-819a-4ba5-8ed0-4e8cf0497860
|
||||
Resource Version: 23050
|
||||
Self Link: /apis/cert-manager.io/v1alpha2/namespaces/cattle-system/certificates/tls-rancher-ingress
|
||||
UID: 6e0886ec-8b2d-4459-8c60-f994f269a146
|
||||
Spec:
|
||||
Dns Names:
|
||||
rancher.example.org
|
||||
Issuer Ref:
|
||||
Group: cert-manager.io
|
||||
Kind: Issuer
|
||||
Name: rancher
|
||||
Secret Name: tls-rancher-ingress
|
||||
Status:
|
||||
Conditions:
|
||||
Last Transition Time: 2020-05-07T23:07:41Z
|
||||
Message: Waiting for CertificateRequest "tls-rancher-ingress-2753661366" to complete
|
||||
Reason: InProgress
|
||||
Status: False
|
||||
Type: Ready
|
||||
Events:
|
||||
Type Reason Age From Message
|
||||
---- ------ ---- ---- -------
|
||||
Normal GeneratedKey 18m cert-manager Generated a new private key
|
||||
Normal Requested 18m cert-manager Created new CertificateRequest resource "tls-rancher-ingress-2753661366"
|
||||
```
|
||||
|
||||
I had some troubles with cert-manager where it wasn't able to access http://<rancher-FQDN> without that it is not able to generate the certificate.
|
||||
|
||||
#### Links
|
||||
|
||||
[kauri.io](https://kauri.io/38-install-and-configure-a-kubernetes-cluster-with/418b3bc1e0544fbc955a4bbba6fff8a9/a) - Some Informations from there
|
||||
[howtoforge Galera](https://www.howtoforge.com/how-to-setup-mariadb-galera-multi-master-synchronous-replication-using-debian-10/) - Install Galera
|
Loading…
Reference in New Issue