12 KiB
date | description | tags | title | |||
---|---|---|---|---|---|---|
2016-04-05T00:17:00+01:00 | Installing Rancher k3s with MariaDB Galera |
|
Howto Install Rancher k3s with MariaDB Galera |
In this blog Post we gonna install a HA Rancher Kubernetes Cluster with a MariaDB Galera Cluster as Datastore.
Outline
- Prepare Ubuntu Bionic Server
- Install MariaDB Galera
- Deploy K3S
- Install kubectl and helm
- Install MetalLB
- Install cert-manager for Let's Encrypt
- Install Rancher
- Forward HTTP/HTTPS to the Rancher Load Balancer IP
Prepare Ubuntu Bionic Server
You need 3 Nodes, 4 CPU, >8GiB RAM, 100GiB Disk, I have 3 Nodes 4 CPU, 24 GiB RAM, 250GiB Disk.
Install Ubuntu on one Server, remove snapd, ufw, cloud-init.
Then clone it and edit /etc/hosts /etc/hostname /etc/netplan/50-cloud-init.yaml and rm -f /etc/ssh/ssh_host_*
- reboot.
Install MariaDB Galera
Install the MariaDB repo on each server.
sudo apt-get install software-properties-common
sudo apt-key adv --fetch-keys 'https://mariadb.org/mariadb_release_signing_key.asc'
sudo add-apt-repository 'deb [arch=amd64,arm64,ppc64el] http://ftp.hosteurope.de/mirror/mariadb.org/repo/10.4/ubuntu bionic main'
sudo apt update
sudo apt install mariadb-server mariadb-client
Secure MariaDB on each Node
Set Password with mysql_secure_installation:
pcdummy@rancher01:~$ sudo mysql_secure_installation
NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!
In order to log into MariaDB to secure it, we'll need the current
password for the root user. If you've just installed MariaDB, and
haven't set the root password yet, you should just press enter here.
Enter current password for root (enter for none):
OK, successfully used password, moving on...
Setting the root password or using the unix_socket ensures that nobody
can log into the MariaDB root user without the proper authorisation.
You already have your root account protected, so you can safely answer 'n'.
Switch to unix_socket authentication [Y/n] n
... skipping.
You already have your root account protected, so you can safely answer 'n'.
Change the root password? [Y/n]
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
... Success!
By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them. This is intended only for testing, and to make the installation
go a bit smoother. You should remove them before moving into a
production environment.
Remove anonymous users? [Y/n] Y
... Success!
Normally, root should only be allowed to connect from 'localhost'. This
ensures that someone cannot guess at the root password from the network.
Disallow root login remotely? [Y/n] n
... skipping.
By default, MariaDB comes with a database named 'test' that anyone can
access. This is also intended only for testing, and should be removed
before moving into a production environment.
Remove test database and access to it? [Y/n] Y
- Dropping test database...
... Success!
- Removing privileges on test database...
... Success!
Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.
Reload privilege tables now? [Y/n] Y
... Success!
Cleaning up...
All done! If you've completed all of the above steps, your MariaDB
installation should now be secure.
Thanks for using MariaDB!
Configure Galera on each Node
Stop Mariadb
sudo systemctl stop mariadb
Liste on all Interfaces (if you want configure it to listen only on a specific address):
sudo sed -i 's/bind-address\t\t= 127.0.0.1/#bind-address\t\t= 127.0.0.1/g' /etc/mysql/my.cnf
Enable Galera, Paste the following into /etc/mysql/mariadb.conf.d/99-cluster.cnf
[galera]
wsrep_on = on
wsrep_provider = /usr/lib/galera/libgalera_smm.so
wsrep_cluster_address = gcomm://10.128.1.17,10.128.1.18,10.128.1.19
wsrep_cluster_name = k3s_cluster_0
default_storage_engine = InnoDB
innodb_autoinc_lock_mode = 2
innodb_doublewrite = 1
binlog_format = ROW
And change the ip addresse for wsrep_cluster_address
Some tuning if you use this Galera cluster for other purposes
sudo nano /etc/mysql/mariadb.conf.d/98-tuning.cnf
[mysqld]
key_buffer_size=256M
thread_stack=192K
thread_cache_size=8
max_connections=1000
innodb_buffer_pool_size=2G
query_cache_limit=2M
query_cache_size=0
query_cache_type=0
table_open_cache=128
join_buffer_size=512k
table_definition_cache=-1
performance_schema=ON
innodb_log_file_size=256M
innodb_buffer_pool_instances=2
tmp_table_size=32M
max_heap_table_size=32M
Bootstrap the cluster
One one node run sudo galera_new_cluster
One the other 2 nodes run: sudo systemctl start mariadb.service
Create the k3s Database
One one node run:
mysql -u root -p
CREATE DATABASE `k3s`;
GRANT ALL PRIVILEGES ON `k3s`.* TO 'k3s'@'%' IDENTIFIED BY '<superSecret>';
Deploy k3s
Install k3s one each nodes, one after another:
curl -sfL https://get.k3s.io | sh -s - server --datastore-endpoint="mysql://k3s:<superSecret>@tcp(localhost:3306)/k3s" --no-deploy servicelb
Check the nodes after.
sudo k3s kubectl get nodes
One one node copy the config (I choose node1 for that):
mkdir ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown -R $(whoami): ~/.kube
Install kubectl and helm
Install kubectl
sudo apt-get update && sudo apt-get install -y apt-transport-https
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubectl
Install helm to ~/bin
wget https://get.helm.sh/helm-v3.2.1-linux-amd64.tar.gz
tar xfz helm-v3.2.1-linux-amd64.tar.gz
mkdir ~/bin
mv linux-amd64/helm linux-amd64/tiller ~/bin
rm -rf linux-amd64
Init helm on the cluster
kubectl --namespace kube-system create serviceaccount tiller;
kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller;
helm init --service-account tiller;
Install MetalLB
Install MetalLB (change the address range!)
helm install stable/metallb \
--name metallb \
--namespace kube-system \
--set configInline.address-pools[0].name=default \
--set configInline.address-pools[0].protocol=layer2 \
--set configInline.address-pools[0].addresses[0]=10.128.3.1-10.128.3.254
Check the deployment
kubectl get pods -n kube-system -l app=metallb -o wide
Install cert-manager for Let's Encrypt
kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.12/deploy/manifests/00-crds.yaml
kubectl create namespace cert-manager
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install jetstack/cert-manager \
--name cert-manager \
--namespace cert-manager \
--version v0.12.0
$ kubectl get pods --namespace cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-6bcdf8c5cc-5bcrg 1/1 Running 0 54s
cert-manager-cainjector-6659d6844d-zrr5h 1/1 Running 0 54s
cert-manager-webhook-547567b88f-ptrlg 1/1 Running 0 54s
Install Rancher
helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
kubectl create namespace cattle-system
helm install rancher-latest/rancher \
--name rancher \
--namespace cattle-system \
--set hostname=rancher.example.org \
--set ingress.tls.source=letsEncrypt \
--set letsEncrypt.email=support@example.org
$ kubectl -n cattle-system rollout status deploy/rancher
Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available...
Waiting for deployment "rancher" rollout to finish: 1 of 3 updated replicas are available...
Waiting for deployment "rancher" rollout to finish: 2 of 3 updated replicas are available...
deployment "rancher" successfully rolled out
Forward HTTP/HTTPS to the Rancher Load Balancer IP
$ kubectl -n kube-system describe service/traefik
Name: traefik
Namespace: kube-system
Labels: app=traefik
chart=traefik-1.81.0
heritage=Helm
release=traefik
Annotations: field.cattle.io/publicEndpoints:
[{"addresses":["10.128.3.1"],"port":80,"protocol":"TCP","serviceName":"kube-system:traefik","allNodes":false},{"addresses":["10.128.3.1"],...
Selector: app=traefik,release=traefik
Type: LoadBalancer
IP: 10.43.47.63
LoadBalancer Ingress: 10.128.3.1
Port: http 80/TCP
TargetPort: http/TCP
NodePort: http 32316/TCP
Endpoints: 10.42.0.6:80
Port: https 443/TCP
TargetPort: https/TCP
NodePort: https 30752/TCP
Endpoints: 10.42.0.6:443
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal IPAllocated 16m metallb-controller Assigned IP "10.128.3.1"
Normal nodeAssigned 3m10s (x5 over 16m) metallb-speaker announcing from node "rancher01"
Here the IP is 10.128.3.1 i forward HTTP (80) and HTTPS (443) to it.
Wait for the Let's Encrypt Cert
$ kubectl -n cattle-system describe certificate
Name: tls-rancher-ingress
Namespace: cattle-system
Labels: app=rancher
chart=rancher-2.4.3
heritage=Tiller
release=rancher
Annotations: <none>
API Version: cert-manager.io/v1alpha2
Kind: Certificate
Metadata:
Creation Timestamp: 2020-05-07T23:07:41Z
Generation: 1
Owner References:
API Version: extensions/v1beta1
Block Owner Deletion: true
Controller: true
Kind: Ingress
Name: rancher
UID: 625bd78c-819a-4ba5-8ed0-4e8cf0497860
Resource Version: 23050
Self Link: /apis/cert-manager.io/v1alpha2/namespaces/cattle-system/certificates/tls-rancher-ingress
UID: 6e0886ec-8b2d-4459-8c60-f994f269a146
Spec:
Dns Names:
rancher.example.org
Issuer Ref:
Group: cert-manager.io
Kind: Issuer
Name: rancher
Secret Name: tls-rancher-ingress
Status:
Conditions:
Last Transition Time: 2020-05-07T23:07:41Z
Message: Waiting for CertificateRequest "tls-rancher-ingress-2753661366" to complete
Reason: InProgress
Status: False
Type: Ready
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal GeneratedKey 18m cert-manager Generated a new private key
Normal Requested 18m cert-manager Created new CertificateRequest resource "tls-rancher-ingress-2753661366"
I had some troubles with cert-manager where it wasn't able to access http:// without that it is not able to generate the certificate.
Links
kauri.io - Some Informations from there howtoforge Galera - Install Galera