__________________________________________________________________________ YAKKO: Yet Another KVM Konfigurator for Openshift __________________________________________________________________________ __________________________________________________________________________ STAGE 1: Gather Cluster Configuration information (Time start: 15:58pm) This section will gather relevant information to build an OpenShift cluster on this host. Questions are separated by '----'. Default options are in [brackets], press enter to accept them. ---- Enter the name of the OpenShift cluster to create ["test"]: ---- Enter the DOMAIN name to setup your cluster under ["home"]: ---- It is recommended that this host have a fixed IP address and a resolvable hostname. Enter the IP address of this host on your network ["192.168.100.2"]: ---- Enter the SUBNET (/24) inside KVM that you want cluster under ["192.168.140"]: ---- YAKKO uses HAproxy to loadbalance access to OpenShift nodes, while also allowing or restricting access to either THIS host only, or clients in your local network. (You can change this later using 'yakko infra changeaccess') Allow 'open' cluster access from other clients (other than this server) (Y/N) [Y]? ---- What release version do you want to install: 1) Latest release available (4.8.4) 2) Releases on disk - 4.8.2 - 4.8.3 - 4.8.4 3) Older releases (hard! you will need to pick image version and installer version) 4) Latest release candidate (good luck!) 5) Latest pre-release nightly (unknown territory...) Pick option number from above (1-5) [1]: ---- Enter the directory where you wish to place the OCP VM disks for this cluster ["/mnt/VMs"]: /mnt/VMs/desktops/fedora-RedHat.qcow2 not reset as customized by admin to system_u:object_r:svirt_image_t:s0:c397,c766 Changing permissions for [/mnt/VMs] and all paths above. All members of the tree need to have read/exec access so that user 'qemu' can run VMs in the destination. ---- How many MASTER nodes do you want to configure (1 or 3) [3]: ---- How much RAM (MiB) should be allocated to MASTER nodes [16000]: ---- Worker nodes can be built at cluster creation or later. To build a cluster with ONLY schedulable MASTER node(s), type '0' How many worker nodes do you want to configure at cluster build time [2]: ---- How much RAM (MiB) should be allocated to WORKER nodes [5000]: ---- Requested OpenShift configuration requires: - RAM: 50000 MiB - DISK: 150 GiB Accept this configuration for cluster build (Y/N) [Y]? ---- Attempt AUTOMATIC creation of cluster "test" (Y/N) [Y]? ---- Although YAKKO automates OpenShift cluster creation, you may want to customise the 'install-config.yaml' configuration file before cluster bootstrap, for example, when wanting to add a proxy server for indirect connection to the internet. Pause for edit of 'install-config.yaml' when file becomes available (Y/N) [N]? __________________________________________________________________________ STAGE 2: Load pull secret (Time start: 15:59pm) Using saved pull secret __________________________________________________________________________ STAGE 3: Obtain OCP binaries (Installer and RHCOS) (Time start: 15:59pm) Using OCP Version 4.8.4 already downloaded... __________________________________________________________________________ STAGE 4: Libvirt package install/start (Time start: 15:59pm) Libvirt is installed and active __________________________________________________________________________ STAGE 5: SSH key configuration (Time start: 15:59pm) Generating public/private rsa key pair. Your identification has been saved in /root/.ssh/id_rsa_ocp.test Your public key has been saved in /root/.ssh/id_rsa_ocp.test.pub The key fingerprint is: SHA256:5/GMDwBlRdH49IlWxkDCF31AcEQLNk2ee62j7/YjYTU root@terminus The key's randomart image is: +---[RSA 4096]----+ | o+==O%B. | | o oo+=*o.| | . + =+o | | . + oE.| | S o. o +| | + = o o | | + + + | | o o.o | | o++.o| +----[SHA256]-----+ Agent pid 289496 __________________________________________________________________________ STAGE 6: Virtual Network Configuration (Time start: 15:59pm) Cleaning up network... Only NAT is supported for now YAKKO will create all infrastructure in the 192.168.140/24 subnet with preallocated IP addresses: Bootstrap: 192.168.140.9 Masters: 192.168.140.10 192.168.140.11 and 192.168.140.12 Defining network at /YAKKO/install-test/net-yakko-test.xml Network net-yakko-test defined from /YAKKO/install-test/net-yakko-test.xml Setting network to start on boot... Network net-yakko-test marked as autostarted Re/starting network to ensure it is operational... Network net-yakko-test started Virtual network is up __________________________________________________________________________ STAGE 7: DNS Configuration (Time start: 15:59pm) Configuring dnsmask plugin in NETWORKMANAGER and adding DNSmasq configuration as /etc/NetworkManager/dnsmasq.d/dnsmasq-yakko-test.conf nameserver 127.0.0.53 ATTENTION: This system is using systemd-resolved. YAKKO needs to add a DNS stub in /etc/systemd/resolved.conf DNS test - from Virtual Network 192.168.140.0: api-int.test.home has address 192.168.140.1 etcd-0.test.home has address 192.168.140.10 etcd-1.test.home has address 192.168.140.11 etcd-2.test.home has address 192.168.140.12 Using domain server: Name: 192.168.140.1 Address: 192.168.140.1#53 Aliases: _etcd-server-ssl._tcp.test.home has SRV record 0 10 2380 etcd-2.test.home. _etcd-server-ssl._tcp.test.home has SRV record 0 10 2380 etcd-1.test.home. _etcd-server-ssl._tcp.test.home has SRV record 0 10 2380 etcd-0.test.home. DNS test - from the host: api.test.home has address 192.168.100.2 etcd-0.test.home has address 192.168.140.10 etcd-1.test.home has address 192.168.140.11 etcd-2.test.home has address 192.168.140.12 Using domain server: Name: 127.0.0.1 Address: 127.0.0.1#53 Aliases: testing.apps.test.home has address 192.168.100.2 __________________________________________________________________________ STAGE 8: Configure HTTP server for installation of all cluster components (Time start: 15:59pm) Using port [8080] for RHCOS image delivery OCP will be made available by HTTP server from directory /YAKKO/images/4.8.4 changing security context of '/YAKKO/images/4.8.2/rhcos-live-rootfs.x86_64.img' changing security context of '/YAKKO/images/4.8.2/openshift-install' changing security context of '/YAKKO/images/4.8.2/oc' changing security context of '/YAKKO/images/4.8.2/rhcos-live-kernel-x86_64' changing security context of '/YAKKO/images/4.8.2/README.md' changing security context of '/YAKKO/images/4.8.2/rhcos-live-initramfs.x86_64.img' changing security context of '/YAKKO/images/4.8.2/rhcos-metal.x86_64.raw.gz' changing security context of '/YAKKO/images/4.8.2/.downloadcomplete' changing security context of '/YAKKO/images/4.8.2/kubectl' changing security context of '/YAKKO/images/4.8.2' changing security context of '/YAKKO/images/4.8.4/rhcos-live-rootfs.x86_64.img' changing security context of '/YAKKO/images/4.8.4/openshift-install' changing security context of '/YAKKO/images/4.8.4/oc' changing security context of '/YAKKO/images/4.8.4/rhcos-live-kernel-x86_64' changing security context of '/YAKKO/images/4.8.4/README.md' changing security context of '/YAKKO/images/4.8.4/rhcos-live-initramfs.x86_64.img' changing security context of '/YAKKO/images/4.8.4/rhcos-metal.x86_64.raw.gz' changing security context of '/YAKKO/images/4.8.4/.downloadcomplete' changing security context of '/YAKKO/images/4.8.4/kubectl' changing security context of '/YAKKO/images/4.8.4' changing security context of '/YAKKO/images/4.8.3/rhcos-live-rootfs.x86_64.img' changing security context of '/YAKKO/images/4.8.3/openshift-install' changing security context of '/YAKKO/images/4.8.3/oc' changing security context of '/YAKKO/images/4.8.3/rhcos-live-kernel-x86_64' changing security context of '/YAKKO/images/4.8.3/README.md' changing security context of '/YAKKO/images/4.8.3/rhcos-live-initramfs.x86_64.img' changing security context of '/YAKKO/images/4.8.3/rhcos-metal.x86_64.raw.gz' changing security context of '/YAKKO/images/4.8.3/.downloadcomplete' changing security context of '/YAKKO/images/4.8.3/kubectl' changing security context of '/YAKKO/images/4.8.3' changing security context of '/YAKKO/images/index.html' changing security context of '/YAKKO/images' __________________________________________________________________________ STAGE 9: Configure Load Balancer (Time start: 15:59pm) Setting up HAproxy with OPEN access You can change this later with 'yakko infra changeaccess' Creating the HA Proxy Config... ● haproxy.service - HAProxy Load Balancer Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2021-08-11 15:59:40 AEST; 10ms ago Process: 290407 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $OPTIONS (code=exited, status=0/SUCCESS) Main PID: 290410 (haproxy) Tasks: 9 (limit: 76991) Memory: 67.0M CPU: 46ms CGroup: /system.slice/haproxy.service ├─290410 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -f /etc/haproxy/yakko └─290413 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -f /etc/haproxy/yakko Aug 11 15:59:40 terminus haproxy[290410]: [WARNING] 222/155940 (290410) : config : missing timeouts for proxy 'test-ingress-router-80'. Aug 11 15:59:40 terminus haproxy[290410]: | While not properly invalid, you will certainly encounter various problems Aug 11 15:59:40 terminus haproxy[290410]: | with such a configuration. To fix this, please ensure that all following Aug 11 15:59:40 terminus haproxy[290410]: | timeouts are set to a non-zero value: 'client', 'connect', 'server'. Aug 11 15:59:40 terminus haproxy[290410]: [WARNING] 222/155940 (290410) : config : missing timeouts for proxy 'test-ingress-router-443'. Aug 11 15:59:40 terminus haproxy[290410]: | While not properly invalid, you will certainly encounter various problems Aug 11 15:59:40 terminus haproxy[290410]: | with such a configuration. To fix this, please ensure that all following Aug 11 15:59:40 terminus haproxy[290410]: | timeouts are set to a non-zero value: 'client', 'connect', 'server'. Aug 11 15:59:40 terminus haproxy[290410]: [NOTICE] 222/155940 (290410) : New worker #1 (290413) forked Aug 11 15:59:40 terminus systemd[1]: Started HAProxy Load Balancer. __________________________________________________________________________ STAGE 10: Configure Firewall (Time start: 15:59pm) Firewall is not running. Configuration is not required. __________________________________________________________________________ STAGE 11: OCP Configuration for Installation (Time start: 15:59pm) Writing ocp-setup-env script for administration. Run "source /YAKKO/ocp-setup-env" to load post-install... The server has [4] CPU cores and [8] threads Generating INSTALL CONFIG file... Making a reference copy of install-config.yaml as install-config.yaml.original __________________________________________________________________________ STAGE 12: OCP Ingest manifest and ignition files (Time start: 15:59pm) Creating manifests... INFO Consuming Install Config from target directory WARNING Making control-plane schedulable by setting MastersSchedulable to true for Scheduler cluster settings INFO Manifests created in: /YAKKO/install-test/manifests and /YAKKO/install-test/openshift Creating OCP Cluster ignition files required for node configuration INFO Consuming Common Manifests from target directory INFO Consuming OpenShift Install (Manifests) from target directory INFO Consuming Worker Machines from target directory INFO Consuming Master Machines from target directory INFO Consuming Openshift Manifests from target directory INFO Ignition-Configs created in: /YAKKO/install-test and /YAKKO/install-test/auth __________________________________________________________________________ STAGE 13: KVM Bootstrap Host Configuration (Time start: 15:59pm) Building OCP node: bootstrap.test.home Configuration: - vCPUs: 2 - Memory: 6000 MiB - MAC Addr: 52:54:00:4a:8c:09 Starting install... Retrieving file rhcos-live-kernel-x86_64... | 9.6 MB 00:00 Retrieving file rhcos-live-initramfs.x86_64.img... | 85 MB 00:00 Allocating 'bootstrap.test.home.qcow2' | 20 GB 00:00 Domain is still running. Installation may be in progress. Waiting for the installation to complete. Domain has shutdown. Continuing. Domain creation completed. Restarting guest. __________________________________________________________________________ STAGE 14: KVM Master Nodes Configuration (Time start: 16:00pm) Building OCP node: master-0.test.home Configuration: - vCPUs: 4 - Memory: 14000 MiB - MAC Addr: 52:54:00:4a:8c:10 Starting install... Retrieving file rhcos-live-kernel-x86_64... | 9.6 MB 00:00 Retrieving file rhcos-live-initramfs.x86_64.img... | 85 MB 00:00 Allocating 'master-0.test.home.qcow2' | 30 GB 00:00 Domain is still running. Installation may be in progress. Waiting for the installation to complete. Domain has shutdown. Continuing. Domain creation completed. Restarting guest. Building OCP node: master-1.test.home Configuration: - vCPUs: 4 - Memory: 14000 MiB - MAC Addr: 52:54:00:4a:8c:11 Starting install... Retrieving file rhcos-live-kernel-x86_64... | 9.6 MB 00:00 Retrieving file rhcos-live-initramfs.x86_64.img... | 85 MB 00:00 Allocating 'master-1.test.home.qcow2' | 30 GB 00:00 Domain is still running. Installation may be in progress. Waiting for the installation to complete. Domain has shutdown. Continuing. Domain creation completed. Restarting guest. Building OCP node: master-2.test.home Configuration: - vCPUs: 4 - Memory: 14000 MiB - MAC Addr: 52:54:00:4a:8c:12 Starting install... Retrieving file rhcos-live-kernel-x86_64... | 9.6 MB 00:00 Retrieving file rhcos-live-initramfs.x86_64.img... | 85 MB 00:00 Allocating 'master-2.test.home.qcow2' | 30 GB 00:00 Domain is still running. Installation may be in progress. Waiting for the installation to complete. Domain has shutdown. Continuing. Domain creation completed. Restarting guest. __________________________________________________________________________ STAGE 15: KVM Worker Node Configuration (Time start: 16:02pm) Restarting virtual network Updated network net-yakko-test persistent config and live state Updating and restarting HAproxy Building OCP node: node-1.test.home Configuration: - vCPUs: 2 - Memory: 4000 MiB - MAC Addr: 52:54:00:4a:8c:20 Starting install... Retrieving file rhcos-live-kernel-x86_64... | 9.6 MB 00:00 Retrieving file rhcos-live-initramfs.x86_64.img... | 85 MB 00:00 Allocating 'node-1.test.home.qcow2' | 30 GB 00:00 Domain is still running. Installation may be in progress. Waiting for the installation to complete. Domain has shutdown. Continuing. Domain creation completed. Restarting guest. Restarting virtual network Updated network net-yakko-test persistent config and live state Updating and restarting HAproxy Building OCP node: node-2.test.home Configuration: - vCPUs: 2 - Memory: 4000 MiB - MAC Addr: 52:54:00:4a:8c:21 Starting install... Retrieving file rhcos-live-kernel-x86_64... | 9.6 MB 00:00 Retrieving file rhcos-live-initramfs.x86_64.img... | 85 MB 00:00 Allocating 'node-2.test.home.qcow2' | 30 GB 00:00 Domain is still running. Installation may be in progress. Waiting for the installation to complete. Domain has shutdown. Continuing. Domain creation completed. Restarting guest. __________________________________________________________________________ STAGE 16: OCP Cluster Bootstrap (Time start: 16:03pm) You can observe the output of the bootstrap node at this stage by issuing: ssh -i /root/.ssh/id_rsa_ocp.test core@bootstrap.test.home sudo journalctl -b -f -u bootkube.service INFO Waiting up to 20m0s for the Kubernetes API at https://api.test.home:6443... INFO API v1.21.1+38b3ecc up INFO Waiting up to 30m0s for bootstrapping to complete... INFO It is now safe to remove the bootstrap resources INFO Time elapsed: 14m45s Domain 'bootstrap.test.home' destroyed Domain 'bootstrap.test.home' has been undefined Volume 'vda'(/mnt/VMs/bootstrap.test.home.qcow2) removed. __________________________________________________________________________ STAGE 17: CSR Background Approval (Time start: 16:18pm) CSR approval task will run in the background and will exit automatically on cluster completion. __________________________________________________________________________ STAGE 18: Prometheus Memory Footprint (Time start: 16:18pm) configmap/cluster-monitoring-config created __________________________________________________________________________ STAGE 19: OCP - Complete Installation (Time start: 16:18pm) Some useful commands while waiting: - tail -f /YAKKO/install-test/.openshift_install.log - source ocp-setup-env ----> For access to "oc" at the command line - oc get clusteroperators -> To check operator progression, can also be "oc get co" - oc get clusterversion ---> On build, it shows you % progression, after that, cluster version - oc get nodes ------------> To see nodes and node status Deleting Prometheus pods for memory reconfiguration CSR Approvals (oc get csr) stopped... __________________________________________________________________________ FINISHED OCP INSTALLATION - Wed 11 Aug 16:35:44 AEST 2021 Time elapsed: 37 mins 13 secs IMPORTANT: - you can bookmark cluster info at http://192.168.100.2:8080 - you DO NOT have a registry -> yakko ops localregistry - you cannot access the cluster from another computer -> yakko infra openaccess - you have no user DB-> yakko ops htpasswd administrator & yakko ops useradd <user> __________________________________________________________________________ CLUSTER: test.home (Ver: 4.8.4 Built: 11-Aug-2021@16:35:44) Active Masters: 3/3 Active Nodes: 2/2 (workers/infra) Active Operators: 31/31 state Web Console: [ ✔ ] https://console-openshift-console.apps.test.home API Server: [ ✔ ] https://api.test.home:6443 Administrator: kubeadmin Password: naW6S-ntkWW-uxhGo-ewVob External access: ENABLED (to change: yakko infra changeaccess) - To use OpenShift's 'oc' command --> source ocp-setup-env (in this shell) - To make infrastructure changes ---> yakko infra <options> - To make operational changes ------> yakko ops <options>