需求檔案

• butane-amd64
  https://mirror.openshift.com/pub/openshift-v4/clients/butane/v0.28.0-1/butane-amd64
• oc-mirror.tar.gz (oc mirror)
• openshift-client-linux-amd64-rhel9-4.21.20.tar.gz (oc指令)
  https://mirror.openshift.com/pub/openshift-v4/amd64/clients/ocp/stable-4.21/openshift-install-rhel9-amd64.tar.gz
• openshift-install-rhel9-amd64.tar.gz
  https://mirror.openshift.com/pub/openshift-v4/amd64/clients/ocp/stable-4.21/openshift-install-rhel9-amd64.tar.gz
• mirror-registry-amd64.tar.gz (離線倉庫)
• 本次範例下載網址: https://developers.redhat.com/content-gateway/file/pub/cgw/mirror-registry/2.0.9/mirror-registry-amd64.tar.gz 詳細步驟
  1. 網址:mirror.openshift.com -> pub -> cgw -> mirror-registry -> 2.0.9
  2. 下載 : mirror-registry.tar.gz
• Pull secret (憑證)
• https://cloud.redhat.com/ 詳細步驟
  1. google: 搜尋 cloud.redhat -> https://cloud.redhat.com/ ->登入帳號
     
  2. 點選OpenShift
     
  3. OpenShift -> Resources -> Downloads
     
  4. Tokens: Pull secret (DownLoad)
     

Firewall 需要開啟的Port

開啟指定Port

firewall-cmd --add-port={53,80,8080,443,6443,22623}/tcp --permanent
firewall-cmd –reload
firewall-cmd --add-port=53/udp --permanent
firewall-cmd --reload

Check

firewall-cmd --list-all

netstat -tlnp

本地軟體組態設定

請先掛載IOS至自訂路近

vim /etc/yum.repos.d/<reponame>.repo #結尾必須.repo

內文如下

	[base]
	name=base
	baseurl=http://data.example.com/BaseOS  
	#若以iso掛載方式請使用file:///<iso位置> baseurl=file:///mnt/iso/BaseOS
	gpgcheck=0
	enabled=1
	
	[app]
	name=app
	baseurl=http://data.example.com/AppStream  
	#若以iso掛載方式請使用file:///<iso位置> 
	#例:baseurl=file:///mnt/iso/AppStream 
	gpgcheck=0
	enabled=1
**驗證方式**
dnf install httpd

• 方法1 若DNS 可以解析可使用http
  baseurl=http://data.example.com/BaseOS

  /etc/hosts 
   192.168.111.111 data.example.com
  

• 方法2 若以本地ISO軟體庫請先mount 軟體庫在進行後續url填寫
  baseurl=file:///mnt/iso/BaseOS

    mkdir /mnt/iso
    mount /dev/sr0 /mnt/iso
  

安裝httpd、vsftpd、haproxy、dnsmasq wget

yum install -y httpd haproxy dnsmasq policycoreutils-python-utils wget net-tools

設定httpd

設定httpd 伺服器(⽤來讓CoreOS 下載ignition file 使⽤，也可以使⽤vsfptd 的⽅式處理，因為
haproxy 已經佔⽤了80 Port所以要將httpd 的預設Port改為8080

vi /etc/httpd/conf/httpd.conf
#將Listen 80 > Listen 8080

systemctl enable --now httpd

安裝 dnsmasq

yum install -y dnsmasq

設定 dnsmasq.d/dns.conf

vi /etc/dnsmasq.d/dns.conf

安裝 dnsmasq

yum install -y dnsmasq

設定網卡dnsmasq.conf

查看IP網卡

ip a
#ens33

vi /etc/dnsmasq.conf

# 搜尋 interface <網卡>
interface=ens33

vi /etc/resolv.conf

編輯resolv.conf設定檔

# Generated by NetworkManager
search kevin.com
nameserver 192.168.100.40   #<- 指向DNServer位置
# nameserver 1.1.1.1

設定Haproxy

vim /etc/haproxy/haproxy.cfg

設定檔

global
  log         127.0.0.1 local2
  pidfile     /var/run/haproxy.pid
  maxconn     4000
  daemon

defaults
  mode                    http
  log                     global
  option                  dontlognull
  option http-server-close
  option                  redispatch
  retries                 3
  timeout http-request    10s
  timeout queue           1m
  timeout connect         10s
  timeout client          1m
  timeout server          1m
  timeout http-keep-alive 10s
  timeout check           10s
  maxconn                 3000

listen api-server-6443 
  bind *:6443
  mode tcp
  option  httpchk GET /readyz HTTP/1.0
  option  log-health-checks
  balance roundrobin
  server bootstrap bootstrap.kevin.com:6443 verify none check check-ssl inter 10s fall 2 rise 3 backup 
  server master1 master1.kevin.com:6443 weight 1 verify none check check-ssl inter 10s fall 2 rise 3
  server master2 master2.kevin.com:6443 weight 1 verify none check check-ssl inter 10s fall 2 rise 3
  server master3 master3.kevin.com:6443 weight 1 verify none check check-ssl inter 10s fall 2 rise 3

listen machine-config-server-22623 
  bind *:22623
  mode tcp
  server bootstrap bootstrap.kevin.com:22623 check inter 1s backup 
  server master1 master1.kevin.com:22623 check inter 1s
  server master2 master2.kevin.com:22623 check inter 1s
  server master3 master3.kevin.com:22623 check inter 1s

listen ingress-router-443 
  bind *:443
  mode tcp
  balance source
  server worker1 worker1.kevin.com:443 check inter 1s
  server worker2 worker2.kevin.com:443 check inter 1s
  
listen ingress-router-80 
  bind *:80
  mode tcp
  balance source
  server worker1 worker1.kevin.com:80 check inter 1s
  server worker2 worker2.kevin.com:80 check inter 1s

setenforce 0

systemctl restart haproxy.service

可預先掛載空間給mirror-registry作為儲存庫使用200G

要修改hostname???

創建目錄

mkdir /mirror-registry
cd /mirror-registry

解壓縮

tar -zxvf mirror-registry-amd64.tar.gz -C /mirror-registry/

安裝mirror-registry

./mirror-registry install --quayHostname  bastion.kevin.com --quayRoot /mirror-quay --quayStorage /mirror-quay/storage --initPassword redhat123

若有錯誤可以使用un卸載

./mirror-registry uninstall

登入測試

podman login -u init -p redhat123 bastion.kevin.com:8443 --tls-verify=false
Login Succeeded!

設置pull-secret

mkdir /root/.docker
cat /root/Download/pull-secret | jq . > /root/.docker/config.json

前置條件

• 以下步驟需再連線環境執行
  oc mirror需在連線環境下才可下載
• 06_pull-secret設定完畢

解壓縮

tar -zxvf oc-mirror.tar.gz -C /usr/local/bin/ 

權限設定

chmod +x /usr/local/bin/oc-mirror 

編輯imageset-config.yaml

mkdir /root/mirror
vim ~/mirror/imageset-config.yaml

imageset-config.yaml

apiVersion: mirror.openshift.io/v2alpha1
kind: ImageSetConfiguration
archiveSize: 5
mirror:
  platform:
    channels:
      - name: stable-4.21
        minVersion: 4.21.20
        maxVersion: 4.21.20
    graph: true
  operators:
    - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.21
      packages:
        - name: kubevirt-hyperconverged
          channels:
            - name: stable
              minVersion: 4.21.10
              maxVersion: 4.21.10
        - name: mtv-operator
          channels:
            - name: release-v2.12
              minVersion: 2.12.1
              maxVersion: 2.12.1
        - name: kubernetes-nmstate-operator
          channels:
            - name: stable
              minVersion: 4.20.0-202606171653
              maxVersion: 4.20.0-202606171653
        - name: netobserv-operator
          channels:
            - name: stable
              minVersion: 1.12.0
              maxVersion: 1.12.0
        - name: cluster-observability-operator
          channels:
            - name: stable
              minVersion: 1.5.0
              maxVersion: 1.5.0
        - name: cluster-logging
          channels:
            - name: stable-6.5
              minVersion: 6.5.1
              maxVersion: 6.5.1
        - name: loki-operator
          channels:
            - name: stable-6.5
              minVersion: 6.5.1
              maxVersion: 6.5.1
        - name: tempo-product
          channels:
            - name: stable
              minVersion: 0.21.0-1
              maxVersion: 0.21.0-1
        - name: opentelemetry-product
          channels:
            - name: stable
              minVersion: 0.150.0-1
              maxVersion: 0.150.0-1
        - name: local-storage-operator
          channels:
            - name: stable
              minVersion: 4.21.0-202606090112
              maxVersion: 4.21.0-202606090112
        - name: node-healthcheck-operator
          channels:
            - name: stable
              minVersion: 0.11.0
              maxVersion: 0.11.0
        - name: node-maintenance-operator
          channels:
            - name: stable
              minVersion: 5.6.1
              maxVersion: 5.6.1
        - name: self-node-remediation
          channels:
            - name: stable
              minVersion: 0.12.1
              maxVersion: 0.12.1
        - name: cincinnati-operator
          channels:
            - name: v1
              minVersion: 5.0.3
              maxVersion: 5.0.3
        - name: openshift-gitops-operator
          channels:
            - name: latest
              minVersion: 1.21.0
              maxVersion: 1.21.0
        - name: advanced-cluster-management
          channels:
            - name: release-2.17
              minVersion: 2.17.0
              maxVersion: 2.17.0
        - name: multicluster-engine
          channels:
            - name: stable-2.17
              minVersion: 2.17.0
              maxVersion: 2.17.0

下載

oc-mirror --v2 --config=/root/mirror/imageset-config.yaml file:///root/mirror

#CC範例
oc-mirror -c /root/install/ocp/imageset-config.yaml file:///root/install/ocp --cache-dir /root/install/ocp/cache --v2

• 驗證下載

cd /root/mirror/
ls -l

#輸出結果
#imageset-config.yaml
#mirror_000001.tar
#mirror_000002.tar

tar czvf /root/openshift-automation.tar.gz -C /root OpenShift-Automation install_source OpenShift-EaaS-Practice

前置條件

• 已完成資料下載
• 以建立05_mirror-registry

憑證設定

cp /mirror-quay/quay-rootCA/rootCA.pem /etc/pki/ca-trust/source/anchors/rootCA.pem
update-ca-trust #更新
trust list      #檢查

上傳

#oc-mirror -c <imageset-config.yaml位置> --from file://<images.tar位置> docker://<倉庫位置> --v2
oc-mirror -c /root/mirror/imageset-config.yaml --from file:///root/mirror/ docker://bastion.kevin.com:8443/ocp421 --v2

創建資料夾

mkdir /root/ocp4
cd /root/ocp4
vim install-config.yaml

內文如下

apiVersion: v1
baseDomain: {{ baseDomain }} #DNS規範要注意
compute:
- hyperthreading: Enabled
  name: worker
  replicas: 0
controlPlane:
  hyperthreading: Enabled
  name: master
  replicas: 3
metadata:
  name: {{ clusterName }}
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  networkType: OVNKubernetes
  serviceNetwork:
  - 172.30.0.0/16
platform:
  none: {}
fips: false
pullSecret: '{"auths":{"{{ bastion.name }}.{{ clusterName }}.{{ baseDomain }}:8443":{"auth":"{{ ('init:'+registryPassword) | b64encode }}"}}}'
sshKey: '{{ sshkey.stdout }}'
additionalTrustBundle: |
{{ (mirrorRegistryCA['content'] | b64decode) | indent(width=2,first=True) }}
imageDigestSources:
- mirrors:
  - {{ bastion.name }}.{{ clusterName }}.{{ baseDomain }}:8443/{{ reponame }}/openshift/release
  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
- mirrors:
  - {{ bastion.name }}.{{ clusterName }}.{{ baseDomain }}:8443/{{ reponame }}/openshift/release-images
  source: quay.io/openshift-release-dev/ocp-release

• mirror創建憑證

podman login -u init -p redhat123 bastion.kevin.com:8443 --tls-verify=false --authfile aaa
cat aaa|jq -c #jq顯示憑證格式

• 取得sshkey

cat /root/.ssh/id_rsa.pub

#輸出結果 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDadWhGXFA26qgzKdSt29UVwdxXfYV2l1I32Dk7i3n64LhkSOMGCh5b25vYPC2avkyKe471Ty9rcWCuI7aSr0rCHdJDH6nU7dcwAi1YgZHM540VzpjjkJJmW3cXHiOWEq1MN2Pi7I79+dTeK2mTpntsH74Q2aPEtrlt/ZN/8R/TieO0oepXpLV2R/O9eAlM5TtZrQ1keyWbv/72+XbRAMznSNmmcF/sdhmTNfx8vEmK3SrvgeK86/iy7OiRPsf+jm2VPndSvvadUivKn8yMOnU3mH2OR9k4hvSHz1fi7o0ixglLv60as5cJopB6g7MSmsGkaHh7B2Pkw97CbjYrPta5aUdKOqQrZd2TUx5EyAKgO7Ypsen50JbhqL8VtO9LfAA0WueEztFLYSiuAB/kixHtpYsG/PMtm+vjP/OUoVVBFu5dLAh8xjkHkoqhvhbU5zX/5QeqOuYqiJtxeamhEJ4FAxT2YHlp/44M9Sr0b6iGx3eISF7GESIyNDtdp1OSmbM= root@ocp.com

• 取得imageDigestSources

#上傳完oc mirror才會有此檔
#可以從這裡查看
vim  /root/mirror/working-dir/cluster-resources/idms-oc-mirror.yaml

內文如下 查詢spec -> imageDigestMirrors -> mirrors :
bastion.kevin.com:8443/ocp421/openshift/release-images

---
apiVersion: config.openshift.io/v1
kind: ImageDigestMirrorSet
metadata:
  annotations:
    createdAt: Wednesday, 01-Jul-26 08:38:35 UTC
    createdBy: oc-mirror v2
    oc-mirror_version: 4.21.0-202606240914.p2.gf981281.assembly.stream.el9-f981281
  name: idms-operator-0
spec:
  imageDigestMirrors:
  - mirrors:
    - bastion.kevin.com:8443/ocp421/migration-toolkit-virtualization
    source: registry.redhat.io/migration-toolkit-virtualization
  - mirrors:
    - bastion.kevin.com:8443/ocp421/container-native-virtualization
    source: registry.redhat.io/container-native-virtualization
  - mirrors:
    - bastion.kevin.com:8443/ocp421/openshift4
    source: registry.redhat.io/openshift4
---
apiVersion: config.openshift.io/v1
kind: ImageDigestMirrorSet
metadata:
  annotations:
    createdAt: Wednesday, 01-Jul-26 08:38:35 UTC
    createdBy: oc-mirror v2
    oc-mirror_version: 4.21.0-202606240914.p2.gf981281.assembly.stream.el9-f981281
  name: idms-release-0
spec:
  imageDigestMirrors:
  - mirrors:
    - bastion.kevin.com:8443/ocp421/openshift/release
    source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
  - mirrors:
    - bastion.kevin.com:8443/ocp421/openshift/release-images
    source: quay.io/openshift-release-dev/ocp-release

這是Kubernetes 建置時候產⽣的那個config.yaml file，就是定義Master 或是Worker 的設定檔。

cp -r ocp4/ ocp4bak

建立manifests

openshift-install create manifests --dir=/root/ocp4

#輸出解果
INFO Consuming Install Config from target directory 
WARNING Making control-plane schedulable by setting MastersSchedulable to true for Scheduler cluster settings 
INFO Successfully populated MCS CA cert information: root-ca 2036-06-28T09:14:41Z 2026-07-01T09:14:41Z 
INFO Successfully populated MCS TLS cert information: root-ca 2036-06-28T09:14:41Z 2026-07-01T09:14:41Z 
INFO Manifests created in: /root/ocp4, /root/ocp4/manifests and /root/ocp4/openshift 

執⾏後會產⽣2 個資料夾
manifests\ ，openshift\

ls /root/ocp4/

修改cluster-scheduler-02-config.yml設定

vim /root/ocp4/manifests/cluster-scheduler-02-config.yml

NTPSERVER 可以預先建置