ubuntu 환경에서 기본 방화벽인 ufw 에 ipset 을 활용하여 kubernetes 서비스 포트들을 등록하는 방법 소개
ufw는 부분적으로 iptables-restore 에 대한 프론트 엔드이며 Uncomplicated Firewall 이라고 부릅니다.
On-premise 환경에서는 보안등의 이유로 firewall 설정이 필요하며, 인터넷에 나와있는 대부분의 kubernetes 환경에서는 firewall 은 배제하고 구성되어 이번에 룰을 구성하면서 정리한 자료를 공유
Iptables Process Flow
iptables 의 경우 Linux 커널의 네트워크 스택에 있는 패킷 필터링 Hook 과 연동되어 동작한다.
Reference : https://kangtaeho.com/66
ufw flow
# ufw 규칙은 before.rules → user.rules → after.rules 순으로 평가된다.
# 기본적으로 ufw command 로 생성되는 규칙은 user.rules 에 들어감
~# tree /etc/ufw/
/etc/ufw/
├── after.init
├── after.rules
├── after6.rules
├── applications.d
│ └── openssh-server
├── before.init
├── before.rules
├── before6.rules
├── sysctl.conf
├── ufw.conf
├── user.rules
└── user6.rules
Bash
복사
규칙 생성
K8S_NODE_NET = MASTER, NODE 들의 IP
K8S_POD_NET = pods 들의 IP 대역
K8S_CLUSTER_NET = CLUSTER IP 대역
K8S_EX_POOL = Public DNS 용
K8S_IN_POOL = Private DNS 용
K8S_ALL_LIST = ipset 초기화를 위한 모든 NET 리스트 추가
K8S_OUT_PORT = kubernetes output port
K8S_IN_PORT = Kubernetes input port
K8S_INTER_PORT = kubernetes 클러스터 내부 master, node 끼리 통신할경우 사용되는 포트
# 추가
K8S_DST_PROTO_HASH = calico 사용시 type 4 가 ufw 에서 차단됨, 추가
Bash
복사
after.rules
#
# rule for kubernetes
#
-A ufw-after-input -o vxlan.calico -m comment --comment "K8S calico shoul be allowed" -j ACCEPT
-A ufw-after-output -o vxlan.calico -m comment --comment "K8S calico shoul be allowed" -j ACCEPT
-A ufw-after-input -o cali+ -m comment --comment "K8S calico shoul be allowed" -j ACCEPT
-A ufw-after-output -o cali+ -m comment --comment "K8S calico shoul be allowed" -j ACCEPT
# enable protocol number rules between nodes
-A ufw-after-input -m set --match-set K8S_NODE_NET src --match-set K8S_DST_PROTO_HASH dst,dst -m comment --comment "K8S accept protocol number rule inbound" -j ACCEPT
-A ufw-after-output -m set --match-set K8S_NODE_NET src --match-set K8S_DST_PROTO_HASH dst,dst -m comment --comment "K8S accept protocol number rule inbound" -j ACCEPT
# enable inter node k8s default ports
-A ufw-after-input -m set --match-set K8S_NODE_NET src -m set --match-set K8S_NODE_NET dst -m set --match-set K8S_INTER-PORT dst -m comment --comment "accept kubernetes node ports" -j ACCEPT
-A ufw-after-output -m set --match-set K8S_NODE_NET src -m set --match-set K8S_NODE_NET dst -m set --match-set K8S_INTER-PORT dst -m comment --comment "accept kubernetes node ports" -j ACCEPT
# whitelist internode from cluster-cidr node address
-A ufw-after-input -m set --match-set K8S_CLUSTER_NET src -m set --match-set K8S_NODE_NET dst -m comment --comment "K8S enable cluster service network access to node network" -j ACCEPT
-A ufw-after-output -m set --match-set K8S_CLUSTER_NET src -m set --match-set K8S_NODE_NET dst -m comment --comment "K8S enable cluster service network access to node network" -j ACCEPT
# enable vip external services
-A ufw-after-input -m set --match-set K8S_EX_POOL dst -m set --match-set K8S_IN_PORT dst -m comment --comment --comment "accept external web services on VIP" -j ACCEPT
-A ufw-after-input -m set --match-set K8S_IN_POOL dst -m set --match-set K8S_IN_PORT dst -m comment --comment --comment "accept external web services on VIP" -j ACCEPT
#
# ends
#
Bash
복사
ipset
ipset 사용이유, iptables 에서 match 를 구성할 경우, 매우 복잡한 iptable 이 생성되는 데 이것을 hashset 으로 만들어서 구성, 속도와 관리를 모두 편리하게 해 줄수 있다.
install
sudo apt install ipset iptables netfilter-persistent ipset-persistent
sudo systemctl enable ipset-persistent
sudo system start ipset-persistent
# 바로 저장
sudo netfilter-persistent save
Bash
복사
ipset 등록을 위한 스크립트
echo "Creating PORT bimmap"
sudo ipset create -! K8S_INTER_PORT bitmap:port range 0-65535 comment
sudo ipset create -! K8S_OUT_PORT bitmap:port range 0-65535 comment
sudo ipset create -! K8S_IN_PORT bitmap:port range 0-65535 comment
# port add
sudo ipset create -! K8S_DST_PROTO_HASH hash:net,port comment
sudo ipset add -! K8S_DST_PROTO_HASH 172.88.10.11/32,4:0 comment "calico ip-in-ip proto 4 type bidrectional"
sudo ipset add -! K8S_DST_PROTO_HASH 172.88.10.12/32,4:0 comment "calico ip-in-ip proto 4 type bidrectional"
sudo ipset add -! K8S_DST_PROTO_HASH 172.88.10.13/32,4:0 comment "calico ip-in-ip proto 4 type bidrectional"
sudo ipset add -! K8S_DST_PROTO_HASH 172.88.10.14/32,4:0 comment "calico ip-in-ip proto 4 type bidrectional"
# metallb
sudo ipset add -! K8S_INTER_PORT 7472 comment "metallb layer2 port"
sudo ipset add -! K8S_INTER_PORT 7946 comment "metallb layer2 port"
# calico
sudo ipset add -! K8S_INTER_PORT tcp:179 comment "calico networking bgp in/out"
sudo ipset add -! K8S_INTER_PORT udp:4789 COMMENT "calico networking with VXLAN in/out"
sudo ipset add -! K8S_INTER_PORT tcp:5473 COMMENT "calico networking with Typha in typha agent host"
sudo ipset add -! K8S_INTER_PORT udp:51820-51821 comment "calico networking with ipvx wireguard in/out"
sudo ipset add -! K8S_INTER_PORT tcp:8080 comment "calico addtional port 8080"
# proto:0 는 별도로 등록 필요 bitmap:port 에서는 등록 불가
#sudo ipset add -! K8S_INTER_PORT 4:0 comment "calico networking ip-in-ip in/out"
# K8S
sudo ipset add -! K8S_INTER_PORT tcp:443 comment "k8s apiserver host 443 6443 in"
sudo ipset add -! K8S_INTER_PORT tcp:6443 comment "k8s apiserver host 443 6443 in"
sudo ipset add -! K8S_INTER_PORT tcp:2379-2380 comment "k8s etcd datastore can vary in"
sudo ipset add -! K8S_INTER_PORT tcp:10250 comment "k8s api used by self and control plane inbound"
sudo ipset add -! K8S_INTER_PORT tcp:10257 comment "k8s control manager inbound"
sudo ipset add -! K8S_INTER_PORT tcp:10259 comment "k8s scheduler self, control plane inbound"
sudo ipset add -! K8S_INTER_PORT tcp:30000-32767 comment "k8s nodeport service all"
# External open (service in ports)
sudo ipset add -! K8S_IN_PORT tcp:80 comment "generic http port inbound"
sudo ipset add -! K8S_IN_PORT tcp:443 comment "generic https port inbound"
# -------------- register network ------------ #
echo "adding node and network"
sudo ipset create -! K8S_NODE_NET hash:net family inet comment
sudo ipset create -! K8S_POD_NET hash:net family inet comment
sudo ipset create -! K8S_CLUSTER_NET hash:net family inet comment
echo "destroying K8S_ALL_NET and recreate as set"
sudo ipset destroy K8S_ALL_LIST
sudo ipset create -! K8S_ALL_LIST list:set comment
sudo ipset create -! K8S_EX_POOL hash:set family inet comment
sudo ipset create -! K8S_IN_POOL hash:set family inet comment
echo "adding K8S-*-NET as member of K8S_ALL_NET"
sudo ipset add -! K8S_ALL_LIST K8S_NODE_NET comment "member K8S_NODE_NET"
sudo ipset add -! K8S_ALL_LIST K8S_POD_NET comment "member K8S_POD_NET"
sudo ipset add -! K8S_ALL_LIST K8S_CLUSTER_NET comment "member K8S_POD_NET"
# K8S node ip
sudo ipset add -! K8S_NODE_NET 172.88.10.11/32 comment "k8s master node"
sudo ipset add -! K8S_NODE_NET 172.88.10.12/32 comment "k8s worker node 1"
sudo ipset add -! K8S_NODE_NET 172.88.10.13/32 comment "k8s worker node 2"
sudo ipset add -! K8S_NODE_NET 172.88.10.14/32 comment "k8s worker node 3"
## pod network
sudo ipset add -! K8S_POD_NET 192.168.0.0/16 comment "pod network cidr"
sudo ipset add -! K8S_CLUSTER_NET 10.96.0.0/12 comment "k8s cluster network cidr"
## vip pool (metallb)
sudo ipset add -! K8S_EX_POOL 172.88.10.15/32 comment "k8s external loadbalancer ip pool"
sudo ipset add -! K8S_IN_POOL 172.88.10.16/32 comment "k8s external loadbalancer ip pool"
# warning
echo "don't forget to remove K8S_ALL_LIST first if you have trouble, \`ipset x K8S_ALL_LIST\` "
echo "otherwise, you can't clear ipset rule"
Bash
복사
