DevOps
docker와 iptables 동시 사용 시 문제
xlwdn98767
2023. 7. 24. 01:35
728x90
iptables
방화벽 도구로써 트래픽을 제어할 수 있다. chain == 방화벽 규칙이다.
ubuntu@ip-172-31-59-210:~$ sudo iptables --help
iptables v1.8.7
Usage: iptables -[ACD] chain rule-specification [options]
iptables -I chain [rulenum] rule-specification [options]
iptables -R chain rulenum rule-specification [options]
iptables -D chain rulenum [options]
iptables -[LS] [chain [rulenum]] [options]
iptables -[FZ] [chain] [options]
iptables -[NX] chain
iptables -E old-chain-name new-chain-name
iptables -P chain target [options]
iptables -h (print this help information)
Commands:
Either long or short options are allowed.
--append -A chain Append to chain
--check -C chain Check for the existence of a rule
--delete -D chain Delete matching rule from chain
--delete -D chain rulenum
Delete rule rulenum (1 = first) from chain
--insert -I chain [rulenum]
Insert in chain as rulenum (default 1=first)
--replace -R chain rulenum
Replace rule rulenum (1 = first) in chain
--list -L [chain [rulenum]]
List the rules in a chain or all chains
--list-rules -S [chain [rulenum]]
Print the rules in a chain or all chains
--flush -F [chain] Delete all rules in chain or all chains
--zero -Z [chain [rulenum]]
Zero counters in chain or all chains
--new -N chain Create a new user-defined chain
--delete-chain
-X [chain] Delete a user-defined chain
--policy -P chain target
Change policy on chain to target
--rename-chain
-E old-chain new-chain
Change chain name, (moving any references)- A 옵션: chain에 append.(뒤에 추가)
- D 옵션: chain 삭제.
- I 옵션: chain에 insert.(앞에 삽입)
- L 옵션: list 조회.(-v와 함께 사용 시 자세히 조회할 수 있음.)
- F 옵션: 전체삭제.
Docker && iptables
docker와 iptables를 동시에 사용하게되면 굉장히 큰 문제가 발생하는데, 이는 docker가 네트워크 트래픽 제어 도구로써 iptables를 사용함에 기인한다.
root@ip-172-31-59-210:/home/ubuntu# iptables -L -v
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy DROP 52 packets, 3584 bytes)
pkts bytes target prot opt in out source destination
93 13651 DOCKER-USER all -- any any anywhere anywhere
93 13651 DOCKER-ISOLATION-STAGE-1 all -- any any anywhere anywhere
51 4615 ACCEPT all -- any docker0 anywhere anywhere ctstate RELATED,ESTABLISHED
2 120 DOCKER all -- any docker0 anywhere anywhere
40 8916 ACCEPT all -- docker0 !docker0 anywhere anywhere
0 0 ACCEPT all -- docker0 docker0 anywhere anywhere
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain DOCKER (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT tcp -- !docker0 docker0 anywhere ip-172-17-0-2.ap-northeast-2.compute.internal tcp dpt:mysql
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
pkts bytes target prot opt in out source destination
40 8916 DOCKER-ISOLATION-STAGE-2 all -- docker0 !docker0 anywhere anywhere
93 13651 RETURN all -- any any anywhere anywhere
Chain DOCKER-ISOLATION-STAGE-2 (1 references)
pkts bytes target prot opt in out source destination
0 0 DROP all -- any docker0 anywhere anywhere
40 8916 RETURN all -- any any anywhere anywhere
Chain DOCKER-USER (1 references)
pkts bytes target prot opt in out source destination
93 13651 RETURN all -- any any anywhere anywhere위 터미널 내용은 단순히 docker가 설치된 환경에서 iptables -L -v를 실행하면 표시되는 화면이다. 내용을 살펴보면 docker 관련된 내용들이 추가됨을 확인할 수 있다.
Test 1
서버에 mysql을 docker로 띄운 뒤 접속을 시도한다.
root@server:/home/ubuntu# docker run -d -p 3306:3306 -e MYSQL_ROOT_PASSWORD=password mysql:8.0connection_test.py
#!/usr/bin/python3
# _*_ coding: utf-8 _*_
import MySQLdb as mdb
import sys
import time
from datetime import datetime
con = mdb.connect('3.38.136.55', 'root', 'password', 'test');
while True:
cur = con.cursor()
cur.execute("SELECT VERSION()")
ver = cur.fetchone()
print("Database versoin: " + ver[0])
time.sleep(60)
ubuntu@client:~$ ./connection_test.py
Database versoin: 8.0.33
DB 서버에 성공적으로 접속한 모습이다.
client에서 netstat으로 조회 시 성공적으로 estblished된 모습이 조회된다.
ubuntu@client:~$ sudo netstat -napo | grep 3306 | grep -i est
tcp 0 0 10.0.0.232:33150 3.38.136.55:3306 ESTABLISHED 107088/python3 keepalive (3.36/0/0)Server -> Client traffic drop
server->client의 트래픽을 drop 시키고 db container를 종료시킨다면 client 측에서는 연결이 해제되었다는 사실을 파악하지 못하고 keepalive 패킷을 지속적으로 전송하므로 established 상태가 지속될 것이다.
root@server:/home/ubuntu# iptables -A OUTPUT -p tcp -d 15.165.19.77 -j DROP
root@server:/home/ubuntu# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8456dcb41e12 mysql:8.0 "docker-entrypoint.s…" 10 minutes ago Up 10 minutes 0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp upbeat_kirch
root@server:/home/ubuntu# docker stop 845
845
root@server:/home/ubuntu# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMESubuntu@client:~$ sudo netstat -napo | grep 3306 | grep -i est
tcp 0 0 10.0.0.232:33150 3.38.136.55:3306 ESTABLISHED 107088/python3 keepalive (14.20/0/0)
ubuntu@client:~$ sudo netstat -napo | grep 3306 | grep -i est
ubuntu@client:~$client 측에서 DB가 종료된 사실을 즉시 인지하였다.
이를 통해 DB가 종료되기 전 자신과 연결을 맺은 client에게 종료 사실을 전달하는 데에 성공하였단 사실을 파악할 수 있다.
service mysql
root@server:/home/ubuntu# docker stop c515
c515
root@server:/home/ubuntu# service mysql start
root@server:/home/ubuntu# service mysql status
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2023-07-23 16:20:17 UTC; 3s ago
Process: 186994 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
Main PID: 187002 (mysqld)
Status: "Server is operational"
Tasks: 38 (limit: 1111)
Memory: 409.5M
CPU: 1.093s
CGroup: /system.slice/mysql.service
└─187002 /usr/sbin/mysqlddocker로 띄우던 DB를 직접 설치하고 iptables를 삭제한다.
root@server:/home/ubuntu# iptables -D OUTPUT -p tcp -d 15.165.19.77 -j DROPubuntu@client:~$ sudo netstat -napo | grep 3306 | grep -i est
tcp 0 0 10.0.0.232:45286 3.38.136.55:3306 ESTABLISHED 107260/python3 keepalive (19.46/0/0)성공적으로 연결된 모습을 확인할 수 있다. 여기서 traffic을 drop시키고 db를 종료시킨다.
root@ubuntu:/home/ubuntu# iptables -A OUTPUT -d 15.165.19.77 -j DROP
root@ubuntu:/home/ubuntu# service mysql stop
root@ubuntu:/home/ubuntu#ubuntu@client:~$ curl 3.38.136.55client에서 db 서버로 요청을 전송하였으나 응답 대기 상태가 지속되므로 트래픽이 정상적으로 drop됨을 확인할 수 있다. 이후 nestat을 통해 조회하면 연결이 est상태로 지속되는 것을 확인할 수 있다.
ubuntu@ip-10-0-0-232:~$ curl 3.38.136.55
^C
ubuntu@ip-10-0-0-232:~$ sudo netstat -napo | grep 3306 | grep -i est
tcp 0 45 10.0.0.232:45286 3.38.136.55:3306 ESTABLISHED 107260/python3 on (78.84/9/0)db가 종료될 때 client로 자신의 죽음을 알렸으나 이번에는 트래픽이 정상적으로 drop되어 client측에서 db의 죽음을 인지하지 못한 것을 확인할 수 있다.
Test 2
root@server:/home/ubuntu# docker run -d -p 8080:8080 bitnami/nginx
Unable to find image 'bitnami/nginx:latest' locally
latest: Pulling from bitnami/nginx
323218fa29f7: Pull complete
Digest: sha256:2d1f6e1612377bbff8f0aa08b1e577da899c16446df5fb47f015a2cc6a54225f
Status: Downloaded newer image for bitnami/nginx:latest
70c432fb4ada324d5369b56c05a47b8a83f589f878e9b4e66bc99f8db221584aubuntu@client:~$ curl 3.38.136.55:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>client에서 server로 정상적으로 요청을 수행할 수 있음을 확인하였다. 이후 iptables -F를 통해 모든 규칙을 삭제하고 위 작업을 반복한다.
root@server:/home/ubuntu# iptables -Fubuntu@cleint:~$ curl 3.38.136.55:8080작업이 완료되지 않고 중지된다.
root@ip-172-31-59-210:/home/ubuntu# iptables -L -v
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy DROP 58 packets, 3944 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain DOCKER (0 references)
pkts bytes target prot opt in out source destination
Chain DOCKER-ISOLATION-STAGE-1 (0 references)
pkts bytes target prot opt in out source destination
Chain DOCKER-ISOLATION-STAGE-2 (0 references)
pkts bytes target prot opt in out source destination
Chain DOCKER-USER (0 references)
pkts bytes target prot opt in out source destination
root@ip-172-31-59-210:/home/ubuntu#iptable docker 규칙 조차 모두 사라져 더 이상 정상적으로 활용할 수 없음을 확인하였다.(실험 이후 삭제 -> 재설치)
Conclusion
docker와 함께 iptables를 사용할 때에는 각별한 주의가 필요하며, 되도록 다른 유틸을 사용하여 제어하길 권장한다.