docker와 iptables 동시 사용 시 문제

iptables

---

방화벽 도구로써 트래픽을 제어할 수 있다. chain == 방화벽 규칙이다.

ubuntu@ip-172-31-59-210:~$ sudo iptables --help
iptables v1.8.7

Usage: iptables -[ACD] chain rule-specification [options]
    iptables -I chain [rulenum] rule-specification [options]
    iptables -R chain rulenum rule-specification [options]
    iptables -D chain rulenum [options]
    iptables -[LS] [chain [rulenum]] [options]
    iptables -[FZ] [chain] [options]
    iptables -[NX] chain
    iptables -E old-chain-name new-chain-name
    iptables -P chain target [options]
    iptables -h (print this help information)

Commands:
Either long or short options are allowed.
  --append  -A chain        Append to chain
  --check   -C chain        Check for the existence of a rule
  --delete  -D chain        Delete matching rule from chain
  --delete  -D chain rulenum
                Delete rule rulenum (1 = first) from chain
  --insert  -I chain [rulenum]
                Insert in chain as rulenum (default 1=first)
  --replace -R chain rulenum
                Replace rule rulenum (1 = first) in chain
  --list    -L [chain [rulenum]]
                List the rules in a chain or all chains
  --list-rules -S [chain [rulenum]]
                Print the rules in a chain or all chains
  --flush   -F [chain]        Delete all rules in  chain or all chains
  --zero    -Z [chain [rulenum]]
                Zero counters in chain or all chains
  --new     -N chain        Create a new user-defined chain
  --delete-chain
         -X [chain]        Delete a user-defined chain
  --policy  -P chain target
                Change policy on chain to target
  --rename-chain
         -E old-chain new-chain
                Change chain name, (moving any references)

• A 옵션: chain에 append.(뒤에 추가)
• D 옵션: chain 삭제.
• I 옵션: chain에 insert.(앞에 삽입)
• L 옵션: list 조회.(-v와 함께 사용 시 자세히 조회할 수 있음.)
• F 옵션: 전체삭제.

Docker && iptables

---

docker와 iptables를 동시에 사용하게되면 굉장히 큰 문제가 발생하는데, 이는 docker가 네트워크 트래픽 제어 도구로써 iptables를 사용함에 기인한다.

root@ip-172-31-59-210:/home/ubuntu# iptables -L -v
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy DROP 52 packets, 3584 bytes)
 pkts bytes target     prot opt in     out     source               destination
   93 13651 DOCKER-USER  all  --  any    any     anywhere             anywhere
   93 13651 DOCKER-ISOLATION-STAGE-1  all  --  any    any     anywhere             anywhere
   51  4615 ACCEPT     all  --  any    docker0  anywhere             anywhere             ctstate RELATED,ESTABLISHED
    2   120 DOCKER     all  --  any    docker0  anywhere             anywhere
   40  8916 ACCEPT     all  --  docker0 !docker0  anywhere             anywhere
    0     0 ACCEPT     all  --  docker0 docker0  anywhere             anywhere

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     tcp  --  !docker0 docker0  anywhere             ip-172-17-0-2.ap-northeast-2.compute.internal  tcp dpt:mysql

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination
   40  8916 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  anywhere             anywhere
   93 13651 RETURN     all  --  any    any     anywhere             anywhere

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       all  --  any    docker0  anywhere             anywhere
   40  8916 RETURN     all  --  any    any     anywhere             anywhere

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination
   93 13651 RETURN     all  --  any    any     anywhere             anywhere

위 터미널 내용은 단순히 docker가 설치된 환경에서 iptables -L -v를 실행하면 표시되는 화면이다. 내용을 살펴보면 docker 관련된 내용들이 추가됨을 확인할 수 있다.

Test 1

---

서버에 mysql을 docker로 띄운 뒤 접속을 시도한다.

root@server:/home/ubuntu# docker run -d -p 3306:3306 -e MYSQL_ROOT_PASSWORD=password mysql:8.0

connection_test.py

#!/usr/bin/python3
# _*_ coding: utf-8 _*_
import MySQLdb as mdb
import sys
import time
from datetime import datetime

con = mdb.connect('3.38.136.55', 'root', 'password', 'test');

while True:

    cur = con.cursor()
    cur.execute("SELECT VERSION()")

    ver = cur.fetchone()

    print("Database versoin: " + ver[0])
    time.sleep(60)

ubuntu@client:~$ ./connection_test.py
Database versoin: 8.0.33

DB 서버에 성공적으로 접속한 모습이다.

client에서 netstat으로 조회 시 성공적으로 estblished된 모습이 조회된다.

ubuntu@client:~$ sudo netstat -napo | grep 3306 | grep -i est
tcp        0      0 10.0.0.232:33150        3.38.136.55:3306        ESTABLISHED 107088/python3       keepalive (3.36/0/0)

Server -> Client traffic drop

---

server->client의 트래픽을 drop 시키고 db container를 종료시킨다면 client 측에서는 연결이 해제되었다는 사실을 파악하지 못하고 keepalive 패킷을 지속적으로 전송하므로 established 상태가 지속될 것이다.

root@server:/home/ubuntu# iptables -A OUTPUT -p tcp -d 15.165.19.77 -j DROP
root@server:/home/ubuntu# docker ps
CONTAINER ID   IMAGE       COMMAND                  CREATED          STATUS          PORTS                                                  NAMES
8456dcb41e12   mysql:8.0   "docker-entrypoint.s…"   10 minutes ago   Up 10 minutes   0.0.0.0:3306->3306/tcp, :::3306->3306/tcp, 33060/tcp   upbeat_kirch
root@server:/home/ubuntu# docker stop 845
845
root@server:/home/ubuntu# docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

ubuntu@client:~$ sudo netstat -napo | grep 3306 | grep -i est
tcp        0      0 10.0.0.232:33150        3.38.136.55:3306        ESTABLISHED 107088/python3       keepalive (14.20/0/0)
ubuntu@client:~$ sudo netstat -napo | grep 3306 | grep -i est
ubuntu@client:~$

client 측에서 DB가 종료된 사실을 즉시 인지하였다.
이를 통해 DB가 종료되기 전 자신과 연결을 맺은 client에게 종료 사실을 전달하는 데에 성공하였단 사실을 파악할 수 있다.

service mysql

---

root@server:/home/ubuntu# docker stop c515
c515
root@server:/home/ubuntu# service mysql start
root@server:/home/ubuntu# service mysql status
● mysql.service - MySQL Community Server
     Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2023-07-23 16:20:17 UTC; 3s ago
    Process: 186994 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
   Main PID: 187002 (mysqld)
     Status: "Server is operational"
      Tasks: 38 (limit: 1111)
     Memory: 409.5M
        CPU: 1.093s
     CGroup: /system.slice/mysql.service
             └─187002 /usr/sbin/mysqld

docker로 띄우던 DB를 직접 설치하고 iptables를 삭제한다.

root@server:/home/ubuntu# iptables -D OUTPUT -p tcp -d 15.165.19.77 -j DROP

ubuntu@client:~$ sudo netstat -napo | grep 3306 | grep -i est
tcp        0      0 10.0.0.232:45286        3.38.136.55:3306        ESTABLISHED 107260/python3       keepalive (19.46/0/0)

성공적으로 연결된 모습을 확인할 수 있다. 여기서 traffic을 drop시키고 db를 종료시킨다.

root@ubuntu:/home/ubuntu# iptables -A OUTPUT -d 15.165.19.77 -j DROP
root@ubuntu:/home/ubuntu# service mysql stop
root@ubuntu:/home/ubuntu#

ubuntu@client:~$ curl 3.38.136.55

client에서 db 서버로 요청을 전송하였으나 응답 대기 상태가 지속되므로 트래픽이 정상적으로 drop됨을 확인할 수 있다. 이후 nestat을 통해 조회하면 연결이 est상태로 지속되는 것을 확인할 수 있다.

ubuntu@ip-10-0-0-232:~$ curl 3.38.136.55
^C
ubuntu@ip-10-0-0-232:~$ sudo netstat -napo | grep 3306 | grep -i est
tcp        0     45 10.0.0.232:45286        3.38.136.55:3306        ESTABLISHED 107260/python3       on (78.84/9/0)

db가 종료될 때 client로 자신의 죽음을 알렸으나 이번에는 트래픽이 정상적으로 drop되어 client측에서 db의 죽음을 인지하지 못한 것을 확인할 수 있다.

Test 2

---

root@server:/home/ubuntu# docker run -d -p 8080:8080 bitnami/nginx
Unable to find image 'bitnami/nginx:latest' locally
latest: Pulling from bitnami/nginx
323218fa29f7: Pull complete
Digest: sha256:2d1f6e1612377bbff8f0aa08b1e577da899c16446df5fb47f015a2cc6a54225f
Status: Downloaded newer image for bitnami/nginx:latest
70c432fb4ada324d5369b56c05a47b8a83f589f878e9b4e66bc99f8db221584a

ubuntu@client:~$ curl 3.38.136.55:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

client에서 server로 정상적으로 요청을 수행할 수 있음을 확인하였다. 이후 iptables -F를 통해 모든 규칙을 삭제하고 위 작업을 반복한다.

root@server:/home/ubuntu# iptables -F

ubuntu@cleint:~$ curl 3.38.136.55:8080

작업이 완료되지 않고 중지된다.

root@ip-172-31-59-210:/home/ubuntu# iptables -L -v
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy DROP 58 packets, 3944 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER (0 references)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER-ISOLATION-STAGE-1 (0 references)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER-ISOLATION-STAGE-2 (0 references)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER-USER (0 references)
 pkts bytes target     prot opt in     out     source               destination
root@ip-172-31-59-210:/home/ubuntu#

iptable docker 규칙 조차 모두 사라져 더 이상 정상적으로 활용할 수 없음을 확인하였다.(실험 이후 삭제 -> 재설치)

Conclusion

---

docker와 함께 iptables를 사용할 때에는 각별한 주의가 필요하며, 되도록 다른 유틸을 사용하여 제어하길 권장한다.