[rabbitmq] Failed to start RabbitMQ broker - 노드가 모두 죽고 기동안됨

airflow 에서 셀러리기반으로 운영할때 DAG 요청을 담는 메시지큐로 rabbitmq 를 주로 사용한다. 그런데 테스트를 위해 모든 노드를 죽였는데 rabbitmq 가 기동을 못하는 상황이 발생되었다. 로그를 확인해보면 다음과 같은 메시지가 error 메시지로 존재했었다.

Feature flag 'quorum_queue': migration function crashed

Jul 12 18:24:03 my-server001 rabbitmq-server[48802]: * suggestion: start the node
Jul 12 18:24:03 my-server001 rabbitmq-server[48802]: rabbit@my-server002:
Jul 12 18:24:03 my-server001 rabbitmq-server[48802]: * connected to epmd (port 4369) on my-server002
Jul 12 18:24:03 my-server001 rabbitmq-server[48802]: * node rabbit@my-server002 up, 'rabbit' application running
Jul 12 18:24:03 my-server001 rabbitmq-server[48802]: Current node details:
Jul 12 18:24:03 my-server001 rabbitmq-server[48802]: * node name: 'rabbit@my-server001'
Jul 12 18:24:03 my-server001 rabbitmq-server[48802]: * effective user's home directory: /var/lib/rabbitmq
Jul 12 18:24:03 my-server001 rabbitmq-server[48802]: * Erlang cookie hash: 0c70pUl+srYqB7wsPB8+/Q==
Jul 12 18:24:06 my-server001 rabbitmq-server[48802]: {"Kernel pid terminated",application_controller,"{application_start_failure,rabbit,{{timeout_waitin
Jul 12 18:24:06 my-server001 rabbitmq-server[48802]: Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{{timeout_waiting
Jul 12 18:24:06 my-server001 rabbitmq-server[48802]: Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done
Jul 12 18:24:06 my-server001 systemd[1]: rabbitmq-server.service: main process exited, code=exited, status=1/FAILURE
Jul 12 18:24:06 my-server001 systemd[1]: Failed to start RabbitMQ broker.
-- Subject: Unit rabbitmq-server.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit rabbitmq-server.service has failed.
--
-- The result is failed.
Jul 12 18:24:06 my-server001 systemd[1]: Unit rabbitmq-server.service entered failed state.
Jul 12 18:24:06 my-server001 systemd[1]: rabbitmq-server.service failed.
Jul 12 18:24:16 my-server001 systemd[1]: Cannot add dependency job for unit dm-event.socket, ignoring: Unit is masked.
Jul 12 18:24:16 my-server001 systemd[1]: Cannot add dependency job for unit nscd.socket, ignoring: Unit is masked.
Jul 12 18:24:16 my-server001 systemd[1]: Cannot add dependency job for unit rpcbind.socket, ignoring: Unit is masked.

해결방법 : force_boot 사용하기

rabbitmq 가 모두 종료될때, 마지막으로 죽었던 노드가 먼저 기동되어야 동작하는 룰이 있다고 한다.

하지만 이걸 무시하고 기동하는것도 가능했다. rabbitmqctl force_boot 를 이용해 기동에 성공했다.

$ sudo /usr/lib/rabbitmq/bin/rabbitmqctl stop_app
Stopping rabbit application on node rabbit@my-server001 ...
Error: unable to perform an operation on node 'rabbit@my-server001'. Please see diagnostics information and suggestions below.
...

$ sudo /usr/lib/rabbitmq/bin/rabbitmqctl force_boot

구글링을 해봤지만, 아래 링크의 글이 나의 케이스에서 잘 해결되었다.

https://pygirl.tistory.com/33

rabbitmq cluster guide