[FLINK] Yarn 기반으로 실행시 오류 - ClassNotFoundException: org.glassfish.jersey.internal.RuntimeDelegateImpl

증상

다음과 같이 Flink 에서 하둡클러스터의 yarn 기반에서 application 잘 invoke 되는지 테스트를 하려고 했는데, 다음과 같은 오류가 나는 경우가 발생했다. 참고로 오류메시지는 다르지만 기존에 "com/sun/jersey/core/util/FeaturesAndProperties" 클래스를 찾을수 없다며 오류를 냈던 케이스와 동일하게 해결이 가능했다.

2022.03.18 - [데이터처리/Flink] - [Flink] yarn-session.sh 오류 : java.lang.NoClassDefFoundError

$ ./bin/flink run-application -t yarn-application ./examples/streaming/TopSpeedWindowing.jar
...생략...
2022-09-01 10:52:03,034 INFO  org.apache.hadoop.yarn.client.api.impl.TimelineReaderClientImpl [] - Initialized TimelineReader URI=http://hadoop-master.host.com:8198/ws/v2/timeline/, clusterId=yarn-cluster
java.lang.ExceptionInInitializerError
        at com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:182)
        at com.sun.jersey.core.spi.factory.MessageBodyFactory.initReaders(MessageBodyFactory.java:175)
        at com.sun.jersey.core.spi.factory.MessageBodyFactory.init(MessageBodyFactory.java:162)
        at com.sun.jersey.api.client.Client.init(Client.java:343)
        at com.sun.jersey.api.client.Client.access$000(Client.java:119)
        at com.sun.jersey.api.client.Client$1.f(Client.java:192)
        at com.sun.jersey.api.client.Client$1.f(Client.java:188)
        at com.sun.jersey.spi.inject.Errors.processWithErrors(Errors.java:193)
        at com.sun.jersey.api.client.Client.<init>(Client.java:188)
        at com.sun.jersey.api.client.Client.<init>(Client.java:171)
        at org.apache.hadoop.yarn.client.api.impl.TimelineConnector.serviceInit(TimelineConnector.java:124)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
        at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:109)
        at org.apache.hadoop.yarn.client.api.impl.TimelineReaderClientImpl.serviceInit(TimelineReaderClientImpl.java:102)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
        at org.apache.hadoop.yarn.client.api.impl.AHSv2ClientImpl.serviceInit(AHSv2ClientImpl.java:63)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
        at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:221)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
        at org.apache.flink.yarn.YarnClusterClientFactory.getClusterDescriptor(YarnClusterClientFactory.java:82)
        at org.apache.flink.yarn.YarnClusterClientFactory.createClusterDescriptor(YarnClusterClientFactory.java:61)
        at org.apache.flink.yarn.YarnClusterClientFactory.createClusterDescriptor(YarnClusterClientFactory.java:43)
        at org.apache.flink.client.deployment.application.cli.ApplicationClusterDeployer.run(ApplicationClusterDeployer.java:63)
        at org.apache.flink.client.cli.CliFrontend.runApplication(CliFrontend.java:214)
        at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1081)
        at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1156)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
        at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
        at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1156)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.glassfish.jersey.internal.RuntimeDelegateImpl
        at javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:130)
        at javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:97)
        at javax.ws.rs.core.MediaType.valueOf(MediaType.java:172)
        at com.sun.jersey.core.header.MediaTypes.<clinit>(MediaTypes.java:65)
        ... 31 more
Caused by: java.lang.ClassNotFoundException: org.glassfish.jersey.internal.RuntimeDelegateImpl
        at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:264)
        at javax.ws.rs.ext.FactoryFinder.newInstance(FactoryFinder.java:87)
        at javax.ws.rs.ext.FactoryFinder.find(FactoryFinder.java:185)
        at javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:112)

해결방법

기존에 해결했던것 처럼 jersey 관련 라이브러리를 flink 의 lib 폴더에 복사하면 해당 문제가 해결된다.

# FLINK_HOME 은 플링크가 설치된 경로를 의미함
cd $FLINK_HOME/lib
wget https://repo1.maven.org/maven2/com/sun/jersey/jersey-client/1.19.4/jersey-client-1.19.4.jar
wget https://repo1.maven.org/maven2/com/sun/jersey/jersey-server/1.19.4/jersey-server-1.19.4.jar
wget https://repo1.maven.org/maven2/com/sun/jersey/jersey-core/1.19.4/jersey-core-1.19.4.jar

위와 같이 라이브러리를 다운로드 받고 다시 실행을 하게되면 다음과 같이 실행에 성공한다.

참고로 flink 대시보드는 로깅된 메시지의 Found Web Interface  다음에 있는 도메인 포트에 접속하면 확인이 가능하다.

$ ./bin/flink run-application -t yarn-application ./examples/streaming/TopSpeedWindowing.jar
...생략...
2022-09-02 17:03:01,401 INFO  org.apache.hadoop.yarn.client.api.impl.TimelineReaderClientImpl [] - Initialized TimelineReader URI=http://hadoop004.com:8198/ws/v2/timeline/, clusterId=yarn-cluster
2022-09-02 17:03:01,453 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Submitting application master application_1661871799627_0050
2022-09-02 17:03:01,458 INFO  org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl    [] - Timeline service address: hadoop003.host.com:8188
2022-09-02 17:03:01,757 INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl        [] - Submitted application application_1661871799627_0050
2022-09-02 17:03:01,757 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Waiting for the cluster to be allocated
2022-09-02 17:03:01,758 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Deploying cluster, current state ACCEPTED
2022-09-02 17:03:08,784 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - YARN application has been deployed successfully.
2022-09-02 17:03:08,785 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Found Web Interface hadoop005.host.com:36346 of application 'application_1661871799628_0051'.