[오류] hive llap 지원 버전 오류 - java.lang.OutOfMemoryError: Java heap space - 컨테이너모드

개요

초창기 hdp 에서 hive 의 성능이 좋아진다면서 llap 를 지원하는 hive를 사용한적이 있다. 이때 llap 모드에서 쿼리가 실패했고, 컨테이너 모드로 바꿔서 실행했는도 다음과 같은 오류가 나서 골치아팠던적이 있다. 컨테이너 사이즈와 힙을 높게 할당했지만 그래도 OutOfMemoryError 가 발생했는데, 결국 삽질끝에 회피 방법을 찾긴 했다.

 

- 오류 메시지 (컨테이너 모드 상태)

Error: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1512714504227_16303_482_00, diagnostics=[Task failed, taskId=task_1512714504227_16303_482_00_000165, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.allocateSpace(PipelinedSorter.java:239)
at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter$SortSpan.end(PipelinedSorter.java:1013)
at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter$SortSpan.next(PipelinedSorter.java:968)
at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.sort(PipelinedSorter.java:307)
at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.collect(PipelinedSorter.java:411)
at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.write(PipelinedSorter.java:367)
at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput$1.write(OrderedPartitionedKVOutput.java:163)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor$TezKVOutputCollector.collect(TezProcessor.java:260)
at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.collect(ReduceSinkOperator.java:562)
at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:407)
at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:955)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:903)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:894)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1047)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.flushHashTable(GroupByOperator.java:1014)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processHashAggr(GroupByOperator.java:822)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:719)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:787)
at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:955)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:903)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:894)
at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:955)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:903)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:136)
at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:149)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:489)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:92)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:76)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:419)

- llap 모드 상태에서 오류 메시지

[Error: Error while running task ( failure ) : org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError: Error while doing final merge 
	at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:320)
	at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:285)
	at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.ClosedByInterruptException
	at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
	at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:293)
	at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.seek(RawLocalFileSystem.java:120)
	at org.apache.hadoop.fs.BufferedFSInputStream.seek(BufferedFSInputStream.java:96)
	at org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:65)
	at org.apache.tez.runtime.library.common.sort.impl.TezMerger$DiskSegment.init(TezMerger.java:380)
	at org.apache.tez.runtime.library.common.sort.impl.TezMerger$MergeQueue.merge(TezMerger.java:731)
	at org.apache.tez.runtime.library.common.sort.impl.TezMerger.merge(TezMerger.java:166)
	at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager.finalMerge(MergeManager.java:1285)
	at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager.close(MergeManager.java:653)
	at org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.callInternal(Shuffle.java:316)
	... 6 more

회피방법

쿼리를 돌리기전에 0.99로 세팅했던값을 default 값인 0.5로 변경하니 동작했다.

set hive.map.aggr.hash.min.reduction=0.5;