[오류] hive 쿼리 에러 해결방법 : SemanticException Exception when trying to remove partition predicates: fail to find child from parent

하이브에서 쿼리를 날릴때 이런 쿼리가 발생했고 구글링 하면 나오는 사이트중 그나마 힌트에 근접한 결과는 아래링크에서 얻을수 있었다.

https://docs.treasuredata.com/display/public/PD/Hive+Known+Limitations 

 

에러의 원인은 뭘까?

원인이 되는 쿼리를 패턴을 찾아보면 "LATERAL VIEW" 와 "UNION ALL" 를 복잡하게 섞었을때 종종 나타나는것 같다.

가끔 beeline 에서 오류나는게 hive 커멘드에서는 재현이 안되는 경우가 있으니 동일한 환경에서 쿼리를 테스트해야 재현되는경우가 있으니 참고하자.

 

상세한 오류(에러)메시지는 아래와 같다. 

참고로 해결방법은 버그가 아닐까 싶을 정도로 단순하며 찜찜하다 ㅋㅋ

etting log thread is interrupted, since query is done!
Error: Error while compiling statement: FAILED: SemanticException Exception when trying to remove partition predicates: fail to find child from parent (state=42000,code=40000)
org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException Exception when trying to remove partition predicates: fail to find child from parent
        at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:262)
        at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:248)
        at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:300)
        at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:241)
        at org.apache.hive.beeline.Commands.execute(Commands.java:871)
        at org.apache.hive.beeline.Commands.sql(Commands.java:729)
        at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1000)
        at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:835)
        at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:793)
        at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:493)
        at org.apache.hive.beeline.BeeLine.main(BeeLine.java:476)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException Exception when trying to remove partition predicates: fail to find child from parent
        at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:380)
        at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:206)
        at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:290)
        at org.apache.hive.service.cli.operation.Operation.run(Operation.java:320)
        at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:530)
        at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:517)
        at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:310)
        at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:530)
        at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)
        at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: Exception when trying to remove partition predicates: fail to find child from parent
        at org.apache.hadoop.hive.ql.exec.Operator.removeChildAndAdoptItsChildren(Operator.java:818)
        at org.apache.hadoop.hive.ql.parse.GenTezUtils.removeUnionOperators(GenTezUtils.java:342)
        at org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:441)
        at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:279)
        at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11273)
        at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:286)
        at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
        at org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:165)
        at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:512)
        at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
        at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1295)
        at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:204)
        ... 15 more

해결방법

UNION ALL 사이에 여러 쿼리 덩어리가 있을텐데 이 순서를 바꾸면 성공한다.

실제로 나같은 경우는 A와 D 쿼리는 LATERAL VIEW 형태로 풀어내는 쿼리였고 나머지는 일반 쿼리였는데 순서를 바꾸니까 성공했는데 재밌는 사실은 LATERAL VIEW 를 연속되게 묶어냈을때 성공한걸 보면, 뭔가 알수없는 영향을 주는건 아닐까? 추측해본다.

 

• 실패 : A + B + C + D
• 실패 : A + B + D + C 
• 성공 : A + D + B  + C
• 성공 : D + A + B + C