cdh6.3.2 configure Hive on Spark

Environment: Dell xps15 (32G memory, 1T solid state, Samsung 1T mobile solid state with external lightning 3 interface, 4T external mechanical hard disk of WD Elements) win10 three Centos7 virtual machines are used to test cdh6.3.2 cluster (the highest version of free community version), self compiled Phoenix 5.1.0, flink1.10.0, elasticsearch6.6.0 and other source codes.

First, the cluster cannot be started after the trial enterprise version function expires Reports Manager Roles (see Figure 1) and Alert Publisher Part of the character's functions are castrated (see Figure 2).

1.1 add services to build Hive on Spark. Because the cdh cluster role of virtual machine on notebook is easy to hang, HDFS is configured as HA high availability mode.

1.2 search spark (see the figure below) in hive service "configuration" and check two configuration items, and the parameter item with small blue back button in the figure is the final configuration after I successfully debug (Note: if the default value of each memory is not modified, all kinds of errors will be reported in the first run of hive)

1.3. When spark is checked in the figure above, you will be prompted to configure Spark On Yarn. Go to the "configuration" of spark service to confirm

1.4. Modify the default resource value on page 10 of "configuration" of Yarn service to avoid the following errors

Among them, yarn.nodemanager.resource.memory-mb and yarn.scheduler.maximum-allocation-mb are required items (from the default value of 1 to 2). In addition, yarn.nodemanager.resource.cpu-vcores and yarn.scheduler.maximum-allocation-vcores are modified from the default value of 2 to 4 to solve the following errors in the yarn list

1.5. Start running hive:

1.5.1. If you directly run hive to enter its client, the disadvantage is that you cannot initialize spark client at the beginning, not to mention that you cannot see the detailed log when you submit it to yarn (see the figure below)

In the original commissioning stage, the Hive command line Set in, but never print logs (see Figure), permissions and error reporting problems

INFO [spark-submit-stderr-redir-6f70c261-ed5e-4af4-a7ea-7b11084e7547 main] client.SparkClientImpl: Exception in                                       thread "main" org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="/user":hdfs:supergr                                      oup:drwxr-xr-x
------------ellipsis------------------------
ERROR [6f70c261-ed5e-4af4-a7ea-7b11084e7547 main] client.SparkClientImpl: Error while waiting for Remote Spark D                                      river to connect back to HiveServer2.
java.util.concurrent.ExecutionException: java.lang.RuntimeException: spark-submit process failed with exit code 1 and error ?
---------------------
ERROR [6f70c261-ed5e-4af4-a7ea-7b11084e7547 main] spark.SparkTask: Failed to execute Spark task "Stage-1"
org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create Spark client for Spark session 6f70c261-ed5e-4af4-a7ea-7b11084e7547_0                                      : java.lang.RuntimeException: spark-submit process failed with exit code 1 and error ?
--------
Caused by: java.lang.RuntimeException: Error while waiting for Remote Spark Driver to connect back to HiveServer2.

So the command to run hive is changed to

In the figure, fruitless means that there is only one line of hive-site.xml log in hive.log file, without any other logs. But the console prints out a lot of useful logs

About this "Timeout" error See the link for details of, and try to change the default value from 60s to 200s to solve the problem of over reporting errors

set hive.spark.client.future.timeout=200;
//Error reporting can be solved:
ERROR client.SparkClientImpl: Timed out waiting for Remote Spark Driver to connect to HiveServer2.
Possible reasons include network issues, errors in remote driver, cluster has no available resources, etc.
Please check YARN or Spark driver's logs for further information.
java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Client 'b1fb6956-93d2-438a-98c2-a9db64024ac4_1' timed out waiting for connection from the Remoark Driver
        at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:41)
        at org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:103)
        at org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:90)
        at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.createRemoteClient(RemoteHiveSparkClient.java:104)
        at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:100)
        at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:77)
        at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:131)
        at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:132)
        at org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:131)
        at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:122)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2200)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1843)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1563)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1339)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1328)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:409)
        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:836)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:772)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:699)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:313)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:227)
Caused by: java.util.concurrent.TimeoutException: Client 'b1fb6956-93d2-438a-98c2-a9db64024ac4_1' timed out waiting for connection from the Remote Spark Driver
        at org.apache.hive.spark.client.rpc.RpcServer$2.run(RpcServer.java:178)
        at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
        at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
        at java.lang.Thread.run(Thread.java:748)
20/05/02 23:37:26 ERROR spark.SparkTask: Failed to execute Spark task "Stage-1"
org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create Spark client for Spark session b1fb6956-93d2-438a-98c2-a9db64024ac4_1: java.util.concurrent.TimeoutExceptioient 'b1fb6956-93d2-438a-98c2-a9db64024ac4_1' timed out waiting for connection from the Remote Spark Driver
        at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.getHiveException(SparkSessionImpl.java:286)
        at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:135)
        at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:132)
        at org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:131)
        at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:122)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2200)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1843)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1563)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1339)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1328)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:409)
        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:836)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:772)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:699)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:313)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:227)
Caused by: java.lang.RuntimeException: Timed out waiting for Remote Spark Driver to connect to HiveServer2.
Possible reasons include network issues, errors in remote driver, cluster has no available resources, etc.
Please check YARN or Spark driver's logs for further information.
        at org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:124)
        at org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:90)
        at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.createRemoteClient(RemoteHiveSparkClient.java:104)
        at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:100)
        at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:77)
        at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:131)
        ... 22 more
Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Client 'b1fb6956-93d2-438a-98c2-a9db64024ac4_1' timed out waiting for connection fre Remote Spark Driver
        at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:41)
        at org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:103)
        ... 27 more
Caused by: java.util.concurrent.TimeoutException: Client 'b1fb6956-93d2-438a-98c2-a9db64024ac4_1' timed out waiting for connection from the Remote Spark Driver
        at org.apache.hive.spark.client.rpc.RpcServer$2.run(RpcServer.java:178)
        at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
        at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
        at java.lang.Thread.run(Thread.java:748)
FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark session b1fb6956-93d2-438a-98c2-a9db644_1: java.util.concurrent.TimeoutException: Client 'b1fb6956-93d2-438a-98c2-a9db64024ac4_1' timed out waiting for connection from the Remote Spark Driver
20/05/02 23:37:26 ERROR ql.Driver: FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark sesb1fb6956-93d2-438a-98c2-a9db64024ac4_1: java.util.concurrent.TimeoutException: Client 'b1fb6956-93d2-438a-98c2-a9db64024ac4_1' timed out waiting for connection from the Remotrk Driver

However, this parameter cannot be set in the same command line as sql (see the figure below)

The final solution is

Note: 1. Only one of the three parameters mentioned in the "timeout" link can be set dynamically in the hql statement to be verified, and the verification results can be seen from the hive console (as shown below)

Spark On Yarn doesn't really end after the hive client runs the correct result

In the above hive, order by has given results, but in the following figure, yarn shows that it is running

count as above

Reference documents:

hive on spark (spark on yarn) of CDH

[hive on spark Error]return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask.

To be practiced:

Troubleshooting for the invalidity of hive replacing the third-party jar package

Special thanks: https://itdiandi.net/view/1431 However, the configuration item in this article is added in the Hive component configuration file of cdh6.3.2 and an error is reported when running the hql statement.

Tags: Java hive Spark Apache

Posted on Tue, 05 May 2020 05:12:05 -0700 by mwichmann4