Spark SQL error report summary

Wrong one: After starting spark shell, query the table information in hive and report an error $SPARK_HOME/bin/spark-shell spark.sql("select * from student.student ").show() Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStor ...

Posted on Wed, 06 Nov 2019 08:12:21 -0800 by Jagarm

I. Flume -- basic principle and use of data collector

I. overview 1. What is flume 1) Flume provides a distributed and reliable service for efficient collection, aggregation and movement of logs with large amount of data. Flume can only run in Linux environment.2) Flume is based on flow architecture, with strong fault tolerance, flexibility and simplicity.3) Flume and Kafka are used for real-time ...

Posted on Tue, 05 Nov 2019 10:15:32 -0800 by menriquez

Analysis of RT thread device driving UART

OS version: RT thread 4.0.0 Chip: STM32F407 RT thread's serial driver framework is familiar with Linux, which is divided into I/O device framework + device bottom driver; 1. Initialization and use of serial device Register the enabled UART obj [] with the device rtthread_startup --> rt_hw_usart_init() --> rt_hw_serial_register --> ...

Posted on Tue, 05 Nov 2019 07:20:53 -0800 by rcatal02

Wechat browser open web page blocked? 366API jump to quickly solve the problem of wechat blocked website resume normal access

When we do marketing activities or promotion, it's easy to encounter links blocked by wechat, leading to a prompt "stop visiting this page" in wechat, so we can't download apps and other situations. At this time, we need to download the app function directly from wechat. Many users don't know about this implementation. Let's explain ...

Posted on Sun, 03 Nov 2019 15:09:17 -0800 by smilley654

III. zookeeper -- implement HA of NN and RM

I. hdfs namenode HA 1. Overview In Hadoop 1.0, there is a single point of failure of the namenode in the hdfs cluster. When the namenode is unavailable, the whole hdfs cluster service will be unavailable. In addition, if you need to temporarily design or operate the namenode, after you stop the namenode, the hdfs cluster cannot be used.Through ...

Posted on Sat, 02 Nov 2019 16:24:43 -0700 by newbie8899

Big data case - MapReduce traffic statistics case - partition

Code download address: https://github.com/tazhigang/big-data-github.git I. demand: output statistical results to different files (zones) according to different provinces where mobile phones belong II. Data preparation Data preparation: phoneData.txt in case 2 According to the first three digits of the phone number III. create maven project P ...

Posted on Fri, 01 Nov 2019 16:50:43 -0700 by biz0r

Deep analysis of JVM source code - creating HotSpot

1 program main entrance src/java.base/share/native/launcher/main.c The main function returns the JLI? Launch() function, located at src/java.base/share/native/libjli/java.c 2 java.c # JLI_Launch() JavaMain() is a native call to a Java main program. In this method, the initialization of the virtual machine will be performed, the main class and ...

Posted on Wed, 30 Oct 2019 13:12:49 -0700 by jonnypixel

12. Serialization of hadoop

I. Basic overview of serialization 1. What is serialization Serialization is the conversion of objects in memory into byte sequences (or according to other data transfer protocols), so as to facilitate persistent storage to disk and network transmission. 2. Why serialization is needed Generally, objects are only stored in local memory, and onl ...

Posted on Mon, 28 Oct 2019 22:13:00 -0700 by magic-eyes

Use of CountDownLatch

Article directory 1. background 2. Usage 3. Business core code 4. Program life 1. background Recently, in the development of big data, there are questions about the competition between threads and the order of execution in JAVA. Therefore, learn the advanced writing method of concurrent programm ...

Posted on Sun, 27 Oct 2019 21:31:48 -0700 by Jakehh

Database query performance LinqDB vs Sql query

Using LinqDB to query Sqlite database data, whether it is big data or a small amount of data, feels particularly time-consuming, especially the first query A data table with 27000 pieces of data First query: Query 27000 pieces of data, taking 1s It also takes 750ms to query 8 pieces of data Secondary query: It takes 475ms to query 2700 ...

Posted on Fri, 25 Oct 2019 05:46:05 -0700 by Digitry Designs