Spark + sbt + IDEA + HelloWorld + MacOS

Build project steps First, install scala, sbt, spark, and know the corresponding version The sbt version can be viewed on the sbt command line using sbtVersion Spark shell can know the version of spark and corresponding scala on the machine. plugin in IDEA to install scala plug-ins pass Modifying the configuration file to change the sl ...

Posted on Tue, 22 Oct 2019 13:54:56 -0700 by Opticon

Can Kafka asynchronous messages also block? Record a Dubbo Frequent Overtime Investigation Process

On-line service A calls service B interface to complete a transaction. After a production change in the evening, system monitoring finds that service B interface frequently timeouts, and then even returns Thread pool is EXHAUSTED, which is a thread pool depletion error. Because service B relies on external interfaces, it was initially mistaken ...

Posted on Tue, 08 Oct 2019 16:08:59 -0700 by jestercobblepot

Summary of Scala Collections

Summary of Scala Collections Recently I had time to look at Scala, summarize the Scala collection and share it with you. 1. Basic concepts Collection classes in scala can be divided into three categories: 1.Seq is an ordered set of elements, such as List, Array 2.Set is a set of elements without d ...

Posted on Thu, 26 Sep 2019 05:31:24 -0700 by cainmi

Apache Flink Zero Foundation Introduction Flink Table API&SQL

What is Flink relational API? Although Flink already supports DataSet and DataStream API s, is there a better way to program without caring about specific API implementations? There is no need to understand the specific implementations of Java and Scala. Flink provides three layered APIs. Each API offers a different trade-off between conci ...

Posted on Wed, 11 Sep 2019 02:54:21 -0700 by navtheace

Introduction to Kafka Streams, the Simplest Stream Processing Engine

Before version 0.10.0.0, Kafka was positioned as a distributed, partitioned, log submission service with backup mechanism. Kafka did not provide data processing services before that. Our stream computing mainly depends on the stream processing frameworks such as Storm, Spark Streaming and Flink. Storm, Spark Streaming and Flink have their own ...

Posted on Thu, 05 Sep 2019 00:43:14 -0700 by nightkarnation

Patterns Matching and Sample Classes of spark Notes

Level has a very powerful pattern matching mechanism, which can be applied to many occasions, such as switch statements, type checking and so on. Level also provides sample classes to optimize pattern matching, which can quickly match.1.1. Matching string package cn.itcast.cases import scala.util.Random   object CaseDemo01 extends App{   v ...

Posted on Thu, 22 Aug 2019 00:15:15 -0700 by atsphpflash

Restapi-rest-mongo application instance: one of the distributed picture management systems, restservice

I agree with a colleague who recently mentioned that he would like to have pictures on the web in MongoDB.MongoDB has many advantages over storing pictures as files in a subdirectory on your hard disk.First, MongoDB is a distributed database where pictures can be stored across servers.In a clustered environment, the technology of replica set, s ...

Posted on Wed, 21 Aug 2019 18:49:20 -0700 by warrior rabbit

Spark Core Custom Sort, Partition

Custom Sorting (Important) Simple data types can be sorted directly in spark, but some complex conditions can be achieved by using custom sorting import org.apache.spark.rdd.RDD import org.apache.spark.{SparkConf, SparkContext} //Custom Sorting obj ...

Posted on Thu, 15 Aug 2019 00:19:09 -0700 by EddieFoyJr

Lambda and Stream Practical Example Collection

Function < T, R > - Take T as input and return R as output Predicate - Take T as input and return Boolean value as output Consumer - Take T as input, perform certain operations and return nothing. Supplier - No input, return T IntConsumer - Take two T as input and return one T as output, which is ...

Posted on Fri, 09 Aug 2019 05:29:03 -0700 by msielski

How to dynamically parse Schema of JSON data from Kafka by schema_of_json method in Structured Streaming

How to parse Schema of JSON data from Kafka in Structured Streaming In actual production, the fields in the message may change, such as adding one more field or something, but the Spark program can't stop. So consider that instead of customizing the Schema in the program, infer the Schema through the ...

Posted on Thu, 08 Aug 2019 23:52:18 -0700 by newbienewbie