Patterns Matching and Sample Classes of spark Notes

Level has a very powerful pattern matching mechanism, which can be applied to many occasions, such as switch statements, type checking and so on. Level also provides sample classes to optimize pattern matching, which can quickly match.1.1. Matching string package cn.itcast.cases import scala.util.Random   object CaseDemo01 extends App{   v ...

Posted on Thu, 22 Aug 2019 00:15:15 -0700 by atsphpflash

Restapi-rest-mongo application instance: one of the distributed picture management systems, restservice

I agree with a colleague who recently mentioned that he would like to have pictures on the web in MongoDB.MongoDB has many advantages over storing pictures as files in a subdirectory on your hard disk.First, MongoDB is a distributed database where pictures can be stored across servers.In a clustered environment, the technology of replica set, s ...

Posted on Wed, 21 Aug 2019 18:49:20 -0700 by warrior rabbit

Spark Core Custom Sort, Partition

Custom Sorting (Important) Simple data types can be sorted directly in spark, but some complex conditions can be achieved by using custom sorting import org.apache.spark.rdd.RDD import org.apache.spark.{SparkConf, SparkContext} //Custom Sorting obj ...

Posted on Thu, 15 Aug 2019 00:19:09 -0700 by EddieFoyJr

Lambda and Stream Practical Example Collection

Function < T, R > - Take T as input and return R as output Predicate - Take T as input and return Boolean value as output Consumer - Take T as input, perform certain operations and return nothing. Supplier - No input, return T IntConsumer - Take two T as input and return one T as output, which is ...

Posted on Fri, 09 Aug 2019 05:29:03 -0700 by msielski

How to dynamically parse Schema of JSON data from Kafka by schema_of_json method in Structured Streaming

How to parse Schema of JSON data from Kafka in Structured Streaming In actual production, the fields in the message may change, such as adding one more field or something, but the Spark program can't stop. So consider that instead of customizing the Schema in the program, infer the Schema through the ...

Posted on Thu, 08 Aug 2019 23:52:18 -0700 by newbienewbie

Scala of Big Data Learning Notes: Chapter 11 Characteristics

Articles Catalogue 11.1 Multiple inheritance is not allowed 11.2 Characteristics of Used as Interfaces 11.3 has the characteristics of concrete realization 11.4 Objects with Traits 11.5 Overlapping Characteristics 11.6 Rewriting abstract methods in features 11.7 as a feature of rich interface usage S ...

Posted on Sun, 04 Aug 2019 23:16:11 -0700 by Nabster

Scala for Large Data Learning Notes: Chapter 7 Category

Articles Catalogue 7.1 Simple Classes and Nonparametric Methods 7.2 Getter and Setter methods 7.3 Object Private Fields 7.4 Bean attributes 7.5 Constructor 7.6 Nested Classes 7.1 Simple Classes and Nonparametric Methods Class is defined by class. class Counter { private var value = 0 // You mus ...

Posted on Sun, 04 Aug 2019 22:33:57 -0700 by phoolster

Detailed description of <objectFactory> elements in Mybatis configuration

In the previous article, we explained the configuration of <type Handlers> and the source code behind it in detail. Today, we will elaborate on the next element <object Factory>. Is this element set when you use mybatis? I've never set it up. Using mybatis as the default implementation we've written can already meet most scenario r ...

Posted on Sun, 30 Jun 2019 15:06:07 -0700 by thegame261

Storage System for spark--BlockManager Source Code Analysis

Based on a series of previous analyses, we have a general understanding of the process of spark jobs from creation, dispatch and distribution, to execution, and finally the results are returned to driver.But there are still a lot of problems in the process of analyzing the source code. The main one is the important basic modules involved in spa ...

Posted on Sun, 09 Jun 2019 10:02:32 -0700 by rdimaggio

spark Task Assignment--TaskSchedulerImpl Source Parsing

TaskSchedulerImpl In the previous article, when DAGScheduler divided the entire calculation chain of a job into multiple stages based on the shuffle dependency, it started submitting the last ResultStage, and because of the dependency between stages, it actually ended up submitting stages from top to bottom along the calculation chain.Each stag ...

Posted on Sun, 02 Jun 2019 12:43:11 -0700 by pleigh