Analysis of spark checkpoint principle and source code

I. overview What is Checkpoint? Spark often faces a lot of RDDS of Tranformation in the production environment (for example, a Job contains 10000 RDDS), or the calculation of RDDS generated by specific Tranformation is particularly complex and time-consuming (for example, the calculation often exce ...

Posted on Sat, 15 Feb 2020 21:22:45 -0800 by jf3000

scala programming idea -- class method

Coverage method So far, none of the classes we inherit actually perform anything that distinguishes them package com.test1 object Scala09_test13 { def main(args: Array[String]): Unit = { //Authentic scala //Define operator //Method names can contain almost all characters. For exampl ...

Posted on Sun, 02 Feb 2020 08:52:06 -0800 by akreation

scala programming ideas -- classes and objects

Object is the foundation of many modern programming languages, including scala. In object-oriented programming language package com.test1 object Scala09_test3 { def main(args: Array[String]): Unit = { val r1 = Range(0,10) //For range, the number of available operations is staggering. The ...

Posted on Sat, 01 Feb 2020 07:48:39 -0800 by digitalflash

The "object" of Scala learning

object It is equivalent to a single instance of class. Usually, some static field s or methods are placed in it. When the method of object is called for the first time, the constructor of object will be executed, that is, the code inside the object that is not in the method. Note that object cannot define the constructor that accepts parameter ...

Posted on Sat, 01 Feb 2020 00:09:09 -0800 by Daen

RDD programming learning note 3 data reading and writing

Local read scala> var textFile = sc.textFile("file:///root/1.txt") textFile: org.apache.spark.rdd.RDD[String] = file:///root/1.txt MapPartitionsRDD[57] at textFile at <console>:24 scala> textFile.saveAsTextFile("file:///root/writeback") scala> textFile.foreach(println) hadoop hello bi ...

Posted on Wed, 29 Jan 2020 05:34:55 -0800 by dizel247

"Class - Basic Concept 3" Learned by Scala

Type judgment using pattern matching In practical development, such as spark's source code, a lot of places use pattern matching to make type judgment, which is more concise and clear, and the code is very maintainable and scalable With pattern matching, functionally, just like isInstanceOf, it is sufficient to judge objects that are predomina ...

Posted on Mon, 27 Jan 2020 19:59:45 -0800 by psyion

scala: Operation hdfs,mapreduce

scala is a scripting language built on Java, and the bottom layer of hadoop is also java code, so it is most appropriate to write mapreduce in Java, and what is scala? Because scala is recommended not to add type declaration, such as val a = "hello", but in hadoop's mapreduce interface, there are generic requirements, ...

Posted on Fri, 13 Dec 2019 11:04:03 -0800 by bocasz

7. Flnk--Asynchronous IO

1. Overview of flink asynchronous IO 1.1 Asynchronous IO Requirements Async I/O was introduced in version 1.2 as a highly vocal feature of Alibaba's contribution to the community.The main purpose is to solve the problem that communication delays (such as waiting for a response from an external system) become bottlenecks when data streams inter ...

Posted on Wed, 11 Dec 2019 01:48:10 -0800 by coowboy

Matrix multiplication with Spark (JAVA version)

In recent projects, because the multiplication operation between large matrices is used, if Java is used to write, the amount of data will consume a lot of resources. Therefore, it is considered to use spark for calculation and processing. The online search data just finds that spark machine learning library spark mllib contains some matrix ope ...

Posted on Mon, 09 Dec 2019 04:27:12 -0800 by gazalec

Schema matching in Scala

1. Typical pattern matching scenario (1) matching string object Test01 { def main(args: Array[String]): Unit = { val arr=Array("aa","bb","cc") //Get any element of the array at random val index=Random.nextInt(3) val value=arr(index) //pattern matching value match{ case "aa" => println("0") case "bb" =&g ...

Posted on Thu, 05 Dec 2019 04:51:29 -0800 by 990805