Mainstream Process Analysis of Spark sql Streaming Processing

Spark sql supports streaming processing, which includes Source and Sink. Source defines the source of the stream, Sink defines the destination of the stream, and the execution of the stream is triggered from Sink. The writeStream of Dataset defines the destination of the stream and triggers the real execution of the stream, so the analysis beg ...

Posted on Sun, 13 Oct 2019 00:55:35 -0700 by BlueKai

Windows operation and groupBy operation

Windows operation import spark.implicits._ val words = ... // streaming DataFrame of schema { timestamp: Timestamp, word: String } // Group the data by window and word and compute the count of each group val windowedCounts = words.groupBy( window($"timestamp", "10 minutes", "5 minutes"), $"word" ).count()   Windows operation ...

Posted on Wed, 09 Oct 2019 13:27:26 -0700 by SulleyMonstersInc

AES Encryption and Decryption Example (walker)

AES introduction Advanced Encryption Standard (AES), also known as Rijndael Encryption, is a block encryption standard adopted by the United States federal government.This standard is used to replace the original DES (Data Encryption Standard), which has been analyzed by many parties and widely used all over the world. After a five-year selecti ...

Posted on Tue, 01 Oct 2019 10:20:53 -0700 by d-woo

Machine Learning for Real-time Computing SparkSpark of Big Data Series-Recommendation Implementation

Recommendation is a very popular technology. Provide the closest recommendation based on previous purchases, clicks and ratings. Amazon uses this technology to show you a list of items you might be interested in, drawing your information from past behavior. The recommendation engine behind captures u ...

Posted on Tue, 01 Oct 2019 04:00:34 -0700 by eva21

Summary of Scala Collections

Summary of Scala Collections Recently I had time to look at Scala, summarize the Scala collection and share it with you. 1. Basic concepts Collection classes in scala can be divided into three categories: 1.Seq is an ordered set of elements, such as List, Array 2.Set is a set of elements without d ...

Posted on Thu, 26 Sep 2019 05:31:24 -0700 by cainmi

Spark Series-Structured API of Spark SQL

1. Creating DataFrame and Dataset 1.1 Create DataFrame The entry point for all functions in Spark is SparkSession, which can be created using SparkSession.builder(). Once created, the application can create a DataFrame from an existing RDD, Hive table, or Spark data source. Examples are as follows: val spark = SparkSession.builder().appName("S ...

Posted on Mon, 23 Sep 2019 08:14:31 -0700 by Vibralux

Deep Understanding of Spark 2.1 Core: TimSort Principle and Source Code Analysis

In the blog Deep Understanding of Spark 2.1 Core (X): Principles and Source Code Analysis of Shuffle Map End We mentioned that: Sort and others are used to sort the data, and TimSort is used. In this blog post, let's take a deeper look at Tim S ...

Posted on Fri, 06 Sep 2019 23:04:59 -0700 by HokieTracks

Spark from zero to Spark API In Java8

                          Spark API In  Java8 1. map, flatMap map is easy to understand. It passes an element of the source JavaRDD into the call method and returns one by one after the algorithm to generate a new JavaRDD. map Sample Code L ...

Posted on Mon, 02 Sep 2019 20:01:48 -0700 by dubhcat

10 Hours Start Big Data: Chapter 6 - Hadoop Project Practice

Overview of User Behavior Log User Behavior Log: All behavioral data (access, browse, search, click, etc.) of the user each time he visits the website User Behavior Trajectory, Traffic Log Why Log User Access Behavior Visits to Web pages Viscosi ...

Posted on Sun, 25 Aug 2019 22:34:59 -0700 by verdrm

Patterns Matching and Sample Classes of spark Notes

Level has a very powerful pattern matching mechanism, which can be applied to many occasions, such as switch statements, type checking and so on. Level also provides sample classes to optimize pattern matching, which can quickly match.1.1. Matching string package cn.itcast.cases import scala.util.Random   object CaseDemo01 extends App{   v ...

Posted on Thu, 22 Aug 2019 00:15:15 -0700 by atsphpflash