python and golang communicate through grpc

gRPC

Advantages of grpc:

  1. gRPC defines the interface through protobuf, so it has more strict interface constraints. 2. At the same time, data can be serialized into binary code through protobuf, which will greatly reduce the amount of data to be transmitted, thus greatly improving performance.
  2. gRPC can easily support streaming communication

The following will implement gRPC between go and python through an example to simply transfer a string

  • python:3.5.9
  • golang:1.13.4
  • protobuf:3.11.3

proto definition rules

ProtoBuf3 data structure definition

message structure type name{

  Field modifier data type field name = field code value;

}
  1. Field modifier
  • The default value of singular. There are only 0 or 1 identity members
  • repeated can repeat 0 or more times
  1. data type

  1. Field code value

The field used by the communication parties to identify each other. The data field corresponding to each coding value through the structure of the same protocol is the same. The value range is 1 ~ 2 ^ 32

Give an example

message String{

    string query = 1;
    int32 page_number = 2; // Which page number do we want
    int32 result_per_page = 3; // Number of results to return per page
    enum Corpus {
        UNIVERSAL = 0;
        WEB = 1;
        IMAGES = 2;
        LOCAL = 3;
        NEWS = 4;
        PRODUCTS = 5;
        VIDEO = 6;
    }
    Corpus corpus = 4;
    repeated int 32 num = 7;

}

Defining services

  1. Simple rpc

The client uses Stub to send the request to the server and wait for the response to return, just like the normal function call, which is a blocking call

Service service type name{
  rpc service name (message type) returns (return value message type);
}
  1. Server side streaming RPC

The client sends a request to the server and gets a stream to read the returned message sequence. The client reads the returned stream until there are no messages in it.

Service service type name{
  rpc service name (message type) returns (stream return value message type);
}
  1. Client streaming RPC

The client writes a sequence of messages and sends it to the server, again using streams. Once the client finishes writing the message, it waits for the server to finish reading and return its response.

Service service type name{
  rpc service name (stream message type) returns (return value message type);
}
  1. Two way flow
Service service type name{
  rpc service name (stream message type) returns (stream return value message type);
}

Reference other proto

// my.proto
import "first.proto";

golang and python compile proto

python compiling proto

First install grpcio tools and grpcio

pip install grpcio -i http://mirrors.aliyun.com/pypi/simple/ --trusted-host mirrors.aliyun.com
pip install grpcio-tools -i http://mirrors.aliyun.com/pypi/simple/ --trusted-host mirrors.aliyun.com

Then python compiles proto

python -m grpc_tools.protoc --python_out=. --grpc_python_out=. -I. rpc.proto

Two files will be generated

  • RPC? Pb2.py is used to interact with protobuf data
  • RPC Pb2 grpc is used to interact with grpc

golang compiling proto

First of all, install grpc. Because of the environmental problems, many go libraries are hard to obtain, so I transferred them to the code cloud, which is very convenient to download

git clone https://gitee.com/daba0007/grpc-go.git $GOPATH/src/google.golang.org/grpc  
git clone https://gitee.com/daba0007/net.git $GOPATH/src/golang.org/x/net  
git clone https://gitee.com/daba0007/text.git $GOPATH/src/golang.org/x/text  
git clone https://gitee.com/daba0007/sys.git $GOPATH/src/golang.org/x/sys
go get -u github.com/golang/protobuf/{proto,protoc-gen-go} 

git clone https://gitee.com/daba0007/go-genproto.git $GOPATH/src/google.golang.org/genproto  

cd $GOPATH/src/  
go install google.golang.org/grpc 
cp $GOPATH/bin/protoc-gen-go /usr/local/bin

Then compile proto

protoc --go_out=plugins=grpc:. rpc.proto

This will generate a file of rpc.pb.go to interact with protobuf data

Demo

Define proto

Define the interface HelloService and data type String through protobuf

// rpc.proto
syntax = "proto3";

package go_protoc;

// Define data type String
message String {
    string value = 1;
}

// Define interface HelloService
service HelloService {
    rpc Hello (String) returns (String);
}

Go as the server, python as the client

  1. go Server
// The compiled proto file is located at $GOPATH/src/go_protoc
package main

import (
    "context"
    "go_protoc"
    "google.golang.org/grpc"
    "log"
    "net"
)


// Define an object to handle the received protobuf
type HelloServiceImpl struct{}

func (p *HelloServiceImpl) Hello(ctx context.Context, args *go_protoc.String,) (*go_protoc.String, error) {
    // Parsing String type data through compiled rpc.pb.go
    reply := &go_protoc.String{Value: "hello:" + args.GetValue()}
    return reply, nil
}


func main() {
    // Define a grpc
    grpcServer := grpc.NewServer()
    // Define a service RegisterHelloServiceServer through the compiled HelloService interface of rpc.pb.go
    go_protoc.RegisterHelloServiceServer(grpcServer, new(HelloServiceImpl))
    // Defining listening port 1234
    lis, err := net.Listen("tcp", "192.168.1.146:1234")
    if err != nil {
        log.Fatal(err)
    }
    // Turn on monitoring
    grpcServer.Serve(lis)
}

  1. python Client
import rpc_pb2
import grpc
import rpc_pb2_grpc

# Connect to rpc server
channel = grpc.insecure_channel('192.168.1.146:1234')
# Call the rpc service, define the helloservicesstub interface through the compiled HelloService interface of rpc ﹣ Pb2 ﹣ grpc, and receive the data from the channel
stub = rpc_pb2_grpc.HelloServiceStub(channel)
# Get String type data through rpc of the interface and get the value
response = stub.Hello(rpc_pb2.String(value='test'))
print("Greeter client received: " + response.value)

python as the server, go as the client

  1. python Server
from concurrent import futures
import time
import rpc_pb2
import grpc
import rpc_pb2_grpc

class Hello(rpc_pb2_grpc.HelloServiceServicer):
    # Implement rpc call defined in proto file
    def Hello(self, request, context):
        return rpc_pb2.String(value = 'hello {msg}'.format(msg = request.value))

# Define to open 4 threads to process received requests
server = grpc.server(futures.ThreadPoolExecutor(max_workers=4))
# Add the add helloserviceserver to server function of the compiled RPC Pb2 grpc to the server
rpc_pb2_grpc.add_HelloServiceServicer_to_server(Hello(), server)
 
# Define server port 1234
server.add_insecure_port('192.168.1.146:1234')
server.start()

# Long term monitoring
try:
    while True:
        time.sleep(60 * 60 * 24)
except KeyboardInterrupt:
    server.stop(0)

  1. go Client
package main

import (
    "fmt"
    "go_protoc"
    "google.golang.org/grpc"
    "log"
    "golang.org/x/net/context"
)

func main() {
    // Connect to the server interface
    conn, err := grpc.Dial("192.168.1.146:1234", grpc.WithInsecure())
    if err != nil {
        log.Fatal(err)
    }
    defer conn.Close()

    // Compile rpc.pb.go to get the NewHelloServiceClient function to handle the connection
    client := go_protoc.NewHelloServiceClient(conn)
    // Send data of String type by compiling the Hello service from rpc.pb.go
    reply, err := client.Hello(context.Background(), &go_protoc.String{Value: "test"})
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(reply.GetValue())
}

Tags: Programming Python git Google pip

Posted on Sat, 07 Mar 2020 22:55:21 -0800 by tecktalkcm0391