Microservice full link tracking: jaeger integrates istio and is compatible with Uber trace ID and b3

Microservice full link tracking: grpc integrated zipkin

Microservice full link tracking: grpc integrated jaeger

Microservice full link tracking: springcloud integrated jaeger

Microservice full link tracking: jaeger integrates istio and is compatible with Uber trace ID and b3

The company has its own paas system based on k8s, and integrated istio. Here is to explain how spring cloud services integrate istio

jaeger cross process delivery

In the distributed call based on HTTP protocol, HTTP Header is usually used to pass the content of SpanContext. Common Wire Protocol includes b3 HTTP header used by Zipkin, Uber trace ID HTTP Header used by Jaeger, x-ot-span-context HTTP Header used by LightStep, etc. Istio1.0 supports b3 header and x-ot-span-context header, which can be connected with Zipkin,Jaeger and LightStep; istio1.4 supports Uber trace ID, please refer to the official instructions of github: https://github.com/istio/istio/issues/12400

uber-trace-id

You can see in the figure that the traceId, spanId and other fields are spliced into a header clock

b3

For details of the b3 head of istio, please refer to: https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/headers#x-request-id b3 didn't splice all the fields. Twenty fields were passed separately. Here's a case:

X-B3-TraceId:427fde2dc7edb084
X-B3-ParentSpanId:427fde2dc7edb084
X-B3-SpanId:827e270489aafbd7
X-B3-Sampled:1

Change jaeger transmission to b3

If you need to integrate the jaeger of istio, you need to change the transmission mode to b3 Microservice full link tracking: springcloud integrated jaeger How to integrate jaeger has been described in this chapter. Only one configuration, enable-b3-propagation, needs to be modified here, as follows

opentracing:
  jaeger:
    enable-b3-propagation: true// Default is false
    udp-sender:
      host: localhost
      port: 6831
    remote-reporter:
      flush-interval: 1000
      max-queue-size: 5000
    log-spans: true
    probabilistic-sampler:
      sampling-rate: 1

In this way, the spring boot service can be linked with jaeger information in istio to form a complete full link.

Compatible with Uber trace ID and b3

Now there is another problem. The company has built a set of full link jaeger for a long time, and has accessed most of the systems. It uses the default header transmission, that is, Uber trace ID However, there are many internal systems of paas in the downstream that are not java, which are inconvenient to access jaeger. They only inject istio, and automatically inject jaeger agent. In this case, b3 head transmission is used, which results in that some links cannot be connected in series. If the unified transmission mode is not practical, first of all, if it is changed to b3, then a lot of connected systems in the upstream need to be modified and configured as b3. If it is changed to Uber trace ID, the current version of istio does not support it. If it is necessary to upgrade istio, kubernetes needs to be upgraded, so the risk is relatively large, so the integration of the two headers is used first according to the actual situation, that is, Uber trac is used in the upstream E-ID: inject the b3 related header manually to the middle tier service as follows.

grpc injection head b3

We need to use the interceptor of grpc

import com.google.common.collect.ImmutableMap;
import io.grpc.*;
import io.opentracing.Span;
import io.opentracing.Tracer;
import io.opentracing.propagation.Format;
import io.opentracing.propagation.TextMap;
import lombok.extern.slf4j.Slf4j;

import javax.annotation.Nullable;
import java.util.Arrays;
import java.util.HashSet;
import java.util.Iterator;
import java.util.Map.Entry;
import java.util.Set;
import java.util.concurrent.TimeUnit;

/** 
 * An intercepter that applies tracing via OpenTracing to all client requests. 
 */
@Slf4j
public class ClientTracingInterceptor implements ClientInterceptor {

    private final Tracer tracer;
    private final OperationNameConstructor operationNameConstructor;
    private final boolean streaming;
    private final boolean verbose;
    private final Set<clientrequestattribute> tracedAttributes;
    private final ActiveSpanSource activeSpanSource;
    private final  Metadata.Key<string> b3TraceIdKey = Metadata.Key.of("X-B3-TraceId", Metadata.ASCII_STRING_MARSHALLER);
    private final  Metadata.Key<string> b3SpanIdKey = Metadata.Key.of("X-B3-SpanId", Metadata.ASCII_STRING_MARSHALLER);
    private final  Metadata.Key<string> b3ParentSpanIdKey = Metadata.Key.of("X-B3-ParentSpanId", Metadata.ASCII_STRING_MARSHALLER);
    private final  Metadata.Key<string> b3SampledKey = Metadata.Key.of("X-B3-Sampled", Metadata.ASCII_STRING_MARSHALLER);
    /**
     * @param
     */
    public ClientTracingInterceptor(Tracer tracer) {
        this.tracer=tracer;
        this.operationNameConstructor = OperationNameConstructor.DEFAULT;
        this.streaming = false;
        this.verbose = false;
        this.tracedAttributes = new HashSet<clientrequestattribute>();
        this.activeSpanSource = ActiveSpanSource.GRPC_CONTEXT;
    }

    private ClientTracingInterceptor(Tracer tracer, OperationNameConstructor operationNameConstructor, boolean streaming,
        boolean verbose, Set<clientrequestattribute> tracedAttributes, ActiveSpanSource activeSpanSource) {
        this.tracer = tracer;
        this.operationNameConstructor = operationNameConstructor;
        this.streaming = streaming;
        this.verbose = verbose;
        this.tracedAttributes = tracedAttributes;
        this.activeSpanSource = activeSpanSource;
    }

    /**
     * Use this intercepter to trace all requests made by this client channel.
     * @param channel to be traced
     * @return intercepted channel
     */ 
    public Channel intercept(Channel channel) {
        return ClientInterceptors.intercept(channel, this);
    }

    @Override
    public <reqt, respt> ClientCall<reqt, respt> interceptCall(
        MethodDescriptor<reqt, respt> method, 
        CallOptions callOptions, 
        Channel next
    ) {
        final String operationName = operationNameConstructor.constructOperationName(method);

        Span activeSpan = this.activeSpanSource.getActiveSpan();
        final Span span = createSpanFromParent(activeSpan, operationName);

        for (ClientRequestAttribute attr : this.tracedAttributes) {
            switch (attr) {
                case ALL_CALL_OPTIONS:
                    span.setTag("grpc.call_options", callOptions.toString());
                    break;
                case AUTHORITY:
                    if (callOptions.getAuthority() == null) {
                        span.setTag("grpc.authority", "null");
                    } else {
                        span.setTag("grpc.authority", callOptions.getAuthority());                        
                    }
                    break;
                case COMPRESSOR:
                    if (callOptions.getCompressor() == null) {
                        span.setTag("grpc.compressor", "null");
                    } else {
                        span.setTag("grpc.compressor", callOptions.getCompressor());
                    }
                    break;
                case DEADLINE:
                    if (callOptions.getDeadline() == null) {
                        span.setTag("grpc.deadline_millis", "null");
                    } else {
                        span.setTag("grpc.deadline_millis", callOptions.getDeadline().timeRemaining(TimeUnit.MILLISECONDS));
                    }
                    break;
                case METHOD_NAME:
                    span.setTag("grpc.method_name", method.getFullMethodName());
                    break;
                case METHOD_TYPE:
                    if (method.getType() == null) {
                        span.setTag("grpc.method_type", "null");
                    } else {
                        span.setTag("grpc.method_type", method.getType().toString());
                    }
                    break;
                case HEADERS:
                	break;
            }
        }

        return new ForwardingClientCall.SimpleForwardingClientCall<reqt, respt>(next.newCall(method, callOptions)) {

            @Override
            public void start(Listener<respt> responseListener, Metadata headers) {
                if (verbose) {
                    span.log("Started call");
                }
                if (tracedAttributes.contains(ClientRequestAttribute.HEADERS)) {
                    span.setTag("grpc.headers", headers.toString()); 
                }

                tracer.inject(span.context(), Format.Builtin.HTTP_HEADERS, new TextMap() {
                    @Override
                    public void put(String key, String value) {
                        log.info("jaeger key:{},value:{}",key,value);
                        Metadata.Key<string> headerKey = Metadata.Key.of(key, Metadata.ASCII_STRING_MARSHALLER);
                        headers.put(headerKey, value);
                        String[] mm=value.split("%3A");
                        if("uber-trace-id".equals(key)&amp;&amp;mm.length==4){
                            headers.put(b3TraceIdKey,mm[0]);
                            log.info("jaeger traceId:{}",mm[0]);
                            headers.put(b3SpanIdKey,mm[1]);
                            headers.put(b3ParentSpanIdKey,mm[2]);
                            headers.put(b3SampledKey,mm[3]);
                        }

                    }
					@Override
					public Iterator<entry<string, string>&gt; iterator() {
						throw new UnsupportedOperationException(
								"TextMapInjectAdapter should only be used with Tracer.inject()");
					}
                });

                Listener<respt> tracingResponseListener = new ForwardingClientCallListener
                    .SimpleForwardingClientCallListener<respt>(responseListener) {

                    @Override
                    public void onHeaders(Metadata headers) {
                        if (verbose) { span.log(ImmutableMap.of("Response headers received", headers.toString())); }
                        delegate().onHeaders(headers);
                    }

                    @Override
                    public void onMessage(RespT message) {
                        if (streaming || verbose) { span.log("Response received"); }
                        delegate().onMessage(message);
                    }

                    @Override 
                    public void onClose(Status status, Metadata trailers) {
                        if (verbose) { 
                            if (status.getCode().value() == 0) { span.log("Call closed"); }
                            else { span.log(ImmutableMap.of("Call failed", status.getDescription())); }
                        }
                        span.finish();
                        delegate().onClose(status, trailers);
                    }
                };
                delegate().start(tracingResponseListener, headers);
            }

            @Override 
            public void cancel(@Nullable String message, @Nullable Throwable cause) {
                String errorMessage;
                if (message == null) {
                    errorMessage = "Error";
                } else {
                    errorMessage = message;
                }
                if (cause == null) {
                    span.log(errorMessage);
                } else {
                    span.log(ImmutableMap.of(errorMessage, cause.getMessage()));
                }
                delegate().cancel(message, cause);
            }

            @Override
            public void halfClose() {
                if (streaming) { span.log("Finished sending messages"); }
                delegate().halfClose();
            }

            @Override
            public void sendMessage(ReqT message) {
                if (streaming || verbose) { span.log("Message sent"); }
                delegate().sendMessage(message);
            }
        };
    }
    
    private Span createSpanFromParent(Span parentSpan, String operationName) {
        if (parentSpan == null) {
            return tracer.buildSpan(operationName).startManual();
        } else {
            return tracer.buildSpan(operationName).asChildOf(parentSpan).startManual();
        }
    }

    /**
     * Builds the configuration of a ClientTracingInterceptor.
     */
    public static class Builder {

        private Tracer tracer;
        private OperationNameConstructor operationNameConstructor;
        private boolean streaming;
        private boolean verbose;
        private Set<clientrequestattribute> tracedAttributes;
        private ActiveSpanSource activeSpanSource;  

        /**
         * @param tracer to use for this intercepter
         * Creates a Builder with default configuration
         */
        public Builder(Tracer tracer) {
            this.tracer = tracer;
            this.operationNameConstructor = OperationNameConstructor.DEFAULT;
            this.streaming = false;
            this.verbose = false;
            this.tracedAttributes = new HashSet<clientrequestattribute>();
            this.activeSpanSource = ActiveSpanSource.GRPC_CONTEXT;
        } 

        /**
         * @param operationNameConstructor to name all spans created by this intercepter
         * @return this Builder with configured operation name
         */
        public Builder withOperationName(OperationNameConstructor operationNameConstructor) {
            this.operationNameConstructor = operationNameConstructor;
            return this;
        } 

        /**
         * Logs streaming events to client spans.
         * @return this Builder configured to log streaming events
         */
        public Builder withStreaming() {
            this.streaming = true;
            return this;
        }

        /**
         * @param tracedAttributes to set as tags on client spans
         *  created by this intercepter
         * @return this Builder configured to trace attributes
         */
        public Builder withTracedAttributes(ClientRequestAttribute... tracedAttributes) {
            this.tracedAttributes = new HashSet<clientrequestattribute>(
                Arrays.asList(tracedAttributes));
            return this;
        }

        /**
         * Logs all request life-cycle events to client spans.
         * @return this Builder configured to be verbose
         */
        public Builder withVerbosity() {
            this.verbose = true;
            return this;
        }

        /**
         * @param activeSpanSource that provides a method of getting the 
         *  active span before the client call
         * @return this Builder configured to start client span as children 
         *  of the span returned by activeSpanSource.getActiveSpan()
         */
        public Builder withActiveSpanSource(ActiveSpanSource activeSpanSource) {
            this.activeSpanSource = activeSpanSource;
            return this;
        }

        /**
         * @return a ClientTracingInterceptor with this Builder's configuration
         */
        public ClientTracingInterceptor build() {
            return new ClientTracingInterceptor(this.tracer, this.operationNameConstructor, 
                this.streaming, this.verbose, this.tracedAttributes, this.activeSpanSource);
        }
    }

    public enum ClientRequestAttribute {
        METHOD_TYPE,
        METHOD_NAME,
        DEADLINE,
        COMPRESSOR,
        AUTHORITY,
        ALL_CALL_OPTIONS,
        HEADERS
    }
}

The main changes are in the header area

feign injection head b3

@Configuration
public class FeignConfig implements RequestInterceptor {
	@Autowired
    private final Tracer tracer;
    @Override
    public void apply(RequestTemplate requestTemplate) {
        ServletRequestAttributes attributes = (ServletRequestAttributes) RequestContextHolder.getRequestAttributes();

        if(attributes!=null&amp;&amp;attributes.getRequest()!=null){
            HttpServletRequest request = attributes.getRequest();

           JaegerSpanContext context=(JaegerSpanContext) tracer.activeSpan().context();
           requestTemplate.header("X-B3-TraceId",String.valueOf(context.getTraceId()));
           requestTemplate.header("X-B3-SpanId", String.valueOf(context.getSpanId()));
           requestTemplate.header("X-B3-ParentSpanId", String.valueOf(context.getParentId()));
           requestTemplate.header("X-B3-Sampled", context.isSampled()?"1":"0");
        }

    }

    @Bean
    Logger.Level feignLoggerLevel() {
        return Logger.Level.FULL;
    }

}

After manual injection, the upstream and downstream can be linked up to achieve the goal temporarily. The later scheme is to unify the transmission mode and upgrade slowly. </clientrequestattribute></clientrequestattribute></clientrequestattribute></respt></respt></entry<string,></string></respt></reqt,></reqt,></reqt,></reqt,></clientrequestattribute></clientrequestattribute></string></string></string></string></clientrequestattribute>

Tags: Programming Java Spring github Kubernetes

Posted on Sun, 26 Apr 2020 02:11:44 -0700 by Alicia