Spring cloud project restart gracefully: tomcat closes the process

This article mainly talks about the process of tomcat "rough stop" and several key information in the solution. The detailed solution steps will be in the following articles.

Overall structure

Let's take a look at the overall architecture of Tomcat to get a general understanding of the relationship between Tomcat components:

Source:< Four pictures show you Tomcat architecture>

Source:< Tomcat system architecture (I)>

Source:< Tomcat system architecture (top): how is the connector designed?>

close

When the spring boot Tomcat server is shut down, it will not be shut down until all the requests are processed, but it will be shut down immediately. We can see the closing process of spring boot Tomcat through the code.
Start with SpringContext:

public abstract class AbstractApplicationContext extends DefaultResourceLoader
  implements ConfigurableApplicationContext {

    public void registerShutdownHook() {
        if (this.shutdownHook == null) {
            // No shutdown hook registered yet.
            this.shutdownHook = new Thread() {
                @Override
                public void run() {
                    synchronized (startupShutdownMonitor) {
                        doClose();
                    }
                }
            };
            Runtime.getRuntime().addShutdownHook(this.shutdownHook);
        }
    }  
    protected void doClose() {
        // Check whether an actual close attempt is necessary...
        if (this.active.get() && this.closed.compareAndSet(false, true)) {
            if (logger.isDebugEnabled()) {
                logger.debug("Closing " + this);
            }

            LiveBeansView.unregisterApplicationContext(this);

            try {
                // Publish shutdown event.
                publishEvent(new ContextClosedEvent(this)); // Publishing events
            }
            catch (Throwable ex) {
                logger.warn("Exception thrown from ApplicationListener handling ContextClosedEvent", ex);
            }

            // Stop all Lifecycle beans, to avoid delays during individual destruction.
            if (this.lifecycleProcessor != null) {
                try {
                    this.lifecycleProcessor.onClose();
                }
                catch (Throwable ex) {
                    logger.warn("Exception thrown from LifecycleProcessor on context close", ex);
                }
            }

            // Destroy all cached singletons in the context's BeanFactory.
            destroyBeans();

            // Close the state of this context itself.
            closeBeanFactory();

            // Let subclasses do some final clean-up if they wish...
            onClose(); // Perform close

            // Reset local application listeners to pre-refresh state.
            if (this.earlyApplicationListeners != null) {
                this.applicationListeners.clear();
                this.applicationListeners.addAll(this.earlyApplicationListeners);
            }

            // Switch to inactive.
            this.active.set(false);
        }
    }        
}  

Note that the publish event (New contextclosed event (this)) is before closing the resource onClose, and the Spring event processing is synchronous by default, which is very important here, and will be used in the scheme mentioned later.

public class ServletWebServerApplicationContext extends GenericWebApplicationContext
  implements ConfigurableWebServerApplicationContext {

    private void stopAndReleaseWebServer() {
        WebServer webServer = this.webServer;
        if (webServer != null) {
            try {
                webServer.stop();
                this.webServer = null;
            }
            catch (Exception ex) {
                throw new IllegalStateException(ex);
            }
        }
    }
            
    protected void onClose() {
        super.onClose();
        stopAndReleaseWebServer();
    }
    
}   

From here on, enter the closure of the Web container.

public class TomcatWebServer implements WebServer {
    public void stop() throws WebServerException {
        synchronized (this.monitor) {
            boolean wasStarted = this.started;
            try {
                this.started = false;
                try {
                    stopTomcat();
                    this.tomcat.destroy();
                }
                catch (LifecycleException ex) {
                    // swallow and continue
                }
            }
            catch (Exception ex) {
                throw new WebServerException("Unable to stop embedded Tomcat", ex);
            }
            finally {
                if (wasStarted) {
                    containerCounter.decrementAndGet();
                }
            }
        }
    }
    
    private void stopTomcat() throws LifecycleException {
        if (Thread.currentThread()
                .getContextClassLoader() instanceof TomcatEmbeddedWebappClassLoader) {
            Thread.currentThread().setContextClassLoader(getClass().getClassLoader());
        }
        this.tomcat.stop();
    }

}

Enter to tomcat.stop() take a look.

public class Tomcat {
    public void stop() throws LifecycleException {
        getServer();
        server.stop();
    }
    
    public Server getServer() {

        if (server != null) {
            return server;
        }

        System.setProperty("catalina.useNaming", "false");

        server = new StandardServer();

        initBaseDir();

        // Set configuration source
        ConfigFileLoader.setSource(new CatalinaBaseConfigurationSource(new File(basedir), null));

        server.setPort( -1 );

        Service service = new StandardService();
        service.setName("Tomcat");
        server.addService(service);
        return server;
    }    
}

Reenter StandardService.stop .

public abstract class LifecycleBase implements Lifecycle {

    public final synchronized void stop() throws LifecycleException {

        if (LifecycleState.STOPPING_PREP.equals(state) || LifecycleState.STOPPING.equals(state) ||
                LifecycleState.STOPPED.equals(state)) {

            if (log.isDebugEnabled()) {
                Exception e = new LifecycleException();
                log.debug(sm.getString("lifecycleBase.alreadyStopped", toString()), e);
            } else if (log.isInfoEnabled()) {
                log.info(sm.getString("lifecycleBase.alreadyStopped", toString()));
            }

            return;
        }

        if (state.equals(LifecycleState.NEW)) {
            state = LifecycleState.STOPPED;
            return;
        }

        if (!state.equals(LifecycleState.STARTED) && !state.equals(LifecycleState.FAILED)) {
            invalidTransition(Lifecycle.BEFORE_STOP_EVENT);
        }

        try {
            if (state.equals(LifecycleState.FAILED)) {
                // Don't transition to STOPPING_PREP as that would briefly mark the
                // component as available but do ensure the BEFORE_STOP_EVENT is
                // fired
                fireLifecycleEvent(BEFORE_STOP_EVENT, null);
            } else {
                setStateInternal(LifecycleState.STOPPING_PREP, null, false);
            }

            stopInternal();

            // Shouldn't be necessary but acts as a check that sub-classes are
            // doing what they are supposed to.
            if (!state.equals(LifecycleState.STOPPING) && !state.equals(LifecycleState.FAILED)) {
                invalidTransition(Lifecycle.AFTER_STOP_EVENT);
            }

            setStateInternal(LifecycleState.STOPPED, null, false);
        } catch (Throwable t) {
            handleSubClassException(t, "lifecycleBase.stopFail", toString());
        } finally {
            if (this instanceof Lifecycle.SingleUse) {
                // Complete stop process first
                setStateInternal(LifecycleState.STOPPED, null, false);
                destroy();
            }
        }
    }
}
public final class StandardServer extends LifecycleMBeanBase implements Server {

    protected void stopInternal() throws LifecycleException {

        setState(LifecycleState.STOPPING);

        if (monitorFuture != null) {
            monitorFuture.cancel(true);
            monitorFuture = null;
        }
        if (periodicLifecycleEventFuture != null) {
            periodicLifecycleEventFuture.cancel(false);
            periodicLifecycleEventFuture = null;
        }

        fireLifecycleEvent(CONFIGURE_STOP_EVENT, null);

        // Stop our defined Services
        for (int i = 0; i < services.length; i++) {
            services[i].stop();
        }

        globalNamingResources.stop();

        stopAwait();
    }
}    

The stop method is in the parent class of LifecycleBase (the components of Tomcat basically inherit LifecycleBase). The StandardServer implements the child method stopInternal, and enters the StandardService.stop Inside. Here are the relationships of the following objects:

  • There is only one Server in a Tomcat.
  • A Server can contain multiple services, such as multiple projects deployed under a Tomcat(Server), that is, multiple services.
  • A Service has only one Container, but it can have multiple connectors. This is because a Service can have multiple connections, such as providing Http and Https links, or providing connections to different ports of the same protocol.
  • There is only one ProtocolHandler under a Connector. Because I/O model and application layer protocol can be freely combined, such as NIO + HTTP or NIO.2 + AJP. The designer of Tomcat considers network communication and application layer protocol analysis together, and designs an interface called ProtocolHandler to encapsulate these two kinds of change points. The combination of various protocols and communication models has corresponding concrete implementation classes. For example: Http11NioProtocol and AjpNioProtocol. The Connector uses the ProtocolHandler to handle the network connection and application layer protocol, which includes two important parts: Endpoint and Processor.
  • There is only one Endpoint under a ProtocolHandler. Endpoint is the communication Endpoint, that is, the interface of communication monitoring, the specific Socket receiving and sending processor, and the abstraction of the transmission layer. Therefore, Endpoint is used to implement the TCP/IP protocol.
public class StandardService extends LifecycleMBeanBase implements Service {
    protected void stopInternal() throws LifecycleException {

        // Pause connectors first
        synchronized (connectorsLock) {
            for (Connector connector: connectors) {
                connector.pause();
                // Close server socket if bound on start
                // Note: test is in AbstractEndpoint
                connector.getProtocolHandler().closeServerSocketGraceful();
            }
        }

        if(log.isInfoEnabled())
            log.info(sm.getString("standardService.stop.name", this.name));
        setState(LifecycleState.STOPPING);

        // Stop our defined Container second
        if (engine != null) {
            synchronized (engine) {
                engine.stop();
            }
        }

        // Now stop the connectors
        synchronized (connectorsLock) {
            for (Connector connector: connectors) {
                if (!LifecycleState.STARTED.equals(
                        connector.getState())) {
                    // Connectors only need stopping if they are currently
                    // started. They may have failed to start or may have been
                    // stopped (e.g. via a JMX call)
                    continue;
                }
                connector.stop();
            }
        }

        // If the Server failed to start, the mapperListener won't have been
        // started
        if (mapperListener.getState() != LifecycleState.INITIALIZED) {
            mapperListener.stop();
        }

        synchronized (executors) {
            for (Executor executor: executors) {
                executor.stop();
            }
        }

    }
}    

Here are two questions related to our above questions:

  1. Connetor.pause Stop receiving new requests.
  2. Connector.stop The thread pool is closed here.
public class Connector extends LifecycleMBeanBase {
    @Override
    protected void stopInternal() throws LifecycleException {

        setState(LifecycleState.STOPPING);

        try {
            if (protocolHandler != null) {
                protocolHandler.stop();
            }
        } catch (Exception e) {
            throw new LifecycleException(
                    sm.getString("coyoteConnector.protocolHandlerStopFailed"), e);
        }
    }

}

public abstract class AbstractProtocol<S> implements ProtocolHandler,
        MBeanRegistration {
    public void stop() throws Exception {
        if(getLog().isInfoEnabled()) {
            getLog().info(sm.getString("abstractProtocolHandler.stop", getName()));
            logPortOffset();
        }

        if (monitorFuture != null) {
            monitorFuture.cancel(true);
            monitorFuture = null;
        }
        stopAsyncTimeout();
        // Timeout any pending async request
        for (Processor processor : waitingProcessors) {
            processor.timeoutAsync(-1);
        }

        endpoint.stop();
    }        
}  

public abstract class AbstractEndpoint<S,U> {

    public final void stop() throws Exception {
        stopInternal();
        if (bindState == BindState.BOUND_ON_START || bindState == BindState.SOCKET_CLOSED_ON_STOP) {
            unbind();
            bindState = BindState.UNBOUND;
        }
    }
    
    public abstract void stopInternal() throws Exception; // Subclass implementation, subclass AprEndpoint and NioEndpoint(Springboot defaults to NioEndpoint) implementation method will call the shutdownExecutor method

    public void shutdownExecutor() {
        Executor executor = this.executor;
        if (executor != null && internalExecutor) {
            this.executor = null;
            if (executor instanceof ThreadPoolExecutor) {
                //this is our internal one, so we need to shut it down
                ThreadPoolExecutor tpe = (ThreadPoolExecutor) executor;
                tpe.shutdownNow();
                long timeout = getExecutorTerminationTimeoutMillis();
                if (timeout > 0) {
                    try {
                        tpe.awaitTermination(timeout, TimeUnit.MILLISECONDS);
                    } catch (InterruptedException e) {
                        // Ignore
                    }
                    if (tpe.isTerminating()) {
                        getLog().warn(sm.getString("endpoint.warn.executorShutdown", getName()));
                    }
                }
                TaskQueue queue = (TaskQueue) tpe.getQueue();
                queue.setParent(null);
            }
        }
    }
    
    /**
     * Time to wait for the internal executor (if used) to terminate when the
     * endpoint is stopped in milliseconds. Defaults to 5000 (5 seconds).
     */
    private long executorTerminationTimeoutMillis = 5000;

    public long getExecutorTerminationTimeoutMillis() {
        return executorTerminationTimeoutMillis;
    }
}      

As you can see above, in connector.stop > protocolHandler.stop > endpoint.stop > NioEndpoint.stopInternal > endpoint.shutdownExecutor In, the thread pool directly and roughly shutdownNow to stop the task. The java thread pool's shutdownNow will try to interrupt the executing thread in the thread pool, and the thread waiting for execution will also be cancelled, but it can't guarantee that the thread in the interrupt thread pool will succeed. All tomcat will wait for 5 seconds through awaitTermination to stop all tasks as much as possible.

From the above analysis, we can see that the main reason is that the thread pool is shut down by Tomcat. In the first chapter, we need to do two steps to deal with this problem:

  1. Pause receiving new requests.
  2. Wait for all tasks to complete before closing the thread pool.

The key to completing these two steps is:

  1. Get connector, Connector.pause Method to pause receiving requests.
  2. Get to thread pool.

Connector

In Tomcat architecture, Connector is mainly responsible for handling communication with clients. The instance of the Connector is used to listen to the port, accept the request from the client and transfer the request to the Engine for processing, and return the reply from the Engine to the client:

  1. Monitor network port
  2. Accept network connection request
  3. Read request network byte stream
  4. Parsing byte stream according to specific application layer protocol (HTTP/AJP) to generate unified Tomcat Request object
  5. Convert Tomcat Request object to standard ServletRequest
  6. Call the Servlet container to get the Servlet response
  7. Convert ServletResponse to Tomcat Response object
  8. Convert Tomcat Response object to network byte stream
  9. Writes the response byte stream back to the server.

Now let's take a look at the steps for spring boot to generate Tomcat.

@Configuration
class ServletWebServerFactoryConfiguration {

    @Configuration
    @ConditionalOnClass({ Servlet.class, Tomcat.class, UpgradeProtocol.class })
    @ConditionalOnMissingBean(value = ServletWebServerFactory.class, search = SearchStrategy.CURRENT)
    public static class EmbeddedTomcat {

        @Bean
        public TomcatServletWebServerFactory tomcatServletWebServerFactory() {
            return new TomcatServletWebServerFactory();
        }

    }

    /**
     * Nested configuration if Jetty is being used.
     */
    @Configuration
    @ConditionalOnClass({ Servlet.class, Server.class, Loader.class,
            WebAppContext.class })
    @ConditionalOnMissingBean(value = ServletWebServerFactory.class, search = SearchStrategy.CURRENT)
    public static class EmbeddedJetty {

        @Bean
        public JettyServletWebServerFactory JettyServletWebServerFactory() {
            return new JettyServletWebServerFactory();
        }

    }

    /**
     * Nested configuration if Undertow is being used.
     */
    @Configuration
    @ConditionalOnClass({ Servlet.class, Undertow.class, SslClientAuthMode.class })
    @ConditionalOnMissingBean(value = ServletWebServerFactory.class, search = SearchStrategy.CURRENT)
    public static class EmbeddedUndertow {

        @Bean
        public UndertowServletWebServerFactory undertowServletWebServerFactory() {
            return new UndertowServletWebServerFactory();
        }

    }

}

You can see that TomcatServletWebServerFactory is generated. TomcatServletWebServerFactory has a createWebServer method to generate Tomcat. When is this method called?

public class ServletWebServerApplicationContext extends GenericWebApplicationContext
  implements ConfigurableWebServerApplicationContext {
    @Override
    protected void onRefresh() {
        super.onRefresh();
        try {
            createWebServer();
        }
        catch (Throwable ex) {
            throw new ApplicationContextException("Unable to start web server", ex);
        }
    }
    
    private void createWebServer() {
        WebServer webServer = this.webServer;
        ServletContext servletContext = getServletContext();
        if (webServer == null && servletContext == null) {
            ServletWebServerFactory factory = getWebServerFactory();
            this.webServer = factory.getWebServer(getSelfInitializer());
        }
        else if (servletContext != null) {
            try {
                getSelfInitializer().onStartup(servletContext);
            }
            catch (ServletException ex) {
                throw new ApplicationContextException("Cannot initialize servlet context",
                        ex);
            }
        }
        initPropertySources();
    }
}    

You can see that the container class of springboot, servletwebserver ApplicationContext, will call - ser when the system starts (onRefresh) vletWebServerFactory.getWebServer `.

public class TomcatServletWebServerFactory extends AbstractServletWebServerFactory
  implements ConfigurableTomcatWebServerFactory, ResourceLoaderAware {
    @Override
    public WebServer getWebServer(ServletContextInitializer... initializers) {
        Tomcat tomcat = new Tomcat();
        File baseDir = (this.baseDirectory != null) ? this.baseDirectory
                : createTempDir("tomcat");
        tomcat.setBaseDir(baseDir.getAbsolutePath());
        Connector connector = new Connector(this.protocol);
        tomcat.getService().addConnector(connector);
        customizeConnector(connector);
        tomcat.setConnector(connector);
        tomcat.getHost().setAutoDeploy(false);
        configureEngine(tomcat.getEngine());
        for (Connector additionalConnector : this.additionalTomcatConnectors) {
            tomcat.getService().addConnector(additionalConnector);
        }
        prepareContext(tomcat.getHost(), initializers);
        return getTomcatWebServer(tomcat);
    }
}        

Entering getWebServer, we see that Tomcat object is being built, in which the step of customize Connector (Connector) has the processing of Connector.

public class TomcatServletWebServerFactory extends AbstractServletWebServerFactory
    implements ConfigurableTomcatWebServerFactory, ResourceLoaderAware { 
    
    protected void customizeConnector(Connector connector) {
        int port = (getPort() >= 0) ? getPort() : 0;
        connector.setPort(port);
        if (StringUtils.hasText(this.getServerHeader())) {
            connector.setAttribute("server", this.getServerHeader());
        }
        if (connector.getProtocolHandler() instanceof AbstractProtocol) {
            customizeProtocol((AbstractProtocol<?>) connector.getProtocolHandler());
        }
        if (getUriEncoding() != null) {
            connector.setURIEncoding(getUriEncoding().name());
        }
        // Don't bind to the socket prematurely if ApplicationContext is slow to start
        connector.setProperty("bindOnInit", "false");
        if (getSsl() != null && getSsl().isEnabled()) {
            customizeSsl(connector);
        }
        TomcatConnectorCustomizer compression = new CompressionConnectorCustomizer(
                getCompression());
        compression.customize(connector);
        for (TomcatConnectorCustomizer customizer : this.tomcatConnectorCustomizers) {
            customizer.customize(connector);
        }
    }
    
    public void addContextCustomizers(
            TomcatContextCustomizer... tomcatContextCustomizers) {
        Assert.notNull(tomcatContextCustomizers,
                "TomcatContextCustomizers must not be null");
        this.tomcatContextCustomizers.addAll(Arrays.asList(tomcatContextCustomizers));
    }
}

Through the above code, we know that: TomcatContextCustomizer can be customized and added through addContextCustomizers method; in the customizeConnector method, all TomcatContextCustomizer objects can be traversed through customizer.customize(connector) method to pass in the Connector object.

Thread pool

On the basis that the connector can be obtained in the previous step, it is very simple to obtain the thread pool: Connector - > protocolhandler - > endpoint - > getexecutor().

Additional:

Spring event handling

public abstract class AbstractApplicationContext extends DefaultResourceLoader
  implements ConfigurableApplicationContext {
  
    protected void publishEvent(Object event, @Nullable ResolvableType eventType) {
        Assert.notNull(event, "Event must not be null");

        // Decorate event as an ApplicationEvent if necessary
        ApplicationEvent applicationEvent;
        if (event instanceof ApplicationEvent) {
            applicationEvent = (ApplicationEvent) event;
        }
        else {
            applicationEvent = new PayloadApplicationEvent<>(this, event);
            if (eventType == null) {
                eventType = ((PayloadApplicationEvent) applicationEvent).getResolvableType();
            }
        }

        // Multicast right now if possible - or lazily once the multicaster is initialized
        if (this.earlyApplicationEvents != null) {
            this.earlyApplicationEvents.add(applicationEvent);
        }
        else {
            getApplicationEventMulticaster().multicastEvent(applicationEvent, eventType);
        }

        // Publish event via parent context as well...
        if (this.parent != null) {
            if (this.parent instanceof AbstractApplicationContext) {
                ((AbstractApplicationContext) this.parent).publishEvent(event, eventType);
            }
            else {
                this.parent.publishEvent(event);
            }
        }
    }
    
}        

You can see that the focus is getapplicationeventmulticaster(). Multicasevent (applicationevent, eventtype); in this step, ApplicationEventMulticaster defaults to SimpleApplicationEventMulticaster.

public class SimpleApplicationEventMulticaster extends AbstractApplicationEventMulticaster {

    public void multicastEvent(final ApplicationEvent event, @Nullable ResolvableType eventType) {
        ResolvableType type = (eventType != null ? eventType : resolveDefaultEventType(event));
        for (final ApplicationListener<?> listener : getApplicationListeners(event, type)) {
            Executor executor = getTaskExecutor();
            if (executor != null) {
                executor.execute(() -> invokeListener(listener, event));
            }
            else {
                invokeListener(listener, event);
            }
        }
    }
}

First, obtain all listeners listening to the corresponding events (this step will not go further, and the logic is relatively simple). Here, the executor has no value by default, so enter the invokeListener method.

public class SimpleApplicationEventMulticaster extends AbstractApplicationEventMulticaster {

    protected void invokeListener(ApplicationListener<?> listener, ApplicationEvent event) {
        ErrorHandler errorHandler = getErrorHandler();
        if (errorHandler != null) {
            try {
                doInvokeListener(listener, event);
            }
            catch (Throwable err) {
                errorHandler.handleError(err);
            }
        }
        else {
            doInvokeListener(listener, event);
        }
    }
    
    private void doInvokeListener(ApplicationListener listener, ApplicationEvent event) {
        try {
            listener.onApplicationEvent(event);
        }
        catch (ClassCastException ex) {
            String msg = ex.getMessage();
            if (msg == null || matchesClassCastMessage(msg, event.getClass())) {
                // Possibly a lambda-defined listener which we could not resolve the generic event type for
                // -> let's suppress the exception and just log a debug message.
                Log logger = LogFactory.getLog(getClass());
                if (logger.isDebugEnabled()) {
                    logger.debug("Non-matching event type for listener: " + listener, ex);
                }
            }
            else {
                throw ex;
            }
        }
    }
}

You can see that invokeListener is a direct call listener.onApplicationEvent method.
So by default (executor is empty), spring's processing is synchronous.

Tags: Java Tomcat Spring network socket

Posted on Sat, 06 Jun 2020 22:58:32 -0700 by HalfaBee