Status providers
The status of a service is the line that systemctl status shows, sent by the application to systemd. A status provider contributes a part of it, implementing StatusProvider: the status is made of the parts of all the status providers registered in the Systemd instance, in their order, separated by commas. A status provider with nothing to report returns an empty string, and is left out. If no status provider is registered at all, no status is sent to systemd.
[root@machine ~]# systemctl status myservice.service
● myservice.service - My Service
Loaded: loaded (/etc/systemd/system/myservice.service; disabled; vendor preset: disabled)
Active: active (running) since Qui 2018-02-15 10:19:28 WET; 1h 24min ago
Main PID: 11142 (java)
Status: "Heap: 139.5 MiB/256.0 MiB, Non-heap: 62.7 MiB/64.1 MiB, Classes: 7915, Threads: 31 (9 runnable; 14 waiting; 8 timed waiting)"
CGroup: /system.slice/myservice.service
└─11142 /opt/jdk17/bin/java -XX:+ExitOnOutOfMemoryError -jar /opt/myservice/myservice.jar
The core library provides status providers for the JVM, which any application can use. The integrations with Spring Boot and Keycloak register them for the application, along with status providers of their own.
Core library
Registering status providers
Register status providers with addStatusProviders(…) on the Systemd instance of the process, returned by Systemd.instance(), and remove them with removeStatusProviders(…). Their parts are shown in the order they were registered, unless added at a given position.
The status is sent to systemd periodically once enabled with enableStatusUpdate(period), and right away with updateStatus(), such as at each milestone of a long startup:
Systemd systemd = Systemd.instance();
systemd.addStatusProviders(new HeapStatusProvider(), new NonHeapStatusProvider(), new ClassLoaderStatusProvider(),
new ThreadStatusProvider(false, false));
systemd.enableStatusUpdate(Duration.ofSeconds(5));
See How to for the full lifecycle, with a complete example.
Built-in status providers
| Class | Purpose | Sample |
|---|---|---|
| HeapStatusProvider | Provides the heap memory used, out of the committed size (not a limit) | Heap: 139.5 MiB/256.0 MiB |
| OldGenerationStatusProvider | Provides the old generation of the heap after the last garbage collection | Old gen after GC: 120.0 MiB/512.0 MiB |
| NonHeapStatusProvider | Provides the non-heap memory used, out of the committed size | Non-heap: 62.7 MiB/64.1 MiB |
| BufferPoolStatusProvider | Provides the memory used by direct and mapped buffers | Buffers: 64.0 MiB/1.0 GiB direct (412); 2.0 GiB mapped (3) |
| GarbageCollectionStatusProvider | Provides the garbage collection pauses since the status was last read | GC: 12 pauses, 0.4% of time |
| ClassLoaderStatusProvider | Provides information regarding the number of loaded classes | Classes: 7915 |
| CpuStatusProvider | Provides the CPU usage of the process and of the system, and the load average | CPU: 12.5% process; 40.1% system; load 1.20 |
| ThreadStatusProvider | Provides the number of threads by state, and of deadlocked threads, if any | Threads: 31 (9 runnable; 14 waiting; 8 timed waiting) |
| FileDescriptorStatusProvider | Provides the open file descriptors, and their health | File descriptors: 812/65536 |
| CgroupMemoryStatusProvider | Provides the memory used by the control group of the process, out of its limit | Cgroup memory: 700.0 MiB/1.0 GiB |
| ExecutorStatusProvider | Provides the busy threads and the waiting tasks of a thread pool | workers: 3/10 threads, 5 queued |
| PhaseStatusProvider | Provides what the application says it is doing | Reindexing 40% |
Most of them are created without arguments. The others are configured when created:
- the ThreadStatusProvider can also detect deadlocked threads, adding their number to the status, such as
2 deadlocked, when created withnew ThreadStatusProvider(true, false); - the FileDescriptorStatusProvider shows the open file descriptors out of the most the process can open, the
LimitNOFILEof the service unit, and is created with the share of that limit from which it reports the application as unhealthy, such asnew FileDescriptorStatusProvider(false, 0.9); - the ExecutorStatusProvider is created with the name to show and the
ThreadPoolExecutorto report, or a supplier of it, for a pool created later; - the PhaseStatusProvider shows nothing until the application sets a phase. When created with
new PhaseStatusProvider(true), every change of phase is sent to systemd right away, rather than with the next status update.
The ThreadStatusProvider and the FileDescriptorStatusProvider can also report the application health, to restart it on deadlocks or when its file descriptors are about to run out. Registering them as status providers only adds them to the status: see Health providers to have their health taken into account.
Reporting what the application is doing
To report what the application is doing without writing a status provider, register a PhaseStatusProvider and set its phase:
PhaseStatusProvider phase = new PhaseStatusProvider(true);
Systemd.instance().addStatusProviders(phase);
phase.setPhase("Reindexing");
try {
// ...
} finally {
phase.clearPhase();
}
Custom status providers
You can create your own status information by implementing StatusProvider, and registering an instance with addStatusProviders(…):
public class QueueStatusProvider implements StatusProvider {
private final Queue<?> queue;
public QueueStatusProvider(Queue<?> queue) {
this.queue = queue;
}
@Override
public String status() {
return "Queue: " + queue.size();
}
}
The status is asked for on every status update, from the thread that sends it: keep it quick, and return an empty string when there is nothing to report. A status provider only reports the status: it must not drive the lifecycle of the application, by calling ready(), stopping() or close() on the Systemd instance.
Under Spring Boot
When using the autoconfiguration class, the built-in status providers are registered for the application, and StatusProvider beans will also be searched in the application context and used to compose an extended status message that systemd will display when using the status verb. The status is sent to systemd once every five seconds, or at the period set with systemd.status-update-period, such as 30s, to call the status providers less often, such as Actuator health indicators that query remote services.
Out of the box this module will show memory (heap, old generation, non-heap, buffers and the memory of the control group of the service), classloader information, threads, file descriptors, the availability of the application, application startup sequence state and bean creation progress, the stage of the shutdown, as well as the application health when using Spring Boot Actuator. The garbage collection pauses and the CPU usage can be shown as well, once enabled.
Additionally, the status of the libraries the application uses is shown when they are available: the connections of the embedded web server (Tomcat, Jetty or Reactor Netty), the HikariCP connection pools, the thread pools of the task executors and schedulers, and the progress of the Liquibase database migrations run on startup.
[root@machine ~]# systemctl status myservice.service
● myservice.service - My Spring Boot Service
Loaded: loaded (/etc/systemd/system/myservice.service; disabled; vendor preset: disabled)
Active: active (running) since Qui 2018-02-15 10:19:28 WET; 1h 24min ago
Main PID: 11142 (java)
Status: "health status: healthy, Liveness: correct; readiness: accepting traffic, Heap: 139.5 MiB/256.0 MiB, Old gen after GC: 48.2 MiB/256.0 MiB, Cgroup memory: 310.4 MiB, Threads: 31 (9 runnable; 14 waiting; 8 timed waiting), File descriptors: 64/524288, http-nio-8080: 8/8192 connections, 3/10 threads, Non-heap: 62.7 MiB/64.1 MiB, Buffers: 1.2 MiB/256.0 MiB direct (14), Classes: 7915"
CGroup: /system.slice/myservice.service
└─11142 /opt/jdk17/bin/java -XX:+ExitOnOutOfMemoryError -jar /opt/myservice/myservice.jar
Spring Boot status providers
On top of the ones of the core library, the Spring Boot module provides the following status providers:
| Class | Purpose | Sample | Details |
|---|---|---|---|
| ApplicationRunStatusProvider | Provides information regarding the application startup sequence state | State: context prepared |
Startup progress |
| ApplicationContextStatusProvider | Provides information regarding the bean creation status of the application context | Creating bean 94 of 472 |
Startup progress |
| TomcatStatusProvider | Provides information regarding Tomcat's connectors | http-nio-8080: 8/8192 connections, 3/10 threads |
tomcat |
| JettyStatusProvider | Provides the connections and threads of the embedded Jetty servers | jetty-8080: 12/1000 connections, 3/10 threads |
jetty |
| NettyStatusProvider | Provides the connections and requests of the embedded Reactor Netty servers | netty-8080: 12 connections, 3 requests |
netty |
| HikariStatusProvider | Provides the HikariCP connection pools, and their health | dataSource: 7/10 connections, 3 waiting |
hikari |
| TaskExecutorStatusProvider | Provides the thread pools of the task executors and schedulers | applicationTaskExecutor: 2/8 threads, 5 queued |
task-executors |
| LiquibaseStatusProvider | Provides the progress of the Liquibase database migrations run on startup | Liquibase: applying changeset 042-add-index (41 applied) |
liquibase |
| AvailabilityStatusProvider | Provides the liveness and readiness of the application | Liveness: correct; readiness: accepting traffic |
availability |
| ShutdownStatusProvider | Provides the stage the application is at while it shuts down | Shutdown: waiting for active requests |
shutdown |
| ActuatorHealthStatusProvider | Provides the application health, based on Spring Boot Actuator health contributors | health status: unhealthy (db) |
actuator-health |
See the Startup progress page for more information regarding ApplicationRunStatusProvider and ApplicationContextStatusProvider.
Enabling and disabling status providers
All built-in status providers are enabled by default, except for garbage-collection and cpu, which are opted into, to keep the status short. Use the systemd.status-providers.*.enabled properties to enable or disable them:
| Property | Status provider | Details |
|---|---|---|
systemd.status-providers.shutdown |
ShutdownStatusProvider | shutdown |
systemd.status-providers.heap |
HeapStatusProvider | |
systemd.status-providers.old-generation |
OldGenerationStatusProvider | |
systemd.status-providers.non-heap |
NonHeapStatusProvider | |
systemd.status-providers.buffer-pools |
BufferPoolStatusProvider | |
systemd.status-providers.garbage-collection |
GarbageCollectionStatusProvider | |
systemd.status-providers.class-loader |
ClassLoaderStatusProvider | |
systemd.status-providers.cpu |
CpuStatusProvider | |
systemd.status-providers.threads |
ThreadStatusProvider | threads |
systemd.status-providers.file-descriptors |
FileDescriptorStatusProvider | file-descriptors |
systemd.status-providers.cgroup-memory |
CgroupMemoryStatusProvider | |
systemd.status-providers.availability |
AvailabilityStatusProvider | availability |
systemd.status-providers.application-run |
ApplicationRunStatusProvider | Startup progress |
systemd.status-providers.application-context |
ApplicationContextStatusProvider | Startup progress |
systemd.status-providers.tomcat |
TomcatStatusProvider | tomcat |
systemd.status-providers.jetty |
JettyStatusProvider | jetty |
systemd.status-providers.netty |
NettyStatusProvider | netty |
systemd.status-providers.hikari |
HikariStatusProvider | hikari |
systemd.status-providers.task-executors |
TaskExecutorStatusProvider | task-executors |
systemd.status-providers.liquibase |
LiquibaseStatusProvider | liquibase |
systemd.status-providers.actuator-health |
ActuatorHealthStatusProvider | actuator-health |
The status providers of a library, such as jetty or hikari, are only set up when the library is available. For example, to stop reacting to bean creation events, and to show the garbage collection pauses:
systemd.status-providers.application-context.enabled=false
systemd.status-providers.garbage-collection.enabled=true
The PhaseStatusProvider is always set up, for the application to report what it is doing (see Custom status providers under Spring Boot below): it shows nothing until the application sets a phase.
Configuring status providers
Some status providers have settings of their own, next to enabled, under the same systemd.status-providers.<key>.
threads
The threads status provider shows the number of threads by state, such as Threads: 31 (9 runnable; 14 waiting; 8 timed waiting). It can also detect deadlocked threads, once opted into: their number is added to the status, such as 2 deadlocked. It is also a health provider, which can restart the application on deadlocks (see Health providers).
| Property | Default | Description |
|---|---|---|
systemd.status-providers.threads.deadlock-detection |
false |
Detect deadlocked threads, and report them in the status. |
file-descriptors
The file-descriptors status provider shows the open file descriptors out of the most the process can open, the LimitNOFILE of the service unit, such as File descriptors: 812/65536. It is also a health provider, which can restart the application when the file descriptors are about to run out (see Health providers).
tomcat
The tomcat status provider shows the usage of each connector of the embedded Tomcat:
- the open connections, out of the maximum the connector accepts (
server.tomcat.max-connections, 8192 by default), which includes idle keep-alive connections; - when the requests are handled by a thread pool, the busy threads, out of the threads in the pool.
So, http-nio-8080: 8/8192 connections, 3/10 threads means that the http-nio-8080 connector has 8 open connections, out of the 8192 it accepts, and that 3 of the 10 threads of its pool are serving requests.
With virtual threads (spring.threads.virtual.enabled=true), the requests are not handled by a thread pool, so only the connections are shown, such as http-nio-8080: 8/8192 connections. When the connector accepts unlimited connections (server.tomcat.max-connections=-1), Tomcat does not count them, so only the threads are shown, such as http-nio-8080: 3/10 threads.
When the management endpoints use a port of their own (management.server.port), the connector of their server is shown as well. The connectors are read directly from the embedded Tomcat, so the Tomcat MBeans do not need to be enabled (server.tomcat.mbeanregistry.enabled).
jetty
The jetty status provider shows the usage of each embedded Jetty server, named after the ports it listens on, such as jetty-8080, or jetty-8080+8443 for a server with several connectors:
- the open connections of all its connectors together, out of the maximum the server accepts when limited (
server.jetty.max-connections, unlimited by default); - when the requests are handled by a thread pool, the threads running requests, out of the threads in the pool. The threads Jetty keeps for itself, to accept connections and wait for their data, never run requests, and are left out of both.
So, jetty-8080: 12/1000 connections, 3/10 threads means that the server has 12 open connections, out of the 1000 it accepts, and that 3 of the 10 threads of its pool that can run requests are running one. Without a limit of connections, only their number is shown, such as jetty-8080: 12 connections, 3/10 threads. With virtual threads (spring.threads.virtual.enabled=true), the requests are not handled by a thread pool, so only the connections are shown.
When the management endpoints use a port of their own (management.server.port), their server is shown as well. A server is only shown once it listens.
netty
The netty status provider shows the usage of each embedded Reactor Netty server, as used by Spring WebFlux, named after the port it listens on: the open connections and the requests being handled, such as netty-8080: 12 connections, 3 requests. Reactor Netty handles requests on a few event loop threads, which never block, so there is no thread pool to report on, nor a maximum of connections.
Reactor Netty does not count its connections and requests unless it records metrics with Micrometer, so the status provider counts them itself, by customizing the servers Spring Boot creates: disabling it also stops counting them. When the management endpoints use a port of their own (management.server.port), their server is shown as well. A server is only shown once it listens.
hikari
The hikari status provider requires HikariCP, the connection pool Spring Boot uses by default. For each pool, named after its data source bean, it shows the connections in use out of the most the pool opens, and the threads waiting for a connection, such as dataSource: 7/10 connections, 3 waiting. A pool is only shown once started, which Hikari does on the first connection.
It is also a health provider, which can restart the application while a pool is starved, which usually means that connections leak (see Health providers).
task-executors
The task-executors status provider shows the thread pools of the task executors and schedulers of the application, the ThreadPoolTaskExecutor and ThreadPoolTaskScheduler beans, such as the ones Spring Boot sets up for @Async and @Scheduled methods. Each is named after its bean, and shows the busy threads out of the threads in the pool, and the tasks waiting for a thread, such as applicationTaskExecutor: 2/8 threads, 5 queued. For a scheduler, the tasks waiting are those scheduled to run, including the periodic ones between their runs, such as taskScheduler: 1/1 threads, 3 scheduled.
The threads in the pool are those it has created so far, not its maximum: a pool creates its threads as tasks arrive, so an idle one may show 0/0 threads. The beans are found once all singleton beans are created, so that lazy ones are not created for the status. With virtual threads (spring.threads.virtual.enabled=true), Spring Boot sets up task executors and schedulers without a thread pool, which are left out.
To show the thread pool of an executor that is not one of these beans, register an ExecutorStatusProvider for it (see Built-in status providers).
availability
The availability status provider shows the liveness and readiness of the application, as Spring Boot reports them, such as Liveness: correct; readiness: accepting traffic. It is also a health provider, which can restart the application while its liveness state is broken (see Health providers).
liquibase
The liquibase status provider requires Liquibase. It shows the progress of the database migrations run on startup by the SpringLiquibase beans, such as the one Spring Boot sets up, as each changeset is applied, such as Liquibase: applying changeset 042-add-index (41 applied). It is cleared once the migrations complete, and when they fail, the changeset that failed is still shown while the application fails to start.
Each changeset also extends the startup timeout (see Startup and shutdown timeouts), so that a migration with many changesets does not exceed the TimeoutStartSec of the service unit. A single changeset that takes longer than the extension still exceeds it: raise TimeoutStartSec for such a migration.
The changesets are reported through the customizer of the SpringLiquibase beans, which keeps the customizer they already have, such as a Customizer<Liquibase> bean of the application. A customizer that sets a ChangeExecListener of its own replaces the one of this provider, in which case only the start of the migrations is shown.
actuator-health
The actuator-health status provider requires Spring Boot Actuator. It shows the application health, as its health contributors report it, the same as the health endpoint (/actuator/health), with the same names, such as db, diskSpace, ping or redis.
It only names the unhealthy health contributors, such as health status: unhealthy (db, redis). Their details are left out, as they often contain sensitive information (error messages, host names, paths, …), and the status can be read by any local user, for instance with systemctl status, regardless of how Spring Boot Actuator exposes health details. The details are logged instead, when the watchdog heartbeat is suppressed.
It is also a health provider, which can restart the application while it is unhealthy. The status shows the health the watchdog takes into account, so the settings of the health provider, such as the health group whose members are taken into account, or the status codes considered as unhealthy, apply to the status as well (see Health providers).
shutdown
The shutdown status provider shows the stage the application is at while it shuts down, so that a slow shutdown can be told apart from a hung one. It shows nothing until the application starts shutting down, and then follows the shutdown of the application context:
Shutdown: stopping, once the context starts closing, while its life cycle beans stop, such as message listeners;Shutdown: waiting for active requests, while the embedded web server waits for the requests being handled to complete, when it shuts down gracefully, as it does by default (unlessserver.shutdown=immediate);Shutdown: destroying beans, once all life cycle beans are stopped, while the beans are destroyed, such as connection pools.
Every change of stage is sent to systemd right away. The status of the other status providers keeps being sent while the application shuts down, such as the connections of the web server, which drop as the requests complete. See Startup and shutdown timeouts for how long systemd lets the shutdown take.
Effects of disabling each status provider
| Key | Effect of disabling |
|---|---|
shutdown |
The stage of the shutdown is no longer part of the status message. Nothing else is affected. |
heap |
The heap memory usage is no longer part of the status message. Nothing else is affected. |
old-generation |
The old generation after garbage collection is no longer part of the status message. Nothing else is affected. |
non-heap |
The non-heap memory usage is no longer part of the status message. Nothing else is affected. |
buffer-pools |
The direct and mapped buffers are no longer part of the status message. Nothing else is affected. |
garbage-collection |
The garbage collection pauses are no longer part of the status message (the default). Nothing else is affected. |
class-loader |
The loaded class count is no longer part of the status message. Nothing else is affected. |
cpu |
The CPU usage is no longer part of the status message (the default). Nothing else is affected. |
threads |
The thread count is no longer part of the status message, and deadlocks are no longer detected, nor restart the application, even if opted into. |
file-descriptors |
The open file descriptors are no longer part of the status message, and running out of them no longer restarts the application, even if opted into. |
cgroup-memory |
The memory of the control group is no longer part of the status message. Nothing else is affected. |
availability |
The liveness and readiness of the application are no longer part of the status message, and a broken liveness state no longer restarts the application, even if opted into. |
tomcat |
Tomcat's connectors are no longer part of the status message. Nothing else is affected. |
jetty |
Jetty's connections and threads are no longer part of the status message. Nothing else is affected. |
netty |
Reactor Netty's connections and requests are no longer part of the status message, and they are no longer counted. Nothing else is affected. |
hikari |
The connection pools are no longer part of the status message, and a starved pool no longer restarts the application, even if opted into. |
task-executors |
The thread pools of the task executors and schedulers are no longer part of the status message. Nothing else is affected. |
liquibase |
The progress of the database migrations is no longer shown, and the startup timeout is no longer extended as changesets are applied. |
application-run |
The startup sequence state is no longer shown, and the startup timeout is no longer extended when the startup sequence changes state. |
application-context |
The bean creation progress is no longer shown, and the startup timeout is no longer extended while beans are being created. No notification is sent to systemd when beans are created during startup. |
actuator-health |
The application health is no longer part of the status message, and is no longer signalled to the watchdog, even if opted into. |
The application-run and application-context providers, along with liquibase while migrations run, are the only mechanism this library uses to extend the startup timeout (see Startup and shutdown timeouts). Disabling them means a slow startup can exceed the TimeoutStartSec of the service unit and be killed by systemd, even if it is making progress. If you disable them, it is your responsibility to ensure that startup completes within the timeout, by increasing TimeoutStartSec in the unit or by extending the timeout yourself, for instance by calling Systemd.extendTimeout() periodically, or Systemd.enableExtendTimeout(period, amount) and Systemd.disableExtendTimeout() on the Systemd instance.
Custom status providers under Spring Boot
Custom status providers, written as described in Custom status providers, are registered by declaring them as beans in any configuration class:
@Configuration
@ConditionalOnSystemd
public class MyConfiguration {
@Bean
MyStatusProvider myStatusProvider() {
return new MyStatusProvider();
}
}
In the example above, the configuration class does not need to be annotated with ConditionalOnSystemd, because these beans are harmless even when not running under systemd. However, this way these extra resources do not have to be built at all. Check the Conditionals page for more information.
The status shows first the startup status providers (ApplicationRunStatusProvider and ApplicationContextStatusProvider), as well as any other status provider registered directly in the Systemd instance before the application context is refreshed, in the order they were registered. Then come the StatusProvider beans of the application context, in the order given by the @Order annotation of their bean methods or classes.
The built-in status provider beans are ordered by how useful their information is when operating a service in production, 1000 apart, so that custom status providers can be placed between them. The critical ones, which tell whether the application is in trouble, use negative orders, and the others positive ones; 0 is left unused:
| Order | Status provider |
|---|---|
| -10000 | ShutdownStatusProvider |
| -9000 | PhaseStatusProvider |
| -8000 | ActuatorHealthStatusProvider |
| -7000 | AvailabilityStatusProvider |
| -6000 | HeapStatusProvider |
| -5000 | OldGenerationStatusProvider |
| -4000 | CgroupMemoryStatusProvider |
| -3000 | HikariStatusProvider |
| -2000 | ThreadStatusProvider |
| -1000 | FileDescriptorStatusProvider |
| 1000 | TomcatStatusProvider |
| 2000 | JettyStatusProvider |
| 3000 | NettyStatusProvider |
| 4000 | TaskExecutorStatusProvider |
| 5000 | CpuStatusProvider |
| 6000 | GarbageCollectionStatusProvider |
| 7000 | NonHeapStatusProvider |
| 8000 | BufferPoolStatusProvider |
| 9000 | ClassLoaderStatusProvider |
The LiquibaseStatusProvider is the exception: as it reports the migrations run while the beans are created, it is registered directly in the Systemd instance as soon as it is created, right after the startup status providers.
To replace a built-in status provider, such as to configure it in a way its properties do not allow, declare a bean of its type: the built-in one is then left out, so that the status does not show the same information twice. Give it the order of the built-in one, from the table above, to keep its place in the status:
@Bean
@Order(-2000)
ThreadStatusProvider myThreadStatusProvider() {
return new ThreadStatusProvider(true, true);
}
The LiquibaseStatusProvider cannot be replaced this way, as a bean of the application would not be registered as soon as it is created; disable it instead.
To report what the application is doing without writing a status provider, inject the PhaseStatusProvider bean and set its phase, which is sent to systemd right away. The bean only exists when running under systemd, so inject it as optional:
@Component
public class Reindexer {
private final ObjectProvider<PhaseStatusProvider> phase;
public Reindexer(ObjectProvider<PhaseStatusProvider> phase) {
this.phase = phase;
}
public void reindex() {
phase.ifAvailable(provider -> provider.setPhase("Reindexing"));
try {
// ...
} finally {
phase.ifAvailable(PhaseStatusProvider::clearPhase);
}
}
}
The Systemd instance can still be used directly, such as to send the status right away, with Systemd.instance().updateStatus().
Under Keycloak
The Keycloak provider registers the status providers of the JVM from the core library, with the same keys and in the same order as under Spring Boot, along with the stage of the startup and of the shutdown. See its page for how to choose which ones are shown.
