Telemetry — Spring Boot Apps

4 min readAug 10, 2020

Telemetry helps us learn what is going on inside the application. Its data can empower both operators and the development team to detect trends or issues in a live application.

The sheer ability to correlate data from different sources in a single dashboard can be a game changer!

For example

What pages in the application do users visit most? Where do they spend most of their time in the application?
Multiple applications are currently showing slowness, is the slowness coming from the OAuth server?
There is a high rate of exception count with our application. This can indicate that something is going wrong, can this trigger an alert for immediate attention by operators?

We will be able to answer these questions with Telemetry data.

Spring Boot Actuator has a solution to Telemetry via Actuator endpoints. Let’s explore it. Sample code for this article in github is here.

Please note: telemetry endpoints that we are going to build in this blog will have tons of sensitive information and should be secured as explained here

Spring’s Actuator

Spring’s Actuator is used to expose operational information about the running application — health, metrics, info, dump, env, etc.
Let’s add the Actuator dependency

implementation 'org.springframework.boot:spring-boot-starter-actuator'

add below entry to application.properties

management.endpoints.web.exposure.include=health,info,metrics,prometheus

this will generate /actuator/metrics endpoint with different types of metrics

{
    "names": [
        "http.server.requests",
        "jvm.buffer.count",
        "jvm.buffer.memory.used",
        "jvm.buffer.total.capacity",
        "jvm.classes.loaded",
        "jvm.classes.unloaded",
        "jvm.gc.live.data.size",
        "jvm.gc.max.data.size",
        "jvm.gc.memory.allocated",
        "jvm.gc.memory.promoted",
        "jvm.gc.pause",
        "jvm.memory.committed",
        "jvm.memory.max",
        "jvm.memory.used",
        "jvm.threads.daemon",
        "jvm.threads.live",
        "jvm.threads.peak",
        "jvm.threads.states",
        "logback.events",
        "process.cpu.usage",
        "process.files.max",
        "process.files.open",
        "process.start.time",
        "process.uptime",
        "system.cpu.count",
        "system.cpu.usage",
        "system.load.average.1m",
        "tomcat.sessions.active.current",
        "tomcat.sessions.active.max",
        "tomcat.sessions.alive.max",
        "tomcat.sessions.created",
        "tomcat.sessions.expired",
        "tomcat.sessions.rejected"
    ]
}

Sample response from /actuator/metrics/http.server.requests endpoint is

{
    "name": "http.server.requests",
    "description": null,
    "baseUnit": "seconds",
    "measurements": [
        {
            "statistic": "COUNT",
            "value": 6.0
        },
        {
            "statistic": "TOTAL_TIME",
            "value": 0.18752140099999998
        },
        {
            "statistic": "MAX",
            "value": 0.0
        }
    ],
    "availableTags": [
        {
            "tag": "exception",
            "values": [
                "None"
            ]
        },
        {
            "tag": "method",
            "values": [
                "GET"
            ]
        },
        {
            "tag": "uri",
            "values": [
                "/book/{id}",
                "/actuator/health",
                "/actuator/metrics"
            ]
        },
        {
            "tag": "outcome",
            "values": [
                "SUCCESS"
            ]
        },
        {
            "tag": "status",
            "values": [
                "200"
            ]
        }
    ]
}

pay close attention to COUNT above, it shows the number of http calls made so far to our application.

Prometheus library

To produce data in Prometheus’s format, add below dependency

implementation 'io.micrometer:micrometer-registry-prometheus'

this creates a new endpoint /actuator/prometheus with below sample response

# HELP http_server_requests_seconds  
# TYPE http_server_requests_seconds summary
http_server_requests_seconds_count{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/book/{id}",} 4.0
http_server_requests_seconds_sum{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/book/{id}",} 0.217755599
http_server_requests_seconds_count{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/actuator/prometheus",} 1.0
http_server_requests_seconds_sum{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/actuator/prometheus",} 0.111382704
http_server_requests_seconds_count{exception="None",method="POST",outcome="SUCCESS",status="200",uri="/book",} 2056.0
http_server_requests_seconds_sum{exception="None",method="POST",outcome="SUCCESS",status="200",uri="/book",} 0.071888396
# HELP http_server_requests_seconds_max  
# TYPE http_server_requests_seconds_max gauge
http_server_requests_seconds_max{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/book/{id}",} 0.206603665
http_server_requests_seconds_max{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/actuator/prometheus",} 0.0
http_server_requests_seconds_max{exception="None",method="POST",outcome="SUCCESS",status="200",uri="/book",} 0.069436448

If we look at this closely, this is showing us that

the number of POST /books made is 2056
max response time of these calls is 0.069436448 seconds.

This information can be visualized on Prometheus server in a graphical form. Let’s install Prometheus in our local.

Visualizing in Prometheus

First we need to download Prometheus docker image

docker pull prom/prometheus

Create a new prometheus.yml file, sample is here. Make sure you replace YOUR_IP_ADDRESS with your workstation’s ip address in this file.
Run the docker container using the below command

docker run -d --name=prometheus -p 9090:9090 -v <PATH_TO_prometheus.yml_FILE>:/etc/prometheus/prometheus.yml prom/prometheus --config.file=/etc/prometheus/prometheus.yml

Make sure to replace the <PATH_TO_prometheus.yml_FILE> with the PATH where you have stored the Prometheus configuration file.
You can now navigate to http://localhost:9090 to explore the Prometheus dashboard.

Custom Metrics

We can also add custom metrics into our application. There are three main types of custom metrics: Timer, Counter and Gauge. There are additional ones too, which we will not be covering in this article. This site covers them well.

Timer

Timers are intended for measuring short-duration latencies.
To add Timer, add Spring AOP dependency

implementation 'org.springframework.boot:spring-boot-starter-aop'

add a new configuration class

@Configuration
@EnableAspectJAutoProxy
public class TimerConfiguration {
    @Bean
    public TimedAspect timedAspect(MeterRegistry registry) {
        return new TimedAspect(registry);
    }
}

now, we can add @Timed annotation to any method.

Let’s add it to a controller method

@Timed("save_book")
public Book saveBook(@RequestBody Book book) {

That’s it, below metrics will appear in /actuator/prometheus endpoint

# HELP save_book_seconds_max  
# TYPE save_book_seconds_max gauge
save_book_seconds_max{class="com.phani.spring.cloudcontracts.BooksController",exception="none",method="saveBook",} 0.00613817
# HELP save_book_seconds  
# TYPE save_book_seconds summary
save_book_seconds_count{class="com.phani.spring.cloudcontracts.BooksController",exception="none",method="saveBook",} 1.0
save_book_seconds_sum{class="com.phani.spring.cloudcontracts.BooksController",exception="none",method="saveBook",} 0.00613817

Gauge

A gauge is to report current value. Typical examples for gauges would be the size of a collection or a map.
Adding Gauge is easy. In the below example, it is used to report the number of books returned in GET /books call

@GetMapping("/books")
public ResponseEntity<List<Book>> searchBooks(@RequestParam String name) {
    List<Book> bookList = findByName(name);
    Gauge.builder("books.list.size", bookList, List::size).register(meterRegistry);
    return ResponseEntity.ok(bookList);
}

After this change, below metric now appears in /actuator/prometheus endpoint

books_list_size 3.0

Counter

Counters can be used to report counts, its interface allows you to increment by a fixed amount. To add Counter to our application, let’s add MeterRegistry bean to the controller.

@Autowired
private MeterRegistry meterRegistry;

Add the counter variable and initialize it

private Counter booksCounter;@PostConstruct
public void initialize() {
  booksCounter = meterRegistry.counter("books_count");
}

We can now use it. In the below example, counter is incremented whenever saveBook() method is called

@PostMapping("/book")
public Book saveBook(@RequestBody Book book) {
    book.setId(++counter);
    bookMap.put(book.getId(), book);
    booksCounter.increment();
    return book;
}

After these changes, below metric now appears in /actuator/prometheus endpoint