How To Set Up OpenTelemetry With Spring Boot For Observability

|

Observability in modern software systems is essential to ensure high performance, reliability, and faster troubleshooting. In this article, we explain how to set up OpenTelemetry with Spring Boot to collect comprehensive metrics, traces, and logs. We also discuss how to integrate these observability signals with Grafana, Loki, and Tempo—tools that provide powerful visualization and analysis capabilities.

OpenTelemetry is an open-source framework that standardizes the collection of telemetry data. It supports multiple signals like metrics, traces, and logs, and enables you to monitor distributed systems comprehensively. With Spring Boot being one of the most popular Java frameworks, integrating OpenTelemetry offers a great way to gain insights into your application’s behavior, identify bottlenecks, and resolve issues promptly.

In the following sections, we will cover:

  • Setting up a basic Spring Boot project

  • Instrumenting your code for metrics, traces, and logs

  • Configuring Grafana along with Loki and Tempo for a unified observability dashboard

  • Practical coding examples that illustrate each step in the process

Getting Started With a Spring Boot Application

The first step is to set up a Spring Boot application if you haven’t already. You can use Spring Initializr to generate a new project. Select dependencies such as:

  • Spring Web (for building REST APIs)

  • Spring Boot Actuator (for exposing application metrics)

  • Spring Boot Logging (for log management)

Once you have your application scaffolded, add the necessary dependencies for OpenTelemetry. In your Maven pom.xml or Gradle build file, include the following dependencies:

Maven Example:

xml
<dependency>
<groupId>io.opentelemetry.instrumentation</groupId>
<artifactId>opentelemetry-spring-boot-starter</artifactId>
<version>1.23.0</version>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-exporter-otlp</artifactId>
<version>1.23.0</version>
</dependency>

Gradle Example:

groovy
implementation 'io.opentelemetry.instrumentation:opentelemetry-spring-boot-starter:1.23.0'
implementation 'io.opentelemetry:opentelemetry-exporter-otlp:1.23.0'

These dependencies bring in the instrumentation libraries for Spring Boot and the OTLP exporter for sending telemetry data.

Instrumenting Metrics in Spring Boot

Spring Boot Actuator already exposes several application metrics. Integrating OpenTelemetry enhances these metrics further. OpenTelemetry can collect custom metrics by using Micrometer, an instrumentation library that acts as a facade for monitoring systems.

Example: Custom Counter Metric

In your Spring Boot application, create a component that registers a custom counter metric:

java
import io.micrometer.core.instrument.MeterRegistry;
import org.springframework.stereotype.Component;import javax.annotation.PostConstruct;@Component
public class MetricInitializer {private final MeterRegistry meterRegistry;public MetricInitializer(MeterRegistry meterRegistry) {
this.meterRegistry = meterRegistry;
}@PostConstruct
public void initMetrics() {
meterRegistry.counter(“custom_application_counter”, “env”, “production”).increment();
}
}

This code creates a custom counter metric that increments when the application starts. OpenTelemetry’s Micrometer integration will ensure that these metrics are automatically exported.

You may also register gauges, timers, and histograms to gain deeper insights into your application’s performance and resource utilization.

Instrumenting Traces in Spring Boot

Tracing is critical for distributed systems as it allows you to follow the path of a request through different microservices. OpenTelemetry makes it straightforward to instrument your code for distributed traces.

Example: Tracing a REST Controller

For example, you can instrument a REST controller as follows:

java
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;@RestController
public class DemoController {private final Tracer tracer;@Autowired
public DemoController(Tracer tracer) {
this.tracer = tracer;
}@GetMapping(“/api/demo”)
public String demoEndpoint() {
// Create a span for the incoming request.
Span span = tracer.spanBuilder(“demoEndpointSpan”).startSpan();
try {
// Simulate some processing within the request
Thread.sleep(50);
return “Demo response”;
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
return “Error occurred”;
} finally {
span.end(); // Ensure that span is ended
}
}
}

This snippet creates a new span when handling a /api/demo request, making it easier to trace the flow of execution and monitor the latency of operations.

Instrumenting Logs with OpenTelemetry

Logging, when combined with tracing and metrics, provides a rich context for debugging and performance monitoring. OpenTelemetry now supports log correlation, which means you can enrich logs with trace identifiers.

Example: Enriching Logs with Trace IDs

In your application.properties, you can add properties to configure logging:

properties
# Enable trace context propagation in log patterns
logging.pattern.console=%d{yyyy-MM-dd HH:mm:ss} - %msg [traceId:%X{traceId}]%n

Then, in your code or logging configuration, ensure that whenever a span starts, its context (specifically the traceId) is injected into the MDC (Mapped Diagnostic Context):

java
import org.slf4j.MDC;
import io.opentelemetry.api.trace.Span;public void logWithTrace() {
Span span = Span.current();
if (span != null) {
MDC.put(“traceId”, span.getSpanContext().getTraceId());
}
// Log as usual
logger.info(“Processing request with trace context”);
MDC.clear();
}

Integrating the trace context into your logs allows you to match logs with trace data, making it easier to analyze individual request flows when an issue occurs.

Integrating Grafana For Visualizing Metrics

Grafana is a powerful dashboarding and visualization tool. It allows you to connect to different data sources to visualize metrics, logs, and traces.

Setting Up Grafana:

  1. Install Grafana:
    You can download and install Grafana using their official installation guide.

  2. Add Data Sources:

    • Prometheus: For metrics collection, configure Prometheus to scrape endpoints exposed by Spring Boot Actuator.

    • Loki: For logs, Loki acts as a log aggregation system.

    • Tempo: For traces, Tempo is designed to work alongside Prometheus and Loki.

  3. Create Dashboards:
    Once data sources are added, you can create custom dashboards in Grafana. For instance, a dashboard can display:

    • Line charts for metrics like application response times, custom counters, and CPU usage.

    • Log panels that show logs enriched with trace identifiers.

    • Trace panels that visualize the distributed trace spans collected from Tempo.

Grafana Panel Example:

json
{
"title": "Application Response Time",
"type": "graph",
"targets": [
{
"expr": "rate(http_server_requests_seconds_sum[1m])",
"legendFormat": "Response Time"
}
],
"datasource": "Prometheus"
}

This Grafana panel configuration showcases how you can visualize the average response time of your application over a one-minute interval.

Configuring Loki for Log Aggregation

Loki is a log aggregation system designed to work seamlessly with Grafana. By integrating Loki with OpenTelemetry, you can view logs that are enriched with trace contexts.

Loki Configuration Example:

  1. Loki Setup:
    Download and install Loki from the official Loki releases. Configure your loki-config.yaml:

    yaml
    auth_enabled: false
    server:
    http_listen_port: 3100
    ingester:
    lifecycler:
    ring:
    kvstore:
    store: inmemory
    replication_factor: 1
    schema_config:
    configs:
    - from: 2020-10-24
    store: boltdb-shipper
    object_store: filesystem
    schema: v11
    index:
    prefix: index_
    period: 24h
    storage_config:
    boltdb_shipper:
    active_index_directory: /tmp/loki/index
    shared_store: filesystem
    filesystem:
    directory: /tmp/loki/chunks

  2. Spring Boot Log Back Configuration:
    Configure logback-spring.xml in your Spring Boot application to output logs in a format that Loki can parse:

    xml
    <configuration>
    <appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender">
    <lokiUrl>http://localhost:3100/loki/api/v1/push</lokiUrl>
    <encoder class="net.logstash.logback.encoder.LogstashEncoder">
    <customFields>{"app":"spring-boot-app"}</customFields>
    </encoder>
    </appender>
    <root level="INFO">
    <appender-ref ref="LOKI" />
    </root>
    </configuration>

This configuration sends application logs directly to Loki, where they can be queried and visualized in Grafana.

Configuring Tempo for Distributed Tracing

Tempo is Grafana’s distributed tracing backend designed to store trace data from applications. Tempo works well with OpenTelemetry by receiving trace spans exported via the OTLP protocol.

Steps to Set Up Tempo:

  1. Install Tempo:
    Tempo can be installed using Docker or via binary releases from the official Tempo GitHub repository. A basic Docker Compose file might look like:

    yaml
    version: '3'
    services:
    tempo:
    image: grafana/tempo:latest
    ports:
    - "3200:3200"
    command: -config.file=/etc/tempo/tempo.yaml
    volumes:
    - ./tempo-config.yaml:/etc/tempo/tempo.yaml

  2. Tempo Configuration Example:
    Create a tempo-config.yaml file:

    yaml
    server:
    http_listen_port: 3200
    distributor:
    receivers:
    otlp:
    protocols:
    grpc: {}
    http: {}
    ingester:
    trace_idle_period: 5m
    storage_config:
    boltdb_shipper:
    active_index_directory: /tmp/tempo/index
    shared_store: filesystem
    filesystem:
    directory: /tmp/tempo/chunks
    limits_config:
    enforce_metric_name: false

  3. Exporting Traces from Spring Boot:
    With the OpenTelemetry starter and OTLP exporter set up earlier in your Spring Boot application, traces will be forwarded to Tempo automatically (assuming your exporter endpoint is configured to point to http://localhost:3200):

    properties
    # application.properties
    otel.exporter.otlp.endpoint=http://localhost:3200
    otel.traces.exporter=otlp
    otel.metrics.exporter=otlp

This configuration allows your Spring Boot application to send trace data to Tempo, where it can later be correlated with logs in Grafana.

Testing Your Observability Setup

After configuring your Spring Boot application and observability tools, it’s essential to perform a series of tests to verify that:

  • Metrics: Check if custom metrics are accessible via the Actuator’s /actuator/prometheus endpoint.

  • Traces: Trigger endpoints in your application and verify that spans appear in Tempo’s UI.

  • Logs: Generate logs and validate that Loki is receiving them with the correct trace identifiers.

Use tools like curl or Postman to simulate HTTP requests. For example:

bash
curl http://localhost:8080/api/demo

Visit your Grafana dashboards to view the collected telemetry data, and use the integrated query tools to cross-reference logs and traces.

Best Practices and Considerations

When setting up observability using OpenTelemetry with Spring Boot, consider the following best practices:

  • Sampling Strategies:
    By default, tracing may collect an enormous amount of data. Configure sampling strategies to capture a representative set of data without overwhelming your storage systems.

  • Centralized Configuration:
    Keep your observability configuration in a central location (such as configuration management systems or environment variables) to ensure consistency across multiple deployments or microservices.

  • Resource Overhead:
    Be aware that adding telemetry can introduce extra overhead. Monitor the performance impact, and adjust the collection frequency or the level of detail as needed.

  • Security and Access Control:
    Ensure that your telemetry endpoints are secured and accessible only by authorized users to prevent unauthorized access or data exfiltration.

  • Integration Testing:
    Include observability configuration in your integration tests to verify that new updates or deployments do not break telemetry pipelines.

Conclusion

In conclusion, setting up OpenTelemetry with Spring Boot for observability is a powerful approach to ensure that your distributed application is fully monitored. With the ability to gather metrics, traces, and logs, you can quickly identify performance issues, track request flows, and gain deep insights into the health and behavior of your system.

We began by outlining the importance of observability and proceeded to configure a Spring Boot application with OpenTelemetry. Detailed examples demonstrated how to instrument your application for metrics, distributed traces, and enriched logging. You learned how to integrate these observability signals with popular tools such as Grafana, Loki, and Tempo—each component playing a crucial role in providing a unified monitoring solution.

Grafana serves as a central dashboard for visualizing metrics collected from Spring Boot Actuator and custom instrumentation. Loki aggregates logs, which are enriched with trace identifiers, thus bridging the gap between logs and traces. Tempo, on the other hand, captures distributed traces to provide a clear picture of the request lifecycle through your system. Together, these tools empower you to monitor your application effectively, troubleshoot problems, and improve overall system performance.

By following this guide, you have set a foundation for proactive monitoring and observability. This architecture not only enhances system reliability but also supports faster incident response, ultimately contributing to a better user experience. With observability integrated into your development lifecycle, you can iterate and improve your services with confidence, backed by clear, actionable telemetry data.

Investing time into establishing a robust observability pipeline using OpenTelemetry with Spring Boot is a step toward future-proofing your application infrastructure. As you scale and add more services, having a centralized observability framework will make it easier to pinpoint issues, maintain high service levels, and ensure an optimal user experience.