Disclaimer: I originally published this article on DZone.
Metrics emitted from applications might contain parameters (i.e., tags or labels) for which a specific metric is measured. Micrometer provides a facade over the instrumentation clients for many widespread monitoring systems like Atlas, Datadog, Graphite, Ganglia, Influx, JMX, and Prometheus. This article's scope explores some extra practices when combining Micrometer and Quarkus to tailor those metrics or give you some ideas of what you could extra measure.
For simplicity and given Prometheus popularity, the Micrometer Prometheus registry will be used as an example to demonstrate most of the use cases.
Prerequisites
You can apply the concepts from this article by generating your project via https://code.quarkus.io/.
JDK 11 installed with JAVA_HOME configured appropriately.
If interested in concepts about working with Quarkus and Micrometer, please take a look at https://quarkus.io/guides/micrometer.
To further observe the metrics in Prometheus and creating Grafana panels, please consider provisioning those tools. However, you can run the code locally as well.
Global Tags Setup
By simply adding micrometer-registry-prometheus extension as a dependency to your Quarkus code, your application code will benefit from some Out Of The Box metrics, accessible on localhost via localhost:8080/q/metrics/q/metrics.
In case your application is deployed across multiple regions, and the business logic inside it needs to consider specific details (like connecting to third-party services that offer data in different formats), you can add your own global tags to help further you inspecting performance statistics per region. In the example below, the global metrics should be all properties prefixed with global and defined separately in a class.
Furthermore, those configuration properties are defined in application.properties file with a default value (for local development work), but the expectation is for them to be configured via environment variables passed from Docker or Kubernetes.
global.region=${REGION:CEE}
global.custom-meter=${CUSTOM_METER:''}The last step was to instruct the Micrometer extension about the CustomConfiguration and to use its MeterFilter CDI beans when initializing MeterRegistry instances:
A transform filter is used to add a name prefix and an additional tag conditionally to meters, starting with the runtime environment's name.
These metrics can help generate insights about how a particular part of your application performed for a region. In Grafana, you can display a graph of database calls via a specific function (tagged by database_calls_find_total ) per region by the query:
sum by(region)(rate(database_calls_find_total[5m]))Measuring Database Calls
If you are trying to measure the database calls, you may want to do the following:
Add @Counted annotation on top of the methods to check the number of invocations.
Add @Timed annotation on top of the expected methods to run long-running operations and deserve a closer inspection.
In the example below, @Counted was added to trace the number of invocations done:
This metric can further help in observing the distribution in a Grafana panel based on the following query:
sum(increase(database_calls_add_total[1m]))In the case of complex database queries or where multiple filters are applied, adding @Timed would give insights into those behaviors under the workload. In the example below, filtering messages by language is expected to be a long-running task:
Measuring Performance of Requests
Depending on the complexity of the methods from your API and how you would like to measure their performance, you may want to do the following:
Add @Counted annotation on top of the methods to check the number of invocations.
Add @Timed annotation on top of the expected methods to run long-running operations and deserve a closer inspection.
Measure code performance with complex business (present either inside those methods or in separate classes); you might want to consider a dynamic tagging strategy.
The simplest scenario for an API method is to be straightforward in its behavior. In such a case, applying the @Counted and/or @Timed should be enough, and of course, more tags can be added, like in the example below:
Yet, implementations with more business logic would need separate metrics with more tags. The goal in such cases is to reuse some metrics and decorate them with more tags. For example, the method below should have additional metrics attached to it to measure the performance of each greeting handled in a given language:
For the previous API request, the average greeting retrieved over time can be measured in a Grafana panel through the query:
avg(increase(greetings_specific_seconds_duration_sum[1m])) without(class, endpoint, pod,instance,job,namespace,method,service,exception,uri)But the above method can also get no result for the given content, and that case should be closely observed. For this extra case, a custom Counter was implemented where firstly the existence of a Counter with similar features is checked and if not, increment a new instance of Counter with additional tags attached:
Note: Similar customization was be applied for DynamicTaggedTimer.
In the ExampleEndpoint, the following changes were added:
Based on the above implementation, the ratio of empty results requests can be outlined in Grafana via the query:
sum(increase(another_requests_count_total{custom_api_greet="empty"}[5m]))/sum(increase(another_requests_count_total[5m]))When multiple tags and values need to be added per action, the previous implementation for DynamicTaggedCounter got additional customizations: DynamicMultiTaggedCounter and DynamicMultiTaggedTimer. The DynamicMultiTaggedTimer below checks, if there is an existing Timer a with the same tags and values, and if it returns a new instance of a Timer with additional tags attached:
The above-implementation is used below to determine the creation of exceptional content of messages in a different language:
A Grafana panel can be created to some exceptional messages by language and content:
sum by (language, custom_api_greet)(increase(other_requests_total{custom_api_greet=~"exceptional.+"}[1m]))Final Thoughts
For sure, you can do many more tweaks with Micrometer, yet I hope this piqued your interest in Quarkus and Micrometer. The code is available at https://github.com/ammbra/micrometering-with-quarkus.