SOLR-18060: Add Prometheus metrics to CrossDC Consumer. #4063

sigram · 2026-01-19T14:10:40Z

This PR replaces Dropwizard JSON metrics with Prometheus metrics in the CrossDC Consumer, using directly the Prometheus client_java API. It also removes the Dropwizard dependency.

mlbiscoc

Can you post a sample of all these metrics? Either dump it here or in a txt file? It would be easier to review the names and labels on the metrics.

gradle/libs.versions.toml

solr/cross-dc-manager/src/test/org/apache/solr/crossdc/manager/SolrAndKafkaIntegrationTest.java

solr/cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/PrometheusMetrics.java

mlbiscoc · 2026-01-20T15:36:52Z

solr/cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/PrometheusMetrics.java

+        Counter.builder()
+            .name("consumer_input_total")
+            .help("Total number of input messages")
+            .labelNames("type", "subtype")


I question most of these metrics really need type label. What is the cardinality of it and possible different combinations? I see in the test UPDATE is one. Is there also QUERY or something along those lines?

Yes, there's ADMIN and CONFIGSET.

Hmmm ok. I am not a fan of naming this label being called type. I think it should have some kind of context what it means as type and subtype can be very generic. Is it an operation or message_type maybe? Then what can subtype be? In core, I made it category but it is debateable if we should just remove that label/attribute all together from metrics. If you move type to something more specific then maybe you can just move off subtype to type. Again seeing an sample text output of these metrics would help if you can.

It's a request type - there are currently three types: UPDATE, ADMIN and CONFIGSET. Sub-type is primarily for UPDATE (add, dbi, dbq) and ADMIN (path).

Here's a sample output:

# HELP crossdc_consumer_input_total Total number of input messages # TYPE crossdc_consumer_input_total counter crossdc_consumer_input_total{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE"} 1.0 # HELP crossdc_consumer_output_total Total number of output requests # TYPE crossdc_consumer_output_total counter crossdc_consumer_output_total{otel_scope_name="org.apache.solr",result="handled",type="UPDATE"} 1.0 # HELP crossdc_consumer_output_batch_size Histogram of output batch sizes # TYPE crossdc_consumer_output_batch_size histogram crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="0.0"} 0 crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="5.0"} 1 crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="10.0"} 1 crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="25.0"} 1 crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="50.0"} 1 crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="75.0"} 1 crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="100.0"} 1 crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="250.0"} 1 crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="500.0"} 1 crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="750.0"} 1 crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="1000.0"} 1 crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="2500.0"} 1 crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="5000.0"} 1 crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="7500.0"} 1 crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="10000.0"} 1 crossdc_consumer_output_batch_size_bucket{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE",le="+Inf"} 1 crossdc_consumer_output_batch_size_count{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE"} 1 crossdc_consumer_output_batch_size_sum{otel_scope_name="org.apache.solr",subtype="add",type="UPDATE"} 1.0 # HELP crossdc_consumer_output_first_attempt_time_nanoseconds Histogram of first attempt request times # TYPE crossdc_consumer_output_first_attempt_time_nanoseconds histogram crossdc_consumer_output_first_attempt_time_nanoseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="0.0"} 0 crossdc_consumer_output_first_attempt_time_nanoseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="5000.0"} 0 crossdc_consumer_output_first_attempt_time_nanoseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="10000.0"} 0 crossdc_consumer_output_first_attempt_time_nanoseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="25000.0"} 0 crossdc_consumer_output_first_attempt_time_nanoseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="50000.0"} 0 crossdc_consumer_output_first_attempt_time_nanoseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="100000.0"} 0 crossdc_consumer_output_first_attempt_time_nanoseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="250000.0"} 0 crossdc_consumer_output_first_attempt_time_nanoseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="500000.0"} 0 crossdc_consumer_output_first_attempt_time_nanoseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="1000000.0"} 0 crossdc_consumer_output_first_attempt_time_nanoseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="2500000.0"} 0 crossdc_consumer_output_first_attempt_time_nanoseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="5000000.0"} 0 crossdc_consumer_output_first_attempt_time_nanoseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="2.5E7"} 0 crossdc_consumer_output_first_attempt_time_nanoseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="1.0E8"} 0 crossdc_consumer_output_first_attempt_time_nanoseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="1.0E9"} 0 crossdc_consumer_output_first_attempt_time_nanoseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="+Inf"} 1 crossdc_consumer_output_first_attempt_time_nanoseconds_count{otel_scope_name="org.apache.solr",type="UPDATE"} 1 crossdc_consumer_output_first_attempt_time_nanoseconds_sum{otel_scope_name="org.apache.solr",type="UPDATE"} 1.7667254470782164E18 # HELP crossdc_consumer_output_time_milliseconds Histogram of output request times # TYPE crossdc_consumer_output_time_milliseconds histogram crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="0.0"} 0 crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="5.0"} 0 crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="10.0"} 0 crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="25.0"} 1 crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="50.0"} 1 crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="75.0"} 1 crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="100.0"} 1 crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="250.0"} 1 crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="500.0"} 1 crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="750.0"} 1 crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="1000.0"} 1 crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="2500.0"} 1 crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="5000.0"} 1 crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="7500.0"} 1 crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="10000.0"} 1 crossdc_consumer_output_time_milliseconds_bucket{otel_scope_name="org.apache.solr",type="UPDATE",le="+Inf"} 1 crossdc_consumer_output_time_milliseconds_count{otel_scope_name="org.apache.solr",type="UPDATE"} 1 crossdc_consumer_output_time_milliseconds_sum{otel_scope_name="org.apache.solr",type="UPDATE"} 13.0 # TYPE target_info gauge target_info{service_name="unknown_service:java",telemetry_sdk_language="java",telemetry_sdk_name="opentelemetry",telemetry_sdk_version="1.56.0"} 1

...cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/KafkaCrossDcConsumer.java

solr/test-framework/src/java/org/apache/solr/util/SolrKafkaTestsIgnoredThreadsFilter.java

solr/cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/PrometheusMetrics.java

...cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/KafkaCrossDcConsumer.java

…ecord counts.

mlbiscoc · 2026-01-26T20:17:42Z

solr/cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/OtelMetrics.java

+    }
+    if (dbqCount > 0) {
+      outputBatchSizeHistogram.record(
+          dbiCount, attr(ATTR_TYPE, type.name(), ATTR_SUBTYPE, "delete_by_query"));


This should be dbqCount.

mlbiscoc · 2026-01-26T20:19:57Z

solr/cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/OtelMetrics.java

+  public void recordOutputBatchSize(MirroredSolrRequest.Type type, SolrRequest<?> solrRequest) {
+    if (type != MirroredSolrRequest.Type.UPDATE) {
+      outputBatchSizeHistogram.record(
+          1, attr(ATTR_TYPE, type.name(), ATTR_SUBTYPE, solrRequest.getPath()));


solrRequest.getPath() -> what is the cardinality of this? Probably not that concerning and I assume it is low but if there is many different paths this can get big.

It should be very small, basically only /admin/collections or /admin/configs. OTOH I can make sure it stays small by using a fixed value for subtype.

OTOH a more useful label here would be the action performed so that you can report what kind of ADMIN / CONFIGSET requests were seen.

mlbiscoc · 2026-01-26T20:22:33Z

solr/cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/OtelMetrics.java

+    outputTimeHistogram =
+        metricsContext.longHistogram(
+            NAME_PREFIX + "output_time",
+            "Histogram of output request times",
+            OtelUnit.MILLISECONDS);
+
+    outputFirstAttemptHistogram =
+        metricsContext.longHistogram(
+            NAME_PREFIX + "output_first_attempt_time",
+            "Histogram of first attempt request times",
+            OtelUnit.NANOSECONDS);


A nitpick but I think we should try to keep request times either one or the other as best as possible unless needed. Making dashboard or aggregations on different metrics of nanoseconds and milliseconds can get annoying. This is not a hard blocker though if you think we need nano and milli.

I changed the first attempt time to use millis, this should be enough.

mlbiscoc · 2026-01-26T20:22:58Z

solr/cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/OtelMetrics.java

+
+    outputBackoffHistogram =
+        metricsContext.longHistogram(
+            NAME_PREFIX + "output_backoff_size", "Histogram of output backoff sleep times");


This is a time but you called it size.

mlbiscoc · 2026-01-26T20:28:42Z

solr/cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/OtelMetrics.java

+    input =
+        metricsContext.longCounter(NAME_PREFIX + "input_total", "Total number of input messages");


I am not liking this metric very much. It is a bit confusing. For any non-update I see above you are just incrementing by 1 but at the same time this metric is incrementing by number of docs and/or deleted docs? The help message doesn't reflect that.

I think it should be consistent otherwise the way these increment is confusing. Make 2 metrics names if one is number of requests and the other is the number of docs/deleted docs. Maybe call it inputItems and other requests/messages

mlbiscoc · 2026-01-26T20:31:35Z

...cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/KafkaCrossDcConsumer.java

+          metrics.incrementOutputCounter(type.name(), "failed-dlq");
        } else {
          kafkaMirroringSink.submit(item);
-          metrics.counter(MetricRegistry.name(type.name(), "failed-resubmit")).inc();
+          metrics.incrementOutputCounter(type.name(), "failed-resubmit");


Do underscores instead of hyphens. I think that is the standard for OTEL and proemtheus

Also output looks to be more of a result rather than type. I think the name for this key should be result

Other thing I you could potentially do is change this metric. Instead have 2 success/failure. If success then successfully_outputs or some name like that then for failures or errors call is failed_outputs with a type label differentiating between the failures. Failed submit or failed rety etc.

gradle/libs.versions.toml

mlbiscoc · 2026-01-26T20:42:05Z

solr/cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/OtelMetrics.java

+  public static final String ATTR_SUBTYPE = "subtype";
+  public static final String ATTR_RESULT = "result";
+
+  protected final Map<String, Attributes> attributesCache = new ConcurrentHashMap<>();


What is this for?

mlbiscoc · 2026-01-26T20:45:38Z

solr/cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/OtelMetrics.java

+  public ConsumerTimer startOutputTimeTimer(final String requestType) {
+    final RTimer timer =
+        new RTimer(TimeUnit.MILLISECONDS) {
+          @Override
+          public double stop() {
+            double elapsedTime = super.stop();
+            outputTimeHistogram.record(
+                Double.valueOf(elapsedTime).longValue(), attr(ATTR_TYPE, requestType));
+            return elapsedTime;
+          }
+        };
+    return () -> timer.stop();
+  }


I had created these wrappers for doing this kind of thing. See AttributedLongTimer. Could you not use this?

mlbiscoc · 2026-01-26T20:46:02Z

solr/cross-dc-manager/src/java/org/apache/solr/crossdc/manager/consumer/Util.java

+    //    List<MetricSnapshot> snapshotList =
+    //        metricManager.getPrometheusMetricReaders().values().stream()
+    //            .flatMap(r -> r.collect().stream())
+    //            .toList();
+    //    MetricSnapshots snapshots = MetricSnapshots.of(snapshotList.toArray(new
+    // MetricSnapshot[0]));
+    //    String output;
+    //    try {
+    //      ByteArrayOutputStream baos = new ByteArrayOutputStream();
+    //      new PrometheusTextFormatWriter(false).write(baos, snapshots);
+    //      output = baos.toString();
+    //    } catch (Exception e) {
+    //      log.error("Error while writing final metrics", e);
+    //      output = snapshots.toString();
+    //    }
+    //    log.info("#### Consumer Metrics: ####\n{}", output);


Delete this?

sigram added 3 commits January 15, 2026 11:47

WIP.

a99dd06

SOLR-18060: Add Prometheus metrics to CrossDC Consumer.

aff9064

Add unit test, fix some bugs.

a42205a

sigram requested a review from mlbiscoc January 19, 2026 14:10

github-actions bot added dependencies Dependency upgrades test-framework tool:build tests labels Jan 19, 2026

sigram mentioned this pull request Jan 19, 2026

SOLR-18060: CrossDC Consumer - add Prometheus metrics. #4044

Closed

sigram added 4 commits January 20, 2026 13:25

Tidy.

5eef271

Add license and notice.

c569ff6

Fix a logging call.

82531d5

Remove unused licenses.

a7eee83

mlbiscoc reviewed Jan 20, 2026

View reviewed changes

Use OTEL API instead of Prometheus API.

13a8a53

github-actions bot removed the test-framework label Jan 22, 2026

sigram added 2 commits January 22, 2026 19:10

Fix first time latency calculation. Clean up metric names. Properly r…

b87ab4b

…ecord counts.

Tidy.

65fa041

github-actions bot added the cat:search label Jan 26, 2026

mlbiscoc reviewed Jan 26, 2026

View reviewed changes

		input =
		metricsContext.longCounter(NAME_PREFIX + "input_total", "Total number of input messages");

SOLR-18060: Add Prometheus metrics to CrossDC Consumer. #4063

Are you sure you want to change the base?

SOLR-18060: Add Prometheus metrics to CrossDC Consumer. #4063

Conversation

sigram commented Jan 19, 2026

Uh oh!

mlbiscoc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants