Duplicate Tags caused by the conversion between '.' and '_'

[zlsmith86]

Describe the bug
If a tag exists a.b and a_b, duplicate labels will be created in the prometheus output.

my_metric_name{a_b="alpha", a_b="alpha"}

Environment

• Micrometer version: 1.11.9
• Micrometer registry: 1.11.9
• OS: docker-alpine, and Windows
• Java version: 17

To Reproduce

@Test
public void TagTest() {
    Tags tags = Tags.of(Tag.of("a.b", "alpha"), Tag.of("a_b", "alpha"));
    assertThat(tags.stream()).hasSize(1);
}

Here is the current workaround that i put in place to account for this issue.

registry.config().meterFilter(
        new MeterFilter() {
            @Override
            public Meter.Id map(Meter.Id id) {

                HashSet<String> tagKeys = new HashSet<>();
                List<Tag> tags = new ArrayList<>();

                for (Tag tag : id.getTagsAsIterable()) {
                    if (!tagKeys.contains(tag.getKey().replace(".", "_"))) {
                        tags.add(tag);
                        tagKeys.add(tag.getKey().replace(".", "_"));
                    }
                    else {
                        System.out.println("Duplicate Tag:  " + tag.getKey());
                    }
                }

                return id.replaceTags(tags);
            }
        });

Expected behavior
duplicates will be detected in both output forms a.b and a_b

[shakuzen]

Do you not control the instrumentation that is defining the duplicate tag names? Note that Micrometer 1.11.x is no longer in OSS support - see our support page. With the newer Prometheus Java client (1.x) used since Micrometer 1.13, an exception will be thrown when scraping:

@Test
void duplicateTagNamesAfterNamingConvention() {
    Counter.builder("counter").tags("a.b", "alpha", "a_b", "alpha").register(registry).increment();
    System.out.println(registry.scrape()); // throws IllegalArgumentException: a_b: duplicate label name
}

The Tags class is not aware of naming conventions by design. The same Tags instance may be used with multiple different registries and naming conventions. Tags does deduplication, but a.b and a_b are different tag names.

If behavior were to change, it would be at the level of the PrometheusMeterRegistry or the Prometheus Java client, I think. It seems better to fix the instrumentation, though. What's the reason you would define tag names like a.b and a_b? For the scrape output with Micrometer 1.12 and earlier, how does the Prometheus server handle the duplicate labels when scraping?

Pinging @fstab in case you have any input on this from the Prometheus Java client's perspective.