# Metrics reference

TAG exposes Prometheus metrics at the `/metrics` endpoint.

## Accessing metrics[​](#accessing-metrics "Direct link to Accessing metrics")

```
# Local

curl http://localhost:8080/metrics



# Kubernetes (port-forward)

kubectl port-forward svc/tag 8080:8080

curl http://localhost:8080/metrics
```

## Request metrics[​](#request-metrics "Direct link to Request metrics")

### tag\_requests\_total[​](#tag_requests_total "Direct link to tag_requests_total")

**Type:** Counter

Total number of requests processed by TAG.

| Label       | Description                                                          |
| ----------- | -------------------------------------------------------------------- |
| `operation` | S3 operation: `GetObject`, `PutObject`, `DeleteObject`, `HeadObject` |
| `status`    | Result: `success`, `error`, `auth_error`, `range_not_satisfiable`    |

```
# Request rate by operation

rate(tag_requests_total[5m])



# Error rate

sum(rate(tag_requests_total{status="error"}[5m])) / sum(rate(tag_requests_total[5m]))



# GetObject success rate

rate(tag_requests_total{operation="GetObject",status="success"}[5m]) /

rate(tag_requests_total{operation="GetObject"}[5m])
```

### tag\_request\_duration\_seconds[​](#tag_request_duration_seconds "Direct link to tag_request_duration_seconds")

**Type:** Histogram

Request duration in seconds.

| Label       | Description  |
| ----------- | ------------ |
| `operation` | S3 operation |

```
# P50 latency

histogram_quantile(0.5, rate(tag_request_duration_seconds_bucket[5m]))



# P99 latency by operation

histogram_quantile(0.99, sum(rate(tag_request_duration_seconds_bucket[5m])) by (operation, le))
```

## Cache metrics[​](#cache-metrics "Direct link to Cache metrics")

### tag\_cache\_hits\_total[​](#tag_cache_hits_total "Direct link to tag_cache_hits_total")

**Type:** Counter — every request served from cache, including range-from-cache hits and conditional 304 (Not Modified) responses. Recorded in lockstep with the `X-Cache: HIT` header.

### tag\_cache\_misses\_total[​](#tag_cache_misses_total "Direct link to tag_cache_misses_total")

**Type:** Counter — total number of cache misses (recorded in lockstep with `X-Cache: MISS`).

These two counters count only `HIT` and `MISS`. `REVALIDATED` responses (object changed on upstream) are neither — they are tracked by the `tag_revalidations_*` metrics — and `BYPASS`/`DISABLED` requests are not counted at all.

### tag\_cache\_operations\_total[​](#tag_cache_operations_total "Direct link to tag_cache_operations_total")

**Type:** Counter

| Label       | Description                               |
| ----------- | ----------------------------------------- |
| `operation` | Operation type: `get`, `put`, `delete`    |
| `result`    | Result: `hit`, `miss`, `success`, `error` |

```
# Cache hit ratio (of hit/miss decisions; excludes REVALIDATED)

rate(tag_cache_hits_total[5m]) /

(rate(tag_cache_hits_total[5m]) + rate(tag_cache_misses_total[5m]))



# Cache operation breakdown

sum by (operation, result) (rate(tag_cache_operations_total[5m]))
```

### tag\_range\_from\_cache\_hits\_total[​](#tag_range_from_cache_hits_total "Direct link to tag_range_from_cache_hits_total")

**Type:** Counter — number of range requests served from cached full objects.

### tag\_cache\_serve\_locality\_total[​](#tag_cache_serve_locality_total "Direct link to tag_cache_serve_locality_total")

**Type:** Counter — cache body reads labeled by whether this node owns the key (`locality="local"`, served from local storage) or had to pull it from a peer over gRPC (`locality="remote"`). In a cluster the single-owner consistent-hash ring routes most reads to the owning node, so a high `remote` share is the cross-node data-plane cost. Single-node mode is always `local`.

| Label      | Description                                                       |
| ---------- | ----------------------------------------------------------------- |
| `locality` | `local` (this node owns the key) or `remote` (pulled from a peer) |

```
# Remote serve ratio (fraction of cache reads pulled cross-node; lower is better).

# Aggregate with sum() so the denominator is local+remote; dividing the raw

# vectors would match locality="remote" against itself and always report 1.

sum(rate(tag_cache_serve_locality_total{locality="remote"}[5m]))

/

sum(rate(tag_cache_serve_locality_total[5m]))



# Cross-node read rate per node

sum by (pod) (rate(tag_cache_serve_locality_total{locality="remote"}[5m]))
```

Only populated when the embedded cache client can report key ownership (cluster mode). When it cannot, the counter stays at 0 rather than guessing a locality.

### tag\_cache\_size\_bytes[​](#tag_cache_size_bytes "Direct link to tag_cache_size_bytes")

**Type:** Gauge — current logical size of **this node's** local cache in bytes (sum of stored object lengths). Per-node; sum across nodes for a cluster-wide total.

```
# Cluster-wide cache size

sum(tag_cache_size_bytes)
```

The embedded cache also exports `ocache_disk_usage_bytes{type="total"}` (same logical size) and `ocache_segment_size_bytes` (physical on-disk segment bytes) directly; `tag_cache_size_bytes` is the stable, TAG-owned name for the logical size.

Logical size is smaller than the space the cache occupies on disk. Objects that are overwritten, deleted, or expired leave their bytes behind inside the storage files until background compaction rewrites those files. To see whether that space is being reclaimed, compare the filesystem usage against this gauge, and watch the compaction metrics below.

### ocache\_segment\_walks\_total[​](#ocache_segment_walks_total "Direct link to ocache_segment_walks_total")

**Type:** Counter — storage files examined to determine how much of each one is still in use. Compaction uses the result to decide what to rewrite.

```
# Examination rate

rate(ocache_segment_walks_total[10m])
```

Zero on a node that has been running for more than two hours suggests compaction is disabled.

### ocache\_recompaction\_segments\_total[​](#ocache_recompaction_segments_total "Direct link to ocache_recompaction_segments_total")

**Type:** Counter — storage files rewritten to reclaim space, with `ocache_recompaction_bytes_freed_total` reporting the bytes recovered.

```
# Space reclaimed per hour

increase(ocache_recompaction_bytes_freed_total[1h])
```

If disk usage grows while this counter stays flat, space is not being reclaimed. See the deployment guide for the alert to configure.

## Block cache metrics[​](#block-cache-metrics "Direct link to Block cache metrics")

Block-aligned caching is on by default, so these apply to most deployments. They describe cache behaviour at block granularity, which is what a range read actually touches.

### tag\_cache\_block\_hits\_total / tag\_cache\_block\_misses\_total[​](#tag_cache_block_hits_total--tag_cache_block_misses_total "Direct link to tag_cache_block_hits_total / tag_cache_block_misses_total")

**Type:** Counters — blocks served from cache, and blocks that had to be fetched. Together they give the ratio that matters for range-read workloads.

```
# Block cache hit ratio

sum(rate(tag_cache_block_hits_total[30m]))

/

(sum(rate(tag_cache_block_hits_total[30m])) + sum(rate(tag_cache_block_misses_total[30m])))
```

A hit ratio that will not rise despite a warm cache usually means `block_size` does not match the workload's read granularity. See [Configuration](/docs/acceleration-gateway/configuration/.md).

### tag\_cache\_block\_bytes\_populated\_total[​](#tag_cache_block_bytes_populated_total "Direct link to tag_cache_block_bytes_populated_total")

**Type:** Counter — bytes fetched from upstream into blocks. Divide by bytes served to clients to measure read amplification:

```
# Upstream read amplification. Aim for <= 1.

sum(rate(tag_cache_block_bytes_populated_total[5m]))

/

sum(rate(tag_bytes_transferred_total{direction="out"}[5m]))
```

Well above 1 means the block size is too large for the read pattern — every miss pulls a full block to serve a small range. Do **not** substitute `tag_bytes_transferred_total{direction="in"}` here: that counts client upload bodies, not upstream fetches.

### tag\_cache\_block\_prefetched\_total[​](#tag_cache_block_prefetched_total "Direct link to tag_cache_block_prefetched_total")

**Type:** Counter, labelled by `trigger` — blocks fetched speculatively rather than because a client asked for them.

```
# Speculative fetching by trigger

sum by (trigger) (rate(tag_cache_block_prefetched_total[30m]))
```

Judge this against the block hit ratio above, not on its own. Volume rising without the hit ratio rising means the speculation is not landing where reads go.

### tag\_cache\_parquet\_footer\_bytes[​](#tag_cache_parquet_footer_bytes "Direct link to tag_cache_parquet_footer_bytes")

**Type:** Histogram — the size of parquet metadata footers observed. Recorded for every parquet object whose trailer is read, including ones that are not prefetched, so it describes the whole population.

```
# Median footer size, in bytes.

histogram_quantile(0.5, sum(rate(tag_cache_parquet_footer_bytes_bucket[1h])) by (le))
```

This is the measurement that tells you whether [Parquet optimization](/docs/acceleration-gateway/parquet-optimization/.md) is worth enabling. Compare it against the tail block rather than the full `block_size` — the tail averages half a block, so `block_size / 2` is the practical yardstick.

## Broadcast metrics[​](#broadcast-metrics "Direct link to Broadcast metrics")

### tag\_broadcast\_shared\_total[​](#tag_broadcast_shared_total "Direct link to tag_broadcast_shared_total")

**Type:** Counter — requests that joined an existing broadcast stream.

### tag\_broadcast\_fetches\_total[​](#tag_broadcast_fetches_total "Direct link to tag_broadcast_fetches_total")

**Type:** Counter — upstream fetches (broadcast initiators).

### tag\_broadcast\_slow\_consumers\_total[​](#tag_broadcast_slow_consumers_total "Direct link to tag_broadcast_slow_consumers_total")

**Type:** Counter — listeners disconnected for being too slow.

### tag\_active\_broadcasts[​](#tag_active_broadcasts "Direct link to tag_active_broadcasts")

**Type:** Gauge — currently active broadcast streams.

```
# Coalescing ratio (higher is better)

rate(tag_broadcast_shared_total[5m]) /

(rate(tag_broadcast_shared_total[5m]) + rate(tag_broadcast_fetches_total[5m]))
```

## Background fetch metrics[​](#background-fetch-metrics "Direct link to Background fetch metrics")

### tag\_background\_fetches\_triggered\_total[​](#tag_background_fetches_triggered_total "Direct link to tag_background_fetches_triggered_total")

**Type:** Counter — background full-object fetches triggered by range requests.

### tag\_background\_fetches\_succeeded\_total[​](#tag_background_fetches_succeeded_total "Direct link to tag_background_fetches_succeeded_total")

**Type:** Counter — background fetches completed successfully.

### tag\_background\_fetches\_failed\_total[​](#tag_background_fetches_failed_total "Direct link to tag_background_fetches_failed_total")

**Type:** Counter — background fetches that failed.

### tag\_active\_background\_fetches[​](#tag_active_background_fetches "Direct link to tag_active_background_fetches")

**Type:** Gauge — currently active background fetches.

### tag\_warm\_on\_write\_triggered\_total[​](#tag_warm_on_write_triggered_total "Direct link to tag_warm_on_write_triggered_total")

**Type:** Counter — cache warms triggered by a successful write (when `cache.warm_on_write` is enabled). The warm's own outcome is recorded by the `tag_background_fetches_*` metrics.

```
# Background fetch success rate

rate(tag_background_fetches_succeeded_total[5m]) /

rate(tag_background_fetches_triggered_total[5m])
```

## Revalidation metrics[​](#revalidation-metrics "Direct link to Revalidation metrics")

### tag\_revalidations\_triggered\_total[​](#tag_revalidations_triggered_total "Direct link to tag_revalidations_triggered_total")

**Type:** Counter — cache revalidation attempts (conditional GET/HEAD to upstream).

### tag\_revalidations\_not\_modified\_total[​](#tag_revalidations_not_modified_total "Direct link to tag_revalidations_not_modified_total")

**Type:** Counter — revalidations where upstream returned 304 Not Modified.

### tag\_revalidations\_updated\_total[​](#tag_revalidations_updated_total "Direct link to tag_revalidations_updated_total")

**Type:** Counter — revalidations where upstream returned 200 with new data.

### tag\_revalidations\_failed\_total[​](#tag_revalidations_failed_total "Direct link to tag_revalidations_failed_total")

**Type:** Counter — revalidations that failed due to errors.

### tag\_revalidations\_stale\_served\_total[​](#tag_revalidations_stale_served_total "Direct link to tag_revalidations_stale_served_total")

**Type:** Counter — times stale cached data was served because revalidation failed.

```
# Revalidation 304 ratio (higher = better cache freshness)

rate(tag_revalidations_not_modified_total[5m]) /

rate(tag_revalidations_triggered_total[5m])



# Stale serve ratio (should be low)

rate(tag_revalidations_stale_served_total[5m]) /

rate(tag_revalidations_triggered_total[5m])
```

## Upstream metrics[​](#upstream-metrics "Direct link to Upstream metrics")

### tag\_upstream\_request\_duration\_seconds[​](#tag_upstream_request_duration_seconds "Direct link to tag_upstream_request_duration_seconds")

**Type:** Histogram — upstream (Tigris) request duration in seconds.

| Label    | Description                                 |
| -------- | ------------------------------------------- |
| `method` | HTTP method: `GET`, `PUT`, `DELETE`, `HEAD` |

### tag\_upstream\_errors\_total[​](#tag_upstream_errors_total "Direct link to tag_upstream_errors_total")

**Type:** Counter — total upstream errors.

| Label    | Description |
| -------- | ----------- |
| `method` | HTTP method |

## Authentication metrics[​](#authentication-metrics "Direct link to Authentication metrics")

### tag\_auth\_failures\_total[​](#tag_auth_failures_total "Direct link to tag_auth_failures_total")

**Type:** Counter

| Label    | Description                                                   |
| -------- | ------------------------------------------------------------- |
| `reason` | Failure reason: `invalid_signature`, `unknown_key`, `expired` |

### tag\_local\_auth\_validations\_total[​](#tag_local_auth_validations_total "Direct link to tag_local_auth_validations_total")

**Type:** Counter — local authentication validation attempts in transparent proxy mode.

| Label    | Description                                                                                                       |
| -------- | ----------------------------------------------------------------------------------------------------------------- |
| `result` | Validation result: `success`, `missing_auth`, `parse_error`, `unknown_key`, `signature_mismatch`, `authz_expired` |

```
# Local auth success rate

rate(tag_local_auth_validations_total{result="success"}[5m]) /

sum(rate(tag_local_auth_validations_total[5m]))



# Auth failure breakdown by reason

sum by (result) (rate(tag_local_auth_validations_total{result!="success"}[5m]))
```

### tag\_derived\_key\_store\_size[​](#tag_derived_key_store_size "Direct link to tag_derived_key_store_size")

**Type:** Gauge — number of derived signing keys currently stored. TAG learns signing keys from Tigris responses and caches them for local SigV4 validation. A value of 0 after receiving requests indicates key learning is not working.

### tag\_authz\_cache\_size[​](#tag_authz_cache_size "Direct link to tag_authz_cache_size")

**Type:** Gauge — number of active per-bucket authorization cache entries (`accessKey × bucket` pairs). Each entry represents a client that has been granted access to a specific bucket.

### tag\_proxy\_signing\_keys\_received\_total[​](#tag_proxy_signing_keys_received_total "Direct link to tag_proxy_signing_keys_received_total")

**Type:** Counter — number of signing key sets received from Tigris responses. Incremented each time Tigris returns an `X-Tigris-Proxy-Signing-Keys` header that TAG uses to enable local validation.

```
# Rate of new key learning events

rate(tag_proxy_signing_keys_received_total[5m])
```

## Connection metrics[​](#connection-metrics "Direct link to Connection metrics")

### tag\_active\_connections[​](#tag_active_connections "Direct link to tag_active_connections")

**Type:** Gauge — number of active connections.

### tag\_bytes\_transferred\_total[​](#tag_bytes_transferred_total "Direct link to tag_bytes_transferred_total")

**Type:** Counter — total bytes transferred.

| Label       | Description                     |
| ----------- | ------------------------------- |
| `direction` | Transfer direction: `in`, `out` |

```
# Throughput (bytes/sec)

rate(tag_bytes_transferred_total[5m])



# Outbound throughput

rate(tag_bytes_transferred_total{direction="out"}[5m])
```

## Prometheus scrape configuration[​](#prometheus-scrape-configuration "Direct link to Prometheus scrape configuration")

```
scrape_configs:

  - job_name: "tag"

    kubernetes_sd_configs:

      - role: pod

    relabel_configs:

      - source_labels: [__meta_kubernetes_pod_label_app]

        action: keep

        regex: tag

      - source_labels: [__meta_kubernetes_pod_container_port_number]

        action: keep

        regex: "8080"
```