Skip to main content

Metrics reference

TAG exposes Prometheus metrics at the /metrics endpoint.

Accessing metrics​

# Local
curl http://localhost:8080/metrics

# Kubernetes (port-forward)
kubectl port-forward svc/tag 8080:8080
curl http://localhost:8080/metrics

Request metrics​

tag_requests_total​

Type: Counter

Total number of requests processed by TAG.

LabelDescription
operationS3 operation: GetObject, PutObject, DeleteObject, HeadObject
statusResult: success, error, auth_error, range_not_satisfiable
# Request rate by operation
rate(tag_requests_total[5m])

# Error rate
sum(rate(tag_requests_total{status="error"}[5m])) / sum(rate(tag_requests_total[5m]))

# GetObject success rate
rate(tag_requests_total{operation="GetObject",status="success"}[5m]) /
rate(tag_requests_total{operation="GetObject"}[5m])

tag_request_duration_seconds​

Type: Histogram

Request duration in seconds.

LabelDescription
operationS3 operation
# P50 latency
histogram_quantile(0.5, rate(tag_request_duration_seconds_bucket[5m]))

# P99 latency by operation
histogram_quantile(0.99, sum(rate(tag_request_duration_seconds_bucket[5m])) by (operation, le))

Cache metrics​

tag_cache_hits_total​

Type: Counter — every request served from cache, including range-from-cache hits and conditional 304 (Not Modified) responses. Recorded in lockstep with the X-Cache: HIT header.

tag_cache_misses_total​

Type: Counter — total number of cache misses (recorded in lockstep with X-Cache: MISS).

These two counters count only HIT and MISS. REVALIDATED responses (object changed on upstream) are neither — they are tracked by the tag_revalidations_* metrics — and BYPASS/DISABLED requests are not counted at all.

tag_cache_operations_total​

Type: Counter

LabelDescription
operationOperation type: get, put, delete
resultResult: hit, miss, success, error
# Cache hit ratio (of hit/miss decisions; excludes REVALIDATED)
rate(tag_cache_hits_total[5m]) /
(rate(tag_cache_hits_total[5m]) + rate(tag_cache_misses_total[5m]))

# Cache operation breakdown
sum by (operation, result) (rate(tag_cache_operations_total[5m]))

tag_range_from_cache_hits_total​

Type: Counter — number of range requests served from cached full objects.

tag_cache_serve_locality_total​

Type: Counter — cache body reads labeled by whether this node owns the key (locality="local", served from local storage) or had to pull it from a peer over gRPC (locality="remote"). In a cluster the single-owner consistent-hash ring routes most reads to the owning node, so a high remote share is the cross-node data-plane cost. Single-node mode is always local.

LabelDescription
localitylocal (this node owns the key) or remote (pulled from a peer)
# Remote serve ratio (fraction of cache reads pulled cross-node; lower is better).
# Aggregate with sum() so the denominator is local+remote; dividing the raw
# vectors would match locality="remote" against itself and always report 1.
sum(rate(tag_cache_serve_locality_total{locality="remote"}[5m]))
/
sum(rate(tag_cache_serve_locality_total[5m]))

# Cross-node read rate per node
sum by (pod) (rate(tag_cache_serve_locality_total{locality="remote"}[5m]))

Only populated when the embedded cache client can report key ownership (cluster mode). When it cannot, the counter stays at 0 rather than guessing a locality.

tag_cache_size_bytes​

Type: Gauge — current logical size of this node's local cache in bytes (sum of stored object lengths). Per-node; sum across nodes for a cluster-wide total.

# Cluster-wide cache size
sum(tag_cache_size_bytes)

The embedded cache also exports ocache_disk_usage_bytes{type="total"} (same logical size) and ocache_segment_size_bytes (physical on-disk segment bytes) directly; tag_cache_size_bytes is the stable, TAG-owned name for the logical size.

Logical size is smaller than the space the cache occupies on disk. Objects that are overwritten, deleted, or expired leave their bytes behind inside the storage files until background compaction rewrites those files. To see whether that space is being reclaimed, compare the filesystem usage against this gauge, and watch the compaction metrics below.

ocache_segment_walks_total​

Type: Counter — storage files examined to determine how much of each one is still in use. Compaction uses the result to decide what to rewrite.

# Examination rate
rate(ocache_segment_walks_total[10m])

Zero on a node that has been running for more than two hours suggests compaction is disabled.

ocache_recompaction_segments_total​

Type: Counter — storage files rewritten to reclaim space, with ocache_recompaction_bytes_freed_total reporting the bytes recovered.

# Space reclaimed per hour
increase(ocache_recompaction_bytes_freed_total[1h])

If disk usage grows while this counter stays flat, space is not being reclaimed. See the deployment guide for the alert to configure.

Block cache metrics​

Block-aligned caching is on by default, so these apply to most deployments. They describe cache behaviour at block granularity, which is what a range read actually touches.

tag_cache_block_hits_total / tag_cache_block_misses_total​

Type: Counters — blocks served from cache, and blocks that had to be fetched. Together they give the ratio that matters for range-read workloads.

# Block cache hit ratio
sum(rate(tag_cache_block_hits_total[30m]))
/
(sum(rate(tag_cache_block_hits_total[30m])) + sum(rate(tag_cache_block_misses_total[30m])))

A hit ratio that will not rise despite a warm cache usually means block_size does not match the workload's read granularity. See Configuration.

tag_cache_block_bytes_populated_total​

Type: Counter — bytes fetched from upstream into blocks. Divide by bytes served to clients to measure read amplification:

# Upstream read amplification. Aim for <= 1.
sum(rate(tag_cache_block_bytes_populated_total[5m]))
/
sum(rate(tag_bytes_transferred_total{direction="out"}[5m]))

Well above 1 means the block size is too large for the read pattern — every miss pulls a full block to serve a small range. Do not substitute tag_bytes_transferred_total{direction="in"} here: that counts client upload bodies, not upstream fetches.

tag_cache_block_prefetched_total​

Type: Counter, labelled by trigger — blocks fetched speculatively rather than because a client asked for them.

# Speculative fetching by trigger
sum by (trigger) (rate(tag_cache_block_prefetched_total[30m]))

Judge this against the block hit ratio above, not on its own. Volume rising without the hit ratio rising means the speculation is not landing where reads go.

Type: Histogram — the size of parquet metadata footers observed. Recorded for every parquet object whose trailer is read, including ones that are not prefetched, so it describes the whole population.

# Median footer size, in bytes.
histogram_quantile(0.5, sum(rate(tag_cache_parquet_footer_bytes_bucket[1h])) by (le))

This is the measurement that tells you whether Parquet optimization is worth enabling. Compare it against the tail block rather than the full block_size — the tail averages half a block, so block_size / 2 is the practical yardstick.

Broadcast metrics​

tag_broadcast_shared_total​

Type: Counter — requests that joined an existing broadcast stream.

tag_broadcast_fetches_total​

Type: Counter — upstream fetches (broadcast initiators).

tag_broadcast_slow_consumers_total​

Type: Counter — listeners disconnected for being too slow.

tag_active_broadcasts​

Type: Gauge — currently active broadcast streams.

# Coalescing ratio (higher is better)
rate(tag_broadcast_shared_total[5m]) /
(rate(tag_broadcast_shared_total[5m]) + rate(tag_broadcast_fetches_total[5m]))

Background fetch metrics​

tag_background_fetches_triggered_total​

Type: Counter — background full-object fetches triggered by range requests.

tag_background_fetches_succeeded_total​

Type: Counter — background fetches completed successfully.

tag_background_fetches_failed_total​

Type: Counter — background fetches that failed.

tag_active_background_fetches​

Type: Gauge — currently active background fetches.

tag_warm_on_write_triggered_total​

Type: Counter — cache warms triggered by a successful write (when cache.warm_on_write is enabled). The warm's own outcome is recorded by the tag_background_fetches_* metrics.

# Background fetch success rate
rate(tag_background_fetches_succeeded_total[5m]) /
rate(tag_background_fetches_triggered_total[5m])

Revalidation metrics​

tag_revalidations_triggered_total​

Type: Counter — cache revalidation attempts (conditional GET/HEAD to upstream).

tag_revalidations_not_modified_total​

Type: Counter — revalidations where upstream returned 304 Not Modified.

tag_revalidations_updated_total​

Type: Counter — revalidations where upstream returned 200 with new data.

tag_revalidations_failed_total​

Type: Counter — revalidations that failed due to errors.

tag_revalidations_stale_served_total​

Type: Counter — times stale cached data was served because revalidation failed.

# Revalidation 304 ratio (higher = better cache freshness)
rate(tag_revalidations_not_modified_total[5m]) /
rate(tag_revalidations_triggered_total[5m])

# Stale serve ratio (should be low)
rate(tag_revalidations_stale_served_total[5m]) /
rate(tag_revalidations_triggered_total[5m])

Upstream metrics​

tag_upstream_request_duration_seconds​

Type: Histogram — upstream (Tigris) request duration in seconds.

LabelDescription
methodHTTP method: GET, PUT, DELETE, HEAD

tag_upstream_errors_total​

Type: Counter — total upstream errors.

LabelDescription
methodHTTP method

Authentication metrics​

tag_auth_failures_total​

Type: Counter

LabelDescription
reasonFailure reason: invalid_signature, unknown_key, expired

tag_local_auth_validations_total​

Type: Counter — local authentication validation attempts in transparent proxy mode.

LabelDescription
resultValidation result: success, missing_auth, parse_error, unknown_key, signature_mismatch, authz_expired
# Local auth success rate
rate(tag_local_auth_validations_total{result="success"}[5m]) /
sum(rate(tag_local_auth_validations_total[5m]))

# Auth failure breakdown by reason
sum by (result) (rate(tag_local_auth_validations_total{result!="success"}[5m]))

tag_derived_key_store_size​

Type: Gauge — number of derived signing keys currently stored. TAG learns signing keys from Tigris responses and caches them for local SigV4 validation. A value of 0 after receiving requests indicates key learning is not working.

tag_authz_cache_size​

Type: Gauge — number of active per-bucket authorization cache entries (accessKey × bucket pairs). Each entry represents a client that has been granted access to a specific bucket.

tag_proxy_signing_keys_received_total​

Type: Counter — number of signing key sets received from Tigris responses. Incremented each time Tigris returns an X-Tigris-Proxy-Signing-Keys header that TAG uses to enable local validation.

# Rate of new key learning events
rate(tag_proxy_signing_keys_received_total[5m])

Connection metrics​

tag_active_connections​

Type: Gauge — number of active connections.

tag_bytes_transferred_total​

Type: Counter — total bytes transferred.

LabelDescription
directionTransfer direction: in, out
# Throughput (bytes/sec)
rate(tag_bytes_transferred_total[5m])

# Outbound throughput
rate(tag_bytes_transferred_total{direction="out"}[5m])

Prometheus scrape configuration​

scrape_configs:
  - job_name: "tag"
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        action: keep
        regex: tag
      - source_labels: [__meta_kubernetes_pod_container_port_number]
        action: keep
        regex: "8080"
Last updated on by Ovais Tariq