Skip to main content

Deployment guide

Ready to run TAG in production? This guide covers sizing, choosing between single-node and cluster topologies, monitoring, upgrading, and troubleshooting. If you just want to try TAG out first, start with the Quick Start.

Single node​

Start here if your working set fits on one machine's storage and a single node's network bandwidth can handle your read throughput. Most workloads start with a single node and scale out only when needed.

Sizing guidelines​

TAG is typically NVMe-bound for large objects and CPU-bound for small objects. Benchmark reference points (single node, cache-warm):

Object SizeOps/secBandwidth
1 KiB~75,000~74 MiB/s
100 KiB~33,000~3.2 GiB/s
1 MiB~11,000~10.7 GiB/s

CPU utilization at peak throughput is around 12%, so a modest machine (4-8 cores) is sufficient for most workloads. Memory is used primarily by RocksDB block cache and in-flight request buffers. 4-8 GiB is a reasonable starting point. NVMe storage is strongly recommended.

For full benchmark methodology, thread scaling, and environment details, see Benchmarks.

Deploy​

See Configuration Reference — Example Configurations for ready-to-use production YAML configs.

Multi-node cluster​

When you outgrow a single node — either you need more cache capacity or higher aggregate throughput — deploy a multi-node cluster. TAG nodes form the cluster automatically via gossip discovery, consistent hashing distributes cache keys, and gRPC forwards requests for remote keys transparently. See Architecture — Cluster Architecture for details on how clustering works.

Cluster Deployment Options​

  • Docker Cluster — 3-node cluster via Docker Compose
  • Kubernetes — StatefulSet with autoscaling (recommended for production clusters)

Block-aligned caching​

Block-aligned caching is on by default (RFC 0001): any object at or above block_size is cached as fixed-size blocks, so a range read fetches and caches only the covering blocks instead of the whole object — ideal for small ranges of large objects (Parquet footers/row-groups, SlateDB/SST blocks, columnar analytics).

The one knob that matters is block_size — size it to your workload's dominant read size:

  • Too large and every cache miss pulls a full block to serve a small range (upstream read amplification). A 4 MiB block serving ~400 KB reads amplifies upstream traffic several-fold and can be worse than whole-object caching.
  • Too small adds per-block bookkeeping and more fetches per read.

The 1048576 (1 MiB) default suits typical analytics footers/row-groups; raise it for larger reads, lower it (e.g. 65536 for 64 KiB reads), or set TAG_CACHE_BLOCK_CACHING_ENABLED=false to cache whole objects. Verify the fit with Prometheus:

  • Upstream read amplification — sum(rate(tag_cache_block_bytes_populated_total[5m])) / sum(rate(tag_bytes_transferred_total{direction="out"}[5m])) (bytes fetched into blocks ÷ bytes served). Aim for ≤ 1; well above 1 means the block size is too large for the read pattern.
  • Block hit ratio — rate(tag_cache_block_hits_total[5m]) / (rate(tag_cache_block_hits_total[5m]) + rate(tag_cache_block_misses_total[5m])).
  • Serve latency — histogram_quantile(0.95, sum(rate(tag_request_duration_seconds_bucket[5m])) by (le)).

TLS​

TAG supports HTTPS with TLS certificates for encrypted client connections. See TLS/HTTPS for setup instructions.

Monitoring​

TAG exposes Prometheus metrics at GET /metrics. For the complete metrics reference and scrape configuration, see Metrics Reference.

Key metrics to alert on​

Error rate:

rate(tag_requests_total{status="error"}[5m])
  / rate(tag_requests_total[5m])

Alert if error rate exceeds 1% sustained over 5 minutes.

Cache hit ratio:

rate(tag_cache_hits_total[5m])
  / (rate(tag_cache_hits_total[5m]) + rate(tag_cache_misses_total[5m]))

A healthy hit ratio depends on your workload. For read-heavy workloads with a bounded working set, expect 80%+ after warmup.

Upstream latency:

histogram_quantile(0.99, rate(tag_upstream_request_duration_seconds_bucket[5m]))

Alert if p99 upstream latency exceeds your SLO, which may indicate Tigris connectivity issues.

Authentication failures:

rate(tag_auth_failures_total[5m])

Spikes indicate credential misconfiguration or unauthorized access attempts.

Unreclaimed cache disk:

sum(kubelet_volume_stats_used_bytes{namespace="tag", persistentvolumeclaim=~"cache-data-tag-.*"})
  - sum(ocache_disk_usage_bytes{namespace="tag", type="total"})

The two metrics come from different exporters and carry different labels, so each side must be aggregated, and both sides must be scoped to the same deployment. Without sum() the subtraction matches no series and returns nothing. Without the matching namespace selector on both sides, a Prometheus that scrapes more than one deployment compares one deployment's volumes against every deployment's cache, which can report a negative or understated gap.

ocache_disk_usage_bytes counts the objects the cache considers live. The filesystem also holds space that overwritten, deleted, and expired objects left behind inside the cache's storage files. Background compaction reclaims that space, so the difference between the two should rise and fall. A difference that only rises means compaction is not keeping up, and the volume will eventually fill even though the cache is under its configured size limit.

Compaction reclaims space in files that are at least two hours old, so expect no reclamation in the first two hours after a restart.

Upgrading​

TAG's on-disk cache is persistent and compatible across versions. Upgrading TAG does not require clearing or rebuilding the cache — the new version picks up where the old one left off.

Troubleshooting​

TAG won't start​

"missing AWS credentials" — Set both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. These are TAG's own credentials, not your application's. In Kubernetes, verify the secret exists: kubectl get secret -n tag tag-credentials

"invalid upstream endpoint" — in transparent proxy mode TAG only allows localhost, *.tigris.dev, or *.storage.dev. Check TAG_UPSTREAM_ENDPOINT, or switch to signing mode (TAG_TRANSPARENT_PROXY=false) to use another S3-compatible endpoint.

"TLS certificate or key file not found" — If either TAG_TLS_CERT_FILE or TAG_TLS_KEY_FILE is set, both must point to valid files.

Connection refused — Verify TAG is running: curl http://localhost:8080/health

Cache not working​

All responses show X-Cache: MISS — Check that caching is enabled (TAG_CACHE_DISABLED is not true) and that the cache directory is writable. Set TAG_LOG_LEVEL=debug and look for cache write errors. In Kubernetes, check logs with kubectl logs -n tag tag-0 and verify the cache PVC is bound with kubectl get pvc -n tag.

Objects not being cached — Objects must return HTTP 200 and be within the size threshold (default 1 GiB). Objects with Cache-Control: no-store are not cached.

Authentication errors​

403 on first request — Verify your client credentials are valid for the requested bucket on Tigris and belong to the same Tigris organization as TAG's credentials. TAG forwards the first request to Tigris, which performs authentication.

403 after credential rotation — After rotating credentials, restart TAG to clear auth-related caches.

Client errors​

405 on bucket creation — You're using virtual-hosted style addressing. TAG requires path-style. Set addressing_style: 'path' in your S3 client config.

Timeout on large files — Increase client-side timeouts. For example, in boto3:

from botocore.config import Config

config = Config(
    connect_timeout=30,
    read_timeout=300,
    s3={'addressing_style': 'path'},
)

Cluster issues​

Nodes not discovering each other — Verify seed nodes are reachable on port 7000 (gossip). In Kubernetes, ensure the headless service resolves correctly:

nslookup tag-headless.tag.svc.cluster.local

gRPC routing failures — Verify port 9000 is open between nodes. Check that TAG_CACHE_ADVERTISE_ADDR is set to an address reachable by other nodes (not localhost).

High latency​

High p99 latency — Check tag_upstream_request_duration_seconds to determine whether latency comes from Tigris or TAG. High request coalescing (tag_broadcast_shared_total) is normal and reduces upstream load. High tag_broadcast_slow_consumers_total indicates clients are reading too slowly. In Kubernetes, also check disk I/O performance on the storage class.

Cache disk keeps growing​

The cache stores objects in large files. When an object is overwritten, deleted, or expires, its bytes stay in the file until background compaction rewrites the file without them. Disk usage that climbs while ocache_disk_usage_bytes stays flat means this space is not coming back.

Check that compaction runs at all:

rate(ocache_segment_walks_total[10m])

A rate of zero on a node older than two hours means compaction is not examining files. Confirm the cache is not configured with recompaction disabled, and confirm the node has more than one storage file.

If compaction runs but reclaims nothing, check whether TAG_CACHE_COMPACTION_BPS is set so low that compaction cannot keep up with the rate at which objects are replaced. One node at 32 MiB/s rewrites about 2.7 TB per day.

TAG versions before v1.17.7 could not reclaim space in files written before the process last restarted. Upgrade first if you see this on an older version.

Debug mode​

Set TAG_LOG_LEVEL=debug for detailed request-level logging. This is verbose; use it only during active debugging.

In Kubernetes, update the StatefulSet:

env:
  - name: TAG_LOG_LEVEL
    value: "debug"
Last updated on by Mantas Mikšys