# Fly.io Replaces Vault With Petsem, Cutting Secret-Read p99 From 10s to 300ms

> Fly.io built Petsem, a home-grown Vault replacement with multi-region read replicas, after Raft latency put a 10-second p99 on the app boot path.

- Publication: Thoai Dispatch
- Canonical URL: https://vokythoai.dev/article/flyio-petsem-vault-replacement
- Section: News
- Author: Fly.io Engineering
- Published: 2026-11-20
- Tags: fly-io, secrets-management, vault, infrastructure, latency, raft, neocloud, security

Fly.io has replaced HashiCorp Vault with an in-house secrets store called Petsem(atary), according to a post on the [Fly.io engineering blog](https://fly.io/blog/petsem/). The company says the new system keeps a write master in IAD but adds read replicas across regions, dropping the p99 for apps loading secrets at boot from 10 seconds to 300 milliseconds.

## Why it matters

Fly.io frames secrets protection as the first-and-foremost risk for any neocloud, and notes a modern twist: attackers are now particularly interested in keys for AI inference providers, not just payment systems or other crown jewels. The post also argues that LLM-powered attackers can explore, iterate, and exploit victims faster than before, tipping the balance of power toward attackers.

The engineering context is the boot path. Fly.io was built on Consul for service discovery, Nomad for fleet orchestration, and Vault for secrets storage. As it scaled, the company hit the edges of what those services could do and began building partial or full replacements: Consul in 2022, Nomad in 2023, and Vault in 2025. The Vault replacement was driven by a specific operational problem rather than a general preference for home-grown software.

## Technical details

Fly.io describes Vault as, more or less, a disk-backed key-value store on a single node, given an API with auth, then split into a consistent and fault-tolerant cluster, perhaps using a storage backend that uses Raft for consensus. Because Raft needs low latency between members for consensus and propagation, the cluster sits in a single region, and a global fleet of readers points at the cluster's active read replica to pull secrets.

The cost of that design showed up at boot. Fly.io reports a p99 of 10 seconds for apps loading secrets when booting, plus a hard dependency on IAD being available and added latency for apps reading from the other side of the globe. The post calls a 10s+ secrets read phase and a single-region dependency on the boot path a huge performance and reliability challenge at scale.

Fly.io says it is not running a bank: for secrets storage it is fine-ish if a read returns an out-of-date value, and the company is willing to give up hard consistency guarantees to get back performance. It considered deploying a multi-region Vault under an enterprise agreement backed by enterprise Consul, but says that would run roughly the amount of a fully loaded engineer's salary per year. That was the main impetus behind Petsem(atary).

The post also references a section titled "A use case for macaroons," though the source material is truncated at that point and does not describe the mechanism. Fly.io notes that it offers secrets on Sprites via [Connectors](https://docs.fly.io/sprites/concepts/connectors) and for Machines via [Tokenizer/SSOkenizer](https://fly.io/blog/tokenized-tokens/). It also points to earlier posts on replacing Consul, [A Foolish Consistency](https://fly.io/blog/a-foolish-consistency/) and [Corrosion](https://fly.io/blog/corrosion/), and on replacing Nomad with flyd in [Carving The Scheduler Out Of Our Orchestrator](https://fly.io/blog/carving-the-scheduler-out-of-our-orchestrator/).

## What to watch

The post's practical framing is that the best case is to not have a key to protect in the first place, citing ambient identity or API gateways. Where that is not an option, it endorses pulling sensitive values from an external source into environment variables rather than hardcoding them in source or config files, and using API-driven secrets managers so operators can roll credentials quickly and log access. Petsem is Fly.io's attempt to keep that centralized model while removing the single-region latency and availability penalty from the boot path. The open question, given the truncated source, is how the macaroon-related design fits into that model.

## Sources

- [Fly.io Engineering](https://fly.io/blog/petsem/) (2026-11-20)
