Vitess in Production: Horizontal MySQL Sharding Without Rebuilding Your Application

I spent eighteen months at a mid-size e-commerce company whose MySQL database grew from 200GB to 3.5TB while I was there. We were on a single r6g.16xlarge RDS instance with read replicas, and for a long time that worked. Then Black Friday 2019 happened and we saturated the primary’s write IOPS at about 11pm. Carts stopped checking out. The primary became unresponsive to binlog replication. It was the kind of night that makes you seriously reconsider your career choices.

We recovered by disabling non-essential writes and routing reads aggressively to replicas. The next morning we started the conversation we should have had six months earlier: what does horizontal sharding actually look like for MySQL? We looked at migrating to PostgreSQL, evaluated CockroachDB, and spent a week on Vitess. We chose Vitess. I have run it at two companies since then, and across twenty years of managing databases at scale, it remains the most pragmatic path to MySQL sharding that I know of.

This is what I wish I had read before we started.

Why Vitess Exists

YouTube built Vitess starting around 2010 because they were running MySQL at a scale where the database team was spending most of their time managing an ever-growing fleet of MySQL instances, connection pooling hacks, and fragile sharding logic spread across application code. They open-sourced it in 2012, donated it to the CNCF, and it graduated to CNCF top-level project status in 2019. PlanetScale, founded by some of the original YouTube/Vitess engineers, is essentially a managed Vitess service.

The core premise of Vitess is that MySQL is a fundamentally good database, but the operational model does not scale horizontally. Every attempt to shard MySQL without tooling ends up with some combination of: sharding logic scattered across application code, inconsistent routing bugs, inability to reshape shards without extended downtime, and connection exhaustion under load. Vitess centralizes that complexity into a separate layer that sits between your application and MySQL. Your application talks to Vitess using standard MySQL protocol. Vitess handles everything else.

The Architecture: Four Components That Matter

Vitess has more components than a first glance suggests, but the core four are what you will interact with day-to-day.

VTGate is the query router. Your application connects to VTGate exactly as it would connect to MySQL, using the standard MySQL wire protocol. VTGate parses every query, consults the topology to determine which shard holds the relevant rows, and routes the query accordingly. It also handles scatter queries (queries that must hit multiple shards), aggregation across shards, and cross-shard transactions. VTGate runs as a stateless service, so you can scale it horizontally behind a load balancer. Each VTGate instance maintains a connection pool to each VTTablet it serves, which is how Vitess solves the connection exhaustion problem: thousands of application connections funnel into a much smaller pool of connections to actual MySQL.

VTTablet runs alongside each MySQL instance. It is a sidecar that manages the MySQL process, handles row-based replication, enforces query rules (timeouts, blacklists, table size limits), handles graceful failover, and exposes metrics. Every MySQL instance in a Vitess cluster is managed by a VTTablet. You never talk to MySQL directly in production; you talk to VTGate, which talks to VTTablet, which talks to MySQL. This layer is where query-level observability lives.

VTorc is the orchestration layer, the successor to Orchestrator. It handles automatic primary failover, manages semi-synchronous replication, and ensures topology consistency. When a primary VTTablet goes unhealthy, VTorc promotes a replica automatically, updates the topology, and VTGate starts routing writes to the new primary without any manual intervention. In my experience this failover happens in 30-45 seconds, which is fast enough for most production workloads.

The Topology Service is usually etcd or ZooKeeper, and stores the global state of the cluster: which keyspaces exist, how they are sharded, which tablet is the primary for each shard, and where VTTablets are running. VTGate reads from topology on startup and caches it, watching for changes. The topology is not in the hot path of queries.

Keyspaces, Shards, and Vindexes

A keyspace in Vitess maps roughly to a MySQL database, but with sharding semantics on top. An unsharded keyspace is just a single MySQL database managed by Vitess, which is a perfectly valid starting point: you get connection pooling, automatic failover, and observability without any sharding complexity. Most teams start here and reshard later.

A sharded keyspace divides the data across N shards using a vindex (virtual index). The primary vindex is the routing key, a hashed or range-based value that determines which shard a row lives on. For most OLTP workloads you want a hash vindex on your primary entity ID (user ID, order ID, account ID), which distributes rows evenly across shards and keeps hotspots manageable.

Understanding sharding vs partitioning is prerequisite knowledge here. Vitess implements horizontal sharding (splitting rows across separate MySQL instances), not partitioning (splitting rows within a single instance). The two are complementary. I have seen teams use both: partition by date within a shard, shard by user ID across the cluster.

The routing table looks like this: given a query with a WHERE clause on the primary vindex column, VTGate computes the keyspace ID for that value, looks up which shard owns that keyspace ID range, and routes the query to that shard’s primary VTTablet. If the query includes no vindex column, Vitess has to do a scatter query, which fans out to all shards and merges the results. Scatter queries are expensive and you want to minimize them.

Secondary vindexes let you route on a second column without scatter. You maintain a lookup table mapping the secondary key to the primary key. A query on email address, for example, hits the lookup table to find the user ID, then routes directly to the correct shard. The lookup table is itself a Vitess table, potentially sharded on its own key.

This is where Vitess introduces a real design constraint: your sharding key matters enormously and is very hard to change later. Pick the wrong primary vindex and you will be doing scatter queries constantly, or you will end up with hot shards. I have seen teams choose customer ID as the sharding key when they should have chosen tenant ID, because their access patterns are almost entirely within a tenant. Take the time to analyze your actual query patterns before committing to a vindex.

VReplication: The Feature That Makes Everything Else Possible

If I had to pick a single Vitess feature that justifies the operational complexity, it would be VReplication. VReplication is Vitess’s internal replication engine, built on MySQL binlog streaming but capable of resharding, moving tables between keyspaces, and applying online schema changes, all without downtime.

Resharding with Vitess means going from, say, 2 shards to 4 shards without taking the database offline. VReplication copies data from the source shards to the target shards, keeps them in sync via binlog streaming while writes continue on the source, and then at switchover time, VTGate switches read traffic, then write traffic, to the target shards in a controlled window measured in seconds. If something goes wrong, you reverse the traffic back to the source. I have done this resharding operation three times in production and never had a user-visible outage.

This is a completely different operational reality from rolling your own MySQL sharding. Without Vitess, resharding requires taking the application offline or accepting dirty reads during a multi-hour migration.

Online Schema Changes (OSC) in Vitess use VReplication under the hood, replacing external tools like gh-ost or pt-online-schema-change. You run ALTER TABLE statements through the vtctldclient command and Vitess handles the migration: it creates a shadow table, copies rows in batches, replays binlog changes, and then swaps the table. The operations are fully integrated with the Vitess topology so every shard runs the migration in parallel.

The zero-downtime database migrations patterns I described previously (expand-contract, shadow tables, gh-ost) are still relevant, but Vitess absorbs the operational tooling. You are still thinking about backward-compatible schema changes, but the mechanics are handled by the platform.

Connection Pooling: Why VTGate Replaces PgBouncer for MySQL

One of the least-discussed benefits of Vitess is that VTGate solves MySQL connection exhaustion by default. MySQL’s per-connection threading model starts to degrade with more than a few hundred active connections on a single instance. Connection poolers like PgBouncer (for PostgreSQL) or similar MySQL connection pools address this, but they are external pieces of infrastructure you need to operate.

VTGate does this natively. You can have 10,000 application connections to VTGate and VTGate will multiplex them into a far smaller connection pool to each VTTablet (typically 100-300 connections per shard). Each VTTablet also maintains its own connection pool to the underlying MySQL process. The connection collapse happens at two levels.

There is a catch: VTGate’s connection pooling only works well for transactions that complete quickly, since connections are returned to the pool after each statement or transaction. Long-running transactions hold the VTTablet connection for their duration. If your workload involves many concurrent long transactions, you will need to tune the pool carefully or architect around this constraint.

Running Vitess on Kubernetes with the Vitess Operator

The recommended production deployment for Vitess today is the Vitess Operator on Kubernetes. The operator manages the lifecycle of all Vitess components: it creates VTGate deployments, VTTablet StatefulSets with PVCs for MySQL data, handles topology registration, and can perform resharding operations via CRDs.

A basic VitessCluster resource defines your keyspaces, shards, and replica counts. The operator reconciles the desired state against the actual cluster state, similar to how any Kubernetes operator works. The topology service (etcd) is typically deployed as a separate StatefulSet, often reusing an existing etcd cluster if you are already running one for other purposes.

The StatefulSet for VTTablets is worth thinking through carefully. Each VTTablet pod needs a local PVC for the MySQL data directory. This means you need a storage class that supports ReadWriteOnce with good IOPS, and you need anti-affinity rules to spread VTTablets across availability zones. Losing two VTTablets on the same shard at the same time (primary and replica both down) means that shard is unavailable, so zone distribution is not optional.

On resource sizing: I run VTGate pods at 2 vCPU / 4GB with horizontal pod autoscaling, since VTGate is stateless and CPU-bound during query routing. VTTablet pods are sized based on MySQL’s needs, typically 4-8 vCPU and 16-32GB for a medium-sized shard. The MySQL processes themselves tend to be the bottleneck, not VTGate or VTTablet.

Resource limits for VTTablet pods matter. I once had a production incident where a VTTablet’s memory limit was too low, MySQL OOMKilled, and the pod restarted mid-transaction. VTorc promoted a replica, but the promotion took 40 seconds during which that shard was read-only. Set memory limits generously or use VPA for the VTTablet/MySQL containers.

When Vitess Is the Right Tool (and When It Is Not)

Vitess is the right choice when:

You are deep in the MySQL ecosystem and cannot migrate. Your application uses MySQL-specific features: JSON functions, full-text search, stored procedures, triggers. Your ORM generates MySQL SQL. Your team knows MySQL inside and out. Migrating to PostgreSQL or a distributed SQL database like CockroachDB or YugabyteDB would be a multi-year rewrite. Vitess lets you scale horizontally without abandoning the ecosystem.

You are approaching the write capacity ceiling of a single MySQL instance. If you are getting close to saturating write IOPS, or your primary CPU is consistently above 70%, and vertical scaling is not giving you enough headroom, horizontal sharding is the right architectural move. Vitess makes that move survivable.

You need automatic MySQL failover. Even without sharding, deploying MySQL under Vitess and VTorc gives you automatic primary promotion when the primary fails, with topology-aware routing so your application is not hardcoded to an IP or DNS name. The unsharded Vitess deployment is worth considering just for this.

You have a multi-tenant SaaS application. Tenant ID is often a natural sharding key. Each shard serves a subset of tenants, hotspot tenants can be resharded to dedicated shards, and VReplication makes tenant moves online. PlanetScale built their managed database product on exactly this pattern.

Vitess is probably the wrong choice when:

Your access patterns are heavily cross-shard. If most of your queries join data across your sharding key, you will be running scatter queries constantly. Vitess can handle scatter queries, but they are expensive and Vitess cannot optimize them the way a shared-nothing distributed SQL database can. CockroachDB and YugabyteDB were designed for distributed joins; Vitess was not.

Your team has no MySQL operational experience. Vitess adds complexity on top of MySQL. If your team is already struggling with MySQL operations, adding Vitess before getting MySQL basics right is going to end badly. Start with managed RDS or Cloud SQL, get your replication and backups solid, understand your query patterns, and then consider Vitess when you genuinely hit the ceiling.

Your dataset is small and you are not growing fast. Vitess has real operational overhead. For a 100GB database that might double in three years, you are solving a problem you do not have yet. Partitioning within a single MySQL instance is almost always sufficient under 500GB.

PlanetScale and the Managed Option

PlanetScale is a managed database service built on Vitess, offering a notably different operational experience. The branching and deploy requests model, where schema changes are applied through a merge request workflow with built-in diff review and zero-downtime deployment, is genuinely excellent. If you are on PlanetScale, you are getting the best parts of Vitess (connection pooling, automatic failover, VReplication-backed schema changes) without running the Kubernetes operators yourself.

The trade-off is cost and control. PlanetScale pricing is based on rows read and written, which can get expensive for analytics-heavy workloads. You also have no access to the underlying MySQL instances and cannot install custom plugins or enable features PlanetScale does not expose. For most product teams, the managed experience is worth it. For teams with unusual requirements or existing MySQL expertise, self-managed Vitess on Kubernetes is worth the extra overhead.

Operational Realities

A few things I have learned the hard way:

Schema changes need discipline. Even with VReplication-backed OSC, you still need to think carefully about backward-compatible changes. Vitess runs migrations per-shard in parallel, but your application code deploys independently. The expand-contract pattern still applies: add the new column first, deploy application code that handles both old and new schema, then remove the old column in a follow-up migration.

Monitor your vindexes. Log scatter queries in production. If scatter query rate starts climbing, something changed in your access patterns or query generation. A scatter query that used to hit 2 shards might hit 16 shards after a resharding event, making a previously-acceptable query suddenly problematic.

Backup strategy is on you. Vitess does not provide a backup solution. Each MySQL instance needs its own backup mechanism: mysqldump, Percona XtraBackup, or snapshot-based backups. On Kubernetes this usually means a CronJob that runs XtraBackup against each VTTablet and uploads to S3. Running databases on Kubernetes means the backup and restore tooling is your responsibility; the operator does not manage it.

VTorc failover is not instant. Thirty to forty-five seconds of primary unavailability during failover is typical, and can be worse if your topology service is slow. Design your application to handle transient MySQL errors gracefully: connection retries with backoff, circuit breakers on the database layer. This is good practice for any database, but Vitess makes you face it explicitly.

The Migration Path

Most teams do not start with Vitess on day one. The typical path I have seen work:

1. Deploy Vitess in unsharded mode in front of your existing MySQL. This gives you connection pooling, automatic failover, and query observability with minimal risk. Spend a month validating that your application works correctly with Vitess in the data path.

2. Move to sharded mode on a new keyspace for new tables. This lets you validate your sharding logic and vindex choices on non-critical data before migrating the core tables.

3. Migrate existing tables using MoveTables (VReplication). Move tables from the unsharded keyspace to the sharded keyspace with zero downtime. VReplication handles the data copy and switchover.

4. Reshard as you grow. Start with 2 or 4 shards. As write load grows, use VReplication to reshard to 8, 16, or more shards without downtime.

This incremental approach is much safer than trying to design the perfect sharding scheme upfront and migrating everything at once. The e-commerce company I mentioned at the start spent eight months on incremental migration. We never had a migration-related production incident.

The Bottom Line

Vitess is a genuinely mature piece of infrastructure, battle-tested at YouTube scale and actively developed. The operational complexity is real: you are adding a stateless routing tier, a sidecar to every MySQL instance, an orchestration daemon, and a topology service. Your team needs to understand VTGate, VTTablet, VTorc, and the Vitess Operator to run this reliably.

But the alternative, rolling your own MySQL sharding, is dramatically worse. The sharding logic ends up scattered across your codebase, schema changes become multi-day operations, resharding requires downtime, and connection pooling is an afterthought. Every team I have seen try to shard MySQL by hand eventually either rewrites to a different database or adopts Vitess.

If you are running MySQL at a scale where you are starting to feel the ceiling, Vitess is how you break through it without starting over.