A vector-database benchmark is almost always a single dot: recall X at Y queries per second, measured once, after an engineer spent a while hand-tuning the index to land there. VectorDBBench is built around that shape -- optimize, then measure once. Our system doesn't have that shape. It refuses to sit still.
What actually happens when I upload a million Cohere vectors: serving in 2.3 minutes, exact brute force, recall 1.0, slow but correct. Then it starts improving on its own. Lossless quantization first, still 1.0. Then ivf, then vamana refining itself, recall settling at 0.9929 while throughput climbs. Zero configuration, zero downtime through every index handoff. The first convergence took 22.6 hours. After July's indexer work, 5.6.
So I stopped measuring the dot and measured the curve -- sample recall, latency, and which index is live every few minutes, plot the improvement over time. index_optimized becomes one annotated point on the curve, not the whole report.
Prod got the same treatment this week. The first 1M curve converged in 8.8 hours with a single worker doing all the serving. Ported dev's placement config so the second worker could join, re-ran from a cold upload: 4.8 hours, recall settling at 0.98, vamana holding all three metric slots. And the curve caught something no dot ever would -- one 6-minute sample at recall 0.63, sitting in the middle of a burst where the auto-promoter was cycling indexes every 30 seconds. Real bug. The promotion ratchet was losing track of its incumbent. Fix shipped the same night, and a converged-state benchmark would have looked immaculate the whole time.