deep-dive

We Built a Clustered Graph Database

Loveliness is our experimental open-source distributed graph database — Raft consensus, Bloom filter routing, Neo4j Bolt compatibility, and a single Go binary. Here's why we built it and how it works.

By John Jansen · · 4 min read

Share

The Problem With Graph Databases at Scale

Graph databases have a scaling problem nobody talks about honestly. The embedded ones are fast and simple but single-node. The distributed ones are complex, expensive, and operationally heavy. There's not much in between.

We kept running into this gap on client work — projects where a single-node Kuzu was performing well but the team needed replication for reliability or horizontal scale for ingest. The answer from the market was basically: pay for Neptune or run JanusGraph on Kubernetes.

We didn't love either of those answers, so we built Loveliness.

What Loveliness Is

Loveliness is experimental. We're using it on real workloads and learning from it, but it's not production-hardened software. We're sharing it because the ideas are interesting and we think others will find it useful — but go in with eyes open.

Loveliness is a clustered graph database built on LadybugDB — a Kuzu fork — the same way Elasticsearch is built on Lucene. A single Loveliness node is a Go binary with no external dependencies. It speaks Neo4j's Bolt protocol on port 7687, which means any existing Neo4j driver works without code changes. We pass 72/72 exhaustive tests against the official Python driver.

The collective noun for a group of ladybugs is a loveliness. That felt right.

How It Works

The distributed layer adds four main things on top of LadybugDB: hash-based sharding, Raft consensus, Bloom filter routing, and edge-cut replication.

Queries arrive over HTTP or Bolt, get parsed by the router, and the Bloom filter decides whether to hit one shard or scatter across all of them. Point lookups and 1-hop traversals usually resolve to a single shard. Aggregations fan out in parallel and merge. Edge-cut replication means both endpoints of any cross-shard edge are stored locally — so 1-hop traversals don't need a network round-trip.

Writes support three consistency levels: ONE (fire-and-forget), QUORUM (primary + one replica, default), and ALL. Replicas replay operations as Cypher statements rather than binary snapshots, which keeps the replication model simple.

One constraint worth knowing upfront: shard count is fixed at cluster creation. If you create a 4-shard cluster, you can run up to 4 nodes. Plan your topology before you start — this isn't something you can change later.

Performance

Against a 15.7M node, 10M edge dataset across 4 shards:

Query type P50 latency QPS
Point lookup 425µs 10,758
1-hop traversal 673µs 1,106
Single write 365µs 2,526
Bulk load 70–190K nodes/sec

The honest caveat: every query crosses the CGo boundary into LadybugDB, which is the primary throughput ceiling. Neo4j has 15 years of cost-based query optimisation we don't have. Multi-hop traversals in particular are slower than you'd get from a native graph engine at equivalent scale. Full comparisons are in docs/benchmarks.md.

Drop-In Neo4j Compatibility

If you've built against the Neo4j Python driver, your session management, retry logic, and transaction handling all work unchanged. Just swap the URL:

# Before
driver = GraphDatabase.driver("bolt://neo4j-host:7687")

# After — everything else is identical
driver = GraphDatabase.driver("bolt://loveliness-host:7687")

Both bolt:// (direct) and neo4j:// (routing with automatic failover) are supported.

Zero-Config Deployment

DNS discovery means loveliness up 3 just works. Nodes find each other on startup, elect a leader via Raft, and start accepting writes. On Fly.io this uses their .internal DNS with no extra configuration. For large datasets, the async ingest queue (/ingest/nodes) accepts CSV, returns a job ID immediately, and processes in the background — better than trying to push 10M nodes over a single HTTP connection.

What We're Using It For

Knowledge graphs, recommendation systems, entity resolution pipelines — workloads where the relationships between data points matter as much as the data itself. Loveliness came out of needing graph traversal performance without the operational weight of the enterprise options.

We're open-sourcing it because the gap between "embedded graph engine" and "full distributed graph platform" is real, and the answer shouldn't always be a managed cloud service. We're still finding the edges of it — the refTracker that keeps cross-shard references in memory doesn't scale to billion-node graphs yet, and there are known rough spots in distributed schema changes. But as a foundation for building on, it's proving useful.

The repo is at github.com/dreamware-nz/loveliness.

Want to discuss this?

We write about what we're actually working on. If this is relevant to something you're building, we'd love to hear about it.