Build Your Own Redis
A single-threaded server that handles thousands of clients without a mutex, a goroutine-per-connection, or the Go net package: just raw kqueue syscalls and a 33-byte byte buffer per client. Every file explained line by line.
// 1. kernel fires kqueue event: fd 7 has data
// 2. serve() calls c.read(store)
// 3. syscall.Read(fd, tmp) → raw bytes land in c.buf
// 4. bytes.NewReader + bufio.NewReader wrap c.buf
// 5. ReadValue() parses RESP → Value{array: [SET, foo, bar]}
// 6. dispatch() routes to handleSet()
// 7. store.Set("foo", "bar") → plain map write
// 8. marshalSimpleString("OK") → "+OK\r\n"
// 9. syscall.Write(fd, response) → bytes back to client
All 9 steps run on one goroutine, in sequence, before we go back to Kevent(). There is no concurrency at the execution level.
This is why the store needs no mutex: two commands from two different clients cannot physically run at the same time. While we're at step 7 executing client A's SET, client B's bytes sit in the kernel's socket buffer waiting their turn.
Side effects: no goroutine-switch overhead, no lock contention, no stack allocated per connection. Thousands of idle clients cost essentially nothing.
package main
func main() {
store := NewStore() // defined in store.go: plain map[string]string, no mutex
serve(store) // defined in server.go: starts kqueue loop, never returns
}The whole program is two calls. store is passed by pointer into serve(): shared state, but safe because only one goroutine ever touches it. serve() enters the infinite event loop and never comes back. All the interesting code is in the files it calls.
// Thread safety is guaranteed by the event loop: only one goroutine ever calls these.
type Store struct {
data map[string]string
}
func NewStore() *Store {
return &Store{data: make(map[string]string)}
}No sync.RWMutex. The event loop guarantees only one goroutine ever calls these methods: thread safety is an architectural property, not something the data structure enforces.
The comment is there to stop a future reader from "fixing" it. A mutex here would be unnecessary and misleading.
func (s *Store) Set(key, value string) { s.data[key] = value }
func (s *Store) Get(key string) (string, bool) {
v, ok := s.data[key]
return v, ok
}Set is one line. Get uses Go's two-return map lookup: ok is whether the key existed at all.
handleGet uses ok to distinguish two different things: key missing ($-1\r\n) vs key exists with empty value ($0\r\n\r\n). Redis treats these differently: so must we.
func (s *Store) Del(keys ...string) int64 {
var count int64
for _, k := range keys {
if _, ok := s.data[k]; ok {
delete(s.data, k)
count++
}
}
return count
}Variadic because DEL key1 key2 key3 is a single Redis command. Returns how many keys were actually deleted: you might ask for 3 but only 2 exist.
The check-before-delete pattern is necessary because Go's delete() on a missing key silently does nothing and returns no indication. We need to count hits, so we check first.
Exists is the same shape: variadic, returns a count (not a bool), because EXISTS k1 k2 k3 counts how many of the given keys exist.
serverFd, err := syscall.Socket(syscall.AF_INET, syscall.SOCK_STREAM, 0)
// AF_INET = IPv4, SOCK_STREAM = TCP, 0 = default protocol for this type
// serverFd is just an int (e.g. 3): we pass it to Bind, Listen, keventAdd belowRaw syscall: no Go net package. The three args tell the kernel what kind of socket to make:
AF_INET: IPv4 (vsAF_INET6for IPv6,AF_UNIXfor local)SOCK_STREAM: TCP: reliable, ordered, byte-stream (vsSOCK_DGRAMfor UDP)0: default protocol for this socket type (TCP)
The return value is an integer: a file descriptor. The kernel owns the actual socket state; you just hold the number and pass it back to interact with it. Nothing is open to the network yet: this just allocates the kernel-side structure.
syscall.SetsockoptInt(serverFd, syscall.SOL_SOCKET, syscall.SO_REUSEADDR, 1)
// SO_REUSEADDR: skip the 60s TIME_WAIT delay on restart
addr := syscall.SockaddrInet4{Port: 6379}
syscall.Bind(serverFd, &addr) // attach serverFd to port 6379
syscall.Listen(serverFd, 128) // tell kernel to accept SYNs; queue up to 128 unaccepted conns
syscall.SetNonblock(serverFd, true)
// without this, Accept() blocks when no client is waiting: freezes the whole event loop
// kqueue tells us when to call Accept(), so it will never block in practice: but nonblocking is a safety net
kq, _ := syscall.Kqueue()
// kq is another fd (e.g. 5): it's the event queue itself
keventAdd(kq, serverFd)
// register serverFd with kqueue: now the event loop will wake when a new client connects
// ↓ client fds are registered the same way in branch 1 of the event loop belowSO_REUSEADDR: when TCP closes, the kernel holds the port in TIME_WAIT for ~60s to absorb delayed packets. Without this, restarting the server immediately gives "address already in use." This option skips the wait.
Bind: claims port 6379. Before this the socket exists but is attached to nothing.
Listen(128): the 128 is the kernel's backlog: how many completed TCP handshakes it will queue waiting for Accept(). Beyond 128 it starts rejecting new connections.
SetNonblock: without this, Accept() blocks when no client is queued, freezing the whole event loop. Non-blocking means it returns EAGAIN immediately instead. Since kqueue already told us a client is waiting before we call Accept(), it will never actually hit EAGAIN: but non-blocking is the correct safety contract.
Kqueue + keventAdd: creates the event queue and registers serverFd as the first thing to watch. All client fds get added the same way later.
// events is make([]syscall.Kevent_t, 64): pre-allocated slice the kernel writes into
n, err := syscall.Kevent(kq, nil, events, nil)
// kq was set up above with Kqueue() + keventAdd(kq, serverFd)
// every Accept()ed client was also added: keventAdd(kq, connFd) in branch 1 below
// kernel parks our process here (zero CPU) until at least one watched fd is ready
// kernel fills events[0..n-1] with what fired, returns n: that's the wakeupThis is the only place the whole program sleeps. "We ask" and "kernel wakes us" are two sides of the same syscall: we hand control to the kernel, it parks our process (zero CPU), and only gives it back when at least one fd is ready. The contrast with polling is stark: polling would be a tight loop calling something like IsReady(fd), burning a full CPU core for nothing. Here, the OS scheduler doesn't even schedule our process between events.
for i := 0; i < n; i++ {
ev := events[i] // one ready event: kernel wrote this into our events slice
fd := int(ev.Ident) // Ident = the fd we passed to keventAdd(): serverFd or a connFd
// ev.Flags has status bits: EV_EOF (client hung up), EV_ERROR (broken socket)
// three possible situations: handled by the branches below
}n is how many fds fired this round. We loop through exactly those: events[n:] is stale from previous iterations and ignored.
ev.Ident is the fd that fired, stored as uint64 (kernel struct) and cast to int (what we use everywhere else).
ev.Flags carries status bits. We only care about two: EV_EOF (client hung up) and EV_ERROR (socket broken). Everything else means data is ready to read.
Each event is processed fully: parse, execute, reply: before moving to i+1. Serial, inline, no scheduling.
if fd == serverFd {
// serverFd was registered above: keventAdd(kq, serverFd)
// "readable" on a listening socket = new TCP handshake completed, client is in accept queue
connFd, _, _ := syscall.Accept(fd) // pop one client from the queue → new fd (e.g. 7)
syscall.SetNonblock(connFd, true) // same reason as serverFd: never block the loop
keventAdd(kq, connFd) // watch this client for future data events
clients[connFd] = newClient(connFd) // newClient() defined in client.go: allocates buf
}kqueue watched two kinds of fds: serverFd and every client fd. When it fires, all we get is a number: we don't know which kind. So we check first.
"Readable" means something different depending on the socket type. On a listening socket: a new TCP handshake completed: a client is in the accept queue. Accept() pops it out and returns a brand-new fd for that specific connection.
We register the new fd with kqueue immediately so future data from this client wakes us up. Without that, the client would connect but we'd never hear from them again.
} else if ev.Flags&syscall.EV_EOF != 0 || ev.Flags&syscall.EV_ERROR != 0 {
// EV_EOF: client sent TCP FIN: connection closed on their side
// EV_ERROR: socket is broken (e.g. network reset)
syscall.Close(fd) // release the kernel socket: without this, fd leaks forever
delete(clients, fd) // remove from the map created in serve(): client struct is GC'd
// kqueue automatically stops watching a closed fd
}EV_EOF means the client sent a TCP FIN: it closed its side. EV_ERROR means the socket is broken. In both cases we must Close(fd). If we don't, the fd number leaks: the kernel keeps the socket alive forever. The OS gives each process a hard limit on open fds (~65535 by default on macOS). Leave enough unclosed and new Accept() calls will start failing.
} else {
// not serverFd, not disconnected → existing client sent data
c := clients[fd] // clients map was populated in branch 1 when Accept() fired
if err := c.read(store); err != nil {
// c.read() is in client.go: returns err only on real socket error or EOF
syscall.Close(fd); delete(clients, fd)
}
// c.read() handles everything: read bytes → append to c.buf → parse → dispatch → write reply
}The happy path. c.read(store) does everything: reads bytes off the socket, appends to c.buf, parses RESP, dispatches to the right handler, writes the reply. It returns a non-nil error only if the socket itself is broken (distinct from a partial or malformed command, which it handles internally). No goroutine is spawned, no lock taken. When c.read() returns we immediately process the next event in the same loop iteration.
func keventAdd(kq, fd int) {
ev := syscall.Kevent_t{
Ident: uint64(fd), // which fd to watch
Filter: syscall.EVFILT_READ, // fire when fd has data to read
Flags: syscall.EV_ADD, // add this fd to the kqueue watch list
}
syscall.Kevent(kq, []syscall.Kevent_t{ev}, nil, nil)
// first arg = changes to apply, second arg = events to receive (nil = don't wait)
// this call returns immediately: it's registering intent, not waiting
}Kevent() has two modes depending on which args are nil:
- Non-nil changes, nil events → registration only, returns immediately (this function)
- Nil changes, non-nil events → blocking wait for notifications (the event loop)
EVFILT_READ means "fire when there are bytes to read." Same filter, different meaning: on serverFd it fires when a new connection is queued; on a client fd it fires when data arrived. That's why the event loop checks fd == serverFd first.
type client struct {
fd int // the fd returned by Accept(): needed to write the reply back
buf []byte // accumulates raw TCP bytes across multiple kqueue wakeups
// starts nil: grows lazily via append in c.read() below
}
// called from server.go branch 1: clients[connFd] = newClient(connFd)
func newClient(fd int) *client {
return &client{fd: fd} // buf starts as nil: first append allocates it
}The entire state for one connection: the socket fd (to write replies back) and a byte buffer.
Why a buffer? TCP is a byte stream, not a message stream. A single SET foo bar is 33 RESP bytes: TCP can deliver them in one packet, or split across two or three. We have no control over this. If we tried to parse on each read, we'd often get an incomplete command with no way to recover the rest.
Instead we accumulate into buf across multiple kqueue wakeups and only parse when ReadValue says we have enough. The buffer is per-client so client A's partial bytes never mix with client B's.
for len(c.buf) > 0 {
br := bytes.NewReader(c.buf) // wrap buf so we can measure how many bytes remain after parsing
reader := bufio.NewReader(br) // bufio reads ahead from br; needed for ReadLine inside ReadValue
val, err := ReadValue(reader) // defined in resp.go: reads one RESP value from the stream
if err != nil { break } // incomplete command: bytes stay in c.buf, wait for next kqueue event
// consumed = total bytes bufio pulled from br minus bytes bufio buffered but ReadValue didn't use
consumed := len(c.buf) - br.Len() - reader.Buffered()
c.buf = c.buf[consumed:] // advance past this command; next iteration handles the next one
response := dispatch(val.array, store) // dispatch() in commands.go: routes to handleSet/Get/etc.
syscall.Write(c.fd, response) // write reply directly to the socket fd
}Fresh readers are created each iteration because the consumed-bytes math requires br to start at the beginning of c.buf each time. bufio wraps br for line-by-line reading inside ReadValue; it also reads ahead, which is why consumed subtracts reader.Buffered().
Pipelining: a client can send 10 commands in one TCP write: they all land in c.buf together. The loop parses and executes them one by one without going back to Kevent().
Incomplete command: if ReadValue fails, there aren't enough bytes yet. We break, leave the partial bytes in c.buf, and wait for the next kqueue event to bring the rest.
bytes.NewReader + bufio.NewReader created per command: one allocation each. A production impl keeps a persistent reader per client. Correctness first.const (
typeSimpleString = '+' // +OK\r\n → marshalSimpleString() / handlePing, handleSet
typeError = '-' // -ERR message\r\n → marshalError() / dispatch default branch
typeInteger = ':' // :42\r\n → marshalInteger() / handleDel, handleExists
typeBulkString = '$' // $3\r\nfoo\r\n → readBulkString() / marshalBulkString()
// $-1\r\n (length=-1) → marshalNull(): signals missing key
typeArray = '*' // *3\r\n + 3 bulk strs → how ALL client commands arrive over the wire
)The first byte tells you everything about how to read the rest. No scanning, no ambiguity: that's why RESP is fast to parse.
+simple string: read until\r\n. Used for short status replies:+OK,+PONG.-error: same wire format as simple string, but signals failure. redis-cli shows these in red.:integer: read a number until\r\n. Used for counts:DEL,EXISTS.$bulk string: length-prefixed: read the number first, then exactly that many bytes. Handles binary data and values containing\r\n.$-1\r\nmeans null (key doesn't exist).*array: how every client command arrives: count, then that many bulk strings.SET foo bar→*3with three$elements.
// called from client.go: val, err := ReadValue(reader)
func ReadValue(r *bufio.Reader) (Value, error) {
b, err := r.ReadByte() // read the type byte: '+', '-', ':', '$', or '*'
if err != nil {
return Value{}, err // includes io.EOF → caller breaks out of parse loop
}
switch b {
case typeSimpleString: return readSimpleString(r) // read until \r\n
case typeError: return readSimpleString(r) // same wire format as simple string
case typeInteger: return readInteger(r) // parse the number after ':'
case typeBulkString: return readBulkString(r) // read length, then exact bytes
case typeArray: return readArray(r) // read count, then that many Values
}
return Value{}, fmt.Errorf("unknown type byte: %q", b)
}Reads one byte and dispatches. The entire parser is one switch.
If there isn't even one byte yet, ReadByte() returns io.EOF → propagates as an error → parse loop breaks and waits. That's how partial-command detection works at the topmost level.
readArray calls ReadValue recursively: once per element: because array elements are themselves full RESP values.
// called by ReadValue() when it sees a '$' type byte
func readBulkString(r *bufio.Reader) (Value, error) {
line, _ := readLine(r) // reads up to \r\n, returns "3" (without the \r\n)
length, _ := strconv.Atoi(line) // "3" → 3
if length == -1 {
// $-1\r\n is the null bulk string: means "key does not exist"
// handleGet returns this; redis-cli displays it as (nil)
return Value{typ: typeBulkString, isNull: true}, nil
}
buf := make([]byte, length+2) // +2 to consume the trailing \r\n after the content
io.ReadFull(r, buf) // reads exactly length+2 bytes, retrying if TCP split them
// plain Read() can return fewer bytes: ReadFull loops until done
return Value{typ: typeBulkString, str: string(buf[:length])}, nil
// strip the \r\n: we only keep buf[:length], not buf[:length+2]
}Read the length line → parse to int → allocate length+2 bytes (+2 for the trailing \r\n) → read exactly that many.
Why length == -1? That's the null bulk string: $-1\r\n. Redis uses it to mean "key doesn't exist": distinct from an empty string ($0\r\n\r\n).
Why io.ReadFull not Read? A normal Read() can return fewer bytes than requested: TCP delivers what's available right now, not what you asked for. io.ReadFull loops internally until every byte is in hand. Without it, large values get silently truncated.
The other read functions: readSimpleString, readInteger: are both just readLine plus optional parse. No interesting edge cases. Bulk string is the one worth studying because it introduces length-prefixing, the null sentinel, and the io.ReadFull contract.
// writing replies back to the client: mirror of the read functions above
func marshalSimpleString(s string) []byte {
return []byte("+" + s + "\r\n") // "+OK\r\n", "+PONG\r\n"
}
func marshalError(s string) []byte {
return []byte("-" + s + "\r\n") // "-ERR unknown command\r\n"
}
func marshalInteger(n int64) []byte {
return []byte(":" + strconv.FormatInt(n, 10) + "\r\n") // ":3\r\n"
}
func marshalBulkString(s string) []byte {
return []byte("$" + strconv.Itoa(len(s)) + "\r\n" + s + "\r\n")
// "$3\r\nbar\r\n" : length prefix, then content, then \r\n
}
func marshalNull() []byte {
return []byte("$-1\r\n") // null bulk string: "key does not exist"
}The write side mirrors the read side: same type bytes, same \r\n terminators.
marshalBulkString is the most-used reply. It length-prefixes so the client knows exactly how many bytes to read: no delimiter scanning needed.
Notice there's no marshalArray. The server only receives arrays (client commands). It never sends them: all replies are scalars.
// called from client.go parse loop: response := dispatch(val.array, store)
// val.array comes from ReadValue() parsing the *N RESP array the client sent
func dispatch(args []Value, store *Store) []byte {
cmd := toUpper(args[0].str) // args[0] = command name bulk string e.g. "set" → "SET"
switch cmd {
case "PING": return handlePing(args[1:]) // args[1:] = [] or ["hello"]
case "SET": return handleSet(store, args[1:]) // args[1:] = ["foo", "bar"]
case "GET": return handleGet(store, args[1:]) // args[1:] = ["foo"]
case "DEL": return handleDel(store, args[1:]) // args[1:] = ["k1", "k2", ...]
case "EXISTS": return handleExists(store, args[1:])
case "COMMAND": return marshalSimpleString("OK")
// redis-cli sends COMMAND on connect to introspect the server: stub it so cli doesn't hang
default: return marshalError("unknown command '" + cmd + "'")
// marshalError → "-unknown command '...'\r\n": redis-cli shows this in red
}
}RESP parsing is done: args is a slice of bulk strings. args[0] is the command name, args[1:] slices it off so each handler only sees its own arguments: handleSet gets [key, value], not [SET, key, value].
Commands are uppercased before switching because Redis is case-insensitive: set, SET, Set all work. toUpper is hand-rolled (subtract 32 for lowercase ASCII) to avoid importing strings for one function.
COMMAND is a real Redis introspection command that redis-cli fires automatically on connect. We stub it with +OK so the client doesn't stall waiting for a full response.
// all handlers receive args[1:] from dispatch: the command name is already stripped
func handlePing(args []Value) []byte {
if len(args) == 0 { return marshalSimpleString("PONG") } // → "+PONG\r\n"
return marshalBulkString(args[0].str) // PING hello → "$5\r\nhello\r\n": actual Redis spec
}
func handleSet(store *Store, args []Value) []byte {
if len(args) < 2 { return marshalError("wrong number of arguments for SET") }
store.Set(args[0].str, args[1].str) // store.Set defined in store.go: plain map write
return marshalSimpleString("OK") // → "+OK\r\n"
}
func handleGet(store *Store, args []Value) []byte {
val, ok := store.Get(args[0].str) // store.Get returns (value, found bool)
if !ok { return marshalNull() } // marshalNull() → "$-1\r\n": redis-cli shows (nil)
// different from empty string: $0\r\n\r\n
return marshalBulkString(val) // e.g. "bar" → "$3\r\nbar\r\n"
}
func handleDel(store *Store, args []Value) []byte {
keys := make([]string, len(args))
for i, a := range args { keys[i] = a.str } // unwrap []Value → []string (store has no RESP types)
return marshalInteger(store.Del(keys...)) // store.Del returns count of keys actually deleted
// → ":2\r\n" if 2 of the requested keys existed
}
func handleExists(store *Store, args []Value) []byte {
keys := make([]string, len(args))
for i, a := range args { keys[i] = a.str }
return marshalInteger(store.Exists(keys...)) // → ":1\r\n" etc.: count, not bool
}Every handler: validate args → call store → marshal result. The marshal call determines the wire type.
- PING →
+PONG\r\n, or echoes the arg as a bulk string (PING hello→$5\r\nhello\r\n) - SET →
+OK\r\n - GET →
$-1\r\nif missing, or$N\r\n{value}\r\nif found - DEL / EXISTS →
:N\r\n: count of keys deleted / found
The store doesn't speak RESP. Handlers are the bridge: they unwrap []Value into plain strings for the store, then wrap the result back into RESP bytes for the client.
EXPIRE, TTL): a background goroutine that tracks deadlines and sends deletes through the same dispatch path.