KDB knowledge

Key-value store (Redis-like)

GET, SET, INCR, EXPIRE, DEL, KEYS, MGET and the other Redis string commands, hashes, sets, sorted sets, lists (with BLPOP), SORT and WATCH / MULTI / EXEC over HTTP (POST /kv, one command or a pipeline) and as SQL functions (kv_get, kv_set, kv_incr); PUBLISH / SUBSCRIBE and keyspace events. Keys are kept in memory and in the table _kv.

Every database has a key-value store that answers Redis's commands with Redis 7's answers: GET, SET (with EX, NX, XX, GET, KEEPTTL), INCR, EXPIRE, TTL, DEL, KEYS, MGET, MSET and more, and hashes, sets, sorted sets and lists. Use it for counters, rate limits, login sessions, one-time codes, caches, leaderboards and job queues.

How
Over HTTPPOST /kv with one command or a list of them (below)
From SQL and the PostgreSQL portSELECT kv_incr('hits'), kv_get('k'), kv_set('k', 'v', 60) (below)
Look at the keysSELECT * FROM _kv (the table)
Publish and subscribePUBLISH news hi, and GET /listen?channel=news to hear it (below)

Why it is fast: the keys are kept in memory, so a command is a look-up, not a SQL statement with a plan and table locks. The changes of many clients go to the disk together: one commit and one disk sync confirm all the changes that arrived meanwhile.

POST /kv

Syntax:

POST /kv[?db=name][&sync=on|off]

The body is one command, or a list of commands (a pipeline: one round trip for all of them). Words are JSON strings; a number may be written as a number.

// one command
await fetch('/kv', { method: 'POST', body: JSON.stringify(['INCR', 'visits']) });   // -> [1]
// a pipeline
await fetch('/kv', { method: 'POST', body: JSON.stringify([
  ['SET', 'otp:1', '123456', 'EX', 120],
  ['GET', 'otp:1'],
  ['TTL', 'otp:1'],
]) });                                                                             // -> ["OK","123456",120]

Plain text works too, one command per line, as you would type it in redis-cli ("..." keeps spaces in a word):

await fetch('/kv', { method: 'POST', body: 'SET greeting "hello world"\nGET greeting' });  // -> ["OK","hello world"]

The answer is always a JSON array with one reply per command:

Redis replyJSON
a value, OK, PONGa string: "123456", "OK"
nil (no such key)null
a numbera number: 8
a list (MGET, KEYS, SCAN)an array
an error of that command{"error": "ERR value is not an integer or out of range"}

A command that fails does not stop the ones after it, as in a Redis pipeline. A body that is not a command list gets 400.

When the answer comes: sync

The answer comesAfter a crash or power cut
default (appendfsync everysec)at once, before the disk sync; the disk follows within millisecondsthe last few milliseconds may be lost (Redis's same default loses up to a second)
?sync=on, or CONFIG SET appendfsync alwaysonce the change is on disknothing confirmed is lost
?sync=offat once, whatever appendfsync saysas the default

The commands

CommandAnswers
GET keythe value, or null
SET key value [NX | XX] [GET] [EX s | PX ms | EXAT unix-s | PXAT unix-ms | KEEPTTL]"OK"; null when NX/XX did not let it set; with GET the old value
SETNX key value1 set, 0 the key exists
SETEX key seconds value, PSETEX key ms value"OK"
GETSET key value, GETDEL keythe old value
GETEX key [EX s | PX ms | EXAT | PXAT | PERSIST]the value; sets or removes its expiry
MGET key ...a list of values (null for missing keys)
MSET key value ..., MSETNX key value ..."OK"; MSETNX: 1, or 0 (nothing set) when any key exists
INCR, DECR, INCRBY key n, DECRBY key nthe new number; a missing key counts as 0; the key keeps its expiry
INCRBYFLOAT key xthe new number as text: "8.5"
APPEND key text, SETRANGE key offset textthe new length
STRLEN key, GETRANGE key start end (SUBSTR)length; part of the value (offsets in bytes, -1 = the last)
LCS key1 key2 [LEN]the longest common subsequence (or its length)
DEL key ..., UNLINK key ...how many keys were deleted
EXISTS key ..., TYPE keyhow many exist; "string", "hash", "set", "zset", "list" or "none"
EXPIRE key s [NX | XX | GT | LT], PEXPIRE, EXPIREAT, PEXPIREAT1 set, 0 not (no key, or the condition said no)
PERSIST key1 when an expiry was removed
TTL key, PTTL keyseconds (ms) left; -1 no expiry; -2 no key
EXPIRETIME key, PEXPIRETIME keywhen it expires (Unix seconds / ms); -1, -2 as above
KEYS patternthe matching keys, sorted (* any text, ? one character, [abc], [^a], [a-z], \ before a special character)
SCAN cursor [MATCH pattern] [COUNT n][next cursor, [keys]]; start at "0", stop when the cursor is "0" again; a key may come twice, but none is missed
RANDOMKEY, DBSIZEa key (or null); the number of keys
RENAME key new, RENAMENX key new, COPY src dst [REPLACE]"OK" / 1 / 0; the value keeps its expiry
FLUSHDB, FLUSHALL"OK": every key of this database is gone
PING [text], ECHO text, TIME"PONG"; the text; [seconds, microseconds]
COMMAND [COUNT | LIST | DOCS name ...], INFOthe commands this server knows; counters of the store (keys, memory, writes)
CONFIG GET pattern, CONFIG SET name value ...the memory cap and its policy (below)
MEMORY USAGE key, MEMORY STATS, MEMORY DOCTORthe bytes a key takes; the store's memory counters

Publish and subscribe

Redis's PUBLISH sends a message to every client that listens on a channel. Here it is the same channel as NOTIFY: a PUBLISH reaches LISTEN on the PostgreSQL port and GET /listen streams, and a NOTIFY reaches a pattern subscriber.

CommandAnswers
PUBLISH channel messagehow many listeners got it (0 when nobody listens)
SPUBLISH channel messagethe same (one server has one shard)
PUBSUB CHANNELS [pattern]the channels someone listens on now
PUBSUB NUMSUB channel ...[channel, count, ...]; pattern subscribers are not counted, as in Redis
PUBSUB NUMPAThow many different patterns are subscribed
PUBSUB SHARDCHANNELS, SHARDNUMSUBthe same as CHANNELS, NUMSUB

To subscribe, open a stream that stays open: a request and its answer cannot carry later messages, so SUBSCRIBE in /kv answers an error that says where to go.

RedisKDB
SUBSCRIBE newsGET /listen?channel=news (server-sent events), or LISTEN news on the PostgreSQL port
PSUBSCRIBE price.*GET /listen?pattern=price.*: each message also says which "pattern" it matched
// client 1: listen
const es = new EventSource('/listen?channel=news&pattern=price.*');
es.onmessage = (e) => console.log(JSON.parse(e.data));
// client 2: publish
await fetch('/kv', { method: 'POST', body: JSON.stringify([['PUBLISH', 'news', 'hi']]) });         // -> [1]
// client 1 gets {"channel":"news","payload":"hi","pid":0}
await fetch('/kv', { method: 'POST', body: 'PUBLISH price.NABIL 512' });                          // -> [1]
// client 1 gets {"channel":"price.NABIL","payload":"512","pid":0,"pattern":"price.*"}

Example: Publish from SQL

SELECT kv('PUBLISH', 'news', 'market opens at 11') AS listeners,
       kv('PUBSUB', 'NUMSUB', 'news') AS numsub;

Result:

listeners (LONG)numsub (STRING)
0["news",0]

(1 row)

Keyspace events

With keyspace events on, every key change is published too, on Redis's channels:

ChannelPayload
__keyspace@0__:<key> (letter K)the event: set, del, expire, expired, ...
__keyevent@0__:<event> (letter E)the key

@0 is Redis's first database, so a Redis client's subscription works unchanged; the channels still belong to one KDB database.

Turn them on with Redis's letters, as a server admin:

await fetch('/kv', { method: 'POST', body: 'CONFIG SET notify-keyspace-events Ex' });   // expired keys, keyevent channel
const es = new EventSource('/listen?channel=__keyevent@0__:expired');                  // payload: the key that expired
LetterEvents
K, Ewhich channels: keyspace, keyevent (at least one is needed)
gdel, expire, persist, rename_from, rename_to, copy_to
$set, incr, decr, incrby, decrby, incrbyfloat, append, setrange
xexpired: the key's time came; sent within about 10 ms of it, whether or not anybody reads the key
eevicted: a key taken by the memory cap
nnew: a key that did not exist was made
l, s, h, zlist, set, hash and sorted set events, with Redis's names: lpush, rpop, sadd, srem, hset, hdel, zadd, zincr, ...
Aevery class above but n (Redis's g$lshzxetd)
t, d, mtaken, as Redis takes them; nothing here sends them yet

Example: Read the setting

SELECT kv('CONFIG', 'GET', 'notify-keyspace-events');

Result:

kv('CONFIG', 'GET', 'notify-keyspace-events') (STRING)
["notify-keyspace-events",""]

(1 row)

Example: Only an admin over HTTP may change it (shows an error)

SELECT kv('CONFIG', 'SET', 'notify-keyspace-events', 'KEA');

Result:

Error: line 1, column 8: kv: NOPERM CONFIG SET notify-keyspace-events changes the whole server: it needs a server admin

A memory cap: CONFIG SET maxmemory

The store can be a cache with a size limit, as Redis with maxmemory:

await fetch('/kv', { method: 'POST', body: JSON.stringify([
  ['CONFIG', 'SET', 'maxmemory', '64mb', 'maxmemory-policy', 'allkeys-lru'],
]) });                                                       // -> ["OK"]
SettingValues
maxmemorythe cap: 64mb, 512kb, 1gb or bytes; 0 = no cap (the default)
maxmemory-policynoeviction (the default), allkeys-lru, volatile-lru, allkeys-random, volatile-random, volatile-ttl
appendonlyyes (the default): keys are written to _kv and survive a restart. no: memory only, a restart starts empty
appendfsynceverysec (the default): an answer comes before the disk sync. always: it waits for it. no: as everysec (sync)

Example: A store of 1 MB that forgets the oldest keys

SELECT kv('CONFIG', 'SET', 'maxmemory', '1mb', 'maxmemory-policy', 'allkeys-lru');
SELECT kv_set('session:1', 'sita'), kv('MEMORY', 'USAGE', 'session:1'), kv_memory();
SELECT kv('CONFIG', 'GET', 'maxmemory*');
SELECT kv('CONFIG', 'SET', 'maxmemory', '0', 'maxmemory-policy', 'noeviction');

Result:

SELECT kv('CONFIG', 'SET', 'maxmemory', '1mb', 'maxmemory-policy', 'allkeys-lru')

kv('CONFIG', 'SET', 'maxmemory', '1mb', 'maxmemory-policy', 'allkeys-lru') (STRING)
OK

(1 row)

SELECT kv_set('session:1', 'sita'), kv('MEMORY', 'USAGE', 'session:1'), kv_memory()

kv_set('session:1', 'sita') (STRING)kv('MEMORY', 'USAGE', 'session:1') (LONG)kv_memory() (LONG)
OK7777

(1 row)

SELECT kv('CONFIG', 'GET', 'maxmemory*')

kv('CONFIG', 'GET', 'maxmemory*') (STRING)
["maxmemory","1048576","maxmemory-policy","allkeys-lru","maxmemory-samples","16"]

(1 row)

SELECT kv('CONFIG', 'SET', 'maxmemory', '0', 'maxmemory-policy', 'noeviction')

kv('CONFIG', 'SET', 'maxmemory', '0', 'maxmemory-policy', 'noeviction') (STRING)
OK

(1 row)

Hashes, sets, sorted sets and lists

A key can hold more than one text. The commands and answers are Redis 7's; COMMAND DOCS HSET shows the syntax of each one.

TypeCommandsUse it for
hashHSET, HGET, HMGET, HGETALL, HDEL, HEXISTS, HLEN, HKEYS, HVALS, HINCRBY, HINCRBYFLOAT, HSETNX, HSTRLEN, HRANDFIELD, HSCANa record: a user, a cart, the fields of one session
setSADD, SREM, SMEMBERS, SISMEMBER, SMISMEMBER, SCARD, SINTER, SUNION, SDIFF (and ...STORE), SINTERCARD, SMOVE, SPOP, SRANDMEMBER, SSCANtags, who is online, unique visitors
sorted setZADD, ZINCRBY, ZREM, ZSCORE, ZMSCORE, ZCARD, ZCOUNT, ZRANK, ZREVRANK, ZRANGE (BYSCORE, BYLEX, REV, LIMIT, WITHSCORES), ZRANGEBYSCORE, ZPOPMIN, ZPOPMAX, ZUNION, ZINTER, ZDIFF (and ...STORE), ZREMRANGEBY..., ZRANDMEMBER, ZSCANleaderboards, things due at a time (score = Unix time)
listLPUSH, RPUSH, LPOP, RPOP, LLEN, LRANGE, LINDEX, LSET, LINSERT, LREM, LPOS, LTRIM, LMOVE, LMPOP, RPOPLPUSHqueues, the last n events
any of themSORT key [BY pattern] [LIMIT o n] [GET pattern ...] [ASC | DESC] [ALPHA] [STORE dst], SORT_ROthe values sorted, or values of other keys they point to
await fetch('/kv', { method: 'POST', body: JSON.stringify([
  ['HSET', 'user:1', 'name', 'a', 'age', '3'],
  ['HINCRBY', 'user:1', 'age', '2'],
  ['HGETALL', 'user:1'],
  ['ZADD', 'board', 10, 'a', 20, 'b'],
  ['ZRANGE', 'board', 0, -1, 'WITHSCORES'],
]) });                                   // -> [2, 5, {"name":"a","age":"5"}, 2, ["a","10","b","20"]]

Example: A hash and a leaderboard from SQL

SELECT kv('HSET', 'user:1', 'name', 'Sita', 'age', '30'), kv('HINCRBY', 'user:1', 'age', '1'), kv('HGETALL', 'user:1');
SELECT kv('ZADD', 'board', '10', 'ram', '25', 'sita'), kv('ZRANGE', 'board', '0', '-1', 'REV', 'WITHSCORES');

Result:

SELECT kv('HSET', 'user:1', 'name', 'Sita', 'age', '30'), kv('HINCRBY', 'user:1', 'age', '1'), kv('H

kv('HSET', 'user:1', 'name', 'Sita', 'age', '30') (LONG)kv('HINCRBY', 'user:1', 'age', '1') (LONG)kv('HGETALL', 'user:1') (STRING)
231{"name":"Sita","age":"31"}

(1 row)

SELECT kv('ZADD', 'board', '10', 'ram', '25', 'sita'), kv('ZRANGE', 'board', '0', '-1', 'REV', 'WITH

kv('ZADD', 'board', '10', 'ram', '25', 'sita') (LONG)kv('ZRANGE', 'board', '0', '-1', 'REV', 'WITHSCORES') (STRING)
2["sita","25","ram","10"]

(1 row)

Waiting for a value: BLPOP

BLPOP key [key ...] timeout takes the first value of the first list that has one. When all are empty it waits until another client pushes, up to timeout seconds (0.5 works; 0 = no limit), then answers null. BRPOP, BLMOVE, BRPOPLPUSH, BLMPOP, BZPOPMIN, BZPOPMAX and BZMPOP wait the same way.

// a worker: waits up to 5 s for a job
const [job] = await (await fetch('/kv', { method: 'POST', body: JSON.stringify(['BLPOP', 'jobs', 5]) })).json();
// -> ["jobs", "job-17"] as soon as someone runs ['RPUSH', 'jobs', 'job-17'], or null after 5 s

WATCH, MULTI and EXEC

MULTI queues the commands after it and EXEC runs them together: no other client's command comes in between. WATCH key ... before MULTI makes EXEC answer null (and run nothing) when another client changed one of those keys meanwhile, so you read again and retry: an optimistic lock.

A Redis connection keeps its WATCH from one command to the next. Over HTTP, give your requests a session name: POST /kv?session=name. Requests with the same name (from the same user) share their WATCH and MULTI; a session ends 10 minutes after its last request. Without session, they last for that one request.

// request 1: read the stock and watch it
POST /kv?session=order-7   [["WATCH","stock"],["GET","stock"]]            // -> ["OK","5"]
// request 2: write it only if nobody changed it since
POST /kv?session=order-7   [["MULTI"],["SET","stock","4"],["EXEC"]]       // -> ["OK","QUEUED",["OK"]], or [..., null] to retry

From SQL: kv_get, kv_set, kv_incr

Syntax:

kv_get(key)                     -- GET
kv_set(key, value [, seconds])  -- SET key value [EX seconds]
kv_incr(key [, n])              -- INCRBY key n (1 when left out)
kv_del(key)                     -- DEL
kv_expire(key, seconds)         -- EXPIRE
kv_ttl(key)                     -- TTL
kv(command, word, ...)          -- any command: kv('MGET', 'a', 'b')

The same store, from any SQL client and the PostgreSQL port. A list answer (MGET, KEYS, LRANGE) comes as JSON text, and HGETALL as a JSON object. A NULL word gives NULL and changes nothing.

Example: A counter and a code that expires

SELECT kv_incr('page:home');
SELECT kv_incr('page:home', 10);
SELECT kv_set('otp:42', '815204', 120), kv_get('otp:42'), kv_ttl('otp:42');
SELECT kv('MGET', 'page:home', 'otp:42', 'nothing');

Result:

SELECT kv_incr('page:home')

kv_incr('page:home') (LONG)
1

(1 row)

SELECT kv_incr('page:home', 10)

kv_incr('page:home', 10) (LONG)
11

(1 row)

SELECT kv_set('otp:42', '815204', 120), kv_get('otp:42'), kv_ttl('otp:42')

kv_set('otp:42', '815204', 120) (STRING)kv_get('otp:42') (STRING)kv_ttl('otp:42') (LONG)
OK815204120

(1 row)

SELECT kv('MGET', 'page:home', 'otp:42', 'nothing')

kv('MGET', 'page:home', 'otp:42', 'nothing') (STRING)
["11","815204",null]

(1 row)

Example: A counter needs a number (shows an error)

SELECT kv_set('name', 'sita'), kv_incr('name');

Result:

Error: line 1, column 32: kv_incr: ERR value is not an integer or out of range

The table _kv

The keys live in the table _kv of the database, made at the first change:

ColumnType
kSTRING PRIMARY KEYthe key
vSTRINGthe value
expires_atTIMESTAMPwhen it expires; NULL = never (row TTL)

Hashes, sets, sorted sets and lists are in a second table, _kv_items, made when the first one is: one row with the key's type (t) and expiry, and one row per field, member or list item (f, v). So HSET of one field writes one row, however big the hash is.

SELECT k, t, f, v FROM _kv_items WHERE k = 'user:1';
-- user:1 | hash | NULL | NULL
-- user:1 | NULL | age  | 31
-- user:1 | NULL | name | Sita

Only the store changes _kv_items too.