Troubleshooting a running server

What to look at, and what to change, when the server is slow, readers wait, memory or disk grows, the start is slow, a timer or a backup does not run, or data must come back.

• Category: Backup, check, repair, restore
• Keywords: troubleshooting, runbook, production problem, incident, server slow, slow server, slow queries, readers wait, reads wait, busy, table is busy, 55P03, 40001, lock timeout, memory high, out of memory, memory full, disk full, disk growing, disk space, trash, keep_versions, last shutdown: not clean, slow start, start-up slow, log replay, timer not running, timer did not run, backup failed, restore, what to do, symptom
• See also: Monitor panel (memory, processor, caches), Health, info, check and backup over HTTP (metrics), Health, info, check and backup over HTTP (slow-queries), Transactions (lock-timeout), kdb serve (memory-limit), Databases and the data folder (purge-trash), Timers, Backups on a schedule, Backup, check, repair and restore (after-damage)

Each part below is one symptom: what you see, where to check, and the fix. Start with the console's Monitor: it shows memory, the statements running now and the slow statements on one page.

The checks use three routes. Send the token as for any route: Authorization: Bearer <token>.

The server is slow

• Fix: give a time range to queries of big time-series tables, add an index for lookups by one column, and see performance tips. Stop a runaway statement with Cancel, and limit them with --statement-timeout (protection).

HTTP: The newest slow statements

GET /api/slow-queries?limit=5

Response:

HTTP 200 (application/json)

{
  "threshold_ms": 1000,
  "recorded": 0,
  "queries": []
}

Example: How much of the table a query reads

Setup:

CREATE TABLE ticks (time TIMESTAMP, symbol SYMBOL, price DOUBLE) TIMESERIES(time) PARTITION BY DAY;
INSERT INTO ticks VALUES ('2026-09-23T10:00:00Z', 'CITY', 510), ('2026-09-24T10:00:00Z', 'CITY', 512);

EXPLAIN SELECT max(price) FROM ticks WHERE time IN '2026-09-24';

Result:

(6 rows)

Readers wait for a writer

A SELECT that is quick at night takes seconds by day, while a load, a trigger or a long CALL writes the same table.

• Fix:
  • Keep reader yield on (--reader-yield on, KDB_READER_YIELD=on; on unless set). Readers then get in between the writer's statements.
  • Turn on snapshot reads (--snapshot-reads on, KDB_SNAPSHOT_READS=on) so readers do not wait even for one long statement, also on tables with an index of up to 1,000,000 rows (--snapshot-index-rows).
  • Split one huge statement (INSERT ... SELECT of millions of rows) into smaller ones: readers can only get in between statements.

Errors 55P03 and 40001: the table is busy

table 'x' is busy: another transaction has held it for N ms means a writer waited the lock timeout (5 seconds unless set) for another writer of the same table. Nothing of the statement was done.

• Fix: keep transactions short. For many small writers of one table that should queue instead of failing, raise the wait for them: SET lock_timeout = '20s' in the session, X-KDB-Lock-Timeout: 20s on one request, or --lock-timeout for the server.

Example: Let this session wait longer for a busy table, then go back to the server's wait

SET lock_timeout = '20s';
SET lock_timeout = DEFAULT;

Result:

SET lock_timeout = '20s'

lock_timeout is 20 s for this session

SET lock_timeout = 'DEFAULT'

lock_timeout is the server's default again

Memory is high

• Fix: allocator big: set GLIBC_TUNABLES as above (see Docker). table_data big: lower --memory. A query that needs much memory: add a time range or a LIMIT. A transaction that fails with transaction too big for memory: split it.

The disk keeps growing

HTTP: What a database takes on disk

Setup:

CREATE TABLE ticks (time TIMESTAMP, price DOUBLE) TIMESERIES(time) PARTITION BY DAY;
INSERT INTO ticks VALUES ('2026-09-24T10:00:00Z', 1.5);

GET /api/disk

Response:

HTTP 200 (application/json)

{
  "database": "demo",
  "data_files": 1,
  "data_bytes": 236,
  "tail_files": 0,
  "tail_bytes": 0,
  "delta_files": 0,
  "delta_bytes": 0,
  "previous_files": 0,
  "previous_bytes": 0,
  "trash_folders": 0,
  "trash_bytes": 0,
  "wal_bytes": 1048640,
  "wal_archive_bytes": 0,
  "total_bytes": 1048876,
  "trash_purged_bytes": 0
}

Example: Free older file versions and the trash

Setup:

CREATE TABLE ticks (time TIMESTAMP, price DOUBLE) TIMESERIES(time) PARTITION BY DAY;
INSERT INTO ticks VALUES ('2026-09-24T10:00:00Z', 1.5);

PURGE VERSIONS KEEP 1;
PURGE TRASH;

Result:

PURGE VERSIONS KEEP 1

0 older file version(s) removed (0 bytes), 1 kept per file; freed 0 bytes from the trash

PURGE TRASH

removed 0 folder(s) from the trash, freed 0 bytes

The start is slow, or says "last shutdown: not clean"

last shutdown: not clean (crash or kill) in the start-up lines means the server was killed, not stopped. That is not damage: KDB replays the log, and every confirmed change comes back. Replaying a long log takes time.

• Fix: stop the server so it can finish: Ctrl+C, docker stop, or a service stop. A Docker container needs --stop-timeout 60 when it is created; without it, Docker Desktop kills it about 1 second after asking it to stop, and every start replays the log (Docker, stopping). Do not use docker rm -f or docker kill on a running server.

A timer does not run

HTTP: Every timer, with its last run and error

GET /api/timers

Response:

HTTP 200 (application/json)

{
  "now_ms": 1792647875758,
  "timers": []
}

A backup failed

Then run one backup now to see that it works: POST /api/backup, or Back up now in the console.

Getting data back

Command line: Check a database, then repair it from the backups (sample data: sample)

kdb check market
kdb repair market --backups backups

Output:

> kdb check market

checked 14 files (76.1 KB) and 1 log segments
OK: no damage found

> kdb repair market --backups backups

manifest copy: a, last shutdown: clean
replayed 0 transactions after LSN 30, last LSN 30

Nothing to repair.