Monitor panel (memory, processor, caches)

What the server is doing right now - memory it holds, processor it uses, what each database keeps in memory, deferred trigger queues, and the statements running this second.

• Category: Web console
• Keywords: monitor, timers, timer failures, late timer, Monitor panel, monitoring, memory, RAM, resident, working set, processor, CPU, threads, uptime, running statements, cancel statement, kill query, result cache, cache hits, cached results, memory budget, loaded partitions, pending rows, checkpoint, open cursors, log position, LSN, deferred trigger queue, trigger lag, failed job, /api/stats, activity, activity log, server log, terminal, what the server is doing, GET /api/activity, --activity-log-size, X-KDB-Quiet, slow statements, slow queries, --slow-query-ms, disk use, on disk, trash, metrics
• See also: Web console, Settings page (data folder, old log, capacity, program files and users), View data page (browse and edit rows), Health, info, check and backup over HTTP, Performance tips, Protection - timeouts, limits, safe updates, read-only, kdb serve, Triggers

The Monitor panel of the web console shows the inside of a running kdb serve: how much memory the process holds, how busy the processor is, what each open database keeps in memory, and which statements are running right now. Open it with the Monitor button on the right side of the console. It is a panel, not a page: pin it to keep it open beside the page you work on.

It reads GET /api/stats every couple of seconds. Everything there is a counter already in memory, so asking often costs almost nothing: no file is read and no database is opened to answer it.

• Any login that may read can see the panel. A read user sees only their own running statements, because statement text can hold values; an admin sees everyone's.
• Refresh at the top picks 2, 5, 10 or 30 seconds, or Off; Now reads the numbers at once. The panel stops asking while it is closed or the browser tab is hidden.
• /api/stats holds more than the panel shows, for example the zone map counters (zone_partition_skips, zone_block_skips), cache_pinned and the memory of each database by kind. Read it from a script for those (GET /api/stats).

Warnings and databases opening

At the top, in a yellow frame, the server's warnings: today, a database far bigger than --memory (why it matters).

Below them, one card per database being opened right now. A big database replays the log written after its last checkpoint, which over a slow disk takes minutes, and its first request waits that long. The card shows what the open does now (checking log, replaying log, loading tables, ...), for how long, how much of the log is replayed, and how many table files it went through. When the numbers move, the open is working; the card goes away when the database is open.

In a database's card, a Checkpoint line shows a checkpoint writing changes to partition files: the table, partitions written of how many, and for how long. Busy means a write or a checkpoint held the database when the panel asked: the memory numbers are live, the tables and queues are from the reading before. The panel never waits for a busy database.

The server process

Memory in RAM is bigger than the memory budget below it: the budget covers table data, while the process also holds indexes, results being built, and the program itself.

Statements running now

One line per statement, over HTTP and the PostgreSQL port together:

Cancel needs a name to cancel by, which is the X-KDB-Request-Id header the client sent. A client that sent none shows no id instead; it still stops on its own statement timeout. The console itself always sends one. There is no Cancel at all on a server started without query protection (can_cancel is false in /api/stats).

A statement that is stuck is usually one of the things in performance tips: a SELECT with no time range on a big time-series table, a join without a matching key, or a GROUP BY over too many groups.

Lanes

Shown while priority lanes are on: the heavy slots in use, how many reports wait for one, how many statement shapes are known to be heavy, and whether heavy statements run at a lower CPU priority here. The table has one line per lane: how many run and wait now, how many started since the server did, how many had to wait and for how long (average and longest), how many were refused because the HTTP line was full (only requests sent with priority low), how many ran at once because it was full (heavy only by history), and how many gave up waiting (a timeout or a cancel).

Slow statements

The newest 20 statements that took at least the slow-query threshold (kdb serve --slow-query-ms, 1 second by default), from GET /api/slow-queries. They are kept in a file on disk, so last night's slow statement is still here in the morning, after a restart too.

The list is read every 10 seconds at most, and at once with Now.

Timers

Every timer of the databases you may read, also of databases not open yet. Timers in trouble come first: failing now, then failed in the last 24 hours, then the latest to start. The title counts the timers with failures.

The numbers come with the rest of /api/stats (timers), from memory: nothing is read from disk for them.

Each database

Only databases that are open are listed. A database opens the first time it is used, so a server that has just started may show none - that is not a fault.

A database with deferred triggers also shows Deferred trigger queues, one line per trigger:

The same numbers come from GET /api/trigger-queue, which can also wait until the jobs are done.

Tables opens a line per table:

This is the quickest way to see which table is using the memory budget, and whether a big load has been written to disk yet.

Server activity

The terminal at the bottom of the Home page shows what the server has done lately, newest at the bottom: every statement that ended, every bulk load and where it came from, timer runs, deferred trigger jobs, logins and backups. It reads GET /api/activity, described here.

• It is kept in memory only: the last 5,000 events (kdb serve --activity-log-size N, 0 turns it off). A restart starts it empty. For a record that is kept, use the audit log.
• Passwords, secrets and tokens in SQL show as '***', and long statements are cut to 300 characters.
• A bulk load is one event per request, never one per row, so logging does not slow loading down.
• Requests with the header X-KDB-Quiet: 1 are not logged. The console sends it with its own background refreshes.

Syntax:

GET /api/activity[?after=<seq>][&limit=<n>][&db=<name>]

The reply is {"seq", "dropped", "events"}. seq is the newest event's number: send it back as after to get only what is new. dropped is true when some events after after were already pushed out of memory.

An admin (or the token) sees every event; any other user sees only their own.

HTTP: Run a statement, then read the newest events

POST /exec?compat=0

SELECT 1 + 1 AS two
###
GET /api/activity?limit=2

Response:

POST /exec?compat=0

HTTP 200 (application/json)

{
  "query": "SELECT 1 + 1 AS two",
  "results": [
    {
      "statement": "SELECT 1 + 1 AS two",
      "columns": [
        {"name": "two", "type": "LONG"}
      ],
      "dataset": [
        [2]
      ],
      "count": 1,
      "truncated": false,
      "rows_affected": 0,
      "message": "",
      "timings": {"parse_us": 35, "execute_us": 68}
    }
  ],
  "timings": {"parse_us": 35, "execute_us": 68}
}

GET /api/activity?limit=2

HTTP 200 (application/json)

{
  "seq": 3,
  "dropped": false,
  "events": [
    {
      "seq": 2,
      "ms": 1792647811577,
      "kind": "query",
      "level": "ok",
      "user": "",
      "client": "127.0.0.1 (http)",
      "database": "h119",
      "text": "CREATE TABLE acct (id INT PRIMARY KEY, name STRING, bal DOUBLE); INSERT INTO acct VALUES (1, 'Asha', 10);",
      "rows": 1,
      "took_ms": 5,
      "error": ""
    },
    {
      "seq": 3,
      "ms": 1792647815921,
      "kind": "query",
      "level": "ok",
      "user": "",
      "client": "127.0.0.1 (http)",
      "database": "demo",
      "text": "SELECT 1 + 1 AS two",
      "rows": 1,
      "took_ms": 0,
      "error": ""
    }
  ]
}

What it does not show

• Numbers of the past: the panel keeps nothing, so a reload starts fresh. For a history, let Prometheus read GET /metrics, or read /api/stats from a script every minute and keep the rows in a table.
• Statements that finished long ago. The activity keeps only the recent ones in memory; the audit log (SHOW AUDIT) keeps who changed what.
• Damage and backups; those are in the Health panel on the right side of the console and in /api/info.

curl -s -H "Authorization: Bearer $KDB_TOKEN" http://127.0.0.1:9900/api/stats