Introduction

mq-db treats Markdown documents as structured, hierarchical databases rather than plain text.

It parses Markdown into a flat block list annotated with an interval index (Nested Set / Pre-Post Order), which turns heading-hierarchy questions like “is this paragraph inside that section?” into a single O(1) integer comparison instead of a tree walk. Documents can be queried with SQL or mq, and persisted to a compact custom page-file format with no SQLite dependency.

This project is under active development and the API may change.

Why Markdown-as-database?

Markdown files already have implicit structure: headings nest sections, code blocks carry a language, front matter carries metadata. mq-db makes that structure queryable directly:

SELECT block_type, count(*) FROM blocks GROUP BY block_type;

.h1

Both engines run against the same underlying block store, so you can pick whichever query language fits the task: SQL for joins, aggregates, and ad-hoc analysis; mq for Markdown-shaped transformations and selectors.

How it fits together

Markdown File(s)
      │  CST Parser (mq-markdown)
      ▼
Block Tree (heading · paragraph · code · list · …)
      │  Interval Index + Secondary Indexes
      ▼
Flat Block Vector (pre/post integers)
      │
      ├── BitmapIndex   (block_type)
      ├── BTreeIndex    (pre / post)
      ├── HashIndex     (content / lang / depth)
      ├── Zone Maps     (per-document stats)
      │
      ├── SQL Engine   (sqlparser, custom native evaluator)
      └── mq Engine    (mq-lang evaluator)

Features

• Flat block storage: every Markdown element becomes a typed Block with row-polymorphic properties
• O(1) hierarchy queries: interval index (pre/post) makes ancestor/descendant checks a single integer comparison
• Four-layer secondary indexes: BitmapIndex (block type), BTreeIndex (pre/post), HashIndex (content/lang/depth), TermIndex (tokenized content, full-text) for fast SQL predicate pushdown
• Zone Maps: per-document statistics skip irrelevant files before scanning any blocks
• Dual query engines: SQL via a custom sqlparser-based evaluator, and mq via mq-lang
• WITH / WITH RECURSIVE support: common table expressions, see CTEs
• Full-text search: match()/score()/bm25() SQL functions, see Full-Text Search
• EXPLAIN / EXPLAIN ANALYZE: inspect and profile a query’s plan, see EXPLAIN
• SQL INSERT/UPDATE/DELETE with write-back: edit blocks and push the change back to the source file, see Write-Back
• DDL support: CREATE TABLE, INSERT INTO, DROP TABLE, and views, see DDL Statements
• Comprehensive SQL function library: string, numeric, null-handling, CASE, and aggregate functions comparable to a general-purpose RDBMS
• mq() scalar function: run an mq program against Markdown content inline in SQL
• read_csv() / read_json() table functions: query external files directly, see External Files
• ATTACH DATABASE / DETACH: query across multiple .mq-db stores, session-scoped like SQLite
• Custom page-file persistence: 8 KB fixed pages, checksums, atomic writes
• CLI + interactive REPL + TUI: full terminal experience

Keep reading in Getting Started, or jump straight to the SQL Reference if you already have a .mq-db store.

Getting Started

This section walks through installing mq-db, indexing Markdown into a store file, and querying it from the CLI, REPL, TUI, HTTP server, or the Rust library directly.

The typical workflow is:

1. Index one or more Markdown files into a .mq-db store file (Install, CLI).
2. Query the store with SQL or mq, either one-shot from the CLI, interactively in the REPL / TUI, or over HTTP via mq-db serve.
3. Optionally, embed mq-db directly with the Library API instead of shelling out to the CLI.

mq-db index docs/ --recursive --output store.mq-db
mq-db sql "SELECT block_type, count(*) FROM blocks GROUP BY block_type" --db store.mq-db

Install

Using the Installation Script (Recommended)

curl -fsSL https://raw.githubusercontent.com/harehare/mq-db/main/bin/install.sh | bash

The installer will:

• Download the latest release for your platform
• Verify the binary with a SHA256 checksum
• Install to ~/.local/bin/
• Update your shell profile (bash, zsh, or fish)

After installation, restart your terminal or run:

source ~/.bashrc  # or ~/.zshrc, or ~/.config/fish/config.fish

Using Cargo

cargo install mq-db

From Source

# Latest development version
cargo install --git https://github.com/harehare/mq-db.git

Supported Platforms

• Linux: x86_64, aarch64
• macOS: x86_64 (Intel), aarch64 (Apple Silicon)
• Windows: x86_64

Verify

mq-db --version

CLI

Every subcommand operates on a .mq-db store file (--db / -d, default store.mq-db). Output-producing commands accept --format / -F: table (default), json, csv, tsv, markdown, html.

mq-db --help
mq-db <command> --help

index

Index Markdown files or directories into a store file.

mq-db index docs/ --recursive --output store.mq-db
mq-db index README.md DESIGN.md
mq-db index docs/ --no-spans   # omit source spans (~21 bytes/block saved)

  ✓ docs/DESIGN.md
  ✓ docs/API.md

Indexed 2 files → store.mq-db

list

List all indexed documents.

mq-db list --db store.mq-db
mq-db list --db store.mq-db --format json

┌──────┬────────────────────────────────────────────────────┬────────┬──────────┐
│   ID │ Path / Title                                       │ Blocks │ Tags     │
├──────┼────────────────────────────────────────────────────┼────────┼──────────┤
│    0 │ docs/DESIGN.md                                     │    142 │          │
│    1 │ docs/API.md                                        │     87 │ api, v2  │
└──────┴────────────────────────────────────────────────────┴────────┴──────────┘
2 documents

find

Full-text search shortcut for match()/bm25(), no SQL needed. See Full-Text Search.

mq-db find "error handling" --db store.mq-db
mq-db find "error handling" --db store.mq-db -n 5 -F json   # top 5, JSON

sql

Run a SQL query over the store. See the Reference for the virtual schema and function library, including CTEs, full-text search, EXPLAIN, external files, and write-back.

mq-db sql "SELECT block_type, count(*) FROM blocks GROUP BY block_type" --db store.mq-db
mq-db sql --file query.sql --db store.mq-db
mq-db sql "SELECT ..." --db store.mq-db --format json

mq

Run an mq query over the store.

mq-db mq ".h1" --db store.mq-db
mq-db mq 'select(.code_lang == "rust")' --db store.mq-db
mq-db mq ".h1" --db store.mq-db --format markdown

repl

Interactive REPL supporting both query modes; switch with .mode.

mq-db repl --db store.mq-db --mode sql

See REPL for the full command list.

lint

Run structural lint checks (currently: a heading at the given depth immediately followed by a list).

mq-db lint --db store.mq-db --depth 2

✗  1 violation  (H2 immediately followed by list)

  file                                      heading
  ────────────────────────────────────────  ──────────────────────────────
  docs/DESIGN.md                            "Quick Start"

stats

Show store-wide statistics: document/block counts, block-type distribution, code-language distribution.

mq-db stats --db store.mq-db

  Documents  5
  Blocks     632

  Block types
  ────────────────────────────────────────────────────────
   ¶  paragraph    ████████████████████░░░░   241  (38%)
   #  heading      ████████░░░░░░░░░░░░░░░░    89  (14%)
  {}  code         ███████░░░░░░░░░░░░░░░░░    73  (12%)
   •  list         ██████░░░░░░░░░░░░░░░░░░    58   (9%)

vacuum

UPDATE/DELETE write-back, DROP TABLE/DROP VIEW, and re-indexing a changed file all replace or remove data by writing a fresh page chain and abandoning the old one, so the .mq-db file only grows. vacuum rewrites the file from scratch (same compaction save/index already do for a brand-new file) and reclaims that dead space:

mq-db vacuum --db store.mq-db

  Pages before   190
  Pages after    123
  Reclaimed      536.0 KB

VACUUM as a SQL statement is recognized but redirects to this command rather than doing something different or silently failing: mq-db’s compaction operates on the whole store file, not per-table, so it doesn’t fit the --write-back/execute_sql_mut path the way UPDATE/DELETE do.

show

Show the full block structure of one document by ID (see list for IDs).

mq-db show 0 --db store.mq-db

  docs/DESIGN.md
  title   Design Document
  blocks  142

  pre   post  type               content
  ────  ────  ────────────────   ──────────────────────────────────────────
     0   141  heading H1         Design Document
     2    55  heading H2         Architecture
     4    21  paragraph          The system is built on…

tui

Launch the interactive TUI. See TUI.

mq-db tui --db store.mq-db

serve

Start an HTTP server exposing SQL/mq query endpoints. See HTTP Server.

mq-db serve --db store.mq-db --host 0.0.0.0 --port 8080

REPL

The interactive REPL supports both query modes in a single session.

mq-db repl --db store.mq-db --mode sql

mq-db  (.help for commands  .quit to exit)
mode: sql  (.mode mq | .mode sql)

sql> SELECT content FROM blocks WHERE block_type = 'heading' LIMIT 3;
┌──────────────────┐
│ content          │
├──────────────────┤
│ Overview         │
│ Architecture     │
│ Query Engine     │
└──────────────────┘
(3 rows)

sql> .mode mq
→ mq mode
mq> .h2
## Architecture
## Query Engine

Dot commands

The initial mode can be set with --mode sql or --mode mq (default sql).

Pass --attach path.mq-db:alias (repeatable) to mount other stores at startup, queryable as alias.blocks/alias.documents; see ATTACH / DETACH.

TUI

mq-db tui opens a full-screen terminal UI for browsing indexed documents and running SQL/mq queries side by side, built with ratatui.

mq-db tui --db store.mq-db

 mq-db  SQL  Tab:switch  i:input  j/k:nav  d/u:scroll  q:quit
┌─ Documents ──────────┬─ SQL ────────────────────────────────────────────────┐
│ DESIGN.md            │ SELECT block_type, count(*) FROM blocks GROUP BY b_  │
│   142 blocks         ├─ Results ────────────────────────────────────────────┤
│ API.md               │ ┌─────────────┬──────────┐                           │
│   87 blocks  API     │ │ block_type  │ count(*) │                           │
│ README.md            │ ├─────────────┼──────────┤                           │
│   34 blocks          │ │ paragraph   │ 48       │                           │
└──────────────────────┴──────────────────────────────────────────────────────┘
 5 docs  632 blocks  3 rows

The left pane lists indexed documents; selecting one shows its full block breakdown (type, pre/post, content preview) in the results pane. The top-right pane accepts a query in the current mode (mq or SQL); running it replaces the results pane with the query output.

Keys

Color scheme

Block types are color-coded for quick scanning (heading, paragraph, code, list, blockquote, table, frontmatter, html, math, …), using the same warm paper/ink/accent palette as the project site.

HTTP Server

mq-db serve starts an HTTP server (built on axum) exposing SQL and mq query endpoints over the indexed store.

mq-db serve --db store.mq-db              # listens on 127.0.0.1:7878
mq-db serve --db store.mq-db --port 8080  # custom port
mq-db serve --db store.mq-db --host 0.0.0.0 --port 8080

Securing the server

--host 0.0.0.0 exposes the query endpoints beyond localhost. When doing so, secure the server with an API key or Basic auth, and consider TLS and a rate limit:

mq-db serve --db store.mq-db --host 0.0.0.0 \
  --api-key "$MQ_DB_API_KEY" \
  --rate-limit 20 \
  --timeout 10 \
  --tls-cert cert.pem --tls-key key.pem

If both --api-key and --basic-auth are set, either credential grants access. --tls-cert and --tls-key must be provided together.

Endpoints

Examples

# Health check
curl http://127.0.0.1:7878/health

# SQL via HTTP
curl -s -X POST http://127.0.0.1:7878/sql \
  -H 'Content-Type: application/json' \
  -d '{"query":"SELECT block_type, count(*) FROM blocks GROUP BY block_type"}'

# mq via HTTP
curl -s -X POST http://127.0.0.1:7878/mq \
  -H 'Content-Type: application/json' \
  -d '{"code":".h1"}'

Library API (Rust)

mq-db is usable directly as a Rust library, without shelling out to the CLI.

[dependencies]
mq-db = "0.1"

#![allow(unused)]
fn main() {
use mq_db::{DocumentStore, SqlEngine, MqEngine, block::BlockType};

// Build in memory
let mut store = DocumentStore::new();
store.add_file("docs/DESIGN.md")?;
store.add_str("# Hello\n\n## Architecture\n\nDetails\n")?;

// Chainable query API: zone-map skip + interval scope + block predicates
let chunks = store.query()
    .documents(|doc| doc.zone_maps.heading_contents.contains("Architecture"))
    .under_heading("Architecture", Some(2))
    .filter(|b| matches!(b.block_type, BlockType::Paragraph | BlockType::Code))
    .blocks();

// SQL engine (custom sqlparser-based evaluator, no SQLite dependency)
let engine = SqlEngine::new(&store)?;
let out = engine.execute(
    "SELECT content FROM blocks WHERE block_type = 'heading' ORDER BY pre"
)?;
print!("{}", out.to_table());

// mq engine
let results = MqEngine::eval_store(".h1", &store)?;

// Structural lint
let violations = store.query().lint_heading_followed_by(2, &[BlockType::List]);

// UPDATE/DELETE with write-back: rewrites the affected block's source file
// (heading/paragraph content only), then re-parses it in place
store.execute_sql_mut(
    "UPDATE blocks SET content = 'New Title' WHERE block_type = 'heading' AND content = 'Old Title'"
)?;

// Persist / load
store.save("store.mq-db")?;

// Full load: all blocks read into memory, indexes built on first SqlEngine use
let store = DocumentStore::load("store.mq-db")?;

// Lazy open: catalog only, call load_all_blocks() + load_all_indexes() before SQL
let mut store = DocumentStore::open("store.mq-db")?;
store.load_all_blocks()?;
store.load_all_indexes()?;

// Catalog-only: for metadata commands (list, stats) that don't need block data
let store = DocumentStore::load_catalog_only("store.mq-db")?;
}

Loading strategies

When using open(), call load_all_blocks() and load_all_indexes() before running any SqlEngine query.

Query builder

store.query() returns a chainable builder that applies the same three index layers used by the SQL engine, in order:

1. .documents(|doc| ...): zone-map predicate, skips whole documents
2. .under_heading(title, depth) / interval-scope helpers: narrows to a (pre, post) range
3. .filter(|block| ...): per-block predicate over the remaining candidates
4. .blocks(): materializes the final Vec<&Block>

See Index Layers for how each layer works.

Reference

Technical reference for the SQL surface and the on-disk/in-memory data model.

• Virtual Schema: the documents and blocks tables
• Built-in Functions: mq-db-specific, string, numeric, null-handling, and aggregate functions
• DDL Statements: CREATE TABLE, INSERT INTO, DROP TABLE, views, ATTACH/DETACH, and friends
• CTEs (WITH / WITH RECURSIVE): named subqueries and recursive queries
• Full-Text Search: match()/score()/bm25() and the find CLI shortcut
• EXPLAIN / EXPLAIN ANALYZE: inspect and profile a query’s index/join plan
• External Files: read_csv()/read_json() table functions
• Write-Back: INSERT/UPDATE/DELETE that edit the source Markdown file
• Example Queries: hierarchy extraction, structural lint, and mixed mq/SQL queries
• Block Model: the Block struct and per-type properties
• Index Layers: zone maps, the interval index, and secondary indexes
• Storage Format: the on-disk page-file layout

For mq language syntax itself (selectors, control flow, pattern matching, …), see the mq documentation.

Virtual Schema

The SQL engine exposes two virtual tables backed directly by the in-memory store; there is no separate schema to migrate.

SELECT id, path, title, tags FROM documents;

SELECT id, document_id, block_type, content, pre, post,
       depth, lang, properties FROM blocks;

documents

blocks

pre/post are the Nested-Set interval-index boundaries described in Index Layers; they encode heading hierarchy as a pure integer range, which is what the under() function operates on.

Built-in Functions

mq-db-specific

-- Hierarchy query: everything nested under a heading
SELECT b.block_type, b.content
FROM blocks b
WHERE under(b.pre, b.post,
  (SELECT pre FROM blocks WHERE block_type = 'heading' AND content = 'Architecture'),
  (SELECT post FROM blocks WHERE block_type = 'heading' AND content = 'Architecture'));

-- Run an mq program inline against block content
SELECT mq('.h1 | to_text', content) AS title
FROM blocks WHERE block_type = 'code' AND lang = 'markdown';

String

Numeric

Date/Time

Null handling & control flow

Aggregates

Usable with GROUP BY:

DDL Statements

mq-db supports a small set of DDL statements for defining custom in-memory tables alongside the built-in documents/blocks virtual tables. Custom tables live only for the process lifetime; they are not persisted to the .mq-db store file.

Examples

# Create from a SELECT result
mq-db sql "CREATE TABLE headings AS SELECT content, depth FROM blocks WHERE block_type = 'heading'" --db store.mq-db

# Create with explicit schema, then insert
mq-db sql "CREATE TABLE notes (id TEXT, body TEXT)" --db store.mq-db
mq-db sql "INSERT INTO notes VALUES ('1', 'Hello world')" --db store.mq-db

# Inspect
mq-db sql "SHOW TABLES" --db store.mq-db
mq-db sql "DESC notes"  --db store.mq-db

# Drop
mq-db sql "DROP TABLE notes" --db store.mq-db

Custom tables can be queried and joined exactly like documents/blocks:

SELECT h.content, n.body
FROM headings h
JOIN notes n ON n.id = h.content;

ATTACH / DETACH

ATTACH DATABASE '<path>' AS <alias> mounts another .mq-db store for the session, queryable as <alias>.blocks, <alias>.documents, or any of its views/custom tables, usable in SELECT, JOIN, subqueries, and CTEs. DETACH <alias> unmounts it. Like SQLite, this is session-scoped only (not saved into either store’s file); pass --attach path.mq-db:alias (repeatable) to sql, repl, or serve to attach automatically on startup.

mq-db repl --db project-a.mq-db --attach project-b.mq-db:b

sql> SELECT a.content, b.content FROM blocks a
     JOIN b.blocks b ON a.block_type = b.block_type
     WHERE a.block_type = 'heading';

Writes (INSERT/UPDATE/DELETE/CREATE TABLE) through an attached alias are rejected, only the local store can be written to.

Views

Unlike CREATE TABLE name AS SELECT … (a frozen snapshot), a view is not materialized: its SELECT re-runs on every reference, so it always reflects current data, including write-back edits and re-indexing. View definitions persist to the .mq-db file, same as custom tables.

mq-db sql "CREATE VIEW headings AS SELECT content, depth FROM blocks WHERE block_type = 'heading'" --db store.mq-db
mq-db sql "SELECT * FROM headings WHERE depth = 1" --db store.mq-db

mq-db sql "SHOW TABLES" --db store.mq-db   # views are listed with kind = "view"
mq-db sql "DESC headings" --db store.mq-db

mq-db sql "CREATE OR REPLACE VIEW headings AS SELECT content FROM blocks WHERE block_type = 'heading'" --db store.mq-db
mq-db sql "DROP VIEW headings" --db store.mq-db

Limitations in this version:

• CREATE VIEW v (a, b) AS ... (explicit column list) is not supported
• A view and a custom table can’t share a name
• A view’s output columns are recovered from their rendered text, same as WITH CTEs, see the CTE page for the trade-off this implies

CTEs (WITH / WITH RECURSIVE)

WITH

Name an intermediate result and reuse it in the main query, a join, or a subquery:

WITH headings AS (SELECT content, pre, post FROM blocks WHERE block_type = 'heading')
SELECT content FROM headings WHERE pre < 10 ORDER BY pre;

A CTE’s (and a view’s) output columns are recovered from their rendered text, so arithmetic/numeric comparisons on them (WHERE pre < 10 above) work as expected, but this is a heuristic, not a real type system: a value that merely looks numeric (e.g. a text cell containing "007") round-trips as a number, and a float’s trailing zeros aren’t preserved.

WITH RECURSIVE

The standard <anchor> UNION [ALL] <recursive term> shape, evaluated by iterative fixed-point: the anchor runs once, then the recursive term re-runs against only the previous iteration’s new rows until it produces none. UNION (not ALL) dedupes against every row produced so far, which is also what makes a query that would otherwise cycle forever terminate. A hard cap of 10,000 iterations guards against a recursive term with no terminating condition. The recursive term must reference the CTE by name exactly once in its FROM; the anchor must not reference it at all.

-- Number sequence: the canonical minimal recursive CTE
WITH RECURSIVE seq AS (
  SELECT 1 AS n
  UNION ALL
  SELECT n + 1 FROM seq WHERE n < 10
)
SELECT n FROM seq ORDER BY n;

-- Heading ancestor chain via interval containment: walk up from a block's
-- pre/post to every heading whose interval contains it
WITH RECURSIVE ancestors AS (
  SELECT pre, post, content FROM blocks WHERE pre = 24
  UNION
  SELECT b.pre, b.post, b.content
  FROM blocks b, ancestors
  WHERE b.pre < ancestors.pre AND ancestors.post < b.post
    AND b.block_type = 'heading'
)
SELECT content FROM ancestors ORDER BY pre;

A plain top-level SELECT ... UNION SELECT ... outside a recursive CTE is not supported. A CTE name identical to blocks, documents, or a custom table shadows it for the duration of the WITH clause’s scope.

Full-Text Search

match()/score()/bm25() are SQL functions backed by a persisted per-document TermIndex (a tokenized inverted index), for index-accelerated relevance search over block content.

SELECT content, bm25(content, 'error handling') AS relevance
FROM blocks
WHERE match(content, 'error handling')
ORDER BY relevance DESC;

• match(content, query): true iff every tokenized term in query appears in content. Index-accelerated when content is a bare column reference and query is a string literal.
• score(content, query): a simple term-frequency relevance score for query against content (no IDF weighting; see Storage Format).
• bm25(content, query): Okapi BM25 relevance score, IDF-weighted across the whole store (k1=1.2, b=0.75, not configurable). Prefer this over score() when documents vary a lot in length or in how common a query term is.

find (CLI shortcut)

mq-db find is a shortcut for match()/bm25(), no SQL needed. It falls back to a case-insensitive substring match too, so partial CJK queries still hit (see tokenize’s known limitations). Results show a snippet centred on the match, with matched terms highlighted when stdout is a terminal (respects NO_COLOR):

mq-db find "error handling" --db store.mq-db
mq-db find "error handling" --db store.mq-db -n 5 -F json   # top 5, JSON

docs/API.md  ¶   1.92  Error handling follows RFC 7807 problem details...
docs/API.md  #   0.71  Error Handling

2 matches

EXPLAIN / EXPLAIN ANALYZE

See which index (zone map, BitmapIndex, BTreeIndex, HashIndex, TermIndex, or full scan) a query’s WHERE/JOIN resolves to, without running it. When more than one index is viable for the same WHERE clause, the choice is cost-based: each candidate’s real matching-block count is read from its index, and the cheapest wins; EXPLAIN shows every candidate considered.

mq-db sql "EXPLAIN SELECT content FROM blocks WHERE block_type = 'code' AND lang = 'json'" --db store.mq-db

┌──────────────────────┬────────────────────────────────────────────────────────────────────────┐
│ step                 │ detail                                                                 │
├──────────────────────┼────────────────────────────────────────────────────────────────────────┤
│ query:from           │ blocks (blocks)                                                        │
│ query:where          │ HashIndex(lang = 'json') used (est. 2 row(s); also considered:          │
│                       │   BitmapIndex(block_type IN (code)) [est. 289])                        │
│ query:zone-map       │ eligible via lang                                                       │
│ query:where-recheck  │ row-by-row (full predicate re-evaluated after scan)                     │
└──────────────────────┴────────────────────────────────────────────────────────────────────────┘

Add ANALYZE to also run the query and report actual row counts, document-skip counts, and timing:

mq-db sql "EXPLAIN ANALYZE SELECT * FROM blocks WHERE match(content, 'error handling')" --db store.mq-db

WITH CTEs are described separately (cte:<name>:... steps) before the outer query; JOINs report whether they resolve to a hash join (equi-join ON) or a nested loop. Only SELECT queries are supported; EXPLAIN on other statements is rejected.

External Files (read_csv / read_json)

read_csv(path) and read_json(path) are table functions, use them anywhere a table name goes (FROM, JOIN, CREATE TABLE ... AS SELECT) to query an external file alongside blocks, no separate import step:

SELECT name, age FROM read_csv('people.csv') WHERE age > 26;
SELECT name FROM read_json('people.jsonl') WHERE active = true;

-- Persist as a custom table
CREATE TABLE people AS SELECT * FROM read_csv('people.csv');

read_csv parses RFC 4180 (quoted fields, embedded commas/quotes/newlines); the first row is the header. read_json expects one JSON object per line (JSON Lines): the column set is the union of every line’s keys, in first-seen order. In both, numeric-looking values support arithmetic/comparison (WHERE age > 26), and a short/missing cell becomes NULL rather than erroring. Parquet is not supported (would need a large parquet/Arrow dependency); an unrecognized table function name (including read_parquet) is rejected with a clear error rather than silently misparsed.

INSERT / UPDATE / DELETE with Write-Back

INSERT/UPDATE/DELETE on blocks write the change back to the document’s source Markdown file (re-parsed in place, same DocumentId), pass --write-back to allow it; without the flag the statement is rejected:

mq-db sql "UPDATE blocks SET content = 'New Title' WHERE block_type = 'heading' AND content = 'Old Title'" \
  --db store.mq-db --write-back

mq-db sql "DELETE FROM blocks WHERE content = 'Outdated paragraph'" \
  --db store.mq-db --write-back

# after_pre anchors the new block right after an existing block's `pre`;
# omit it to append at the end of the document.
mq-db sql "INSERT INTO blocks (document_id, block_type, content, depth, after_pre) VALUES (0, 'heading', 'New Section', 2, 4)" \
  --db store.mq-db --write-back

mq-db sql "INSERT INTO blocks (document_id, block_type, content) VALUES (0, 'paragraph', 'Appended at the end')" \
  --db store.mq-db --write-back

Limitations in this version

• UPDATE ... SET content and INSERT INTO blocks only support heading/paragraph blocks (not tables, code, lists, …)
• INSERT INTO blocks requires an explicit column list drawn from document_id, block_type, content, depth (required for heading, 1-6), after_pre (optional); INSERT ... SELECT is not supported, only VALUES
• Only documents indexed with spans (the default; not --no-spans) and from a real file (not added via the library’s add_str) are eligible
• Not available over serve’s HTTP endpoint or from mq-mcp: CLI (sql/repl with --write-back) and the library (DocumentStore::execute_sql_mut) only

Example Queries

-- All text/code under a specific section (RAG extraction)
SELECT b.block_type, b.content
FROM blocks b
WHERE under(b.pre, b.post,
  (SELECT pre FROM blocks WHERE block_type = 'heading' AND content = 'Architecture'),
  (SELECT post FROM blocks WHERE block_type = 'heading' AND content = 'Architecture'))
  AND b.block_type IN ('paragraph', 'code')
ORDER BY b.pre;

-- Extract H1 title from code block content via the mq() scalar function
SELECT mq('.h1 | to_text', content) AS title
FROM blocks
WHERE block_type = 'code' AND lang = 'markdown';

-- H2 headings immediately followed by a list (structural lint)
SELECT d.path, h.content AS heading
FROM blocks h
JOIN blocks nxt ON nxt.document_id = h.document_id AND nxt.pre = h.pre + 1
JOIN documents d ON d.id = h.document_id
WHERE h.block_type = 'heading' AND depth = 2 AND nxt.block_type = 'list';

-- Documents containing Python code
SELECT DISTINCT d.path
FROM documents d JOIN blocks b ON b.document_id = d.id
WHERE b.block_type = 'code' AND lang = 'python';

-- Bucket headings by depth and summarize with string/numeric functions
SELECT
  CASE WHEN depth <= 1 THEN 'top-level' ELSE 'nested' END AS bucket,
  count(*),
  group_concat(initcap(trim(content)), ', ') AS headings
FROM blocks
WHERE block_type = 'heading'
GROUP BY CASE WHEN depth <= 1 THEN 'top-level' ELSE 'nested' END;

Mixing mq and SQL

The mq() scalar function lets a SQL query delegate per-row Markdown transformation to mq, which is convenient when a block’s content is itself a Markdown snapshot (e.g. a fenced code block containing Markdown, as in the to_text example above).

From the CLI you can also move between the two engines freely, since both mq-db sql and mq-db mq support --format markdown/--format json, so the output of one can feed a pipeline built around the other.

Block Model

Every Markdown element becomes a Block:

#![allow(unused)]
fn main() {
struct Block {
    id: u32,
    document_id: u32,
    block_type: BlockType,  // Heading, Paragraph, Code, List, …
    content: String,
    span: Option<Span>,     // line/column for editor sync
    pre: u32,               // interval index pre-order
    post: u32,              // interval index post-order
    properties: Properties, // row-polymorphic extra attributes
}
}

properties is row-polymorphic: different block_types carry different keys.

Block types

BlockType covers every CST node mq-markdown produces:

Heading, Paragraph, Code, List, TableCell, TableRow, TableAlign, Blockquote, HorizontalRule, Html, Yaml, Toml, Math, Definition, Footnote.

In SQL, block_type is exposed as the lowercase, snake_case string form (e.g. 'table_cell', 'horizontal_rule').

See Storage Format for the exact on-disk wire encoding of a Block and its properties.

Index Layers

mq-db applies three complementary index layers, cheapest-first:

SQL Query
   │
   ▼
Layer 1: Zone Maps (document skip) ────skip───▶ ✗ irrelevant docs
   │ relevant docs
   ▼
Layer 2: Interval Index (section scope)
   │ candidate blocks
   ▼
Layer 3: Secondary Indexes (block lookup)
   │  BitmapIndex · BTreeIndex · HashIndex
   │  (no hint ──▶ Full Scan)
   ▼
Result Rows

Layer 1: Zone Maps (document-level skip)

Built once per document and stored in the .mq-db file. Checked before any block is read.

Via SQL: SqlEngine derives a skip automatically from the WHERE clause, for a single, non-JOINed SELECT ... FROM blocks:

Via the Rust API: store.query().documents(|doc| ...) lets you filter on any zone-map field yourself (heading_slugs, frontmatter_keys, title, tags, …), not just the patterns SqlEngine recognizes automatically.

Layer 2: Interval Index (section hierarchy)

Heading hierarchy is encoded as (pre, post) pairs via Pre-Post Order (Nested Set) traversal:

# Doc                 pre=0  · post=11
├── ## Section A      pre=2  · post=7
│   ├── Paragraph     pre=3  · post=4
│   └── Code          pre=5  · post=6
└── ## Section B      pre=8  · post=11
    └── Paragraph     pre=9  · post=10

A is_under B ↔ B.pre < A.pre AND A.post < B.post: O(1), no tree traversal. This is exactly what the SQL under() function and the Rust .under_heading() query-builder method check.

Layer 3: Secondary Indexes (block-level fast lookup)

SQL predicate pushdown picks an IndexHint based on the shape of the WHERE predicate:

MQDB Storage Format

Overview

mq-db persists documents in a fixed-size page file. Every file is split into 8192-byte pages. Page 0 is the file header, page 1 is the catalog root, and all remaining pages are used for document block data or overflow chains.

+-----------+-----------+-----------+-----------+-----------+
| Page 0    | Page 1    | Page N    | Page N+1  | Page ...  |
| FileHeader| Catalog   | BlockData | Overflow  | Free/Future|
+-----------+-----------+-----------+-----------+-----------+

Multi-page values are stored as singly linked page chains using the next_page field in the page header.

Page Layout

Each page is exactly 8192 bytes.

Page header (16 bytes)

Page body (8176 bytes)

File Header Page (page 0)

Page 0 always has page_type = 1 and page_id = 0.

File header body

Catalog Pages

The catalog always starts at page 1. If the serialized catalog exceeds one page body, additional catalog pages are linked by next_page.

Page 1 (Catalog) --> Page 12 (Catalog) --> Page 18 (Catalog) --> 0

Catalog body format

Block Data Pages

A document is serialized as concatenated encoded blocks. The byte stream is cut into 8176-byte chunks.

• The first chunk is stored in a page with page_type = 3 (BlockData).
• Continuation chunks are stored in pages with page_type = 4 (Overflow).
• The last page in the chain has next_page = 0.

first_block_page
      |
      v
+-------------------+    +-------------------+    +-------------------+
| type=BlockData    | -> | type=Overflow     | -> | type=Overflow     |
| body bytes 0..8175|    | next chunk        |    | final chunk       |
+-------------------+    +-------------------+    +-------------------+

Unused bytes at the end of the final page body are zero-filled.

Block Wire Format

Each block is encoded independently and concatenated without separators.

Block type mapping

PropertyValue Encoding

Each property value starts with a one-byte type tag.

Arrays are recursive: each element is another complete PropertyValue.

ZoneMap Encoding

Zone maps are encoded independently and embedded as opaque bytes inside catalog entries.

Sets are serialized as sorted UTF-8 strings for deterministic output.

Checksum Algorithm

The checksum is a simple wrapping sum over every byte in the 8192-byte page except the checksum field itself (page[4..8]).

Pseudo-code:

checksum = 0u32
for i in 0..8192:
    if i in [4, 5, 6, 7]:
        continue
    checksum = checksum.wrapping_add(page[i] as u32)

Verification recomputes the checksum and compares it to the stored checksum field.

Multi-Page Chains

Large catalog payloads and large document block streams are stored as chains.

+---------+      +---------+      +---------+
| page_id |----->| page_id |----->| page_id |
| next=42 |      | next=77 |      | next=0  |
+---------+      +---------+      +---------+

The body bytes of each page are concatenated in chain order to reconstruct the original serialized byte stream.

Atomic Write Procedure

DocumentStore::save() writes atomically using a sibling temporary file:

1. Create path.tmp.
2. Write page 0 file header and page 1 empty catalog.
3. Append all document block chains.
4. Serialize and write the final catalog chain.
5. Close the temporary file.
6. Rename path.tmp to path.

Because the final rename is atomic on the same filesystem, readers either observe the old file or the new complete file, never a partially written database image.