Store queue payload references instead of oversized payload blobs

By Batu · English technical notes

Also published in our Blogger archive.

Store a payload locator in the queue record

payload_ref holds a synthetic object-style locator rather than a document body. The bound value round-trips by job id and is short in this fixture, making the indirection visible to a worker that resolves it after claiming. Queue state can remain independent of the payload format.

The table does not prove a target exists, is immutable, or is readable. The length assertion is not a database limit; reference validation and lifecycle belong with the storage system.

Example

import sqlite3
from pathlib import Path
from tempfile import TemporaryDirectory

with TemporaryDirectory() as directory:
    connection = sqlite3.connect(Path(directory) / "queue.db")
    connection.execute("CREATE TABLE job(id INTEGER PRIMARY KEY, payload_ref TEXT NOT NULL)")
    reference = "objects/2026/receipt-17.json"
    connection.execute("INSERT INTO job VALUES (?, ?)", (1, reference))
    row = connection.execute("SELECT payload_ref FROM job WHERE id = 1").fetchone()[0]
    assert row == reference
    assert len(row) < 100
    print("payload=reference-only")

Expected stdout:

payload=reference-only

Sources

- SQLite Limits

- sqlite3.Connection.execute

Prepared with AI assistance. The example uses synthetic data; its stated limits apply.