About Ajika

Ajika is a library that lets your application to remember computations and reuse them when inputs do not change. Moreover, it lets you to store these computations to the disk, allowing you to offload computations from memory when they are not needed, load them on demand, and reuse computations across the application restarts.

In other words, Ajika is a persistent incrementality library.

Ajika is heavily inspired by Salsa, with a different focus. Salsa is generally oriented to provide maximum efficiency, and is memory-first. Ajika is the opposite: it accepts a performance hit (often not very big in practice, especially for heavy computations) to offload data from memory and load it from disk on demand. It might be a good choice if you want to use Salsa approach but reduce the memory usage.

You can also keep reusable work in RAM with a memory database. For frequent unsaved edits, a session overlay can reuse results in memory while preserving saved answers on disk. The query functions stay the same in each case; the following chapters show how to choose where their results live.

LLM usage disclaimer

This book was written with use of LLMs, but it has plenty a human touch. I’ve spend hours on this text to make sure that it’s something I am comfortable reading myself, so, I hope, it will be a good read for you too.

Regardless, I am very open for feedback, so if some parts (or even the book as a whole) do not feel good, please open an issue and I’ll work on it.

Eventually I do plan to rewrite the documentation myself from scratch, but it’s a very big amount of work and there are only so many hours in a day.

About this book

This book tries to teach you Ajika through a prism of a very naive query in a text editor. The request is very stupid: how many words are there in the document.

In real world, the queries are typically much heavier, when your task is implementing an LSP server, a compiler, or a build system. The editor we’re discussing is, obviously, not a project where Ajika shines (because with such simple queries the persistence overhead will dominate, and the memory usage is low anyways), but a project that is useful for understanding basic concepts.

Regardless, let’s think how we would implement such a request.

From functions to queries

In ordinary Rust, we could write:

struct File {
    text: String,
}

fn words(file: &File) -> Vec<String> {
    file.text.split_whitespace().map(str::to_owned).collect()
}

fn word_count(file: &File) -> usize {
    words(file).len()
}

Each call to word_count runs words again, even if the text hasn’t changed. With Ajika, we can describe the text as an input and these functions as queries:

extern crate ajika;
extern crate anyhow;
use std::{sync::Arc, time::Duration};

use ajika::{Database, Db, Inputs, Retention};

#[ajika::input(key = String)]
struct File {
    #[input(arc)]
    text: String,
}

#[ajika::query]
fn words(db: &Db<'_>, path: String) -> ajika::Result<Vec<String>> {
    let text = File::at(path).text(db)?;
    Ok(text.split_whitespace().map(str::to_owned).collect())
}

#[ajika::query]
fn word_count(db: &Db<'_>, path: String) -> ajika::Result<usize> {
    Ok(words(db, path)?.len())
}

// This convenience query reuses the words query's allocation.
#[ajika::query(forward)]
fn note_words(db: &Db<'_>) -> ajika::Result<Arc<Vec<String>>> {
    words(db, "notes.txt".into())
}

fn main() -> anyhow::Result<()> {
    let limits = Retention {
        max_entries: 10_000,
        max_bytes: 256 * 1024 * 1024,
        max_idle: Duration::from_secs(24 * 60 * 60),
    };
    let mut database = Database::memory(Inputs::default(), (), limits);
    let file = File::at("notes.txt".into());
    database.update_inputs(|update| file.set_text(update, "hello world".to_owned()))?;

    let count = database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(*count, 2);
    assert_eq!(database.statistics().executed, 2);

    database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(database.statistics().executed, 0);

    database.update_inputs(|update| file.set_text(update, "hello    world".to_owned()))?;
    database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(database.statistics().executed, 1); // Only words ran.

    database.update_inputs(|update| file.set_text(update, "hello new world".to_owned()))?;
    let count = database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(*count, 3);
    assert_eq!(database.statistics().executed, 2);
    database.run(|db| {
        let words = words(db, "notes.txt".into())?;
        assert!(Arc::ptr_eq(&words, &note_words(db)?));
        Ok(())
    })?;
    Ok(())
}

The function bodies are still ordinary Rust. The extra declarations let Ajika remember which inputs and other query answers each function used.

Suppose we change hello world to hello world, widening the gap to four spaces. Ajika reruns words because the text changed, but its answer is still ["hello", "world"]. Since word_count depends on that vector, Ajika can reuse the count without running its function again. Counting words is cheap, but the same boundary could protect an expensive analysis from unnecessary work.

How to read this book

The tutorial turns the queries above into a runnable program. We’ll supply text, edit it, reuse results after reopening, and separate words from their source locations. Later chapters cover interning, loading inputs on demand, unsaved edits, parallel work, and memory and storage budgets.

The conceptual reference explains the choices behind the public interface: what belongs in an input or a query key, what callers depend on, how to organize a request, and what to keep between requests. Read it when designing your own integration, or return to individual chapters as those questions arise. It does not require following the tutorial.

The internal walkthrough follows query calls through the implementation. It introduces the objects after explaining the problem they solve, then traces validation, publication, and storage.

Where the boundary lies

Ajika manages derived work. Your application still decides what counts as an input, when those inputs form a coherent view, and when a result is fit to publish to a user. Source files, configuration, and editor buffers remain application concerns.

Ajika’s disk store is a disposable cache. It is not the authoritative copy of those inputs, a general database for user data, or a way to query arbitrary historical revisions. A corrupted or incompatible cache can be discarded and rebuilt.

The project is under active development. APIs and cache formats can change.

About this book

This book is obviously inspired by the Salsa book.

Your first queries

We will count words in a captured text file. This is small enough to follow in full, but has the same useful boundary as a larger analyzer: changing whitespace changes the text without changing the words.

The complete program is the word-count example. From this repository, run it with:

cargo run -p ajika-word-count

Describe the question

An input is data supplied from outside the query graph. Here its key is a path and its value is the captured text. A query computes an answer from inputs or other queries.

extern crate ajika;
extern crate anyhow;
use std::{sync::Arc, time::Duration};

use ajika::{Database, Db, Inputs, Retention};

#[ajika::input(key = String)]
struct File {
    #[input(arc)]
    text: String,
}

#[ajika::query]
fn words(db: &Db<'_>, path: String) -> ajika::Result<Vec<String>> {
    let text = File::at(path).text(db)?;
    Ok(text.split_whitespace().map(str::to_owned).collect())
}

#[ajika::query]
fn word_count(db: &Db<'_>, path: String) -> ajika::Result<usize> {
    Ok(words(db, path)?.len())
}

// This convenience query reuses the words query's allocation.
#[ajika::query(forward)]
fn note_words(db: &Db<'_>) -> ajika::Result<Arc<Vec<String>>> {
    words(db, "notes.txt".into())
}

fn main() -> anyhow::Result<()> {
    let limits = Retention {
        max_entries: 10_000,
        max_bytes: 256 * 1024 * 1024,
        max_idle: Duration::from_secs(24 * 60 * 60),
    };
    let mut database = Database::memory(Inputs::default(), (), limits);
    let file = File::at("notes.txt".into());
    database.update_inputs(|update| file.set_text(update, "hello world".to_owned()))?;

    let count = database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(*count, 2);
    assert_eq!(database.statistics().executed, 2);

    database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(database.statistics().executed, 0);

    database.update_inputs(|update| file.set_text(update, "hello    world".to_owned()))?;
    database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(database.statistics().executed, 1); // Only words ran.

    database.update_inputs(|update| file.set_text(update, "hello new world".to_owned()))?;
    let count = database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(*count, 3);
    assert_eq!(database.statistics().executed, 2);
    database.run(|db| {
        let words = words(db, "notes.txt".into())?;
        assert!(Arc::ptr_eq(&words, &note_words(db)?));
        Ok(())
    })?;
    Ok(())
}

File::at(path) addresses one input; it does not read a file from the filesystem. The generated text(db) getter reads the text that the application supplied.

The bodies return owned values through Result<T>. The query macro wraps them in Arc<T>, so callers receive Result<Arc<Vec<String>>> from words and Result<Arc<usize>> from word_count.

The context, &Db, connects their reads to the active operation. The remaining arguments identify the question: words("notes.txt") is one call; words("other.txt") is another.

Calling words inside word_count also tells Ajika what the latter relied on. You do not register the relationship separately. It follows the calls that actually run, including calls made conditionally.

Supply inputs and ask

The long-lived Database owns the inputs and reusable work. Its run method starts an operation: a scope in which query calls can share computed and loaded values.

extern crate ajika;
extern crate anyhow;
use std::{sync::Arc, time::Duration};

use ajika::{Database, Db, Inputs, Retention};

#[ajika::input(key = String)]
struct File {
    #[input(arc)]
    text: String,
}

#[ajika::query]
fn words(db: &Db<'_>, path: String) -> ajika::Result<Vec<String>> {
    let text = File::at(path).text(db)?;
    Ok(text.split_whitespace().map(str::to_owned).collect())
}

#[ajika::query]
fn word_count(db: &Db<'_>, path: String) -> ajika::Result<usize> {
    Ok(words(db, path)?.len())
}

// This convenience query reuses the words query's allocation.
#[ajika::query(forward)]
fn note_words(db: &Db<'_>) -> ajika::Result<Arc<Vec<String>>> {
    words(db, "notes.txt".into())
}

fn main() -> anyhow::Result<()> {
    let limits = Retention {
        max_entries: 10_000,
        max_bytes: 256 * 1024 * 1024,
        max_idle: Duration::from_secs(24 * 60 * 60),
    };
    let mut database = Database::memory(Inputs::default(), (), limits);
    let file = File::at("notes.txt".into());
    database.update_inputs(|update| file.set_text(update, "hello world".to_owned()))?;

    let count = database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(*count, 2);
    assert_eq!(database.statistics().executed, 2);

    database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(database.statistics().executed, 0);

    database.update_inputs(|update| file.set_text(update, "hello    world".to_owned()))?;
    database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(database.statistics().executed, 1); // Only words ran.

    database.update_inputs(|update| file.set_text(update, "hello new world".to_owned()))?;
    let count = database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(*count, 3);
    assert_eq!(database.statistics().executed, 2);
    database.run(|db| {
        let words = words(db, "notes.txt".into())?;
        assert!(Arc::ptr_eq(&words, &note_words(db)?));
        Ok(())
    })?;
    Ok(())
}

The unit value () means this application has no extra host services. The retention limits are example budgets, not prescribed settings. For a memory database, entry count and age apply; it does not serialize values merely to enforce the byte budget.

update_inputs publishes a batch before the operation starts. Query code can read those inputs, but cannot replace them while the operation is running.

The first request executes both functions. Each caller receives an Arc so it can share an immutable answer. Keeping the returned Arc keeps that answer alive; it does not keep a live view of future edits.

Change the text without changing the answer

First ask the same question again, then widen the gap to four spaces:

extern crate ajika;
extern crate anyhow;
use std::{sync::Arc, time::Duration};

use ajika::{Database, Db, Inputs, Retention};

#[ajika::input(key = String)]
struct File {
    #[input(arc)]
    text: String,
}

#[ajika::query]
fn words(db: &Db<'_>, path: String) -> ajika::Result<Vec<String>> {
    let text = File::at(path).text(db)?;
    Ok(text.split_whitespace().map(str::to_owned).collect())
}

#[ajika::query]
fn word_count(db: &Db<'_>, path: String) -> ajika::Result<usize> {
    Ok(words(db, path)?.len())
}

// This convenience query reuses the words query's allocation.
#[ajika::query(forward)]
fn note_words(db: &Db<'_>) -> ajika::Result<Arc<Vec<String>>> {
    words(db, "notes.txt".into())
}

fn main() -> anyhow::Result<()> {
    let limits = Retention {
        max_entries: 10_000,
        max_bytes: 256 * 1024 * 1024,
        max_idle: Duration::from_secs(24 * 60 * 60),
    };
    let mut database = Database::memory(Inputs::default(), (), limits);
    let file = File::at("notes.txt".into());
    database.update_inputs(|update| file.set_text(update, "hello world".to_owned()))?;

    let count = database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(*count, 2);
    assert_eq!(database.statistics().executed, 2);

    database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(database.statistics().executed, 0);

    database.update_inputs(|update| file.set_text(update, "hello    world".to_owned()))?;
    database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(database.statistics().executed, 1); // Only words ran.

    database.update_inputs(|update| file.set_text(update, "hello new world".to_owned()))?;
    let count = database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(*count, 3);
    assert_eq!(database.statistics().executed, 2);
    database.run(|db| {
        let words = words(db, "notes.txt".into())?;
        assert!(Arc::ptr_eq(&words, &note_words(db)?));
        Ok(())
    })?;
    Ok(())
}

The second operation with unchanged inputs executes neither function. After the edit, Ajika must run words because it read the changed text. Its new vector still contains ["hello", "world"], so Ajika preserves the answer’s version.

word_count depended on that answer, not directly on the file text. Its dependency still matches, so its function does not run. The executed statistic counts all query executions in the latest operation, including children.

Now insert a word:

extern crate ajika;
extern crate anyhow;
use std::{sync::Arc, time::Duration};

use ajika::{Database, Db, Inputs, Retention};

#[ajika::input(key = String)]
struct File {
    #[input(arc)]
    text: String,
}

#[ajika::query]
fn words(db: &Db<'_>, path: String) -> ajika::Result<Vec<String>> {
    let text = File::at(path).text(db)?;
    Ok(text.split_whitespace().map(str::to_owned).collect())
}

#[ajika::query]
fn word_count(db: &Db<'_>, path: String) -> ajika::Result<usize> {
    Ok(words(db, path)?.len())
}

// This convenience query reuses the words query's allocation.
#[ajika::query(forward)]
fn note_words(db: &Db<'_>) -> ajika::Result<Arc<Vec<String>>> {
    words(db, "notes.txt".into())
}

fn main() -> anyhow::Result<()> {
    let limits = Retention {
        max_entries: 10_000,
        max_bytes: 256 * 1024 * 1024,
        max_idle: Duration::from_secs(24 * 60 * 60),
    };
    let mut database = Database::memory(Inputs::default(), (), limits);
    let file = File::at("notes.txt".into());
    database.update_inputs(|update| file.set_text(update, "hello world".to_owned()))?;

    let count = database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(*count, 2);
    assert_eq!(database.statistics().executed, 2);

    database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(database.statistics().executed, 0);

    database.update_inputs(|update| file.set_text(update, "hello    world".to_owned()))?;
    database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(database.statistics().executed, 1); // Only words ran.

    database.update_inputs(|update| file.set_text(update, "hello new world".to_owned()))?;
    let count = database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(*count, 3);
    assert_eq!(database.statistics().executed, 2);
    database.run(|db| {
        let words = words(db, "notes.txt".into())?;
        assert!(Arc::ptr_eq(&words, &note_words(db)?));
        Ok(())
    })?;
    Ok(())
}

The words differ, so word_count runs as well. If a third query used the count, its reuse would depend on whether that count changed.

This is the central programming model: choose meaningful intermediate answers, call queries normally, and give each answer an equality relation that covers everything consumers can observe. Next, we will look more closely at input changes.

Changing inputs

A query can only react to changes Ajika knows about. Reading the current directory, an environment variable, or a file directly inside a query hides that observation from the runtime. A reused query does not execute those reads again.

Put answer-affecting data in typed inputs. The application decides when to acquire new data and publish it.

Fields are tracked separately

Suppose the analyzer also needs a language setting:

extern crate ajika;

#[ajika::input(key = String)]
struct File {
    #[input(arc)]
    text: String,
    language: String,
}

Reading File::at(path).text(db) depends on the text field. It does not depend on language. A query that reads both fields depends on both.

#[input(arc)] shares the field’s value through an Arc. Setters and initializers accept an owned value or an existing Arc; getters return Result<Arc<T>>. Stamp and policy callbacks inspect &T. Use it for large values such as source text; ordinary fields return their declared type by copying or cloning it.

The declaration describes Ajika-managed fields, not ordinary fields on a Rust struct. Its getters, setters, and field selectors are public; the declaration’s visibility controls access to the handle. There is no need to write pub on each field.

Publish one coherent batch

Group related changes in one update:

extern crate ajika;
use ajika::{Database, Result};
#[ajika::input(key = String)]
struct File { #[input(arc)] text: String, language: String }
fn publish_files(database: &mut Database, changed: Vec<(String, String)>) -> Result<()> {
    database.update_inputs(|update| {
        for (path, text) in changed {
            let file = File::at(path);
            file.set_text(update, text)?;
            file.set_language(update, "plain-text".into())?;
        }
        Ok(())
    })
}

The batch becomes visible together. An error while preparing it leaves the old inputs installed. Batching is also cheaper: a changed field table is copied once for the batch, rather than once per setter in a sequence of separate updates.

The host may want to inspect a batch before making it active. database.inputs().edit(...) prepares a candidate Inputs view for those checks while the database continues to use its installed inputs. Once the host accepts the candidate, database.replace_inputs(candidate) installs it.

Keeping a clone of the previous Inputs view lets the host restore it later. Cloning shares captured data, while editing builds a separate set of changes.

Publishing changed fingerprints or policies advances the database’s input revision. Publishing equivalent observations does not. Starting another operation or saving query results does not itself advance the revision. Restoring an older snapshot is another input replacement, never a rewind of the database’s validation history. See replace_inputs.

Fingerprints describe observations

By default, the input macro computes a Stamp from the field’s serialized value. Ajika stores that stamp beside the captured value and later compares it with the stamp a query recorded.

Use a custom #[input(stamp = function)] when a field already carries a content fingerprint or is an opaque source object. The function receives a reference to the field and returns Result<Stamp>. It must identify everything query code can observe. A modification time alone is not necessarily a content fingerprint.

Reproducible encoding helps reuse: two logically equal unordered maps can otherwise produce different stamps. Prefer a stable representation or a stamp computed from the semantic content. A changed stamp can cause extra work; incorrectly reusing a stamp can return a stale answer.

Durability is a shortcut, not permanence

Source text usually changes often; configuration may change rarely:

extern crate ajika;
#[ajika::input(key = ())]
struct Configuration {
    #[input(durability = High)]
    case_sensitive: bool,
}

Fields default to Low durability. If only low-durability fields change, Ajika can leave configuration-only queries verified without inspecting their dependencies. A query that also reads source text inherits the lower class.

High durability does not make a field immutable, and it does not decide whether data reaches disk. Changing a high-durability field invalidates the corresponding shortcut automatically. Input lifetime is the separate decision about whether derived work may survive the operation, session, or restart.

The input implementation is described in snapshots and revisions.

Keeping results across restarts

The first example used a memory database. Switching to disk storage changes the owner’s construction, not the query functions or their callers.

The persistence example opens a temporary cache, computes a word count, closes it, then opens it again. Run it with cargo run -p ajika-persistence.

Open, compute, close

extern crate ajika;
extern crate anyhow;
extern crate tempfile;
use std::time::Duration;

use ajika::{Database, Db, Inputs, Retention};

#[ajika::input(key = String)]
struct File {
    #[input(arc)]
    text: String,
}

#[ajika::query]
fn words(db: &Db<'_>, path: String) -> ajika::Result<Vec<String>> {
    let text = File::at(path).text(db)?;
    Ok(text.split_whitespace().map(str::to_owned).collect())
}

#[ajika::query]
fn word_count(db: &Db<'_>, path: String) -> ajika::Result<usize> {
    Ok(words(db, path)?.len())
}

fn main() -> anyhow::Result<()> {
    // A real application chooses a persistent cache directory.
    let directory = tempfile::tempdir()?;
    let path = directory.path().join("queries");
    let limits = Retention {
        max_entries: 10_000,
        max_bytes: 256 * 1024 * 1024,
        max_idle: Duration::from_secs(24 * 60 * 60),
    };

    let mut database = Database::open(&path, "word-count-v1", Inputs::default(), (), limits)?;
    database.update_inputs(|update| {
        File::at("notes.txt".into()).set_text(update, "hello world".to_owned())
    })?;
    assert_eq!(*database.run(|db| word_count(db, "notes.txt".into()))?, 2);
    assert_eq!(database.statistics().executed, 2);
    drop(database);

    // Simulate fresh acquisition by the application, without reusing the old Inputs.
    let inputs = Inputs::default()
        .edit(|update| File::at("notes.txt".into()).set_text(update, "hello world".to_owned()))?;
    let mut database = Database::open(&path, "word-count-v1", inputs, (), limits)?;
    assert_eq!(*database.run(|db| word_count(db, "notes.txt".into()))?, 2);
    assert_eq!(database.statistics().executed, 0);
    assert_eq!(database.statistics().validated, 2);

    database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(database.statistics().executed, 0);
    assert_eq!(database.statistics().validated, 0);
    assert_eq!(database.statistics().decoded, 1);
    Ok(())
}

path names a directory the application has chosen for rebuildable query data. The example uses a temporary directory so running it leaves no permanent cache. The operation checkpoints successful work before returning. No per-query save call is needed, and dropping the owner releases its directory lock.

Only one owner may open that storage directory at a time. Parallel query calls share one owner through an operation; opening another database for each worker is not the concurrency model.

Reopening still needs fresh inputs

extern crate ajika;
extern crate anyhow;
extern crate tempfile;
use std::time::Duration;

use ajika::{Database, Db, Inputs, Retention};

#[ajika::input(key = String)]
struct File {
    #[input(arc)]
    text: String,
}

#[ajika::query]
fn words(db: &Db<'_>, path: String) -> ajika::Result<Vec<String>> {
    let text = File::at(path).text(db)?;
    Ok(text.split_whitespace().map(str::to_owned).collect())
}

#[ajika::query]
fn word_count(db: &Db<'_>, path: String) -> ajika::Result<usize> {
    Ok(words(db, path)?.len())
}

fn main() -> anyhow::Result<()> {
    // A real application chooses a persistent cache directory.
    let directory = tempfile::tempdir()?;
    let path = directory.path().join("queries");
    let limits = Retention {
        max_entries: 10_000,
        max_bytes: 256 * 1024 * 1024,
        max_idle: Duration::from_secs(24 * 60 * 60),
    };

    let mut database = Database::open(&path, "word-count-v1", Inputs::default(), (), limits)?;
    database.update_inputs(|update| {
        File::at("notes.txt".into()).set_text(update, "hello world".to_owned())
    })?;
    assert_eq!(*database.run(|db| word_count(db, "notes.txt".into()))?, 2);
    assert_eq!(database.statistics().executed, 2);
    drop(database);

    // Simulate fresh acquisition by the application, without reusing the old Inputs.
    let inputs = Inputs::default()
        .edit(|update| File::at("notes.txt".into()).set_text(update, "hello world".to_owned()))?;
    let mut database = Database::open(&path, "word-count-v1", inputs, (), limits)?;
    assert_eq!(*database.run(|db| word_count(db, "notes.txt".into()))?, 2);
    assert_eq!(database.statistics().executed, 0);
    assert_eq!(database.statistics().validated, 2);

    database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(database.statistics().executed, 0);
    assert_eq!(database.statistics().validated, 0);
    assert_eq!(database.statistics().decoded, 1);
    Ok(())
}

The cache contains the answer and the observations that justified it. It does not contain the application’s input snapshot. The application supplies fresh inputs on startup, either eagerly or through lazy acquisition.

This new owner has no proof that the stored answer matches its inputs. It follows the count’s dependency on words, checks the words’ saved input stamp, and finds the same text. It can then load the count without executing either function or decoding the words.

Valid does not mean resident

Ask again under the same inputs:

extern crate ajika;
extern crate anyhow;
extern crate tempfile;
use std::time::Duration;

use ajika::{Database, Db, Inputs, Retention};

#[ajika::input(key = String)]
struct File {
    #[input(arc)]
    text: String,
}

#[ajika::query]
fn words(db: &Db<'_>, path: String) -> ajika::Result<Vec<String>> {
    let text = File::at(path).text(db)?;
    Ok(text.split_whitespace().map(str::to_owned).collect())
}

#[ajika::query]
fn word_count(db: &Db<'_>, path: String) -> ajika::Result<usize> {
    Ok(words(db, path)?.len())
}

fn main() -> anyhow::Result<()> {
    // A real application chooses a persistent cache directory.
    let directory = tempfile::tempdir()?;
    let path = directory.path().join("queries");
    let limits = Retention {
        max_entries: 10_000,
        max_bytes: 256 * 1024 * 1024,
        max_idle: Duration::from_secs(24 * 60 * 60),
    };

    let mut database = Database::open(&path, "word-count-v1", Inputs::default(), (), limits)?;
    database.update_inputs(|update| {
        File::at("notes.txt".into()).set_text(update, "hello world".to_owned())
    })?;
    assert_eq!(*database.run(|db| word_count(db, "notes.txt".into()))?, 2);
    assert_eq!(database.statistics().executed, 2);
    drop(database);

    // Simulate fresh acquisition by the application, without reusing the old Inputs.
    let inputs = Inputs::default()
        .edit(|update| File::at("notes.txt".into()).set_text(update, "hello world".to_owned()))?;
    let mut database = Database::open(&path, "word-count-v1", inputs, (), limits)?;
    assert_eq!(*database.run(|db| word_count(db, "notes.txt".into()))?, 2);
    assert_eq!(database.statistics().executed, 0);
    assert_eq!(database.statistics().validated, 2);

    database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(database.statistics().executed, 0);
    assert_eq!(database.statistics().validated, 0);
    assert_eq!(database.statistics().decoded, 1);
    Ok(())
}

No dependency walk is needed: this owner already verified the answer for its current revision. But the previous operation released its decoded working set, so this one decodes the requested value again.

Validation can check a child’s output version without loading its payload. It does not imply decoding the whole reachable graph.

Values and schema changes

Persistent query values need PartialEq and Ajika’s Persist serialization bound. For your own value types, deriving wincode::SchemaRead and SchemaWrite normally supplies Persist. Query keys must also be serializable. All query keys and values must meet the thread-safety and ownership bounds described in query boundaries.

The "word-count-v1" argument is an application epoch. Change it when a new query implementation, key encoding, or result schema makes old stored work unsafe to reuse. Ajika tracks input changes; it cannot discover that you edited a function’s Rust code. A different epoch discards the old cache on opening rather than migrating it.

Recovering a cache

A run that encounters a damaged value returns a storage error and leaves repair pending. The next uncancelled run on the same database attempts repair before invoking its callback. If repair fails, the caller receives that error. Maintenance can also discover damage after a successful run, so database.recovery_pending() reports whether repair is due between runs.

Live repair keeps the installed inputs and interner, as does database.invalidate() when the host chooses to clear saved work. If corruption forces Ajika to discard results, it logs a warning and recomputes them on demand. A new process gets its identity assignments from disk, so it starts with a fresh interner if the saved identity table is lost.

A checkpoint is not application acceptance

The host should reject incoherent input capture before returning successfully from run. Ajika can then abandon the operation’s new query results.

The host still owns publication of an editor response or project generation. Cancellation can arrive after a checkpoint has committed. Stored results remain associated with the observations they actually used. See Database::run and checkpointing.

Separating outputs

A text analyzer may need both words and their byte ranges. Those are produced by the same scan, but they do not change together. Inserting a space preserves the words and moves some ranges.

Returning both in one large value makes every consumer load both and depend on their combined equality. Named outputs let one computation publish separately readable answers.

Produce a set of output handles

The outputs example extends our analyzer with a set of output handles:

extern crate ajika;
extern crate anyhow;
use std::time::Duration;

use ajika::{Database, Db, Inputs, Retention};

#[ajika::input(key = String)]
struct File {
    #[input(arc)]
    text: String,
}

#[ajika::output]
struct FileOutputs {
    words: Vec<String>,
    locations: Vec<(usize, usize)>,
}

#[ajika::query]
fn analyze(db: &Db<'_>, path: String) -> ajika::Result<FileOutputs> {
    let text = File::at(path).text(db)?;
    let mut words = Vec::new();
    let mut locations = Vec::new();
    let mut cursor = 0;
    for word in text.split_whitespace() {
        let start = cursor + text[cursor..].find(word).expect("word comes from text");
        let end = start + word.len();
        words.push(word.to_owned());
        locations.push((start, end));
        cursor = end;
    }
    FileOutputs::new(db, words, locations)
}

#[ajika::query]
fn word_count(db: &Db<'_>, path: String) -> ajika::Result<usize> {
    let outputs = analyze(db, path)?;
    let words = outputs.words.value(db)?;
    Ok(words.len())
}

fn main() -> anyhow::Result<()> {
    let mut database = Database::memory(
        Inputs::default(),
        (),
        Retention {
            max_entries: 10_000,
            max_bytes: 256 * 1024 * 1024,
            max_idle: Duration::from_secs(24 * 60 * 60),
        },
    );
    let file = File::at("notes.txt".into());
    database.update_inputs(|update| file.set_text(update, "hello world".to_owned()))?;
    database.run(|db| word_count(db, "notes.txt".into()))?;
    let locations = database.run(|db| Ok(analyze(db, "notes.txt".into())?.locations.clone()))?;

    database.update_inputs(|update| file.set_text(update, "hello  world".to_owned()))?;
    assert_eq!(*database.run(|db| word_count(db, "notes.txt".into()))?, 2);
    assert_eq!(database.statistics().executed, 1); // analyze ran; word_count stayed valid.
    assert_eq!(
        *database.run(|db| locations.value(db))?,
        vec![(0, 5), (7, 12)]
    );
    Ok(())
}

The macro turns words: Vec<String> into a public Output<Vec<String>> field. FileOutputs::new accepts the actual values, emits them, and returns the handles. Its arguments can also be existing Arcs.

The returned output handles contain addresses, not the words or locations. Equality compares those addresses. When the same query/key emits the same slots again, the returned set can stay equal even if one payload changes.

Read the fact you need

extern crate ajika;
extern crate anyhow;
use std::time::Duration;

use ajika::{Database, Db, Inputs, Retention};

#[ajika::input(key = String)]
struct File {
    #[input(arc)]
    text: String,
}

#[ajika::output]
struct FileOutputs {
    words: Vec<String>,
    locations: Vec<(usize, usize)>,
}

#[ajika::query]
fn analyze(db: &Db<'_>, path: String) -> ajika::Result<FileOutputs> {
    let text = File::at(path).text(db)?;
    let mut words = Vec::new();
    let mut locations = Vec::new();
    let mut cursor = 0;
    for word in text.split_whitespace() {
        let start = cursor + text[cursor..].find(word).expect("word comes from text");
        let end = start + word.len();
        words.push(word.to_owned());
        locations.push((start, end));
        cursor = end;
    }
    FileOutputs::new(db, words, locations)
}

#[ajika::query]
fn word_count(db: &Db<'_>, path: String) -> ajika::Result<usize> {
    let outputs = analyze(db, path)?;
    let words = outputs.words.value(db)?;
    Ok(words.len())
}

fn main() -> anyhow::Result<()> {
    let mut database = Database::memory(
        Inputs::default(),
        (),
        Retention {
            max_entries: 10_000,
            max_bytes: 256 * 1024 * 1024,
            max_idle: Duration::from_secs(24 * 60 * 60),
        },
    );
    let file = File::at("notes.txt".into());
    database.update_inputs(|update| file.set_text(update, "hello world".to_owned()))?;
    database.run(|db| word_count(db, "notes.txt".into()))?;
    let locations = database.run(|db| Ok(analyze(db, "notes.txt".into())?.locations.clone()))?;

    database.update_inputs(|update| file.set_text(update, "hello  world".to_owned()))?;
    assert_eq!(*database.run(|db| word_count(db, "notes.txt".into()))?, 2);
    assert_eq!(database.statistics().executed, 1); // analyze ran; word_count stayed valid.
    assert_eq!(
        *database.run(|db| locations.value(db))?,
        vec![(0, 5), (7, 12)]
    );
    Ok(())
}

This records a dependency on the set returned by analyze, then on the words output read through its handle. It does not read or depend on the locations payload.

Use output.value_ref(db) to borrow an output for the lifetime of the query context. It records the same dependency as output.value(db), which gives you an Arc you can keep after the operation.

Try the whitespace edit:

extern crate ajika;
extern crate anyhow;
use std::time::Duration;

use ajika::{Database, Db, Inputs, Retention};

#[ajika::input(key = String)]
struct File {
    #[input(arc)]
    text: String,
}

#[ajika::output]
struct FileOutputs {
    words: Vec<String>,
    locations: Vec<(usize, usize)>,
}

#[ajika::query]
fn analyze(db: &Db<'_>, path: String) -> ajika::Result<FileOutputs> {
    let text = File::at(path).text(db)?;
    let mut words = Vec::new();
    let mut locations = Vec::new();
    let mut cursor = 0;
    for word in text.split_whitespace() {
        let start = cursor + text[cursor..].find(word).expect("word comes from text");
        let end = start + word.len();
        words.push(word.to_owned());
        locations.push((start, end));
        cursor = end;
    }
    FileOutputs::new(db, words, locations)
}

#[ajika::query]
fn word_count(db: &Db<'_>, path: String) -> ajika::Result<usize> {
    let outputs = analyze(db, path)?;
    let words = outputs.words.value(db)?;
    Ok(words.len())
}

fn main() -> anyhow::Result<()> {
    let mut database = Database::memory(
        Inputs::default(),
        (),
        Retention {
            max_entries: 10_000,
            max_bytes: 256 * 1024 * 1024,
            max_idle: Duration::from_secs(24 * 60 * 60),
        },
    );
    let file = File::at("notes.txt".into());
    database.update_inputs(|update| file.set_text(update, "hello world".to_owned()))?;
    database.run(|db| word_count(db, "notes.txt".into()))?;
    let locations = database.run(|db| Ok(analyze(db, "notes.txt".into())?.locations.clone()))?;

    database.update_inputs(|update| file.set_text(update, "hello  world".to_owned()))?;
    assert_eq!(*database.run(|db| word_count(db, "notes.txt".into()))?, 2);
    assert_eq!(database.statistics().executed, 1); // analyze ran; word_count stayed valid.
    assert_eq!(
        *database.run(|db| locations.value(db))?,
        vec![(0, 5), (7, 12)]
    );
    Ok(())
}

analyze runs because the text changed. Its words compare equal and keep their version; its locations differ and get a new version. The returned set also stays equal, so word_count needs no execution.

The previously retained locations handle reads the new locations. A handle names a slot under the active inputs. It is not an immutable snapshot of its old payload. A retained Arc<T>, by contrast, still contains the old value.

What splitting does and does not buy

Outputs provide separate loading and separate change propagation. They do not make the producer partially executable: when analyze must run, its whole body runs and produces both values. Use separate query functions when computations themselves should be requested independently.

All outputs of a producer share its dependency list and reuse policy. A session input read anywhere in that producer restricts the producer’s outputs together. The evaluator compares their payloads independently after execution.

While constructing outputs, keep using your local values. A producer cannot read its own output handle before it finishes. Its return value and emitted slots become available to waiting callers together.

Optional and dynamically keyed outputs

extern crate ajika;
#[ajika::output]
struct FileOutputs {
    words: Vec<String>,
    #[output(optional)]
    locations: Vec<(usize, usize)>,
}

The optional field becomes Option<Output<Vec<(usize, usize)>>>. Its constructor argument is Option<Arc<Vec<(usize, usize)>>>. Passing None emits no slot. By contrast, an ordinary field locations: Option<Vec<(usize, usize)>> always emits one slot whose payload may be None.

For a collection known only during execution, call db.output(("body", body_key), value_arc) directly. Names and keys identify slots; emission order does not. Each slot may be emitted once by that execution. Field names provide fixed slot names, and #[output(name = "locations")] can choose a name explicitly.

If a later execution omits a slot, a parent validating its old read is made uncertain and can run a different branch. Directly reading an old handle to that removed slot returns an error. Read a producer’s current output handles when membership can change.

The Output implementation and publication walkthrough explain how addresses and versions make these behaviors fit together.

Choosing query boundaries

A query is a reusable question, not a requirement to wrap every helper function. Start with work that is expensive, reused, or useful as a boundary where equality can stop changes. Keep ordinary local computation inside it when no caller benefits from retaining the intermediate answer.

For our analyzer, a words query is useful because many text edits preserve the word vector. A query for every byte would add keys, records, and checks while doing almost no work per call.

Keys identify questions; reads identify observations

Use a stable key such as a file identity or an owner identity. Read changing content through inputs or queries. Putting the entire current source text or revision in the key turns each edit into a new question; Ajika cannot find the old answer to compare.

Context belongs in the key when it changes the question. If one file is analyzed under two language editions, a query keyed only by the file would conflate them. Include the relevant interpretation context, or read it through an input keyed in a way that identifies that interpretation.

Keys are owned and implement Clone + Eq + Hash + Send + Sync + 'static + Persist. The context is the first argument; every argument after it contributes to the key. Values are Send + Sync + 'static; persistent values also need Persist + PartialEq. These are the ordinary Query bounds, whether the implementation is handwritten or generated.

Return a value or reuse an Arc

Declare a query with Result<T> and return an owned T. The macro wraps successful results in Arc<T>, so the generated function returns Result<Arc<T>> to callers.

Sometimes the answer already belongs to an Arc: another query, a captured input, or a named output may have produced it. Use forward to return that Arc directly, without copying the payload or allocating another wrapper. The word-count example includes a query for the fixed notes.txt path:

extern crate ajika;
extern crate anyhow;
use std::{sync::Arc, time::Duration};

use ajika::{Database, Db, Inputs, Retention};

#[ajika::input(key = String)]
struct File {
    #[input(arc)]
    text: String,
}

#[ajika::query]
fn words(db: &Db<'_>, path: String) -> ajika::Result<Vec<String>> {
    let text = File::at(path).text(db)?;
    Ok(text.split_whitespace().map(str::to_owned).collect())
}

#[ajika::query]
fn word_count(db: &Db<'_>, path: String) -> ajika::Result<usize> {
    Ok(words(db, path)?.len())
}

// This convenience query reuses the words query's allocation.
#[ajika::query(forward)]
fn note_words(db: &Db<'_>) -> ajika::Result<Arc<Vec<String>>> {
    words(db, "notes.txt".into())
}

fn main() -> anyhow::Result<()> {
    let limits = Retention {
        max_entries: 10_000,
        max_bytes: 256 * 1024 * 1024,
        max_idle: Duration::from_secs(24 * 60 * 60),
    };
    let mut database = Database::memory(Inputs::default(), (), limits);
    let file = File::at("notes.txt".into());
    database.update_inputs(|update| file.set_text(update, "hello world".to_owned()))?;

    let count = database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(*count, 2);
    assert_eq!(database.statistics().executed, 2);

    database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(database.statistics().executed, 0);

    database.update_inputs(|update| file.set_text(update, "hello    world".to_owned()))?;
    database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(database.statistics().executed, 1); // Only words ran.

    database.update_inputs(|update| file.set_text(update, "hello new world".to_owned()))?;
    let count = database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(*count, 3);
    assert_eq!(database.statistics().executed, 2);
    database.run(|db| {
        let words = words(db, "notes.txt".into())?;
        assert!(Arc::ptr_eq(&words, &note_words(db)?));
        Ok(())
    })?;
    Ok(())
}

Both words and note_words return Result<Arc<Vec<String>>> to their callers. note_words reuses the allocation returned by words.

The forward option works on associated queries too.

Borrow the answer for an operation

Use returns(ref) when callers only need to read the answer during the operation. The body still returns Result<T>, and the generated function returns Result<&'db T>.

extern crate ajika;
use ajika::{Db, Result};

#[ajika::query(returns(ref))]
fn label(_db: &Db<'_>, id: u32) -> Result<String> {
    Ok(format!("file-{id}"))
}

fn label_text<'db>(db: &Db<'db>, id: u32) -> Result<&'db str> {
    Ok(label(db, id)?.as_str())
}

returns(arc) spells out the default for callers that need an owned answer. Both modes work on free and associated queries and can be combined with forward.

Call a query explicitly

A free #[ajika::query] function also generates a query type with the same name. Pass that type to db.query::<Q>(key) for an Arc or to db.query_ref::<Q>(key) for a borrow. Both calls use the same cached answer and track the same dependency. They let you choose the return mode at the call site.

extern crate ajika;
use std::sync::Arc;
use ajika::{Db, Result};

#[ajika::query(returns(ref))]
fn location(_db: &Db<'_>, file: u32, line: u32) -> Result<String> {
    Ok(format!("{file}:{line}"))
}

fn keep_location(db: &Db<'_>) -> Result<Arc<String>> {
    db.query::<location>((4, 12))
}

fn borrow_location<'db>(db: &Db<'db>) -> Result<&'db String> {
    db.query_ref::<location>((4, 12))
}

Use () for a query with no key arguments, the argument itself for one, and a tuple in argument order for several. For a handwritten query, use the type that implements Query and pass its Query::Key. For associated queries, call the generated associated function with its declared return mode.

Put a query beside a natural owner

An associated query can make a constructor-like computation easier to find:

extern crate ajika;
use ajika::{Db, Result};

struct WordIndex(Vec<String>);

#[ajika::query]
impl WordIndex {
    #[ajika::query(transient)]
    fn from_text(_db: &Db<'_>, text: String) -> Result<Self> {
        Ok(Self(text.split_whitespace().map(str::to_owned).collect()))
    }

    fn len(&self) -> usize {
        self.0.len()
    }
}

This small example takes text directly as an owned key to demonstrate the signature. A file-backed index would normally key by the file and read its text through an input.

Mark the concrete inherent impl and each query function. Ordinary methods remain ordinary methods, and Self works in signatures and bodies. Query functions have no receiver. Generic and trait impls are not supported.

The owner also becomes part of the query’s default name, so WordIndex::from_text gets a name such as module::WordIndex::from_text. The query attribute documentation lists the available overrides and other declaration options.

Choose what survives an operation

#[ajika::query] retains its result according to the database backend. #[ajika::query(transient)] releases the primary result when the operation ends. The latter is useful for an intermediate that should not need serialization or equality.

A transient query’s dependency record can still be retained. A stored parent may validate through it without asking for its value. If someone does need that value again, its function runs to reconstruct it.

When the old dependencies are still valid, that reconstruction can preserve the known output version. If its dependencies changed, there is no PartialEq comparison available for the transient result, so Ajika conservatively assigns a fresh version. Use a persistent semantic result when equality is an important invalidation boundary.

A memory database does not make transient query values survive operations. A serializable result depending on session input does not become durable.

Equality is part of the contract

After recomputation, PartialEq decides whether consumers may keep using the previous answer. It must include every property they can observe. Ajika may retain the old serialized bytes when values compare equal.

If locations should change without invalidating semantic consumers, split them into outputs; do not hide observable locations from equality. Likewise, answer-affecting reads through services or mutable global state would break determinism. Host services are for facilities such as progress reporting, not an alternative input channel.

Keep output granularity proportional to how callers read data. Reading an Output<T> decodes the whole T, not selected fields. An operation also retains its touched results until it finishes. A finer query boundary can reduce the data requested; it does not eliminate the cost of a large operation that requests everything.

Interning objects

Interning gives equal immutable objects the same compact ID. An interned declaration describes the fields that identify an object; an Interner assigns their IDs. This keeps query keys small when many calls refer to the same object.

For example, a file path can identify a file while its contents change:

extern crate ajika;
use ajika::{Database, Result};

#[ajika::interned(name = "source-files")]
struct FileId {
    path: String,
}

fn intern_file(database: &Database, canonical_path: String) -> Result<FileId> {
    let interner = database.interner();
    let first = FileId::new(&interner, canonical_path.clone())?;
    let second = FileId::new(&interner, canonical_path)?;
    assert_eq!(first, second);
    let path = first.path(&interner)?;
    assert!(!path.is_empty());
    Ok(first)
}

The generated ID is copyable, comparable, hashable, and serializable. You do not convert integers or manually encode keys. Each declaration has one numbering space per database. By default, Ajika names that space after the module and type. Use name = "..." to keep the same numbering space across a Rust rename.

Choose what to intern

Intern the immutable facts that identify an entity. If you identify a file by its canonical path, canonicalize it before calling new. Ajika has no filesystem policy: two differently encoded paths are different keys.

If package or language context belongs to interpretation rather than identity, keep it in the semantic query’s key. If it really distinguishes the entity itself, put it in the interned fields.

Editable text is usually an input value, not an interned field. Interning a new complete text on every edit would retain a growing history of interned objects.

Use the database’s interner

database.interner() gives the host access to the database’s assignments, including any restored from disk. The host can use this handle while constructing inputs, and queries can reach the same interner through db.interner(). Clones share assignments, so two workers interning the same path still get the same ID.

An ID doesn’t record which database created it. Looking it up through an unrelated interner can resolve a different entity. Keep the owner relationship in the application’s design.

Field lookup decodes the immutable key and returns an owned value. It records no query dependency because that assignment does not change. File existence, contents, and membership are separate observations and belong in tracked inputs.

What survives

Assignments are never recycled within a live interner. Query collection and Database::invalidate preserve them. New assignments become durable alongside a successful durable query checkpoint; merely allocating an ID or running an operation with no durable changes is not a persistence barrier.

Reopening a compatible cache restores saved assignments before handing out the interner. A full reset can discard them. This includes epoch changes and the interner size limit checked at open, so IDs must not become an application’s authoritative identifiers outside the cache lifecycle.

See Interner for allocation and checkpointing, and storage reclamation for the distinction between query collection and identity reset.

Acquiring inputs on demand

An application can know that thousands of files exist without needing all their text. An input initializer lets it capture a field when the first query reads it. Later reads of that captured field receive the same observation.

Capture once, then read

The initializer receives the input key and an InputInit context. For a file keyed by an interned ID, use the context’s interner to look up its path.

extern crate ajika;
use ajika::{InputInit, Result};

#[ajika::interned]
struct FileId {
    path: String,
}

fn acquire_text(id: &FileId, cx: &dyn InputInit) -> Result<String> {
    cx.check_cancelled()?;
    let path = id.path(&cx.interner())?;
    std::fs::read_to_string(path).map_err(ajika::Error::application)
}

#[ajika::input(key = FileId)]
struct File {
    #[input(arc, initialize = acquire_text)]
    text: String,
}

Treat lazy acquisition as an optimization. A read or cache validation can run acquire_text whenever it needs an uncaptured key. A successful capture becomes visible to the application, including through views that share that input.

A setter can install the field before the initializer is needed. For an arc field, an initializer can also return an existing Arc<T>.

The initializer runs outside the input table’s lock. If two successful initializations race for the same entry, both reads return the first value installed.

An initializer captures external data. During a query read or cache validation, check_cancelled() polls the active operation. For long acquisitions, call it between chunks and return the error to abandon the capture.

The host decides when to reacquire

Filesystem notifications are an application concern. On a change, either install newly captured text in an input batch or forget the old capture:

extern crate ajika;
use ajika::{Database, InputInit, Result};
fn acquire_text(path: &String, _cx: &dyn InputInit) -> Result<String> {
    std::fs::read_to_string(path).map_err(ajika::Error::application)
}
#[ajika::input(key = String)]
struct File { #[input(arc, initialize = acquire_text)] text: String }
fn refresh_later(database: &mut Database, path: String) -> Result<()> {
    database.update_inputs(|update| {
        update.forget(File::TEXT, path);
        Ok(())
    })
}

The next read can initialize that field again. Use File::at(path).forget_all(update) to forget every field for one path, or update.forget_all(File::TEXT) to forget the text for every path. If absence is a meaningful result, capture it explicitly, for example as an Option value, so queries can depend on absence too.

Prefer one batch for a group of file changes. It defines the next input view and avoids copying a field table separately for every file.

A captured view is not a filesystem snapshot

Ajika keeps a captured field stable. It cannot make separate filesystem reads atomic. If a file changes while an operation discovers several files, the host must decide whether that combined view is acceptable.

An editor or indexer can attach an immutable source observation to a field instead of storing text directly. That object may acquire text lazily, but it must keep the text consistent with the field’s stamp. It cannot simply expose “whatever is on disk now” under an old stamp. Verify acquisition against the expected content, or reject the operation and capture a new view.

Do not mutate answer-affecting contents of a captured object in place. Ajika only sees published stamps and reads through its input interface.

Inspecting inputs from the host

The generated field selector, such as File::TEXT, also works with Inputs::get, for example inputs.get(File::TEXT, &id, &database). You can also pass &interner directly, using the same interner that created the ID. Both host contexts use a no-op cancellation check. peek and entries inspect already captured data without triggering acquisition.

Inside a query, use the tracked getter with its Db. A host-side peek hidden in a service would bypass dependency tracking.

The capture and sharing rules live in input tables. The internal input chapter follows how snapshots share these observations while edits remain isolated.

Working with unsaved edits

An editor needs to reuse work between keystrokes without replacing saved results on disk. Mark the edited observations as session inputs. Queries still use the same functions and keys; their input reads determine where results may be kept.

Lifetime follows the observations

Each field observation has one of three lifetimes:

A query inherits the shortest lifetime it reads, including through child queries. If a saved project depends on one unsaved file, affected work becomes session work; unaffected saved queries remain durable.

Lifetime does not describe how often data changes. That is the separate durability classification.

Keep the saved view and override one field

The editor-session example stores text together with its retention policy:

extern crate ajika;
extern crate anyhow;
extern crate tempfile;
use std::{sync::Arc, time::Duration};

use ajika::{Database, Db, Inputs, Lifetime, Retention, Stamp};

#[derive(Clone)]
struct Text {
    contents: Arc<String>,
    lifetime: Lifetime,
}

impl Text {
    fn stamp(&self) -> ajika::Result<Stamp> {
        Ok(Stamp::of(self.contents.as_bytes()))
    }

    fn lifetime(&self) -> Lifetime {
        self.lifetime
    }
}

#[ajika::input(key = String)]
struct File {
    #[input(stamp = Text::stamp, lifetime = Text::lifetime)]
    text: Text,
}

#[ajika::query]
fn word_count(db: &Db<'_>, path: String) -> ajika::Result<usize> {
    Ok(File::at(path).text(db)?.contents.split_whitespace().count())
}

fn main() -> anyhow::Result<()> {
    let directory = tempfile::tempdir()?;
    let mut database = Database::open(
        directory.path().join("queries"),
        "editor-v1",
        Inputs::default(),
        (),
        Retention {
            max_entries: 10_000,
            max_bytes: 256 * 1024 * 1024,
            max_idle: Duration::from_secs(24 * 60 * 60),
        },
    )?;
    let file = File::at("notes.txt".into());
    database.update_inputs(|update| {
        file.set_text(
            update,
            Text {
                contents: Arc::new("hello world".into()),
                lifetime: Lifetime::Persistent,
            },
        )
    })?;
    database.run(|db| word_count(db, "notes.txt".into()))?;

    let saved = database.inputs().clone();
    let edited = Arc::new("hello new world".to_owned());
    let editor = saved.overlay().edit(|update| {
        file.set_text(
            update,
            Text {
                contents: edited.clone(),
                lifetime: Lifetime::Session,
            },
        )
    })?;
    database.replace_inputs(editor);
    assert_eq!(*database.run(|db| word_count(db, "notes.txt".into()))?, 3);
    assert_eq!(database.statistics().bytes_written, 0);

    database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(database.statistics().executed, 0);
    assert_eq!(database.statistics().decoded, 0);

    // The host has saved these exact bytes successfully.
    let saved = saved.edit(|update| {
        file.set_text(
            update,
            Text {
                contents: edited,
                lifetime: Lifetime::Persistent,
            },
        )
    })?;
    database.replace_inputs(saved);
    database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(database.statistics().executed, 0);
    assert_eq!(database.statistics().session_queries, 0);
    Ok(())
}

The custom stamp identifies the text; the lifetime callback supplies its policy. Changing policy is still an input change even if the stamp stays equal.

After computing the saved file, the application creates a sparse input overlay:

extern crate ajika;
extern crate anyhow;
extern crate tempfile;
use std::{sync::Arc, time::Duration};

use ajika::{Database, Db, Inputs, Lifetime, Retention, Stamp};

#[derive(Clone)]
struct Text {
    contents: Arc<String>,
    lifetime: Lifetime,
}

impl Text {
    fn stamp(&self) -> ajika::Result<Stamp> {
        Ok(Stamp::of(self.contents.as_bytes()))
    }

    fn lifetime(&self) -> Lifetime {
        self.lifetime
    }
}

#[ajika::input(key = String)]
struct File {
    #[input(stamp = Text::stamp, lifetime = Text::lifetime)]
    text: Text,
}

#[ajika::query]
fn word_count(db: &Db<'_>, path: String) -> ajika::Result<usize> {
    Ok(File::at(path).text(db)?.contents.split_whitespace().count())
}

fn main() -> anyhow::Result<()> {
    let directory = tempfile::tempdir()?;
    let mut database = Database::open(
        directory.path().join("queries"),
        "editor-v1",
        Inputs::default(),
        (),
        Retention {
            max_entries: 10_000,
            max_bytes: 256 * 1024 * 1024,
            max_idle: Duration::from_secs(24 * 60 * 60),
        },
    )?;
    let file = File::at("notes.txt".into());
    database.update_inputs(|update| {
        file.set_text(
            update,
            Text {
                contents: Arc::new("hello world".into()),
                lifetime: Lifetime::Persistent,
            },
        )
    })?;
    database.run(|db| word_count(db, "notes.txt".into()))?;

    let saved = database.inputs().clone();
    let edited = Arc::new("hello new world".to_owned());
    let editor = saved.overlay().edit(|update| {
        file.set_text(
            update,
            Text {
                contents: edited.clone(),
                lifetime: Lifetime::Session,
            },
        )
    })?;
    database.replace_inputs(editor);
    assert_eq!(*database.run(|db| word_count(db, "notes.txt".into()))?, 3);
    assert_eq!(database.statistics().bytes_written, 0);

    database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(database.statistics().executed, 0);
    assert_eq!(database.statistics().decoded, 0);

    // The host has saved these exact bytes successfully.
    let saved = saved.edit(|update| {
        file.set_text(
            update,
            Text {
                contents: edited,
                lifetime: Lifetime::Persistent,
            },
        )
    })?;
    database.replace_inputs(saved);
    database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(database.statistics().executed, 0);
    assert_eq!(database.statistics().session_queries, 0);
    Ok(())
}

The overlay refers to the saved base and stores only its overrides. It does not copy all saved sources. Missing fields can still be acquired through the base. The application retains the saved view so it can discard or save editor changes later.

The first operation computes the edited count. The second reuses it directly from the session store. Neither writes that edited payload to disk.

This is one installed input view and one set of latest session results per owner. It is not a facility for querying several historical revisions concurrently.

Saving can reuse the same computation

After the host successfully saves those exact bytes, it publishes the same content as persistent:

extern crate ajika;
extern crate anyhow;
extern crate tempfile;
use std::{sync::Arc, time::Duration};

use ajika::{Database, Db, Inputs, Lifetime, Retention, Stamp};

#[derive(Clone)]
struct Text {
    contents: Arc<String>,
    lifetime: Lifetime,
}

impl Text {
    fn stamp(&self) -> ajika::Result<Stamp> {
        Ok(Stamp::of(self.contents.as_bytes()))
    }

    fn lifetime(&self) -> Lifetime {
        self.lifetime
    }
}

#[ajika::input(key = String)]
struct File {
    #[input(stamp = Text::stamp, lifetime = Text::lifetime)]
    text: Text,
}

#[ajika::query]
fn word_count(db: &Db<'_>, path: String) -> ajika::Result<usize> {
    Ok(File::at(path).text(db)?.contents.split_whitespace().count())
}

fn main() -> anyhow::Result<()> {
    let directory = tempfile::tempdir()?;
    let mut database = Database::open(
        directory.path().join("queries"),
        "editor-v1",
        Inputs::default(),
        (),
        Retention {
            max_entries: 10_000,
            max_bytes: 256 * 1024 * 1024,
            max_idle: Duration::from_secs(24 * 60 * 60),
        },
    )?;
    let file = File::at("notes.txt".into());
    database.update_inputs(|update| {
        file.set_text(
            update,
            Text {
                contents: Arc::new("hello world".into()),
                lifetime: Lifetime::Persistent,
            },
        )
    })?;
    database.run(|db| word_count(db, "notes.txt".into()))?;

    let saved = database.inputs().clone();
    let edited = Arc::new("hello new world".to_owned());
    let editor = saved.overlay().edit(|update| {
        file.set_text(
            update,
            Text {
                contents: edited.clone(),
                lifetime: Lifetime::Session,
            },
        )
    })?;
    database.replace_inputs(editor);
    assert_eq!(*database.run(|db| word_count(db, "notes.txt".into()))?, 3);
    assert_eq!(database.statistics().bytes_written, 0);

    database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(database.statistics().executed, 0);
    assert_eq!(database.statistics().decoded, 0);

    // The host has saved these exact bytes successfully.
    let saved = saved.edit(|update| {
        file.set_text(
            update,
            Text {
                contents: edited,
                lifetime: Lifetime::Persistent,
            },
        )
    })?;
    database.replace_inputs(saved);
    database.run(|db| word_count(db, "notes.txt".into()))?;
    assert_eq!(database.statistics().executed, 0);
    assert_eq!(database.statistics().session_queries, 0);
    Ok(())
}

The text stamp matches. Ajika refreshes the derived lifetime and can promote the retained result to durable storage without executing the query again. This reuse depends on the session result still being retained; eviction may require computation.

If an edit is discarded, install the saved input view instead. That is a new revision, not a rewind of the runtime. Validation decides which saved answers match it.

Budget the session separately

Database::set_session_retention controls retained session records and changed payloads. The default budget is 16,384 records, 64 MiB of estimated encoded data, and five minutes since last use. These are cache limits, not a measurement of allocator memory.

The session store can refer back to an equal durable payload rather than keep a second decoded copy. Changed payloads remain typed values in memory. Transient query results still expire with the operation.

clear_session drops retained session query work. It does not close editor buffers or replace input overlays; the host owns those actions. Keeping many returned Arc values alive also keeps their allocations alive independently of these budgets.

Checkpointing explains how Ajika separates session overrides from the durable cache.

Parallel work, errors, and cancellation

An operation is a useful unit of application work: an indexing pass, one editor request, or a batch of related requests. Queries within it share decoded and computed values. When it finishes, Ajika checkpoints eligible work and releases that working set.

Choose the boundary with both reuse and peak memory in mind. Starting an operation for every tiny helper loses sharing. Keeping one operation open indefinitely keeps everything it touched alive.

Run independent work together

Query calls are synchronous. A parent waits for the children it calls, and two workers requesting the same call share its computation.

Use Db::parallel_map for a batch of independent work:

extern crate ajika;
use ajika::{Db, Result};

#[ajika::input(key = String)]
struct File { text: String }
#[ajika::query]
fn word_count(db: &Db<'_>, path: String) -> Result<usize> {
    Ok(File::at(path).text(db)?.split_whitespace().count())
}

#[ajika::query]
fn total_words(db: &Db<'_>, files: Vec<String>) -> Result<usize> {
    let counts = db.parallel_map(&files, |db, path| word_count(db, path.clone()))?;
    Ok(counts.iter().map(|count| **count).sum())
}

Each callback receives a context for the same operation. Its reads become reads of total_words; there is no extra query declaration for a callback. Results keep input order even when work finishes out of order.

Database::set_parallelism limits workers, including the calling thread. Nested batches share that budget. All scoped workers finish before parallel_map returns, including after failure.

The supported helper uses scoped threads that keep making progress while a query waits. Arbitrary work-stealing pools can suspend a query owner beneath a task waiting for that owner. Do not assume that substituting a Rayon batch preserves this scheduling property.

The database owner itself is exclusive: run and input replacement require &mut Database. To react to an edit during work, cancel or finish the operation, then install the next inputs.

Keep host services separate from semantic inputs

The second argument to Database::memory, or the services argument to open, becomes available through db.services(). It may hold progress reporting, tracing facilities, or other host integration.

For a Services type, queries take &Db<'_, Services>. Services are untracked: changing them does not invalidate queries. Values that affect an answer belong in inputs, even if the application also stores them elsewhere.

Queries must not rely on execution for required side effects. A cache hit skips the function. For example, progress reporting may describe work that actually runs, but saving a user document cannot be hidden inside a query.

Errors abandon an operation’s pending work

A query error is an operation failure. It prevents new work from being checkpointed and wakes workers waiting for that work. Catching a child query’s error inside another query does not make the operation healthy again.

Represent expected domain problems—such as a syntax error with useful partial analysis—as successful result data. Reserve Err for work the operation cannot complete. A host can also return an error from the run callback to reject a candidate before checkpointing.

No fixed-point recovery is provided for query cycles. Ajika returns a QueryCycle containing a closed chain of calls, including cross-worker cycles. Diagnostic context preserves the typed cycle:

extern crate ajika;
use ajika::{Error, ErrorKind};

fn report_cycle(error: &Error) {
    if let ErrorKind::Cycle(cycle) = error.kind() {
        eprintln!("{cycle}");
    }
}

Queries and Database::run use the same ajika::Result<T>, a standard Result<T, ajika::Error>. Error::is_cancelled() identifies an expected stop. Error::kind() exposes cancellation, cycle, unavailable-data, persistence, application, contract, and unexpected-failure categories. Precise causes remain available through std::error::Error::source.

Use Error::application(source) to preserve a host-specific failure from a provider. This extension point is explicit: an existing Ajika error should be propagated with ?, so it keeps its classification. ResultExt::context adds reporting context without changing a failure’s kind or a cancellation checkpoint. Display prints the current summary; an application reporter such as anyhow can include the full source chain. A run callback uses ordinary ? and Ok(...) without an error-type annotation. It can reject a candidate with Err(Error::application(source)). This accepts concrete errors, boxed standard errors, and anyhow::Error reports. Concrete errors remain downcastable through ErrorKind::Application. An anyhow report’s standard boxed conversion preserves its diagnostic chain but may erase typed downcasts; pass the concrete error when the host needs typed recovery.

Classify provider-side parsing and serialization with .map_err(Error::application)?. Ajika classifies its own codec failures internally.

Recursive domain algorithms can still live inside a query. A recursive call to another key is also fine when it terminates; a call graph that returns to an unfinished key is the cycle that cannot complete.

Cancellation is cooperative

Obtain a Cancellation handle from database.cancellation() before starting work. Another thread can call cancel(). Query boundaries and parallel workers check it, and long local loops should call db.check_cancelled()? themselves.

Handle cancellation once at the request’s publication boundary. Other helpers propagate it with ordinary ?:

extern crate ajika;
fn publish<T>(_: T) {}
fn finish<T>(result: ajika::Result<T>) -> ajika::Result<()> {
    match result {
        Ok(value) => publish(value),
        Err(error) if error.is_cancelled() => {},
        Err(error) => return Err(error),
    }
    Ok(())
}

A recorded worker failure takes precedence over cancellation, even if the run callback catches that failure and returns cancellation. Cancelled::at lets an external cancellation system retain a named checkpoint when adapting its signal with Err(ajika::Cancelled::at("request superseded").into()). The checkpoint is available through ErrorKind::Cancelled.

Cancellation does not forcibly interrupt arbitrary Rust code or an I/O syscall. The owner issues a fresh token after the operation or input replacement; retain the token for the work you mean to cancel, not for all future operations.

A failed operation releases its working set and claims. A cache checkpoint may already have committed before a late cancellation, so the host still decides whether to publish the response. Automatic maintenance failures are logged.

A capacity refusal still permits reads and reclamation. Storage errors can schedule cache repair for the next run.

Understanding cost and reuse

Avoiding execution is only one kind of reuse. A request may still walk dependencies, read metadata, or decode a large stored answer. Ajika exposes these costs separately so you can choose query boundaries from evidence.

Start with operation statistics

After run, inspect database.statistics(). These counters cover the latest operation and its nested calls, not just the root request.

For example, zero executions with a large bytes_read suggests that the request loads more data than it needs. Many unchanged recomputations show useful equality boundaries, but may also expose an expensive producer that could be split.

These are not wall-clock timings or exact RSS measurements. bytes_written counts encoded payload bytes, even if the final commit fails. Metadata-only updates can still happen when it is zero. The Statistics fields specify the individual accounting rules.

Ask why a particular call ran

Ajika emits diagnostic events through the tracing target ajika::diagnostics. Enable it with this filter:

ajika::diagnostics=debug

debug reports executions and the first dependency that failed validation. trace also reports reuse decisions and individual output outcomes.

An illustrative whitespace edit looks like this, with some fields omitted:

op=17 revision=3 query=analyze call=ab12cd34
  invalidated dependency=File::text reason=input_stamp_changed
  execute reason=dependency_invalid
  output=words result=equal evidence=comparison version=preserved
  output=locations result=unequal evidence=comparison version=advanced
op=17 revision=3 query=word_count call=ef56ab78
  reuse reason=dependencies_valid

Every event includes a process-local operation ID and its input revision. Query and input families have names. call is a shortened address hash; call_id retains the full hash for correlation. Named slots include their name; keyed slots show the name and an opaque key hash. No Debug bounds on application keys or values are required.

A new version doesn’t always mean the value changed. The previous value may be unavailable for comparison; that outcome is comparison_unavailable. Conversely, reconstructing a transient result with still-valid dependencies can preserve a version by proof without comparing payloads.

Events describe decisions made during evaluation, before checkpointing. Enabling them does not force additional reads or comparisons. The diagnostic implementation only renders evidence the evaluator already has.

Separate cache budgets from validity

Retention::max_idle controls when unused stored work becomes eligible for collection. It is not how Ajika detects stale answers. Input stamps and dependency versions do that on demand, even one millisecond after an edit.

Entry, age, and byte budgets reclaim whole query records with their outputs. Successful operations collect periodically or when over budget; collect() lets the host request an idle-boundary pass. There is no background timer.

Logical retained bytes, compressed payload bytes, physical files, and allocator memory are different measurements. storage_statistics() exposes physical storage information for a disk backend. compact() performs a full physical maintenance pass; it can take much longer than regular maintenance.

set_compression selects no compression, LZ4, or Zstandard for new payloads. It does not rewrite existing payloads or alter query equality. Measure with representative data: serialization, compression, validation, and computation are different costs.

The storage chapter explains where space is used and how logical eviction becomes physical reclamation.

The application’s model

Ajika lets an application ask the same questions as its data changes. A request might ask for a module’s declarations, an image’s thumbnail, or the diagnostics for one function. The application describes how to compute the answer; Ajika decides when a previous answer is still usable.

This section explains the public concepts and the choices they support. You can read it without the tutorial or knowledge of the runtime’s implementation. The type and method names give you entrypoints to look up in the API documentation when you need exact signatures.

Separate supplied facts from derived answers

An input is a fact supplied by the application: file text, language settings, or the list of files in a project. A query computes an answer by reading inputs and calling other queries.

For example, a declaration query can read source text and an edition setting. Ajika remembers those observations. Editing another file doesn’t necessarily affect that answer. An edit to this file requires checking it, but may still leave its declarations equal.

The question needs an identity that survives those edits. A query key says which question is being asked—perhaps a file and its project context. The changing text belongs in an input. Otherwise every edit would become an unrelated question, with no previous answer to compare.

The declaration macros express those roles:

They are useful independently. A query can use ordinary keys and return one ordinary value. It doesn’t need interned IDs or output handles.

Give the host and queries different jobs

The host owns a long-lived Database. It supplies inputs and starts an operation with Database::run. The operation is a scope of work such as serving an editor request.

The callback and its queries receive a Db. It connects their reads to the installed inputs and lets queries share work. Passing it to a child query is how the child participates in the same operation.

host publishes inputs
        ↓
Database::run
        ├─ requested query
        │    ├─ input reads
        │    └─ child queries
        └─ result returned to the host

Ordinary query calls return Arc<T>. Callers can work with the answer without knowing whether it was just computed or reused. When the operation finishes, Ajika retains eligible results for later use and releases its temporary working set. The host decides what to publish to the user.

Choose the part you need

• Inputs, interning, and context explains how compact IDs, supplied values, and project settings work together.
• Queries and observable change explains where reuse boundaries belong and what makes an answer safe to reuse.
• Values and output handles explains the difference between keeping an answer and keeping a way to ask for it again.
• Operations explains how requests, input updates, and failures fit into application control flow.
• Where results live separates memory sharing, persistence, session reuse, and eviction.
• Owning the cache covers startup, schema changes, and maintenance.

The API documentation gives the detailed contracts for these entrypoints. From a checkout, cargo doc -p ajika --no-deps --open opens that documentation for the version you are using.

Inputs, interning, and context

A source file has a path, text that changes as the user edits it, and settings under which it is analyzed. The same file may belong to two projects with different settings. Keeping these roles separate helps queries reuse work across edits and projects.

Interning gives objects short IDs

Plain strings already work as query keys. In a large graph, though, the same file path may appear in many keys and results. Interning stores that path once and lets those references carry a compact ID. The rest of the graph can copy, hash, and compare these IDs as numbers rather than string contents.

The file’s path can stay the same while its text changes. We can intern the path and keep the text in an input addressed by that ID:

extern crate ajika;

#[ajika::interned(name = "source-files")]
struct FileId {
    path: String,
}

#[ajika::input(key = FileId)]
struct Source {
    #[input(arc)]
    text: String,
}

FileId::new(&interner, path) returns the same ID for equal identifying fields in that interner. Updating the text through Source::at(file) keeps the file’s ID, so queries about that file can find their previous answers after an edit.

Fields are compared as supplied. If your application treats two path spellings as the same file, normalize them before interning.

The host gets the shared Interner through Database::interner(); queries use Db::interner(). Keep an ID with the database that assigned it: another database can assign the same number to a different path.

Interned IDs can be part of persistent query keys and results. A durable query checkpoint saves new ID assignments alongside the query data, and reopening the same cache restores them. Query collection and Database::invalidate preserve those assignments. A full cache reset can discard both the assignments and the query data that refers to them; see Owning the cache.

Inputs hold the values that change

FileId lets queries refer to the same file; the Source input supplies its text. If we also give the file a display-name field, changing that label should not make a parser reconsider the text. Ajika tracks input fields separately, so a query depends only on the fields it reads.

When the user edits the file, the host supplies the new text through a generated setter in Database::update_inputs. The batch publishes a new Inputs view. Queries read the text through their Db, which records what each answer relied on.

Before reusing an answer, Ajika needs to recognize whether that supplied text has changed. It records a Stamp when the input is supplied or first acquired, then compares stamps during validation. Default stamps use the encoded value. A custom stamp can use an existing content fingerprint, but must cover everything queries can observe. Mutating the text in place without publishing an update can leave the recorded stamp unchanged and make reused answers stale.

Inputs also describe facts such as existence, membership, and failed lookups. For example, resolving an optional configuration file depends on its absence until it appears. Model that absence as a supplied value, perhaps None, so Ajika can notice the transition. Forgetting an input instead asks to acquire it again; it does not establish that the file is absent.

Acquire inputs on first use

Some files may never be requested, so the host can use an initializer to load their text on first use. Later reads in that input view reuse the captured text. When the host learns that a file has changed, it can supply new text through an input update, or forget the old value so the next read invokes the initializer again.

The lazy-input chapter covers initializers, forgetting captured values, and keeping file acquisition consistent.

Context can change the answer

Now suppose the same file belongs to two projects using different language editions. They can share its FileId and source text, but a semantic query may produce different answers in each project. Such a query can take (ProjectId, FileId) and read the settings input for that project. A query that only needs source text can still take just FileId.

Keeping project context in the semantic query’s key lets source-only queries share their work. Giving the file a different ID in each project would separate those reads too, even when they use exactly the same text.

Read those settings through tracked inputs. Db::services is useful for facilities such as progress reporting, but its values are untracked: changing them does not invalidate answers.

Queries and observable change

A query defines a question that remains useful to ask after inputs change. The function describes how to answer it; calling the function asks Ajika for the current answer. Calling and executing are different events.

That distinction is why query code should describe a computation, rather than a step that must happen every time a request arrives.

Keep the question stable

The query family and its key identify the question. All arguments after the Db contribute to the key. For a declaration query, a file and interpretation context are plausible keys. The current source revision usually is not.

Including a revision or entire changing input in the key creates a new call after each change. That can be useful when content really is the computation’s identity, but it gives up comparing successive answers about the same entity.

Read changing state through inputs and child queries instead. Then Ajika knows both which previous answer to consider and which observations could make it stale.

A dependency means “my answer relied on this”

If a completion query reads an import table, it depends on the import table’s answer. It does not automatically depend directly on every source file used to build that table. A source edit that leaves the table equal can leave completion reusable.

Relationships follow the reads that actually occur, including conditional ones. You do not need to predict a query’s full dependency set in its declaration.

A view built before the completion query runs might already hold an Arc<ImportTable>. Reading that Arc through a helper does not tell Ajika that the completion query depends on the table. If the view keeps a query key or output handle instead, the helper can acquire the table through the caller’s Db. Each query using the view then records its own read.

That tracked read depends on the whole table, even if the caller only uses one field. A smaller query result or named output can provide a narrower boundary when callers benefit from it.

Equality describes interchangeable answers

Persistent query results use PartialEq to decide whether dependent work can keep its old answer. Equal results must be interchangeable for their consumers. Ajika may keep the old stored bytes instead of saving the newly computed value.

Suppose declarations contain both signatures and source locations. Defining equality to ignore locations is wrong if callers can read those locations. Separate signatures and locations into independently comparable results instead.

The same principle applies to custom input stamps: they describe equivalence of observations, not just the fields an author happens to find important.

A query can reach the same answer through different reads. For example, it can select a different child after an edit and still return the same value. Future reuse will follow the new reads, while callers may continue reusing the equal answer.

Choose boundaries that callers can benefit from

A good query boundary often serves one of three purposes: sharing expensive work, letting callers request a useful subset, or stopping change propagation at an answer that often stays equal.

A query per trivial helper can cost more in metadata and acquisition than it saves. A single project-wide query can force small requests to load and compare far more than they use. Look at the questions features ask and the kinds of edits that should leave their answers unchanged.

Named outputs help when one computation naturally produces several independently used values. Separate queries help when the computations themselves should run independently. Those are different ways to improve granularity.

A helper that only assembles a small value may be clearer as an ordinary function. Use a query declaration when callers benefit from remembering its answer.

Execution must remain optional

The same observations must yield an equivalent answer. Time, random values, or mutable services must not secretly determine the result. Required side effects, such as saving a document, belong in host control flow because a reused query skips execution.

Cycles between unfinished calls return QueryCycle; Ajika does not solve them as fixed points. A recursive domain algorithm can run inside one query, and recursive queries over different keys can work when they terminate.

Values and output handles

There are two useful things a caller can keep: an answer it already received, or the information needed to ask for an answer later. Ajika represents them differently.

An Arc<T> owns a particular answer. An Output<T> identifies one result produced by a query and can acquire that result under the active inputs.

Keep a value while working with it

Suppose a query produces diagnostics for an editor response. The host may need to format or send them after Database::run finishes. Query calls return Arc<T>, so the host can keep that answer beyond the operation. Repeated callers within the operation can share the same allocation.

Query declarations normally return an owned T through Result<T>; the macro supplies the Arc for callers. A body that already has an Arc can use #[ajika::query(forward)] to return it directly.

That retained value doesn’t update when inputs change. It still reflects the inputs from when you requested it. This is useful for finishing a response or comparing old and new answers, but it is not a way to issue more queries against the old input view.

Keeping the Arc also keeps its memory alive. Database retention settings cannot reclaim values still owned by the application.

Keep a handle when acquisition should happen later

For later requests, a symbol index may need a way to retrieve names without keeping every name loaded. It can retain output handles and acquire the values when a request needs them.

A query can emit additional values with Db::output, then return handles to them. #[ajika::output] declares a set of output handles.

For example, one declaration pass may produce a name table and a source-location table. Completion needs names; navigation needs locations. Separate handles let those consumers acquire and depend on the relevant table.

output.value(db) asks for the value under that operation’s inputs. It may share, load, or compute the result, validating previous work as needed.

Two handles compare equal when they name the same producer and slot. That does not mean the payloads are unchanged. Reading a handle is the separate observation of its current payload.

Handles belong to the database that produced them. Like interned IDs, they should not be mixed across unrelated databases or treated as identifiers independent of the cache’s lifecycle.

Acquire through the consumer’s context

The symbol index can expose names through a view that keeps their handle:

extern crate ajika;
use std::sync::Arc;
use ajika::{Db, Output, Result};

struct Symbols {
    names: Output<Vec<String>>,
}

impl Symbols {
    fn names(&self, db: &Db<'_>) -> Result<Arc<Vec<String>>> {
        self.names.value(db)
    }
}

Each calling query passes its own context, so its read becomes its own dependency. A query that instead uses an Arc acquired before it started has not made that tracked read. Helpers and views should keep this distinction clear.

Reading a handle gets the whole value. Selecting one element from the returned vector still records a dependency on the entire vector. Split the stored result further only when callers benefit from that smaller boundary.

Separate loading and execution are different choices

Named outputs are produced together. Splitting names from locations lets readers load them separately and lets their equality affect consumers separately. It does not make either half of the producing function lazy.

Use separate query functions when computations should be independently requested. Use outputs when a shared computation produces several independently consumed answers. They can be combined: a query can return output handles or keys for more granular queries.

During production, use the local values being built. Newly emitted handles become readable when the producer completes, not halfway through its execution.

A handle doesn’t guarantee that the output still exists

Suppose a file’s analysis emits a body output for each function. After a function is deleted, a caller may still hold its body handle, but the producer no longer emits that slot. Reading the old handle then fails.

Ask for the producer’s current output handles or use a membership query to discover which functions remain. An optional output represents absence by omitting the slot. An output whose payload is Option<T> still exists and can itself return None.

Keeping a handle also does not pin a stored record against eviction. Ajika can recompute an evicted result when requested; the handle does not keep its data loaded.

Operations and application state

The host owns a long-lived Database; an operation is one bounded period of work on its installed inputs. This separates the lifetime of the application from the lifetime of the data needed to answer a request.

Database::run is the entrypoint. Its callback receives a Db that can call queries, read inputs, and share work with other calls in that operation.

Choose a useful unit of work

An editor request is often a natural operation. Several related features can also share one operation when they should use the same inputs and loaded values. An indexing pass may ask many queries in one operation.

Large operations improve sharing, but retain more touched data until they finish. Very small operations release memory earlier, but later operations may need to load the same values again. Start from the application’s actual unit of work, then measure acquisition and peak memory before changing that boundary.

An operation is also the checkpoint boundary. Successful query results are usable by other queries immediately; eligible changed work is saved after the callback finishes successfully. Query authors do not manage a separate commit for each call.

Operations and revisions are not the same thing

publish input view A
    run request 1
    run request 2
publish input view B
    run request 3

Requests 1 and 2 can use the same input revision. An operation does not itself advance the revision. Publishing changed input stamps or policies does; publishing equivalent observations does not.

The revision is Ajika’s bookkeeping. A document version or project generation can have different meaning in the application. The host remains responsible for deciding whether a result belongs to the generation it wants to expose.

Replacing inputs and running operations require exclusive access to the database owner. There is one installed input view at a time. Cloned Inputs can describe proposed edits or previously used snapshots, but queries run against the installed view.

Publish a view, then decide whether to accept its results

An editor may want to analyze proposed edits before accepting them, while keeping its last accepted inputs available. Inputs::edit prepares that candidate view without switching the database to it; replace_inputs installs it for queries. For ordinary updates, update_inputs prepares and publishes the edit batch in one call.

Once installed, that candidate remains the database’s input view even if the following operation fails. The host can restore its accepted view or install a new candidate before trying again.

Returning an error from the run callback abandons pending derived work before checkpointing. A late cancellation can still return an error after cache data was committed. Automatic maintenance failures are logged separately.

This is safe for reusable derived work because later requests validate it against their inputs. Publishing an editor response or accepting a project generation remains a separate application decision.

Parallel work stays inside the operation

Use Db::parallel_map to request independent work concurrently. Its callbacks share inputs and query results, and their reads belong to the calling query. Database::set_parallelism bounds its workers across nested batches.

The helper joins workers before returning. It does not create independent input revisions or detached tasks. Ordinary child query calls remain synchronous. Do not assume that an arbitrary work-stealing executor can replace this helper: its scheduling must keep a query’s owner able to finish while another call waits.

Failures and cancellation stop the work

Use successful result data for expected domain problems, such as diagnostics for broken source code. A query returning Err fails the operation; catching that error inside a parent does not permit a partial successful checkpoint.

For cancellation, obtain a fresh Cancellation token from the owner for each operation. Another thread can signal it. Query boundaries notice cancellation; long local work should call Db::check_cancelled too. Cancellation is cooperative, so external blocking work may still need to finish.

After work stops, the host can install the next inputs and start another operation. The normal lifecycle stays synchronous even when some computation uses workers.

Where results live

Reusing an answer does not require keeping its decoded Rust value in memory. Ajika can remember that an answer is valid, load its value when requested, and release that value when the operation ends.

It helps to make two separate decisions: whether an answer is still correct, and whether retaining its data is worth the memory or storage cost.

First choose what a query produces

Consider parsing a file and extracting its signatures. The signatures are a compact answer worth reusing across requests; the parse tree may be large or awkward to serialize.

A default #[ajika::query] suits the signatures. Its result must support serialization and equality, so Ajika can retain it between operations and compare it after edits. The parse query can use #[ajika::query(transient)] instead. Its value doesn’t need to support serialization or equality, and Ajika only keeps it for the current operation.

Ajika can still retain the parse query’s dependency information. If the source has not changed, saved signatures can remain valid without rebuilding the tree. Requesting the tree itself in a later operation requires reconstruction.

If the source changes, checking the signatures requires rebuilding the tree and extracting them again. The old tree is unavailable for comparison, but the new signatures can still be compared with the saved ones. When signatures stay equal, queries that use them can keep their answers.

Then decide where observations allow work to survive

The same parse and signature queries can analyze saved files or unsaved editor buffers. Draft results should be reusable across editor requests while leaving the saved disk answers available. Marking the draft input as Session gives its derived results that restriction.

An input observation’s Lifetime sets this limit:

Restrictions flow through child queries. A result depending on one session input cannot overwrite the saved disk result, even if its other inputs are persistent. All outputs of one producer share that restriction.

The host can keep the saved view and create an Inputs::overlay for a draft. Only the edited text needs a session override; unchanged inputs remain shared. When the host saves that same text and publishes it with persistent lifetime, Ajika can promote results that are still in the session store without recomputing them. The unsaved-edits chapter shows this lifecycle.

Lifetime is an upper limit on retention; cache budgets can evict a result sooner. A transient query still releases its primary value at operation end.

The backend supplies the storage medium

Database::open keeps eligible results on disk. Database::memory keeps eligible values as Rust objects for the owner’s lifetime, subject to retention. Both expose the same query interface.

These choices combine as follows:

The word “persistent” in a query policy means eligible for backend retention. Choosing a memory database does not write files, and choosing a disk database does not make session inputs durable.

Durability answers a different question

Source text may change on every keystroke while compiler settings stay the same for hours. Durability lets the host describe that difference. With settings marked High and source text marked Low, a source edit doesn’t force Ajika to recheck queries that depend only on settings.

This reduces validation work; Lifetime and the backend still determine where results can survive. Use the default low class until the application has a useful distinction to make.

Retention controls cost, not correctness

Over time, an editor may request data for many files it no longer needs. Retention places age, count, and byte budgets on that retained work. set_session_retention controls the session store separately. These are cache budgets: eviction can require recomputation later, but does not make answers stale. The age limit is not a delay before detecting input changes.

clear_session lets the host drop derived session results as part of closing its drafts. The editor’s buffers and input views remain owned by the host, which can close or replace them as part of the same cleanup.

Those input views and any query results the host keeps in Arcs have their own lifetimes. They stay alive for as long as the host keeps them, even after an operation ends or a retention pass trims the query cache. The cost and reuse guide explains cache statistics and their relationship to process memory.

Owning the cache

A persistent Ajika database is a way to reuse derived work across process lifetimes. The application must still be able to acquire its inputs and compute its answers if the cache disappears.

This gives the host a simpler responsibility than maintaining an authoritative database: decide which inputs are current, which stored formats are compatible, and how much space reusable work may occupy.

Reopen the cache before supplying inputs

When the application starts, Database::open restores saved query work and interner state. That gives Ajika previous answers to consider, but files may have changed while the application was closed. The host supplies current inputs so Ajika can tell which saved answers still apply.

Those inputs can be supplied up front or acquired through initializers when first read. Ajika checks saved answers as queries are requested, so startup doesn’t require loading every input or validating every cached query.

If your inputs refer to files by interned IDs, those IDs have to match the ones in saved query keys. The restored interner keeps that mapping, so you can use the handle from Database::interner() while building the inputs. Each file will then keep its earlier ID even if you discover the files in a different order.

Only one owner can open a storage directory at a time. Workers share the owner’s operation; they do not each open the directory as a separate database.

Change the epoch when the meaning of saved work changes

Ajika can detect changed tracked inputs. It cannot infer that editing a query’s Rust implementation should invalidate yesterday’s answer.

For example, changing a tokenizer’s rules may produce different tokens for the same source text, even though the input and result types stay the same. Reusing old tokens would hide that change from the queries that consume them.

The application’s epoch supplied to open marks compatible saved work. Change it when query semantics or encoded keys and values become incompatible. A changed epoch rebuilds the cache rather than migrating old results.

Use input changes for ordinary edits. Resetting the entire database for every file update loses the reuse that query dependencies provide.

Distinguish eviction, invalidation, and a full reset

Eviction forgets selected reusable results to stay within budgets. A later request can compute them again. It is not the same as deciding that a changed input makes an answer outdated.

Database::invalidate discards retained query work while preserving installed inputs and interner state. The next operation computes against those inputs without old query answers.

Live recovery preserves interned IDs even if it has to rebuild the disk cache. At startup, an incompatible epoch or unreadable identity table can start a new interner. Keep interned IDs and saved output handles within the originating cache’s lifecycle.

Schedule maintenance around application work

Successful operations already perform periodic or budget-triggered collection and bounded physical maintenance. A host can call collect at another idle boundary. There is no background timer advancing collection while the application does nothing.

Eviction can remove records while leaving their storage files partly in use. The remaining records must be moved before those files can be deleted, so physical disk usage may fall later than logical cache usage. Routine maintenance does this gradually; compact finishes reclamation and shrinks the index. It can take longer than routine maintenance because it completes that work in one call.

If disk usage stays high after eviction, statistics().stored_bytes shows how much logical data remains, while storage_statistics() reports physical disk usage. The cost and reuse guide explains how to read those measurements and how compression affects stored size.

See recovering a cache for how startup and later operations handle storage failures.

Follow one result

This part follows how Ajika serves queries using the same text analyzer as the tutorial. It reads a file’s text, splits it into words, and counts them:

word_count("notes.txt")
  └─ words("notes.txt")
       └─ File::text("notes.txt")

Suppose the text is hello world. Ajika must remember both the count, 2, and why that count is reusable. Keeping only 2 would not tell it what to do after the file changes. Keeping every decoded answer and every dependency in memory would make idle memory grow with the amount of analysis.

The runtime separates the temporary work of answering a request from the stored evidence needed to answer the next one.

Three different objects

While words("notes.txt") is executing, it has a frame: a temporary log of the reads made by this execution. Its frame records the file’s input stamp. The parent has a separate frame that records the words answer’s version.

When the function finishes, the evaluator builds a record: a completed description of this call’s dependencies and output versions. Payloads, such as the vector of words, are stored separately from that record.

During the operation, workers asking for the same query and key share a memo: the working place where they find the record, its validation status, and any loaded or newly computed values. A memo may begin empty, acquire an old record, validate it, and eventually hold a replacement answer.

These are different jobs, not three interchangeable caches.

One cold operation

The host enters Database::run. It creates an operation and a root Db. The root has no query of its own, so its reads are not saved as another dependency graph node.

Calling word_count enters the evaluator through the macro-generated provider. No answer exists, so the evaluator reserves this call and executes the function with its own frame. Calling words repeats that process for the child.

The child reads the captured text, returns a word vector, and receives an output version. Its record and value become available to other callers in this operation. The parent records that version, counts the words, and completes in the same way.

Nothing has to reach disk before the parent can use the child. The operation’s working state is the place where completed computation becomes immediately shared.

When the host callback finishes successfully, checkpointing saves eligible changed records and payloads together. The operation then releases its memos and decoded values. The backend keeps compact process-local validity and usage information; the detailed dependency records live in storage. Any Arc returned to the host remains alive for as long as the host retains it.

The next operation

With the same inputs, the backend can remember that this call was already verified at the current revision. A new operation retrieves the stored output metadata and loads the requested count. It doesn’t need to check words again.

After an input change, that shortcut may no longer apply. The evaluator follows the old dependencies to see whether their observed answers still match. This is the red-green check described in evaluation.

After a process restart, old process-local revision proofs are gone altogether. Records and payloads remain useful, but must be checked against the new owner’s captured inputs.

The implementation boundaries

application: acquire inputs, publish revisions, choose operations
    │
    ▼
declarations: typed wrappers and replay registrations
    │
    ▼
runtime: track reads, validate, compute, share values
    │
    ▼
backend: checkpoint records, retain session work, collect queries
    │
    ▼
ajika_storage: index and payload files, atomic publication, reclamation

The macros do not decide whether a query runs. The storage crate does not know what a dependency means. Those boundaries keep declaration ergonomics, incremental evaluation, and file management independently understandable.

Calls, addresses, and registration

A normal Rust call knows which function to invoke. A dependency saved yesterday must describe that call as data, so a new process can make it again.

For words("notes.txt"), that description contains two things:

• The query family’s stable name, such as example::words.
• The encoded key, "notes.txt".

Together they form an Address. Inputs use the same kind of address for a field family and key.

The same question can have a new answer

The storage ID is a 32-byte hash of that address. It identifies the question, not the answer’s contents. Changing the file replaces work at the same logical address instead of creating a new historical entry for every text version.

The ordinary return value uses the call’s ID. A named output derives another ID from the producer’s ID and its encoded slot. The name "locations" on two different producers therefore identifies two different outputs.

Ajika keeps the full address in saved dependencies as well as using hashes for lookup. If the child’s record is collected, the parent still has the key needed to recompute that child.

The address identifies the question; other identifiers describe what happened when answering it:

Output versions are fresh 16-byte random tokens when equality cannot be established. They are neither content hashes nor revision counters. Random tokens avoid accidentally repeating a forgotten version after a record has been collected.

From a declaration to a callable provider

Query is the evaluator’s interface to a computation. #[ajika::query] generates a provider type implementing that trait, and leaves a calling wrapper where the function was declared. The wrapper passes the provider and owned key to the evaluator; the provider’s execute method invokes the original computation.

The body declares Result<T>, while the wrapper returns Result<Arc<T>>. The provider wraps successful values in an Arc. With forward, the body already returns Result<Arc<T>>, and the provider passes that Arc through.

One key argument is used directly; zero or multiple key arguments become a tuple. Associated queries use the same machinery. Their computation stays in the owning impl so Self and helper calls keep normal Rust meaning.

For a saved address, the runtime has only a family name and encoded key, so it also needs a way to find this provider and decode its arguments. The macro emits a registration gathered by inventory when the program links. At database construction, Registry::linked collects the query implementations for that database’s services type, alongside the input declarations.

This means replay works before the application has directly called every query function. There is no “warm up registrations by executing queries” step. Handwritten Query implementations use #[ajika::register] for the same purpose.

Replay enters the normal evaluator

When validating word_count, Ajika checks whether words still has the version the count depended on. Comparing those versions does not require the word vector itself. Registry::replay resolves the saved family name, decodes the key, and enters the same evaluator used by a typed query call.

If the child’s dependencies still match, the evaluator can establish its current version without loading that vector. Changed dependencies may require executing words first. The evaluation chapter follows both paths.

If the current program has no registration for the saved query name, Ajika cannot check that old dependency. It reruns the parent instead, letting its function collect fresh reads. An error from a registered provider fails the operation.

Two different implementations cannot share a family name within the same registry. In-process TypeId checks detect that collision; TypeId is not part of the persistent address.

Changing query code or encoded types still needs an application epoch change when stored results are no longer compatible. Registration describes how to call code; it does not fingerprint the implementation.

Interning is separate from query registration

The registry connects query names to executable code. The interner instead gives application values, such as file paths, compact IDs. An interned FileId can appear inside many query keys, each of which identifies a different question about that file.

The interner is shared for the database’s lifetime and restored before queries intern new objects. Each declaration has its own numbering space. Looking up an ID’s fields does not add a query dependency because the assignment is immutable.

See Interner for assignment ownership and the user chapter for its reset boundary. The interner must remain consistent with every saved key that contains its IDs.

Input snapshots and revisions

Queries read a captured input view. The host builds another view when files, configuration, or editor buffers change, then installs it between operations.

The useful distinction is between replacing an observation and capturing an observation for the first time. Replacement belongs to the host. Lazy capture may happen while queries run.

Sharing a view

Inputs contains typed field tables. A field table maps keys to captured values and their stamps, durability, and lifetime. For a file declaration, text and language are separate tables because they are separately tracked fields.

Cloning Inputs shares those tables. If neither view has read notes.txt yet, the first read through either can capture its text for both. Once captured, that entry supplies the same observation to later readers.

Replacing that text must leave the old view usable. Inputs::edit creates a private table map for the candidate, then copies the text field’s table on its first write. Subsequent writes in the batch reuse that private table; other fields remain shared. Table copies clone each value. For #[input(arc)] field: T, the generated Input::Value is Arc<T>, so copying the table or reading the field clones the pointer while keeping the payload in place.

The edit returns a candidate only after its closure succeeds. Database::update_inputs installs that result; a failed batch leaves the installed view alone.

Sparse editor views

For a large source inventory, copying the text table to edit one file can still be costly. An overlay adds small local tables over a base Inputs instead. A draft can override just notes.txt; reads of other files fall through to the saved base.

If the host forgets notes.txt in the overlay to reacquire it, deleting the local entry would expose the base’s old text again. A forgotten entry therefore masks that key in the base until the overlay acquires a replacement. The table keeps this distinction between an unseen key and a forgotten observation.

First acquisition also needs a destination. If the overlay has no local table for a field, it captures unseen keys through the base, so the host sharing that base sees discovered files too. Once the field has a local table, new captures for it stay local. Inputs::read chooses where to acquire; lookup in the same module checks overrides and forgotten entries before falling through.

Building each successive draft over the saved base keeps lookup from walking a growing history of edits. Overlay composition retains that chain of views rather than flattening it.

First acquisition

When notes.txt has no captured text, Ajika calls the field’s initializer without holding the table lock. It computes the stamp and policy from the result, then installs the observation unless another reader already installed one. Two workers can therefore acquire the file concurrently, but both receive the winning observation.

This keeps file I/O out of critical sections while giving the view one answer per field/key. Different files can still be acquired at different times. If the application needs them to describe one filesystem revision, its acquisition layer must verify that condition or reject the operation.

To validate a query that read this text, Ajika only needs the captured stamp and policies. Inputs::record, next to read, returns that metadata without cloning the input value. It triggers acquisition only if the field/key is still unseen.

Publishing a new revision

replace_inputs compares the old and new view’s stamps and policies. Shared tables can be skipped; sparse views can compare their differences over a shared base.

If all stamps and policies match, the revision stays the same. Otherwise the owner advances its process-local revision and records which durability classes may have been affected. Both the old and new policies matter when a field changes class.

For example, a query depending only on high-durability configuration can remain verified after low-durability source edits. A query that reads both configuration and text inherits the lower class. If configuration changes, the high-durability shortcut is invalidated too.

The new revision tells the evaluator that an earlier validity check may no longer be enough. Dependency validation decides which functions actually need to run; an equal child answer can still spare its parents from execution.

Policy changes also flow through equal answers

Imagine saving an editor buffer without changing its text. The stamp stays equal, but its lifetime changes from session to persistent. A parent can still reuse the same answer while refreshing its inherited policy.

Validation therefore gathers the current durability and lifetime even when all observed stamps and versions match. Updating that policy may require a checkpoint without executing the function. Treating “equal answer” as “no metadata changed” would leave saved work stranded in the session store.

Remembering what a query read

Ajika does not ask query authors to list their dependencies. It records observations as they happen. This matters because a query may choose different children for different input values.

The observation must belong to the calling query, even if another worker computed the value or storage supplied it. Sharing a value does not mean sharing the caller’s dependency log.

Direct reads preserve useful boundaries

In the word-count example:

words frame:       File::text("notes.txt") at stamp S1
word_count frame:  words("notes.txt") at version W1

The parent does not inherit the child’s text dependency. If it did, a whitespace edit would invalidate the parent directly and there would be no chance for equal words to stop that propagation.

Db carries both the shared operation and the active Frame. When the evaluator enters a child function, it creates a child frame. After the child is ready, the read is recorded on the caller’s frame using the child’s output version.

An input getter records the field address and stamp instead. An Output::value read records the producer address, selected slot, and that slot’s version. These cases become the two variants of Dependency.

The frame also accumulates the least durability and lifetime of its observations. Those policies flow transitively through children without copying their entire dependency lists into the parent.

The order of reads matters

Suppose a query first reads which file is selected, then reads that file:

old execution: selection → file A
new execution: selection → file B

If selection changes, validation must stop there and let the function run again. Trying to validate every old dependency first could access file A after it has been deleted, even though the new computation would never read it.

Reads::depend keeps the first-read order and deduplicates repeated observations. Validation examines that sequence and stops at the first mismatch.

When a new execution completes, its reads replace the old list. Even if A and B contain equal text and the final answer stays equal, future validation must follow B. Output equality preserves a version, not the old explanation of that version.

Repeated acquisition within a function

A function may ask for the same child or output many times. Its frame remembers already acquired values, so later reads can return the value without repeating acquisition or adding the same dependency again.

Now suppose two different queries call words("notes.txt"). The operation’s memo lets both receive the same decoded vector, but each caller’s frame must record its own dependency on W1. Reusing the first caller’s log would leave the second query with no record of that read.

At the host root, the frame has no query owner. Host reads do not create another stored producer, though queries called by the host still track their own reads.

Parallel branches are parts of their caller

parallel_map gives each callback its own frame with the same query owner. Workers can collect reads independently without racing to append a single log. After joining, branch merging combines their logs in input order, deduplicating shared observations.

The callbacks are not separate persistent queries. If each item needs its own reuse boundary, call a declared query from the callback.

At the end of execution, Frame::finish transfers the collected reads and pending outputs to the evaluator. It does not write storage. The evaluator still has to compare answers and publish the completed record.

Deciding whether to execute

A stored answer can be correct without being loaded, and a loaded old answer can be wrong for new inputs. The evaluator treats validity and residency separately.

A direct call to word_count needs the count itself. To validate another query that used the count, Ajika may only need its version. Both enter Db::ensure; the internal want_value argument tells the evaluator whether to acquire the primary value as well as establish its current version.

Start with the cheapest available proof

The output metadata is grouped in a RecordHeader: slots, versions, stored payload lengths, and inherited durability and lifetime. It leaves out the dependency list and the values themselves. Once a call is known to be valid, this header is enough to select and load one of its outputs.

After finding the operation’s memo for the query/key, the evaluator checks whether it already has what the caller needs at the current revision. This is the shared working-set fast path.

Otherwise it acquires a claim for this call and checks again: another worker may have finished while this one waited. It then obtains the saved record or header and the backend’s verified revision: the input revision at which that record was last proved valid in this process.

There are three ways to establish validity without executing:

1. The call was verified at this input revision.
2. Its inherited durability proves that no relevant class of input changed since its previous verification.
3. Rechecking its saved dependencies finds the same observations.

The first two only need the output header. The third needs the full dependency record. After reopening, the verified revision is zero, so a saved answer cannot take a process-local shortcut before checking fresh inputs.

RecordHeader comes first in the record’s encoding so the runtime can decode it without allocating the dependency vector. Storage may still fetch the whole encoded row; the saving here is in constructing Rust objects from those bytes.

Red-green validation

“Red-green” describes the distinction between an answer that may have changed and one proved unchanged. Ajika expresses that proof through output versions.

For each saved dependency, validate does the following:

• For an input, acquire its current stamp if needed and compare it with the saved stamp.
• For a child output, replay the child to establish its current versions, then compare the selected slot’s version with the saved version.

It stops at the first mismatch. A missing registration or a removed output also prevents the old proof from succeeding. Errors from acquisition or a registered query fail the operation; they are not converted into evidence that the data changed.

Consider the extra-space edit:

File::text:  stamp S1 → S2
words:       executes, ["hello", "world"] == previous value → keeps W1
word_count:  observed W1, still sees W1 → reuses C1

The parent discovers that it is still valid by making its child current. It does not need to load the child’s vector merely to compare W1 with W1. The child may have loaded its own old vector to establish equality during recomputation.

There is no eager walk marking every downstream node invalid when an input changes. Work starts from requested roots and follows the evidence needed for those roots.

Obtain a value only when needed

Once validity is established, a validation-only request can return the memo’s header. A normal call still needs an Arc<Q::Value>.

For a persistent result, the evaluator loads the selected payload through its codec and runs the query’s optional structural validate hook. For a transient result, no retained primary payload exists, so the function runs to reconstruct it.

For example, suppose words were declared transient. With unchanged text, a later request for its vector still has to run the function. The dependency check has already established that it will read the same observations, so determinism lets the evaluator preserve W1 without an old vector to compare against. If the text changed, that proof would no longer apply; the evaluator would need the comparison rules below.

Execute and compare

compute runs the provider in a fresh frame. After success, the evaluator considers the primary result and every named output separately:

A fresh version therefore means that equality was not established. It does not always mean the values were observed to differ.

The memo is marked dirty when its completed record or payloads need checkpointing. New reads or policies can require this even when every output version is preserved. Conversely, reconstructing a valid transient value with identical metadata doesn’t necessarily cause a write.

Only after all of this succeeds does the evaluator install the record, values, and verified revision together in the memo. Waiting callers can then use them. Saving those results beyond the operation is a separate checkpoint step.

Publishing independent outputs

The tutorial’s output example extends the text analyzer to produce words and their byte locations in one analyze(file) call. Its return value is a set of output handles naming words and locations; the handles do not contain those payloads.

This separates the cost of producing data from the decision to load or depend on each part.

Emission remains local until completion

Db::output derives a slot address from the running producer and the supplied name or name/key pair. It puts the typed value in that producer’s frame and returns an Output<T> handle.

The #[ajika::output] constructor is a convenience wrapper around these calls. It does not register another kind of query or write a storage row itself.

The function can pass handles back in its return value, but must finish before other queries can read its new outputs. Reading its own pending handle would require that unfinished producer to complete, so it is rejected.

If the function fails, its frame and pending outputs are discarded. If it succeeds, the evaluator compares all outputs and publishes their versions, values, and replacement dependency record together. Waiting callers can then read the completed set.

One record, several versions

For the initial text, the record might look like:

analyze("notes.txt")
  reads: File::text("notes.txt") at S1
  outputs:
    <return>    version H1   (the set of output handles)
    words       version W1
    locations   version L1

After inserting a space, the new input stamp is S2. The handles still contain the same addresses, words are equal, and locations differ:

reads:   S1 → S2
versions: H1, W1, L2

A consumer of words can stay valid. A consumer of locations must recheck its answer. Both depend on versions in the same producer’s output metadata.

All these outputs share the execution’s dependency list, durability, and lifetime. Independent versions do not imply independent execution or retention policies.

Reading a retained handle

A retained words handle already identifies analyze(file) and its words slot. Reading it does not require loading the producer’s returned set of handles just to find that same slot again.

output_value uses replay to establish the producer’s current versions without requesting its primary return value, just as dependency validation does. It then selects the slot and loads that payload if needed.

The read records the chosen slot’s version on the calling query’s frame. The operation and frame can then share that acquired Arc with repeated callers.

If the producer needs to execute, it still executes in full. If the selected slot is gone afterward, a direct handle read fails. In dependency validation, that same absence instead requires the parent to execute again, so it can discover which outputs the new inputs call for.

The returned set records membership

For dynamically keyed outputs, the returned handles also describe which slots exist. A code analyzer might emit one output per function body. If an edit deletes function B, a producer that previously returned body handles A and B should now return only A. Consumers observe that membership change before requesting individual bodies.

Keeping B’s old handle does not establish that B still exists. Handles name slots under the active input view; they are not historical snapshots.

Omitting a formerly emitted slot removes it from the new output metadata. Checkpointing then removes its durable location or masks the durable slot with the session producer’s new metadata. Old file bytes may remain until storage maintenance, but query readers can no longer observe them through that slot.

When adding a new output-producing abstraction, preserve these two boundaries: the producer owns publication, and every consumer records its own selected read. An accessor that hides either boundary can accidentally make the whole returned set a dependency or lose a dependency altogether.

Coordinating workers

Two workers may reach words("notes.txt") at the same time. They should share both expensive computation and less visible work such as validating dependencies or decoding a stored answer.

The memo holds the answer. A separate claim decides who may do unfinished work on that call.

Reserve work, not a mutex

Coordination::claim reserves a call ID. If another worker owns it, the caller waits. When the owner releases the claim, the waiter checks the memo again and can use the completed answer.

The claim guard does not keep the coordinator’s mutex locked. Provider code, recursive queries, storage reads, and value comparisons run outside it. The memo’s state mutex likewise protects short reads and publication, not the whole computation. Unrelated calls can make progress concurrently.

A claim covers validation and loading as well as execution. A warm database can therefore avoid duplicate decoding even when no query function needs to run.

Track active calls to detect cycles

Thread-local recursion checks are insufficient when one worker owns A and another owns B:

worker 1: owns A, requests B
worker 2: owns B, requests A

Both would wait forever unless the runtime sees the relationship across workers. The coordinator keeps an active parent-to-child call graph. Before adding an edge A → B, it checks whether B already reaches A.

This graph includes running nested calls and calls waiting for claims. It is different from the saved dependency graph: it describes unfinished work, not observations used by completed answers.

Parallel branches can produce the same edge more than once. Edges are reference counted so one branch completing does not erase another branch’s relationship. When the graph reveals a cycle, QueryCycle reports the calls in order, including the repeated first call that closes the chain, such as A → B → A.

Failure must wake waiters

The coordinator retains the operation’s first failure and notifies waiting workers. They receive a shared error that preserves the original error chain and application context.

If provider code panics before publishing an answer, the waiting worker still needs to wake up. The claim guard releases ownership and its call edge during unwinding, and records an operation failure. Waiters then receive an error instead of retrying the abandoned computation. Cancellation similarly wakes waiters and is checked at query boundaries.

Cancellation is cooperative. A worker in a long loop or blocking external call still needs to return or check the token. The coordinator does not suspend or kill application threads.

Bounded parallel batches

Db::parallel_map uses scoped threads. The caller participates, while extra threads consume permits from the operation’s shared worker budget. A nested batch with no spare permits runs locally instead of creating more threads.

Workers claim successive item indices for load balancing. Each saves its result and frame at the input index; joining then preserves result and dependency order. The helper also carries the caller’s tracing dispatcher into its workers.

The scoped-thread model keeps each query’s owner able to finish its call stack. In a work-stealing executor, a worker waiting for a batch might start an unrelated task. If that task asks for a query owned by the suspended stack beneath it, it can block the very worker needed to finish that query. This can happen even when the active call graph has no cycle, so the coordinator can’t detect this scheduler deadlock. Integrating another executor requires preserving progress for query owners too.

Checkpointing an operation

During evaluation, completed answers are visible to other queries in the operation. They are not yet part of the retained backend state. This lets a parent use a new child immediately while still allowing the host to reject the whole operation.

Database::run starts checkpointing only after the callback returns successfully, scoped workers have finished, and cancellation has been checked.

Decide what can survive

Backend::checkpoint gathers memos marked dirty: they have record or payload changes for checkpointing to handle. It prepares replacements for retained work. The query’s inherited lifetime determines where Ajika can keep the record and its values:

operation lifetime  → no retained replacement
session lifetime    → bounded in-memory override
persistent lifetime → durable backend

The destination applies to the dependency record as well as eligible values. A transient primary result has no retained payload, but its dependency record can still survive in the session store or durable backend. Its serializable named outputs can survive there too.

If nothing is dirty, there is no checkpoint work. In particular, allocating an interned ID by itself does not force a durable transaction; pending assignments are included when query work is durably checkpointed.

Reuse durable payloads where versions match

For a disk database, the durable baseline is the record and payloads already saved on disk. Checkpointing compares the completed output metadata with that baseline. An output that kept its version and already has stored bytes can retain that location. A removed slot loses its location. New or changed persistent values need encoding.

In the whitespace edit, words kept W1, so its saved vector can stay where it is. Its record must still change: the function now depends on text stamp S2 instead of S1. Leaving S1 in the record would make a later dependency check rediscover the same change and recompute words again. No payload bytes need rewriting for this change.

Many outputs may change in one operation. PendingValue::write_all prepares bounded batches so checkpointing does not keep an encoded copy of every changed value at once. Workers serialize and compress a batch in parallel; the caller passes its buffers to the storage writer and releases them before the next batch. Limits cover raw size and value count. One oversized value runs alone and still requires a complete buffer.

The writer publishes records, output locations, and new interner assignments in one storage commit. Opening the writer at this stage keeps query execution outside the durable transaction.

Session work overlays the saved baseline

A session record contains the producer’s complete output metadata and dependency list. Its changed persistent-policy payloads remain typed values in memory. Equal durable payloads can be read from their saved locations instead of being duplicated in the session store.

Consider the body outputs A and B from the output chapter. If an unsaved edit deletes B, the session record must list only A. Falling through to the saved output metadata would incorrectly make B visible again. A surviving slot such as A can still use its saved payload when the versions match.

An operation containing only session changes does not open a durable writer. If the host later saves that exact draft, it publishes the same input content with persistent lifetime. Validation can reuse the session answer, and checkpointing promotes its retained data without rerunning the query.

The typed memory backend uses the same logical decisions, but keeps eligible Rust values directly rather than serializing them.

Publish bookkeeping after storage succeeds

Only after a successful checkpoint does the backend replace retained session work and update its usage inventory. The owner then remembers verified revisions for accepted records. Otherwise, a later operation could skip validation based on a replacement that existed only in the failed operation’s working memos.

The working memos leave with the operation on both success and failure. Retained host Arc values are outside that cleanup.

An aborted write leaves the previous checkpoint intact. After an uncertain commit, either checkpoint could be visible. Ajika resolves it before invoking the next uncancelled run’s callback. See storage recovery for how it decides what to keep.

Cache commit and host acceptance differ

There is a cancellation check after commit, so run can fail even though a valid cache checkpoint was published. Automatic maintenance failures are logged and can schedule recovery for the next run.

The host should not infer acceptance of a project generation or editor response from the mere existence of those files. Stored results retain their actual input observations, so they can be validated later, regardless of whether the host used the response.

After the working set is released, routine retention and physical maintenance can run. That ordering keeps maintenance from carrying the operation’s decoded query values into an otherwise idle period.

Storage and reclamation

The runtime understands queries, versions, and dependencies. ajika_storage understands opaque record bytes and value bytes. It provides a committed view of those bytes without retaining all decoded query state.

The division lets Ajika use a database for atomic indexing while putting large payloads in a layout suited to replacing whole query outputs.

What lives where

A disk cache has three main parts:

owner.lock    one owner for this cache directory
index.redb    records, interner state, value locations, pack accounting
packs/       larger payloads and their headers

A pack is a file holding many encoded values. Each value occupies a storage frame: its payload plus a header containing its ID, lengths, compression format, and integrity information. These are file entries; the query Frame described earlier is the runtime’s temporary log of reads and outputs.

The redb index supplies atomic publication and lookup. Values up to 1 KiB stay inline; larger values get index locations pointing into packs. Storage can read one frame without decoding its neighbors. It checks the frame and decompresses the payload, then the runtime’s serialization codec reconstructs an owned Rust value from those bytes. The temporary byte buffer is released before returning the query result.

Packs target 64 MiB, allowing a larger individual frame to exceed that size. Replacing a value changes its index location rather than overwriting its old frame. The old space becomes reclaimable after the replacement commits.

The storage database owns the index and pack readers. FrameHeader defines the information needed to read or relocate one stored value.

Publish files before references to them

A committed index location must point to a fully written payload. A write starts from the previous committed index, appends prepared payloads to private packs, and builds new index entries. At commit, it finishes and synchronizes the packs before publishing their locations in redb.

Dropping an uncommitted write preserves the previous checkpoint. If commit itself fails with an uncertain outcome, the owner stops serving the cache and leaves cleanup to reopening, which can inspect the committed catalog.

Read views pin their backing files. The storage API requires their scopes to end before writing; returned byte buffers are owned and do not pin a read view. Ajika uses short reads while evaluating, then writes at the operation boundary.

The transaction implementation contains the commit sequence that makes pack locations visible. On Unix, it synchronizes directory entries as well as files.

Compression belongs to individual values

The runtime first serializes a Rust value into bytes. Compression reduces those bytes for storage; reading reverses compression before Rust decoding. The evaluator decides equality using Rust values; these transformations determine how those values are stored.

LZ4 is the default. The alternatives are raw bytes and level-three Zstandard. Small inline values stay raw, and larger values also stay raw if compression saves less than one eighth. Each frame identifies its compression format, so changing policy affects new writes without invalidating old frames.

Maintenance can move already compressed frames without reconstructing their Rust values or comparing them again.

Logical eviction and physical reclamation

The runtime first decides which query records to forget, based on last use, entry count, and logical byte budgets. It removes each selected record, its outputs, and its process-local validity proof together.

Reads update last-use ages in memory. Collection batches those age updates into storage writes, avoiding a metadata write for every read-only operation.

This is not reachability collection. A retained parent may still name an evicted child; its full replay address allows reconstruction, at the cost of lost reuse. Removing source entities does not immediately discover every now-unreachable query. Budgets bound retained work instead.

Removing an index location does not necessarily free a whole pack: other live values may share the file. Routine maintenance copies live frames out of selected sparse packs, commits their new locations and scan position, then removes unreferenced files. A restart resumes from committed progress. Small packs are consolidated in groups to avoid leaving a file for every small checkpoint.

Normal passes have byte and frame-count budgets and finish an individual frame once started. That bounds work, not exact latency. compact() finishes reclamation and shrinks the index without the normal pass budget.

Physical headroom is separate from query retention. At storage capacity, a Capacity error refuses new payloads while allowing reads, removals, and maintenance. Reclamation cannot make a live dataset smaller; the host may need to evict work or change its storage allocation.

What remains in memory

Between operations, the disk backend keeps compact usage entries: call ID, last-use time, logical size, and verified revision. It does not retain every query’s encoded key, dependency vector, output metadata, and mutex.

On reopening, Backend::open rebuilds that inventory from record metadata without decoding query payloads. Verified revisions begin at zero. Full records and headers are acquired into operation memos when needed.

The storage layer also keeps a small cache of open pack files to avoid reopening them for every read, and redb has its own page cache. These remain useful between operations, even though the runtime has released its decoded query values.

Inputs, interner state, bounded session values, and answers kept by the host add to that memory use. Releasing query working state does not guarantee an allocator immediately returns all freed pages to the OS.

Interned keys outlive many operations. If temporary discovery data was allocated beside them, releasing that data can leave allocator pages occupied by a few long-lived keys. Collection relocates the keys into fresh allocations after releasing query working values, allowing those old pages to become unused.

Identity assignments have a wider lifetime than query retention: IDs cannot be recycled while saved keys or live callers may contain them. Ordinary collection keeps them. A reopen-time size safeguard can reset interner state and the query cache together before handing either out.

Rebuildable, with explicit boundaries

Recovery holds owner.lock while reopening or replacing the index. Storage prepares a replacement catalog at a temporary path and renames it to index.redb. On Unix it syncs the directory before removing packs the old catalog could still reference if a crash rolls back the rename.

Ajika checks recovered IDs against the live interner before reusing queries. A rebuild saves the live IDs with an empty query catalog.

The format is for application-owned, throwaway cache data, not hostile input or authoritative user documents. Recovery aims to retain a coherent checkpoint or rebuild, rather than salvage every possible cached answer.

For lower-level API examples and exact maintenance thresholds, see the storage crate guide.