Separating outputs

A text analyzer may need both words and their byte ranges. Those are produced by the same scan, but they do not change together. Inserting a space preserves the words and moves some ranges.

Returning both in one large value makes every consumer load both and depend on their combined equality. Named outputs let one computation publish separately readable answers.

Produce a set of output handles

The outputs example extends our analyzer with a set of output handles:

extern crate ajika;
extern crate anyhow;
use std::time::Duration;

use ajika::{Database, Db, Inputs, Retention};

#[ajika::input(key = String)]
struct File {
    #[input(arc)]
    text: String,
}

#[ajika::output]
struct FileOutputs {
    words: Vec<String>,
    locations: Vec<(usize, usize)>,
}

#[ajika::query]
fn analyze(db: &Db<'_>, path: String) -> ajika::Result<FileOutputs> {
    let text = File::at(path).text(db)?;
    let mut words = Vec::new();
    let mut locations = Vec::new();
    let mut cursor = 0;
    for word in text.split_whitespace() {
        let start = cursor + text[cursor..].find(word).expect("word comes from text");
        let end = start + word.len();
        words.push(word.to_owned());
        locations.push((start, end));
        cursor = end;
    }
    FileOutputs::new(db, words, locations)
}

#[ajika::query]
fn word_count(db: &Db<'_>, path: String) -> ajika::Result<usize> {
    let outputs = analyze(db, path)?;
    let words = outputs.words.value(db)?;
    Ok(words.len())
}

fn main() -> anyhow::Result<()> {
    let mut database = Database::memory(
        Inputs::default(),
        (),
        Retention {
            max_entries: 10_000,
            max_bytes: 256 * 1024 * 1024,
            max_idle: Duration::from_secs(24 * 60 * 60),
        },
    );
    let file = File::at("notes.txt".into());
    database.update_inputs(|update| file.set_text(update, "hello world".to_owned()))?;
    database.run(|db| word_count(db, "notes.txt".into()))?;
    let locations = database.run(|db| Ok(analyze(db, "notes.txt".into())?.locations.clone()))?;

    database.update_inputs(|update| file.set_text(update, "hello  world".to_owned()))?;
    assert_eq!(*database.run(|db| word_count(db, "notes.txt".into()))?, 2);
    assert_eq!(database.statistics().executed, 1); // analyze ran; word_count stayed valid.
    assert_eq!(
        *database.run(|db| locations.value(db))?,
        vec![(0, 5), (7, 12)]
    );
    Ok(())
}

The macro turns words: Vec<String> into a public Output<Vec<String>> field. FileOutputs::new accepts the actual values, emits them, and returns the handles. Its arguments can also be existing Arcs.

The returned output handles contain addresses, not the words or locations. Equality compares those addresses. When the same query/key emits the same slots again, the returned set can stay equal even if one payload changes.

Read the fact you need

extern crate ajika;
extern crate anyhow;
use std::time::Duration;

use ajika::{Database, Db, Inputs, Retention};

#[ajika::input(key = String)]
struct File {
    #[input(arc)]
    text: String,
}

#[ajika::output]
struct FileOutputs {
    words: Vec<String>,
    locations: Vec<(usize, usize)>,
}

#[ajika::query]
fn analyze(db: &Db<'_>, path: String) -> ajika::Result<FileOutputs> {
    let text = File::at(path).text(db)?;
    let mut words = Vec::new();
    let mut locations = Vec::new();
    let mut cursor = 0;
    for word in text.split_whitespace() {
        let start = cursor + text[cursor..].find(word).expect("word comes from text");
        let end = start + word.len();
        words.push(word.to_owned());
        locations.push((start, end));
        cursor = end;
    }
    FileOutputs::new(db, words, locations)
}

#[ajika::query]
fn word_count(db: &Db<'_>, path: String) -> ajika::Result<usize> {
    let outputs = analyze(db, path)?;
    let words = outputs.words.value(db)?;
    Ok(words.len())
}

fn main() -> anyhow::Result<()> {
    let mut database = Database::memory(
        Inputs::default(),
        (),
        Retention {
            max_entries: 10_000,
            max_bytes: 256 * 1024 * 1024,
            max_idle: Duration::from_secs(24 * 60 * 60),
        },
    );
    let file = File::at("notes.txt".into());
    database.update_inputs(|update| file.set_text(update, "hello world".to_owned()))?;
    database.run(|db| word_count(db, "notes.txt".into()))?;
    let locations = database.run(|db| Ok(analyze(db, "notes.txt".into())?.locations.clone()))?;

    database.update_inputs(|update| file.set_text(update, "hello  world".to_owned()))?;
    assert_eq!(*database.run(|db| word_count(db, "notes.txt".into()))?, 2);
    assert_eq!(database.statistics().executed, 1); // analyze ran; word_count stayed valid.
    assert_eq!(
        *database.run(|db| locations.value(db))?,
        vec![(0, 5), (7, 12)]
    );
    Ok(())
}

This records a dependency on the set returned by analyze, then on the words output read through its handle. It does not read or depend on the locations payload.

Use output.value_ref(db) to borrow an output for the lifetime of the query context. It records the same dependency as output.value(db), which gives you an Arc you can keep after the operation.

Try the whitespace edit:

extern crate ajika;
extern crate anyhow;
use std::time::Duration;

use ajika::{Database, Db, Inputs, Retention};

#[ajika::input(key = String)]
struct File {
    #[input(arc)]
    text: String,
}

#[ajika::output]
struct FileOutputs {
    words: Vec<String>,
    locations: Vec<(usize, usize)>,
}

#[ajika::query]
fn analyze(db: &Db<'_>, path: String) -> ajika::Result<FileOutputs> {
    let text = File::at(path).text(db)?;
    let mut words = Vec::new();
    let mut locations = Vec::new();
    let mut cursor = 0;
    for word in text.split_whitespace() {
        let start = cursor + text[cursor..].find(word).expect("word comes from text");
        let end = start + word.len();
        words.push(word.to_owned());
        locations.push((start, end));
        cursor = end;
    }
    FileOutputs::new(db, words, locations)
}

#[ajika::query]
fn word_count(db: &Db<'_>, path: String) -> ajika::Result<usize> {
    let outputs = analyze(db, path)?;
    let words = outputs.words.value(db)?;
    Ok(words.len())
}

fn main() -> anyhow::Result<()> {
    let mut database = Database::memory(
        Inputs::default(),
        (),
        Retention {
            max_entries: 10_000,
            max_bytes: 256 * 1024 * 1024,
            max_idle: Duration::from_secs(24 * 60 * 60),
        },
    );
    let file = File::at("notes.txt".into());
    database.update_inputs(|update| file.set_text(update, "hello world".to_owned()))?;
    database.run(|db| word_count(db, "notes.txt".into()))?;
    let locations = database.run(|db| Ok(analyze(db, "notes.txt".into())?.locations.clone()))?;

    database.update_inputs(|update| file.set_text(update, "hello  world".to_owned()))?;
    assert_eq!(*database.run(|db| word_count(db, "notes.txt".into()))?, 2);
    assert_eq!(database.statistics().executed, 1); // analyze ran; word_count stayed valid.
    assert_eq!(
        *database.run(|db| locations.value(db))?,
        vec![(0, 5), (7, 12)]
    );
    Ok(())
}

analyze runs because the text changed. Its words compare equal and keep their version; its locations differ and get a new version. The returned set also stays equal, so word_count needs no execution.

The previously retained locations handle reads the new locations. A handle names a slot under the active inputs. It is not an immutable snapshot of its old payload. A retained Arc<T>, by contrast, still contains the old value.

What splitting does and does not buy

Outputs provide separate loading and separate change propagation. They do not make the producer partially executable: when analyze must run, its whole body runs and produces both values. Use separate query functions when computations themselves should be requested independently.

All outputs of a producer share its dependency list and reuse policy. A session input read anywhere in that producer restricts the producer’s outputs together. The evaluator compares their payloads independently after execution.

While constructing outputs, keep using your local values. A producer cannot read its own output handle before it finishes. Its return value and emitted slots become available to waiting callers together.

Optional and dynamically keyed outputs

extern crate ajika;
#[ajika::output]
struct FileOutputs {
    words: Vec<String>,
    #[output(optional)]
    locations: Vec<(usize, usize)>,
}

The optional field becomes Option<Output<Vec<(usize, usize)>>>. Its constructor argument is Option<Arc<Vec<(usize, usize)>>>. Passing None emits no slot. By contrast, an ordinary field locations: Option<Vec<(usize, usize)>> always emits one slot whose payload may be None.

For a collection known only during execution, call db.output(("body", body_key), value_arc) directly. Names and keys identify slots; emission order does not. Each slot may be emitted once by that execution. Field names provide fixed slot names, and #[output(name = "locations")] can choose a name explicitly.

If a later execution omits a slot, a parent validating its old read is made uncertain and can run a different branch. Directly reading an old handle to that removed slot returns an error. Read a producer’s current output handles when membership can change.

The Output implementation and publication walkthrough explain how addresses and versions make these behaviors fit together.