decisions/learning

What the record is read for besides queues: agreement and accuracy per recommender version, the export of training examples, and the replay of a candidate model against recorded decisions. Each is a fold over occurrences; none writes, and none needs a store.

import decisions/learning

let assert Ok(examples) =
  list.try_map(recorded, learning.example(_, as_of: now, from: archive))
let scored = learning.replay(examples, candidate, same: fn(a, b) { a == b })
// scored.tally.agreed of scored.tally.judged

Target-independent: builds for Erlang and JavaScript.

Types

Where an example’s parts are resolved from at export time.

pub type Archive {
  Archive(
    render: fn(vocabulary.Manifest) -> Result(String, String),
    reads: fn(vocabulary.Explanation) -> List(Read),
    content: fn(vocabulary.Ref) -> Result(String, model.Failure),
  )
}

Constructors

  • Archive(
      render: fn(vocabulary.Manifest) -> Result(String, String),
      reads: fn(vocabulary.Explanation) -> List(Read),
      content: fn(vocabulary.Ref) -> Result(String, model.Failure),
    )

    Arguments

    render

    The renderer that built the model’s prompt from a context. It fails for a renderer version it does not have. It sees only the manifest: when the real prompt was captured as evidence (one provider call’s exact request), name that evidence among the context’s references and return its content here through the same resolver.

    reads

    Reads retained elsewhere, such as in the run an explanation points into. Return [] when the recommendation’s own reads are the whole story.

    content

    The content of a reference to immutable data or evidence: the same resolver a card uses.

How triggers fared before a recommendation, for coverage: of the records read, how many began with an expectation (decisions.expect or decisions.expect_until), how many of those a newer event superseded before anything was recommended, how many expired at their own deadline before anything was recommended (the expiry written: a record lapsed by the clock whose wake-up has not run yet still counts as expected), and how many are still expected with nothing recommended (work in flight, or a worker that died for good). A recommendation recorded after the deadline began the record, so it counts only as expected.

pub type Coverage {
  Coverage(
    expected: Int,
    superseded_before: Int,
    lapsed_before: Int,
    still_expected: Int,
  )
}

Constructors

  • Coverage(
      expected: Int,
      superseded_before: Int,
      lapsed_before: Int,
      still_expected: Int,
    )

One training example: an occurrence as it was known at a stated time, with every input resolved. It needs nothing else to be replayed.

pub type Example(output) {
  Example(
    as_of: timestamp.Timestamp,
    known: occurrence.Occurrence(output),
    prompt: String,
    context: List(#(vocabulary.Ref, String)),
    reads: List(#(Read, String)),
    expansions: List(#(String, String)),
  )
}

Constructors

  • Example(
      as_of: timestamp.Timestamp,
      known: occurrence.Occurrence(output),
      prompt: String,
      context: List(#(vocabulary.Ref, String)),
      reads: List(#(Read, String)),
      expansions: List(#(String, String)),
    )

    Arguments

    known

    The occurrence folded from only the entries up to as_of.

    expansions

    What validators opened beyond the card, as #(read, content).

Why an example could not be exported.

pub type ExportError {
  NotYetRecommended
  RenderFailed(reason: String)
  MissingContent(ref: vocabulary.Ref, failure: model.Failure)
}

Constructors

  • NotYetRecommended

    The occurrence had not been recommended at the stated time.

  • RenderFailed(reason: String)

    The archive has no renderer for the context.

  • MissingContent(ref: vocabulary.Ref, failure: model.Failure)

    The archive cannot resolve a reference the record names.

A read a model made, with the evidence it returned.

pub type Read {
  Read(name: String, argument: String, result: vocabulary.Ref)
}

Constructors

What a candidate model would have scored on recorded occurrences.

pub type Replay {
  Replay(
    tally: Tally,
    unanswered: List(#(String, model.Failure)),
  )
}

Constructors

  • Replay(tally: Tally, unanswered: List(#(String, model.Failure)))

    Arguments

    unanswered

    Occurrences the candidate could not answer, as #(occurrence, failure): a read the record does not hold, for one.

Agreement is agreed of judged, from verdicts and reviews. Accuracy is right of assessed, from assessments.

pub type Tally {
  Tally(
    agreed: Int,
    judged: Int,
    rejected: Int,
    right: Int,
    assessed: Int,
  )
}

Constructors

  • Tally(
      agreed: Int,
      judged: Int,
      rejected: Int,
      right: Int,
      assessed: Int,
    )

    Arguments

    rejected

    Occurrences people rejected outright: nothing was let through, so they are not judged, and counted here instead.

Values

pub fn answer_under(
  example: Example(o),
  policy candidate: policy.Policy,
  point point: decisions.Point(o),
) -> policy.Outcome

What policy would have said about a recorded example when it was recommended: the point’s facts of the recommended output, the recommender’s signals and what consulted points returned, evaluated under the candidate policy with no time waited. Use it beside replay to see, before publishing a version, how many recorded occurrences it would have let proceed and how many of those people corrected or rejected.

pub fn candidate_under(
  example: Example(o),
  candidate candidate: model.Model(o),
  policy policy_version: policy.Policy,
  point point: decisions.Point(o),
) -> Result(
  #(vocabulary.Recommendation(o), policy.Outcome),
  model.Failure,
)

What policy would say about a candidate model’s answer on a recorded example: the candidate runs on the example’s recorded sources (as in replay), and its recommendation is evaluated as answer_under evaluates the recorded one, with the facts consulted points returned. Returns the candidate’s recommendation with the policy’s answer, or why the candidate could not answer from the record.

pub fn coverage(records: List(decisions.Record(o))) -> Coverage

Count Coverage over records read with decisions.read_record.

pub fn example(
  occurrence: occurrence.Occurrence(o),
  as_of as_of: timestamp.Timestamp,
  from archive: Archive,
) -> Result(Example(o), ExportError)

Export an occurrence as it was known at as_of, resolving every input once.

pub fn example_to_json(
  example: Example(o),
  output: wire.Codec(o),
) -> Result(json.Json, String)

The example as a versioned "decision-example.v1" document, as a durable job appends it to a training lake. An error means the codec cannot encode an output the example carries.

pub fn merge(replays: List(Replay)) -> Replay

Add up replays that ran apart, keeping their order.

pub const nothing: Tally

A tally of nothing.

pub fn replay(
  examples: List(Example(o)),
  candidate: model.Model(o),
  same same: fn(o, o) -> Bool,
) -> Replay

Run candidate over examples through decisions/model.run. Its whole world is each example. It calls no gated tool, writes no record, and has no path to a network that decisions hands it. A candidate that raises is reported as unanswered, not as a crash.

What is judged follows the rule tally states, so for the same occurrences replay(..).tally.judged equals the recorded tally’s judged: an occurrence is judged when people let an output through, and the candidate agrees when its output is same as that one. An occurrence that was only rejected is judged by neither. Accuracy is scored against what proved true: the truth an assessment names, or the final output when it proved right.

pub fn replay_one(
  example: Example(o),
  candidate: model.Model(o),
  same same: fn(o, o) -> Bool,
) -> Replay

Replay one example: the pure half of replay, for callers that spread examples over processes and merge the results.

pub fn sources(example: Example(o)) -> model.Sources

A candidate’s whole world for one example: the recorded context and the recorded reads.

pub fn tally(
  occurrences: List(occurrence.Occurrence(o)),
) -> List(#(vocabulary.Recommender, Tally))

Agreement and accuracy per recommender version.

Agreement follows one rule, shared with replay so the two figures have the same denominator: an occurrence is judged when people let an output through to acceptance (it was accepted, held, released, or accepted and then refused by the tool), and that output is what agreement compares with. Walking its verdicts and then its reviews in order, an approval lets through the output that stands, a correction lets through its own output, and a rejection lets nothing through. A person’s own act counts as their judgment of the output they proposed. Each judged occurrence counts once, however many verdicts it has. Here it agrees when the output let through is the one recommended; in replay, when it is the candidate’s output.

So an occurrence that was rejected, or whose review rejected it, is not judged by either: nothing was let through to compare with. Both count it in rejected instead, so neither figure hides it. Not judged and not counted at all: an output the policy accepted with no verdict and no review; a partial quorum that expired (approvals that never reached acceptance are not a judgment); a withdrawn occurrence (the decider took it back during its hold, which says they changed their mind, not what the right output was); and an acceptance a newer event superseded before its release, for the same reason as a withdrawal: the world moved on, which says nothing about whether the output was right. Each assessment counts toward accuracy.

pub fn total(tallies: List(#(a, Tally))) -> Tally

The tallies of several recommenders added together.

pub fn widened(example: Example(o)) -> Example(o)

The example with what validators opened beyond the card added to what a candidate may read: the context as the validator saw it. What a validator opened is later knowledge, so it answers a read of the same name and argument.

✨ Search Document