decisions/learning
What the record is read for besides queues: agreement and accuracy per recommender version, the export of training examples, and the replay of a candidate model against recorded decisions. Each is a fold over occurrences; none writes, and none needs a store.
- Agreement is how often validators let the recommendation through: an approval agrees; a correction or a rejection does not. An output the policy accepted has no verdict and counts toward no agreement.
- Accuracy is how often the final output proved right, from assessments.
- Replay runs a candidate model on self-contained examples. Its whole world is the example: the recorded context and the recorded reads, so it needs no network and calls no gated tool.
import decisions/learning
let assert Ok(examples) =
list.try_map(recorded, learning.example(_, as_of: now, from: archive))
let scored = learning.replay(examples, candidate, same: fn(a, b) { a == b })
// scored.tally.agreed of scored.tally.judged
Target-independent: builds for Erlang and JavaScript.
Types
Where an example’s parts are resolved from at export time.
pub type Archive {
Archive(
render: fn(vocabulary.Manifest) -> Result(String, String),
reads: fn(vocabulary.Explanation) -> List(Read),
content: fn(vocabulary.Ref) -> Result(String, model.Failure),
)
}
Constructors
-
Archive( render: fn(vocabulary.Manifest) -> Result(String, String), reads: fn(vocabulary.Explanation) -> List(Read), content: fn(vocabulary.Ref) -> Result(String, model.Failure), )Arguments
- render
-
The renderer that built the model’s prompt from a context. It fails for a renderer version it does not have. It sees only the manifest: when the real prompt was captured as evidence (one provider call’s exact request), name that evidence among the context’s references and return its content here through the same resolver.
- reads
-
Reads retained elsewhere, such as in the run an explanation points into. Return
[]when the recommendation’s own reads are the whole story. - content
-
The content of a reference to immutable data or evidence: the same resolver a card uses.
How triggers fared before a recommendation, for coverage: of the
records read, how many began with an expectation (decisions.expect or
decisions.expect_until), how many of those a newer event superseded
before anything was recommended, how many expired at their own
deadline before anything was recommended (the expiry written: a record
lapsed by the clock whose wake-up has not run yet still counts as
expected), and how many are still expected with
nothing recommended (work in flight, or a worker that died for good). A
recommendation recorded after the deadline began the record, so it
counts only as expected.
pub type Coverage {
Coverage(
expected: Int,
superseded_before: Int,
lapsed_before: Int,
still_expected: Int,
)
}
Constructors
-
Coverage( expected: Int, superseded_before: Int, lapsed_before: Int, still_expected: Int, )
One training example: an occurrence as it was known at a stated time, with every input resolved. It needs nothing else to be replayed.
pub type Example(output) {
Example(
as_of: timestamp.Timestamp,
known: occurrence.Occurrence(output),
prompt: String,
context: List(#(vocabulary.Ref, String)),
reads: List(#(Read, String)),
expansions: List(#(String, String)),
)
}
Constructors
-
Example( as_of: timestamp.Timestamp, known: occurrence.Occurrence(output), prompt: String, context: List(#(vocabulary.Ref, String)), reads: List(#(Read, String)), expansions: List(#(String, String)), )Arguments
- known
-
The occurrence folded from only the entries up to
as_of. - expansions
-
What validators opened beyond the card, as
#(read, content).
Why an example could not be exported.
pub type ExportError {
NotYetRecommended
RenderFailed(reason: String)
MissingContent(ref: vocabulary.Ref, failure: model.Failure)
}
Constructors
-
NotYetRecommendedThe occurrence had not been recommended at the stated time.
-
RenderFailed(reason: String)The archive has no renderer for the context.
-
MissingContent(ref: vocabulary.Ref, failure: model.Failure)The archive cannot resolve a reference the record names.
A read a model made, with the evidence it returned.
pub type Read {
Read(name: String, argument: String, result: vocabulary.Ref)
}
Constructors
-
Read(name: String, argument: String, result: vocabulary.Ref)
What a candidate model would have scored on recorded occurrences.
pub type Replay {
Replay(
tally: Tally,
unanswered: List(#(String, model.Failure)),
)
}
Constructors
-
Replay(tally: Tally, unanswered: List(#(String, model.Failure)))Arguments
- unanswered
-
Occurrences the candidate could not answer, as
#(occurrence, failure): a read the record does not hold, for one.
Agreement is agreed of judged, from verdicts and reviews. Accuracy
is right of assessed, from assessments.
pub type Tally {
Tally(
agreed: Int,
judged: Int,
rejected: Int,
right: Int,
assessed: Int,
)
}
Constructors
-
Tally( agreed: Int, judged: Int, rejected: Int, right: Int, assessed: Int, )Arguments
- rejected
-
Occurrences people rejected outright: nothing was let through, so they are not judged, and counted here instead.
Values
pub fn answer_under(
example: Example(o),
policy candidate: policy.Policy,
point point: decisions.Point(o),
) -> policy.Outcome
What policy would have said about a recorded example when it was
recommended: the point’s facts of the recommended output, the
recommender’s signals and what consulted points returned, evaluated
under the candidate policy with no time waited. Use it beside replay
to see, before publishing a version, how many recorded occurrences it
would have let proceed and how many of those people corrected or
rejected.
pub fn candidate_under(
example: Example(o),
candidate candidate: model.Model(o),
policy policy_version: policy.Policy,
point point: decisions.Point(o),
) -> Result(
#(vocabulary.Recommendation(o), policy.Outcome),
model.Failure,
)
What policy would say about a candidate model’s answer on a recorded
example: the candidate runs on the example’s recorded sources (as in
replay), and its recommendation is evaluated as answer_under
evaluates the recorded one, with the facts consulted points returned.
Returns the candidate’s recommendation with the policy’s answer, or why
the candidate could not answer from the record.
pub fn coverage(records: List(decisions.Record(o))) -> Coverage
Count Coverage over records read with decisions.read_record.
pub fn example(
occurrence: occurrence.Occurrence(o),
as_of as_of: timestamp.Timestamp,
from archive: Archive,
) -> Result(Example(o), ExportError)
Export an occurrence as it was known at as_of, resolving every input
once.
pub fn example_to_json(
example: Example(o),
output: wire.Codec(o),
) -> Result(json.Json, String)
The example as a versioned "decision-example.v1" document, as a
durable job appends it to a training lake. An error means the codec
cannot encode an output the example carries.
pub fn replay(
examples: List(Example(o)),
candidate: model.Model(o),
same same: fn(o, o) -> Bool,
) -> Replay
Run candidate over examples through decisions/model.run. Its whole
world is each example. It calls no gated tool, writes no record, and
has no path to a network that decisions hands it. A candidate that
raises is reported as unanswered, not as a crash.
What is judged follows the rule tally states, so for the same
occurrences replay(..).tally.judged equals the recorded tally’s
judged: an occurrence is judged when people let an output through,
and the candidate agrees when its output is same as that one. An
occurrence that was only rejected is judged by neither. Accuracy is
scored against what proved true: the truth an assessment names, or the
final output when it proved right.
pub fn replay_one(
example: Example(o),
candidate: model.Model(o),
same same: fn(o, o) -> Bool,
) -> Replay
Replay one example: the pure half of replay, for callers that spread
examples over processes and merge the results.
pub fn sources(example: Example(o)) -> model.Sources
A candidate’s whole world for one example: the recorded context and the recorded reads.
pub fn tally(
occurrences: List(occurrence.Occurrence(o)),
) -> List(#(vocabulary.Recommender, Tally))
Agreement and accuracy per recommender version.
Agreement follows one rule, shared with replay so the two figures
have the same denominator: an occurrence is judged when people let
an output through to acceptance (it was accepted, held, released, or
accepted and then refused by the tool), and that output is what
agreement compares with. Walking its verdicts and then its reviews in
order, an approval lets
through the output that stands, a correction lets through its own
output, and a rejection lets nothing through. A person’s own act counts
as their judgment of the output they proposed. Each judged occurrence
counts once, however many verdicts it has. Here it agrees when the
output let through is the one recommended; in replay, when it is the
candidate’s output.
So an occurrence that was rejected, or whose review rejected it, is not
judged by either: nothing was let through to compare with. Both count
it in rejected instead, so neither figure hides it. Not judged and
not counted at all: an output the policy accepted with no verdict and
no review; a partial quorum that expired (approvals that never reached
acceptance are not a judgment); a withdrawn occurrence (the decider
took it back during its hold, which says they changed their mind, not
what the right output was); and an acceptance a newer event superseded
before its release, for the same reason as a withdrawal: the world
moved on, which says nothing about whether the output was right. Each assessment counts toward accuracy.