Module: experiments

​

References

​

defineExperiment

Re-exports defineExperiment

---

​

ExperimentDefinition

Re-exports ExperimentDefinition

---

​

ExperimentProtocol

Re-exports ExperimentProtocol

---

​

ScenarioCollectionBudget

Re-exports ScenarioCollectionBudget

​

Type Aliases

​

ExperimentCandidate

Ƭ ExperimentCandidate: Object A captured program, optionally with exact model replacements for its candidate decisions. Keys are model.dependency... paths, so {"policy.demand": demandV2} changes only that dependency: every other part of the program, its fixed environment and its evaluator stay the same.

​

Type declaration

---

​

CapturedExperimentCandidate

Ƭ CapturedExperimentCandidate: Object Not exported: an exported type refers to it.

​

Type declaration

---

​

ExperimentCapture

Ƭ ExperimentCapture: Object

​

Type declaration

---

​

ExperimentMetricSummary

Ƭ ExperimentMetricSummary: { case: string } & { status: "complete" ; unit: string | null ; difference: PairedSummary } | { status: "unavailable" ; reason: string }

---

​

ExperimentAttributionOptions

Ƭ ExperimentAttributionOptions: Object

​

Type declaration

---

​

ExperimentRun

Ƭ ExperimentRun: Omit<ScenarioCollection, "capture" | "order" | "select" | "start" | "request" | "training"> & { capture: Readonly<ExperimentCapture> | null ; recover: (signal?: AbortSignal) => ReturnType<ScenarioCollection["start"]> ; order: (signal?: AbortSignal) => ReturnType<ScenarioCollection["order"]> ; select: (signal?: AbortSignal) => ReturnType<ScenarioCollection["select"]> ; compare: (options: { baseline: string ; candidate: string ; metric: string ; bootstrapDraws?: number }, signal?: AbortSignal) => Promise<ExperimentMetricSummary[]> ; recompute: (signal?: AbortSignal) => Promise<ExperimentRun> ; resample: (seed: string, signal?: AbortSignal) => Promise<ExperimentRun> ; attribute: (options: ExperimentAttributionOptions, signal?: AbortSignal) => Promise<ExperimentRun> } One admitted experiment: its paired runs, their order and selection.

---

​

PairedSummary

Ƭ PairedSummary: Object Exact paired differences; the interval describes sampled simulation variation.

​

Type declaration

---

​

ScenarioCollection

Ƭ ScenarioCollection: Object Not exported: an exported type refers to it.

​

Type declaration

​

Variables

​

MAX_COLLECTION_RUNS

• Const MAX_COLLECTION_RUNS: 64

---

​

MAX_COLLECTION_CONCURRENCY

• Const MAX_COLLECTION_CONCURRENCY: 16

---

​

MAX_COLLECTION_WORK

• Const MAX_COLLECTION_WORK: 1073741824

---

​

MAX_COLLECTION_DURATION_MS

• Const MAX_COLLECTION_DURATION_MS: 1800000

---

​

COLLECTION_BRANCH_BYTES

• Const COLLECTION_BRANCH_BYTES: 8589934592

---

​

MAX_SCENARIO_CANDIDATES

• Const MAX_SCENARIO_CANDIDATES: 16

---

​

MAX_CASE_WEIGHT

• Const MAX_CASE_WEIGHT: 1000 The largest weight one case contributes to a summed metric.

​

Functions

​

createExperimentClient

▸ createExperimentClient(call, readProtocol): Readonly<{ promote: (request: ScenarioPromotionRequest, signal?: AbortSignal) => Promise<ScenarioPromotionStatus> = scenarios.promote; promotion: (request: ScenarioPromotionRequest, signal?: AbortSignal) => Promise<ScenarioPromotionStatus> = scenarios.promotion; get: (experiment: string | ExperimentCapture) => ExperimentRun ; start: (name: string, options: { candidates: readonly ExperimentCandidate[] ; simulation: ScenarioSimulation ; from?: SnapshotRef ; start?: ScenarioStart ; retain?: boolean ; budget?: ScenarioCollectionBudget ; protocol?: ExperimentProtocol ; collectionId?: string ; training?: ScenarioSelectionReference ; signal?: AbortSignal }) => Promise<ExperimentRun> }> Authoring and observation only: native collections own admission, execution and selection.

​

Parameters

​

Returns

Readonly<{ promote: (request: ScenarioPromotionRequest, signal?: AbortSignal) => Promise<ScenarioPromotionStatus> = scenarios.promote; promotion: (request: ScenarioPromotionRequest, signal?: AbortSignal) => Promise<ScenarioPromotionStatus> = scenarios.promotion; get: (experiment: string | ExperimentCapture) => ExperimentRun ; start: (name: string, options: { candidates: readonly ExperimentCandidate[] ; simulation: ScenarioSimulation ; from?: SnapshotRef ; start?: ScenarioStart ; retain?: boolean ; budget?: ScenarioCollectionBudget ; protocol?: ExperimentProtocol ; collectionId?: string ; training?: ScenarioSelectionReference ; signal?: AbortSignal }) => Promise<ExperimentRun> }>

---

​

summarizePairs

▸ summarizePairs(differences, seed, draws?): PairedSummary Deterministic bootstrap over complete paired repetitions of one declared case.

​

Parameters

​

Returns

PairedSummary