Algorithms¶

This section contains the API documentation for solution algorithms, neural network components, and grid tools used to solve dynamic stochastic optimization problems.

Value Function Iteration (VFI)¶

The value function iteration (VFI) algorithm derives arrival value functions from a continuation value function and the stage dynamics of model blocks.

Value function iteration (VFI).

Derive a decision rule and a gridded value function for a single BellmanPeriod by value function iteration: at each point of a grid over the period’s arrival states, solve an exact scipy.optimize.minimize() for the control that maximizes the period reward plus the discounted continuation value.

skagent.algos.vfi.AxisSpec¶

One coordinate vector per variable, whose cartesian product is the lattice a value array is tabulated over. Unrelated to skagent.grid.Grid, which is a batch of scattered points rather than a per-axis specification.

alias of Mapping[str, Sequence]

skagent.algos.vfi.ar_from_data(da)¶

Build a decision rule from a fitted policy DataArray.

The returned rule follows the library’s decision-rule calling convention: it takes the control’s information-set values as positional arguments, in the order of da.dims (which solve() aligns to control.iset). This matches how block.transition invokes a rule, dr(*[vals[v] for v in iset]), so a VFI-fitted rule is a drop-in for the rest of the stack.

Interpolation runs in numpy/xarray space. For a torch-tensor interface, wrap the result with tensor_decision_rule().

Parameters:: da (xarray.DataArray) – The fitted policy, with one dimension per information-set variable in control.iset order. A zero-dimensional array encodes a constant rule (empty information set).
Returns:: A rule ar(*args) taking one positional argument per dimension of da. Scalar arguments return a Python scalar; array-like arguments are interpolated pointwise (not as an outer product) and return a numpy array.
Return type:: callable
Raises:: TypeError – If the number of positional arguments does not match da.ndim.

skagent.algos.vfi.get_action_rule(action)¶

Build a constant decision rule that ignores its inputs.

Parameters:: action (Any) – The fixed value the rule returns.
Returns:: A zero-argument function ar() returning action. Used to wrap a candidate action as a decision rule during the per-point optimization.
Return type:: callable

skagent.algos.vfi.grid_to_data_array(grid={})¶

Construct a zero-valued DataArray over the coordinates of a grid.

Parameters:: grid (Mapping[str, Sequence]) – A mapping from variable labels to a sequence of numerical values. An empty mapping yields a zero-dimensional array.
Returns:: An array whose dimensions and coordinates are those of grid.
Return type:: xarray.DataArray

skagent.algos.vfi.solve_bellman(bp, state_grid, *, continuation_vf=None, agent=None, scope={}, disc_params={}, tol=1e-06, max_iter=100, x0=1.0, raise_on_nonconvergence=False, artificial_borrowing_constraint=False)¶

Solve a recurring BellmanPeriod by value-function iteration.

Iterates solve_step() to a fixed point: each backup uses the previous iterate’s value grid as its continuation (rebuilt via value_array_to_function()) and offers the previous iterate’s policy_array as a per-point warm start — one multi-start candidate among the others, so a collapsed iterate cannot entrench itself by seeding the next backup at its own bound. It stops when the sup-norm change in the value grid falls below tol, or after max_iter iterations.

Iteration 1 uses the terminal (zero) continuation, so solve_bellman(..., max_iter=1) reproduces solve_step() under a terminal continuation. For an infinite-horizon problem the loop converges geometrically (modulus the discount factor) to the stationary solution; for a finite horizon of length T set max_iter=T.

The stopping rule assumes an infinite horizon. A finite-horizon problem is structurally indistinguishable from an infinite-horizon one, so a run that reaches max_iter is reported as non-convergence in both cases: at a caller-specified horizon that report is expected rather than a failure, and the residual is a sup-norm change between iterates, not an error against the T-period answer.

A period with no arrival states is not a dynamic problem — nothing carries between periods — and is refused. Such a block is solved statically, by skagent.algos.tabular.TabularBestResponseSolver or skagent.solver.NeuralBestResponse, driven by skagent.solver.solve_in_order(). Every arrival state must be an axis of state_grid, since the continuation is rebuilt from the value grid and cannot represent dependence on a variable the grid has no axis for.

Shocks are discretized internally: disc_params is threaded into every backup (hidden shocks integrated inside the max) and into value_array_to_function() (observed-shock axes integrated into the arrival value between iterations).

Parameters:

bp (BellmanPeriod) – The recurring period providing the model mechanics.
state_grid (Mapping[str, Sequence]) – A grid over the value function’s domain (arrival states and/or observed shocks); see solve_step().
continuation_vf (Optional[Callable]) – Initial continuation guess continuation_vf(states, shocks, parameters). Defaults to the terminal (zero) continuation.
agent (str | None) – If given, the period reward sums only this agent’s reward symbols.
scope (Mapping) – Fixed non-shock exogenous values (and, in this scope, any hidden-shock realization) merged into the model parameters.
disc_params (Mapping) – Per-shock discretization arguments, threaded into each backup (for hidden shocks) and into value_array_to_function() (for observed shocks); see solve_step().
tol (float) – Convergence tolerance on the sup-norm change in the value grid.
max_iter (int) – Maximum number of backups.
x0 (float) – Modest multi-start seed candidate passed to solve_step().
raise_on_nonconvergence (bool) – If True, raise RuntimeError when the loop hits max_iter with the sup-norm change still above tol; otherwise emit a warnings.warn and return the last iterate (the scipy OptimizeResult.success convention, O5). At a finite horizon set through max_iter this condition is expected, so leave it False.
artificial_borrowing_constraint (bool) – Forwarded to solve_step(): confine next-period arrival states to the state grid (grid edge = slack artificial borrowing limit), so the rebuilt continuation is never extrapolated off-grid.

Return type:

tuple[dict[str, Callable], DataArray, dict[str, DataArray]]

Returns:

dr_from_data (dict of callable) – One decision rule per control at the fixed point (see solve_step()).
value_array (xarray.DataArray) – The converged value grid. Its attrs carry n_iter, converged (bool), and residual (the final sup-norm change).
policy_array (dict of xarray.DataArray) – The gridded optimal control(s) at the fixed point.

Raises:

ValueError – If max_iter is less than 1, or if the period has no arrival states.
RuntimeError – If raise_on_nonconvergence is True and the loop does not converge.

skagent.algos.vfi.solve_step(bp, continuation_vf, state_grid, *, agent=None, control=None, scope={}, disc_params={}, decision_rules=None, x0=1.0, x0_policy=None, artificial_borrowing_constraint=False)¶

One exact value backup over state_grid.

At each grid point the optimal control is found with scipy.optimize.minimize(), maximizing the period reward plus the discounted continuation value of the resulting arrival states. Under a terminal (zero) continuation, continuation_vf = lambda s, sh, p: 0.0, the result is the single-step solution; the value-iteration wrapper solve_bellman() iterates it to a fixed point.

Applies to static and dynamic periods alike, which is why it is not named for Bellman: a period naming no discount variable discounts by 1.0, and a static block has no arrival states for a continuation to read, so the backup reduces to maximizing the period reward.

Speaks the BellmanPeriod protocol the rest of the torch stack uses, with an explicit discount factor and multi-reward summation, and is empty-shock-safe.

Shocks are handled by their information role, which is derived from the block’s own structure (skagent.relevance) rather than inferred from the grid the caller supplies. A shock some control’s information set accounts for becomes a grid axis over its discretization nodes, so its pre-state and bounds are computed per realization; the rest are integrated out inside the per-point max via internal discretization (_discretize_shocks() + expected), so an optimum characterized by an expectation over an unobserved shock is in scope. Any shock may instead be pinned to a fixed realization by supplying it in scope, which takes precedence over both.

Note

Current scope: one or more controls, jointly optimized by scipy.optimize.minimize() over the stacked control vector with per-control bounds — restarted from each seed candidate, keeping the best optimum — each policy then reprojected onto its own information set (_project_to_iset()). A control’s pre-state and bounds are evaluated with each integrated-out shock fixed at its (discretized) mean, since a single value is required there even though the objective integrates the shock. That is exact when the pre-state does not depend on such a shock, and a shock the pre-state does depend on is a grid axis rather than an integrated one, so it does not arise for a well-formed block.

Parameters:

bp (BellmanPeriod) – The recurring period providing the model mechanics.
continuation_vf (Callable) – The continuation value function, called continuation_vf(states, shocks, parameters) on the next-period arrival states (the bp.compute_value convention). Terminal continuation is lambda s, sh, p: 0.0.
state_grid (Mapping[str, Sequence]) – The shared backup grid over arrival states: one axis per variable. Axes for the shocks an information set accounts for are added automatically from their discretization nodes, so supplying one is optional and idempotent. This grid covers the variables the Bellman loop iterates over and is not necessarily equal to any individual control’s information set (a control’s iset may be a strict subset). For an empty grid, pass {} — the empty mapping, not skagent.grid.Grid, whose from_config({}) raises.
agent (str | None) – If given, the period reward sums only this agent’s reward symbols.
scope (Mapping) – Fixed non-shock exogenous values merged into the model parameters. A shock supplied here is pinned to that fixed realization instead of being gridded or integrated. Note this fixes a value for the solve; it does not make the symbol a model parameter, so it does not affect any shock’s information role.
decision_rules (Optional[Mapping[str, Callable]]) – Rules for the controls not being optimized here, which is what reduces a multi-control block to one control’s problem. Supplied per call, since which rules the others are held at is a property of the question rather than of the model; the controls being optimized are pinned to their trial values and ignore any rule given for them.
disc_params (Mapping) – Per-shock discretization arguments, keyed by shock symbol (e.g. {"theta": {"N": 7}}), forwarded to that shock’s Distribution.discretize whether it becomes a grid axis or is integrated out. A shock without an entry uses its distribution’s default discretization (exact for already-discrete shocks).
x0 (float) – A modest optimizer seed, clamped into each control’s bounds. One of the multi-start candidates at every point, not just a fallback: it is what covers a box whose midpoint is far from the optimum, e.g. the natural borrowing limit’s [0, m + H].
x0_policy (Optional[Mapping[str, DataArray]]) – Warm-start seeds keyed by control symbol (e.g. a previous iterate’s policy_array); supplies the first multi-start candidate at each grid point, and wins ties. Supplied by solve_bellman().
artificial_borrowing_constraint (bool) – When True, tighten each control’s bounds so the next-period arrival state stays inside the state grid (_tighten_bounds_to_grid()), an artificial state (borrowing) limit at the grid’s lower edge. This keeps the continuation interpolated rather than extrapolated past the grid edges, so value iteration cannot ride a control bound by over-crediting off-grid successors. Single control with an affine successor only (raises otherwise). The limit must be slack at the states of interest (it is just the grid floor), or it biases the policy where it binds.
control (str | None)

Return type:

tuple[dict[str, Callable], DataArray, dict[str, DataArray]]

Returns:

dr_from_data (dict of callable) – One decision rule per control, keyed by control symbol; each takes its information-set values as positional arguments in control.iset order.
value_array (xarray.DataArray) – The gridded optimized decision value over the state grid.
policy_array (dict of xarray.DataArray) – The gridded optimal control(s) over the state grid, keyed by control symbol (a dict for forward-compatibility with multi-control, O1).

skagent.algos.vfi.tensor_decision_rule(np_rule, dtype=None, device=None)¶

Wrap a numpy-space decision rule so it speaks torch tensors.

Adapts a rule (e.g. from ar_from_data(), including the rules returned by solve()) for interop with the torch solving stack (BellmanPeriod / loss / solver). Inputs may be torch tensors or numpy/scalars; outputs are torch tensors.

Because interpolation runs in numpy, the autograd graph is severed: the returned controls are detached. This rule is therefore valid as a fixed / ground-truth / warm-start policy (e.g. an other_dr or in a value-residual loss), but not as a trainable policy in a loss that differentiates through the control (FOC-weighted Bellman, Euler).

Parameters:

np_rule (callable) – A numpy-space decision rule taking positional information-set arguments and returning numpy/scalar values.
dtype (torch.dtype, optional) – Output tensor dtype. Defaults to torch.float32 to match grid tensors (grid.py builds them with torch.FloatTensor).
device (torch.device, optional) – Output tensor device. Defaults to the stack’s device (skagent.grid.device).

Returns:

A rule tdr(*args) accepting torch tensors (or numpy/scalars) and returning a detached torch tensor on device with dtype dtype.

Return type:

callable

skagent.algos.vfi.value_array_to_function(value_array, bp, disc_params={})¶

Rebuild a continuation value function from an iterate’s value grid.

solve_bellman() feeds iteration n’s value grid back as iteration (n+1)’s continuation. This wraps the grid as a callable in the bp.compute_value convention wf(states, shocks, parameters), so it drops straight into solve_step()’s continuation_vf slot.

The decision value grid ranges over arrival states and any observed-shock axes. Those shock axes are first integrated out into the arrival value W(s) = E_obs[V(s, obs)], using the weights of the same discretized distribution that produced the axis nodes; a grid built from other node values raises ValueError. wf then interpolates linearly over the remaining arrival-state axes and extrapolates linearly past the grid edges (via scipy.interpolate.RegularGridInterpolator), so an off-grid next-period state during the backup gets a finite, sloped continuation rather than NaN (which breaks the optimizer) or a flat boundary clamp (which zeroes the marginal value of saving and collapses the policy onto its bound).

When the value grid has no observed-shock axes (the deterministic and hidden-shock-only cases, where the backup already integrated any hidden shocks), the expectation step is a no-op and the value grid over arrival states is W.

wf reads only the axes it has, so it raises ValueError rather than silently discard an arrival state that the value grid has no axis for — with D-3’s survival state liv left off the grid, the mortality discount cancels out of the backup and the loop converges to the no-mortality policy, a plausible value for a different model.

Parameters:

value_array (DataArray) – A gridded decision value function, e.g. the value_array returned by solve_step(); its axes are arrival states and any observed-shock nodes.
bp (BellmanPeriod) – The recurring period; used to identify and discretize the shock axes.
disc_params (Mapping) – Per-shock discretization arguments for the observed-shock axes, keyed by shock symbol. A shock axis without an entry uses its distribution’s default discretization.

Returns:

wf(states, shocks, parameters) returning the interpolated arrival value at states. shocks and parameters are accepted for the bp.compute_value calling convention but unused.

Return type:

Callable

Core VFI Functions¶

One exact value backup over state_grid.

Speaks the BellmanPeriod protocol the rest of the torch stack uses, with an explicit discount factor and multi-reward summation, and is empty-shock-safe.

Note

Parameters:

bp (BellmanPeriod) – The recurring period providing the model mechanics.
continuation_vf (Callable) – The continuation value function, called continuation_vf(states, shocks, parameters) on the next-period arrival states (the bp.compute_value convention). Terminal continuation is lambda s, sh, p: 0.0.
state_grid (Mapping[str, Sequence]) – The shared backup grid over arrival states: one axis per variable. Axes for the shocks an information set accounts for are added automatically from their discretization nodes, so supplying one is optional and idempotent. This grid covers the variables the Bellman loop iterates over and is not necessarily equal to any individual control’s information set (a control’s iset may be a strict subset). For an empty grid, pass {} — the empty mapping, not skagent.grid.Grid, whose from_config({}) raises.
agent (str | None) – If given, the period reward sums only this agent’s reward symbols.
scope (Mapping) – Fixed non-shock exogenous values merged into the model parameters. A shock supplied here is pinned to that fixed realization instead of being gridded or integrated. Note this fixes a value for the solve; it does not make the symbol a model parameter, so it does not affect any shock’s information role.
decision_rules (Optional[Mapping[str, Callable]]) – Rules for the controls not being optimized here, which is what reduces a multi-control block to one control’s problem. Supplied per call, since which rules the others are held at is a property of the question rather than of the model; the controls being optimized are pinned to their trial values and ignore any rule given for them.
disc_params (Mapping) – Per-shock discretization arguments, keyed by shock symbol (e.g. {"theta": {"N": 7}}), forwarded to that shock’s Distribution.discretize whether it becomes a grid axis or is integrated out. A shock without an entry uses its distribution’s default discretization (exact for already-discrete shocks).
x0 (float) – A modest optimizer seed, clamped into each control’s bounds. One of the multi-start candidates at every point, not just a fallback: it is what covers a box whose midpoint is far from the optimum, e.g. the natural borrowing limit’s [0, m + H].
x0_policy (Optional[Mapping[str, DataArray]]) – Warm-start seeds keyed by control symbol (e.g. a previous iterate’s policy_array); supplies the first multi-start candidate at each grid point, and wins ties. Supplied by solve_bellman().
artificial_borrowing_constraint (bool) – When True, tighten each control’s bounds so the next-period arrival state stays inside the state grid (_tighten_bounds_to_grid()), an artificial state (borrowing) limit at the grid’s lower edge. This keeps the continuation interpolated rather than extrapolated past the grid edges, so value iteration cannot ride a control bound by over-crediting off-grid successors. Single control with an affine successor only (raises otherwise). The limit must be slack at the states of interest (it is just the grid floor), or it biases the policy where it binds.
control (str | None)

Return type:

tuple[dict[str, Callable], DataArray, dict[str, DataArray]]

Returns:

dr_from_data (dict of callable) – One decision rule per control, keyed by control symbol; each takes its information-set values as positional arguments in control.iset order.
value_array (xarray.DataArray) – The gridded optimized decision value over the state grid.
policy_array (dict of xarray.DataArray) – The gridded optimal control(s) over the state grid, keyed by control symbol (a dict for forward-compatibility with multi-control, O1).

Solve a recurring BellmanPeriod by value-function iteration.

Parameters:

bp (BellmanPeriod) – The recurring period providing the model mechanics.
state_grid (Mapping[str, Sequence]) – A grid over the value function’s domain (arrival states and/or observed shocks); see solve_step().
continuation_vf (Optional[Callable]) – Initial continuation guess continuation_vf(states, shocks, parameters). Defaults to the terminal (zero) continuation.
agent (str | None) – If given, the period reward sums only this agent’s reward symbols.
scope (Mapping) – Fixed non-shock exogenous values (and, in this scope, any hidden-shock realization) merged into the model parameters.
disc_params (Mapping) – Per-shock discretization arguments, threaded into each backup (for hidden shocks) and into value_array_to_function() (for observed shocks); see solve_step().
tol (float) – Convergence tolerance on the sup-norm change in the value grid.
max_iter (int) – Maximum number of backups.
x0 (float) – Modest multi-start seed candidate passed to solve_step().
raise_on_nonconvergence (bool) – If True, raise RuntimeError when the loop hits max_iter with the sup-norm change still above tol; otherwise emit a warnings.warn and return the last iterate (the scipy OptimizeResult.success convention, O5). At a finite horizon set through max_iter this condition is expected, so leave it False.
artificial_borrowing_constraint (bool) – Forwarded to solve_step(): confine next-period arrival states to the state grid (grid edge = slack artificial borrowing limit), so the rebuilt continuation is never extrapolated off-grid.

Return type:

tuple[dict[str, Callable], DataArray, dict[str, DataArray]]

Returns:

dr_from_data (dict of callable) – One decision rule per control at the fixed point (see solve_step()).
value_array (xarray.DataArray) – The converged value grid. Its attrs carry n_iter, converged (bool), and residual (the final sup-norm change).
policy_array (dict of xarray.DataArray) – The gridded optimal control(s) at the fixed point.

Raises:

ValueError – If max_iter is less than 1, or if the period has no arrival states.
RuntimeError – If raise_on_nonconvergence is True and the loop does not converge.

skagent.algos.vfi.get_action_rule(action)

Build a constant decision rule that ignores its inputs.

Parameters:: action (Any) – The fixed value the rule returns.
Returns:: A zero-argument function ar() returning action. Used to wrap a candidate action as a decision rule during the per-point optimization.
Return type:: callable

skagent.algos.vfi.ar_from_data(da)

Build a decision rule from a fitted policy DataArray.

Interpolation runs in numpy/xarray space. For a torch-tensor interface, wrap the result with tensor_decision_rule().

Parameters:: da (xarray.DataArray) – The fitted policy, with one dimension per information-set variable in control.iset order. A zero-dimensional array encodes a constant rule (empty information set).
Returns:: A rule ar(*args) taking one positional argument per dimension of da. Scalar arguments return a Python scalar; array-like arguments are interpolated pointwise (not as an outer product) and return a numpy array.
Return type:: callable
Raises:: TypeError – If the number of positional arguments does not match da.ndim.

skagent.algos.vfi.grid_to_data_array(grid={})

Construct a zero-valued DataArray over the coordinates of a grid.

Parameters:: grid (Mapping[str, Sequence]) – A mapping from variable labels to a sequence of numerical values. An empty mapping yields a zero-dimensional array.
Returns:: An array whose dimensions and coordinates are those of grid.
Return type:: xarray.DataArray

Tabular Best Response¶

The tabular best-response solver solves one decision at a time from a tabulated payoff table. For every value of what the decision-maker observes, it reports the action that maximizes the payoff of the agent making that decision, conditional on the observation and on a supplied rule for every other decision. The order in which the decisions are solved belongs to a schedule rather than to the method. skagent.solver.solve_in_relevance_order() supplies one such schedule, which takes components from the block’s relevance graph (see Model Analysis and Visualization). Acyclic components are solved once. Cyclic components use simultaneous best-response iteration and either reach a pure-strategy fixed point or report non-convergence. Mixed-strategy equilibria and recurring cyclic games are not supported.

Best responses from a tabulated payoff table.

The solver here handles one decision at a time. For every value of what the decision-maker observes, it reports the action that maximizes their expected payoff, conditional on that observation and on a supplied rule for every other decision. The order in which the decisions are solved, and whether the result is iterated to a fixed point, belong to a schedule rather than to this module; skagent.solver.solve_in_relevance_order() is the sweep this module used to carry.

Expectations are estimated by drawing shock_samples realizations of the block’s shocks once, at construction, and reusing the same draws for every candidate action (common random numbers), so that comparisons between actions carry far less error than their levels do. Conditioning is done by grouping the simulated samples, which makes the beliefs at a decision the posterior induced by the other decision rules in the profile.

The representation is tabular, which sets the limits of this module: a decision’s information set must take finitely many values under the profile being played, since a cell with no repeats has no conditional expectation to estimate, and the actions searched are a finite set. Shocks may be continuous – they are integrated over, never conditioned on. For continuous observations, or where a differentiable policy is wanted, use skagent.solver.NeuralBestResponse in the same schedule.

class skagent.algos.tabular.ConditionalPayoffs(cells, counts, actions, payoff)¶

Expected payoffs by information cell and action.

cellsnumpy.ndarray: Shape (n_cells, len(iset)); the distinct observed values of the decision’s information set. Shape (1, 0) for an empty information set.
countsnumpy.ndarray: Shape (n_cells,); how many samples support each cell.
actionsnumpy.ndarray: Shape (n_actions,); the candidate actions searched.
payoffnumpy.ndarray: Shape (n_cells, n_actions); the estimated expected payoff of the decision’s agent in each cell for each action.

actions¶: Alias for field number 2

cells¶: Alias for field number 0

counts¶: Alias for field number 1

payoff¶: Alias for field number 3

class skagent.algos.tabular.TabularBestResponseSolver(ground, *, actions=None, action_count=21, shock_samples=100000, rng=None, max_cells=1024, samples=None)¶

Solve a block’s decisions by best response, in relevance-component order.

Parameters:

ground (GroundedBlock) – The block-and-calibration pair to solve. The block’s dynamics must be declared in topological order, as the library requires of any block, and its controls must carry the agent attribution needed to tell whose payoff each maximizes when the block’s utilities are owned by more than one agent. The calibration supplies both the arguments the block’s shocks are constructed from and the values any parameter the dynamics refer to.
actions (array_like, optional) – Candidate actions to search, shared by every decision. Defaults to action_count points spanning [0, 1].
action_count (int, optional) – Number of candidate actions when actions is not given.
shock_samples (int, optional) – Number of shock realizations drawn at construction. This is the axis the expectations are estimated over – the integral over the block’s declared shocks that a decision’s objective is defined by, taken by Monte Carlo rather than by quadrature. Grouping splits these across a decision’s information cells, so each cell’s expectation rests on the samples that reached it rather than on all of them.
rng (numpy.random.Generator, optional) – Generator for the shock draws, overriding the one ground carries. When neither supplies a generator, an unseeded one is used.
max_cells (int, optional) – Upper limit on the number of information cells a single decision may have. Exceeding it raises, since grouping samples by observed value only estimates a conditional expectation when observations repeat.

Notes

Constructing the solver draws the block’s shocks from ground. The block itself is left as its author wrote it, so one block may be solved at several calibrations by grounding it against each.

Expectations are Monte Carlo estimates; a solved rule is exact only up to sampling error, which falls as shock_samples rises.

best_response(decision, policies)¶

The payoff-maximizing rule for decision against policies.

Parameters:

decision (str) – The control to solve.
policies (Mapping[str, Callable]) – A decision rule for every control of the block; the profile the best response is computed against.

Returns:

One action per information cell.

Return type:

TabulatedRule

conditional_payoffs(decision, policies)¶

Estimate decision’s payoffs by information cell and action.

Parameters:

decision (str) – The control to evaluate.
policies (Mapping[str, Callable]) – A decision rule for every control of the block, including decision itself: the profile the expectation is taken under.

Return type:

ConditionalPayoffs

initial_policies()¶: A spread rule for every decision in the block.

payoff(vals, agent)¶: The sum of agent’s utility nodes, per sample.

rule_distance(new_rule, old_rule, iset)¶: Largest action difference over the new tabulated rule’s cells.

spread_rule(weights=None)¶

A rule spreading the candidate actions across the samples.

This rule is held by the decisions that are not yet solved. Conditional payoffs are estimated by grouping simulated samples, so a placeholder playing one constant action would induce only the observations that action produces. Every cell reachable only under another action would then be empty, and its conditional expectation undefined. Spreading the actions over the sample axis is what keeps every cell populated.

This is coverage, not randomization: the assignment is deterministic and the sample axis holds shock draws rather than repeated plays, so the share of samples playing an action is not a probability of playing it. A control that genuinely randomizes declares that on itself and draws from a shock.

Parameters:: weights (array_like, optional) – Relative shares of the samples per action, one per candidate action. Defaults to equal shares.
Returns:: A decision rule returning one action per sample.
Return type:: callable

class skagent.algos.tabular.TabulatedRule(iset, cells, actions)¶

A decision rule tabulated over the cells of an information set.

Follows the library’s decision-rule calling convention: the information-set values are passed as positional arguments in iset order. An observation is answered with the action of the nearest tabulated cell, so a rule remains total when queried away from the values it was tabulated on.

For a rule tabulated on a full grid, and where interpolation between grid points is wanted, use skagent.algos.vfi.ar_from_data() instead.

Parameters:

iset (sequence of str) – The information set, in the order the cell columns are given.
cells (array_like) – Shape (n_cells, len(iset)); the tabulated observations.
actions (array_like) – Shape (n_cells,); the action for each cell.

to_dict(decimals=3)¶: The rule as {observed values: action}, rounded for display.

Core Best-Response Classes¶

class skagent.algos.tabular.TabularBestResponseSolver(ground, *, actions=None, action_count=21, shock_samples=100000, rng=None, max_cells=1024, samples=None)

Solve a block’s decisions by best response, in relevance-component order.

Parameters:

ground (GroundedBlock) – The block-and-calibration pair to solve. The block’s dynamics must be declared in topological order, as the library requires of any block, and its controls must carry the agent attribution needed to tell whose payoff each maximizes when the block’s utilities are owned by more than one agent. The calibration supplies both the arguments the block’s shocks are constructed from and the values any parameter the dynamics refer to.
actions (array_like, optional) – Candidate actions to search, shared by every decision. Defaults to action_count points spanning [0, 1].
action_count (int, optional) – Number of candidate actions when actions is not given.
shock_samples (int, optional) – Number of shock realizations drawn at construction. This is the axis the expectations are estimated over – the integral over the block’s declared shocks that a decision’s objective is defined by, taken by Monte Carlo rather than by quadrature. Grouping splits these across a decision’s information cells, so each cell’s expectation rests on the samples that reached it rather than on all of them.
rng (numpy.random.Generator, optional) – Generator for the shock draws, overriding the one ground carries. When neither supplies a generator, an unseeded one is used.
max_cells (int, optional) – Upper limit on the number of information cells a single decision may have. Exceeding it raises, since grouping samples by observed value only estimates a conditional expectation when observations repeat.

Notes

Constructing the solver draws the block’s shocks from ground. The block itself is left as its author wrote it, so one block may be solved at several calibrations by grounding it against each.

Expectations are Monte Carlo estimates; a solved rule is exact only up to sampling error, which falls as shock_samples rises.

best_response(decision, policies)

The payoff-maximizing rule for decision against policies.

Parameters:

decision (str) – The control to solve.
policies (Mapping[str, Callable]) – A decision rule for every control of the block; the profile the best response is computed against.

Returns:

One action per information cell.

Return type:

TabulatedRule

conditional_payoffs(decision, policies)

Estimate decision’s payoffs by information cell and action.

Parameters:

decision (str) – The control to evaluate.
policies (Mapping[str, Callable]) – A decision rule for every control of the block, including decision itself: the profile the expectation is taken under.

Return type:

ConditionalPayoffs

initial_policies(): A spread rule for every decision in the block.

payoff(vals, agent): The sum of agent’s utility nodes, per sample.

rule_distance(new_rule, old_rule, iset): Largest action difference over the new tabulated rule’s cells.

spread_rule(weights=None)

A rule spreading the candidate actions across the samples.

Parameters:: weights (array_like, optional) – Relative shares of the samples per action, one per candidate action. Defaults to equal shares.
Returns:: A decision rule returning one action per sample.
Return type:: callable

class skagent.algos.tabular.TabulatedRule(iset, cells, actions)

A decision rule tabulated over the cells of an information set.

For a rule tabulated on a full grid, and where interpolation between grid points is wanted, use skagent.algos.vfi.ar_from_data() instead.

Parameters:

iset (sequence of str) – The information set, in the order the cell columns are given.
cells (array_like) – Shape (n_cells, len(iset)); the tabulated observations.
actions (array_like) – Shape (n_cells,); the action for each cell.

to_dict(decimals=3): The rule as {observed values: action}, rounded for display.

Maliar-Style Algorithms¶

Neural network-based solution methods following Maliar et al.

Tools for the implementation of the Maliar, Maliar, and Winant (JME ‘21) method.

This method relies on a simpler problem representation than that elaborated by the skagent Block system.

Note

Shock draws here are Monte Carlo only; structured draws (e.g. exact discretizations) could be supported via BellmanPeriod.

skagent.algos.maliar.generate_givens_from_states(states, bellman_period, shock_copies)¶

Generate omega_i values of the MMW JME ‘21 method.

Parameters:

states (Grid) – A grid of starting state values (exogenous and endogenous).
bellman_period (BellmanPeriod) – The period whose shocks are drawn. It resolves them against its own calibration, so the draws follow its generator.
shock_copies (int) – Number of copies of the shocks to be included. Must be >= 1.

Returns:

Grid containing states augmented with shock copies.

Return type:

Grid

skagent.algos.maliar.maliar_training_loop(bellman_period, loss_function, states_0_n, parameters, shock_copies=2, max_iterations=5, tolerance=1e-06, random_seed=None, simulation_steps=1, network_width=16, epochs_per_iteration=250, lr=0.001)¶

Run the Maliar, Maliar, and Winant (JME ‘21) training loop.

Trains a single neural network policy to minimize empirical risk (loss) on a panel of states drawn forward through the model dynamics. This helper constructs and trains a BlockPolicyNet internally and does not currently accept a pre-built shared-backbone BlockPolicyValueNet. If value-aware training is needed (e.g. for a Bellman residual loss with a value head), call train_block_nn() directly on a BlockPolicyValueNet; a future refactor may add value-network support here.

The loop maps onto the MMW JME’21 algorithm steps as follows: _validate_training_inputs() and the network construction below cover Step 1 (initialize topology and coefficients); the per-iteration train_block_nn() call is Step 2 (minimize the empirical risk \(\Xi^n(\theta)\)); the returned network is the Step 3 trained approximation \(\varphi(\cdot, \theta)\).

Parameters:

bellman_period (BellmanPeriod) – A model definition containing block dynamics and transitions.
loss_function (Callable) – The empirical risk function \(\Xi^n\) from MMW JME’21. This function is passed to the neural network training routine as loss_function(decision_function, input_grid) -> loss_tensor.
states_0_n (Grid) – A panel of starting states for training. Must contain at least one state.
parameters (dict) – Given parameters for the model.
shock_copies (int) – Number of shock copies to include in the training set \(\{\omega_i\}\). Must match the expected number of shock copies in the loss function. Must be >= 1. Default is 2.
max_iterations (int) – Maximum number of training loop iterations before stopping. Must be >= 1. Default is 5.
tolerance (float) – Convergence tolerance. Training stops when either the L2 norm of parameter changes or the absolute difference in loss is below this threshold. Satisfying either criterion alone is sufficient. Must be > 0. Default is 1e-6.
random_seed (Optional[int]) – Random seed for reproducibility. Default is None.
simulation_steps (int) – Number of time steps to simulate forward when determining the next training set \(\{\omega_i\}\). Higher values let the training states explore more of the state space at higher computational cost. Must be >= 1. Default is 1.
network_width (int) – Width of hidden layers in the policy neural network. Must be >= 1. Default is 16.
epochs_per_iteration (int) – Number of training epochs per iteration. Must be >= 1. Default is 250.
lr (float) – Learning rate for the internal Adam optimizer. The optimizer is created once and reused across iterations to preserve momentum. Must be > 0. Default is 0.001.

Returns:

(trained_policy_network, training_states) where trained_policy_network is the trained BlockPolicyNet and training_states is the Grid of states from the final iteration (the convergence point if training converged early, otherwise the states after max_iterations steps).

Return type:

tuple

Raises:

ValueError – If max_iterations < 1, tolerance <= 0, shock_copies < 1, simulation_steps < 1, network_width < 1, epochs_per_iteration < 1, or states_0_n contains no states.
TypeError – If bellman_period is None or loss_function is not callable.

skagent.algos.maliar.simulate_forward(states_t, bellman_period, decision_function, parameters, big_t)¶

Simulate the model forward for a specified number of periods.

Parameters:

states_t (Grid | dict) – Initial state values.
bellman_period (BellmanPeriod) – The Bellman period containing model dynamics.
decision_function (Callable) – Function mapping (states, shocks, parameters) to controls.
parameters (dict) – Model parameters.
big_t (int) – Number of time periods to simulate forward. If 0, returns the initial states unchanged.

Returns:

Final state values after big_t periods.

Return type:

dict

Raises:

ValueError – If big_t < 0 or if states_t is an empty dict.

Reinforcement Learning (Stable-Baselines3)¶

Proximal Policy Optimization (PPO) for BellmanPeriod models, via a Stable-Baselines3 backend. The agent wraps a model in a gymnasium environment (see Environments), trains PPO, and emits a standard skagent decision rule.

Stable Baselines3 wrappers for BellmanPeriod models.

Provides PPOAgent, a thin wrapper around SB3’s PPO that:

builds a skagent.env.GymEnv from a BellmanPeriod + initial state distribution,
delegates training to stable_baselines3.PPO.learn,
exposes a PPOAgent.decision_rule() that returns the trained policy as a skagent-style {control_sym: callable} dict — i.e. the same shape consumed by skagent.env.Environment and the rest of the skagent decision-rule API. Actions are unscaled back to real units via GymEnv.unscale_action(), so downstream code does not see the [-1, 1] SB3 representation.

class skagent.algos.sb3.PPOAgent(bp, initial, *, max_episode_steps=200, seed=None, gym_kwargs=None, ppo_kwargs=None, policy='MlpPolicy', device='cpu', verbose=0)¶

Train SB3’s PPO on a BellmanPeriod and emit a skagent decision rule.

Parameters:

bp (BellmanPeriod) – Model definition.
initial (dict) – Maps arrival-state symbols to skagent Distribution objects, used by GymEnv to sample fresh initial states on reset.
max_episode_steps (int) – Episode horizon for the underlying GymEnv. Default 200.
seed (Optional[int]) – Seed for both the environment and the PPO algorithm.
gym_kwargs (Optional[dict]) – Extra keyword arguments forwarded to GymEnv (e.g. default_lower, default_upper, bound_clearance, control_sym).
ppo_kwargs (Optional[dict]) – Extra keyword arguments forwarded to stable_baselines3.PPO (e.g. n_steps, batch_size, learning_rate, n_epochs, policy_kwargs). gamma defaults to bp.calibration[bp.discount_variable] if it is a finite scalar; callers can override by passing gamma here.
policy (str) – SB3 policy class string. Default "MlpPolicy".
device (str) – Torch device for PPO. Default "cpu" (SB3’s recommended default for MlpPolicy).
verbose (int) – Verbosity passed to PPO. Default 0.

env¶

The constructed gymnasium environment.

Type:: GymEnv

model¶

The SB3 model. None until learn() is called the first time (constructed lazily so callers can inspect env without paying PPO’s setup cost).

Type:: stable_baselines3.PPO

decision_rule(deterministic=True)¶

Return a skagent decision rule that uses the trained policy.

The returned dict has the form {control_sym: callable} where the callable accepts positional arguments matching the control’s iset order (i.e. the same signature skagent Environment.step and BellmanPeriod.decision_function call). The callable’s output is a torch.Tensor of unscaled action values; the inputs may be scalars, numpy arrays, or torch tensors of compatible length.

Parameters:: deterministic (bool) – Whether to use a deterministic (mean) policy. Default True — matches typical skagent decision-rule semantics.
Return type:: dict[str, Callable]

learn(total_timesteps, callback=None, **kwargs)¶

Run PPO.learn. Returns self.

Every completed episode’s undiscounted reward is appended to self.episode_rewards via an internal SB3 callback; repeated learn calls accumulate. A user-supplied callback is merged with the internal one via CallbackList. Extra **kwargs forward to model.learn.

Parameters:

total_timesteps (int)
callback (Any)
kwargs (Any)

Return type:

PPOAgent

predict_unscaled(obs, deterministic=True)¶

Predict an unscaled action for obs.

obs may be a single observation (shape (|iset|,)) or a batch ((N, |iset|)). Returns a 1-D array of shape (N,).

Parameters:: deterministic (bool)
Return type:: ndarray

snapshot()¶

Capture the current trained policy as a frozen PolicySnapshot.

The snapshot holds an independent copy of the policy network, so it is unaffected by later learn() calls. This is the supported way to retain the policy at intermediate points during training (e.g. to compare checkpoints) without re-running training or re-implementing the unscaling logic.

Return type:: PolicySnapshot

class skagent.algos.sb3.PolicySnapshot(policy, env)¶

Frozen copy of a trained policy, decoupled from further training.

Returned by PPOAgent.snapshot(). Holds a deep copy of the policy network taken at snapshot time, so subsequent learn calls on the source agent do not change its predictions. Exposes the same predict_unscaled() and decision_rule() interface as PPOAgent.

The GymEnv is shared with the source agent (not copied): it is used only for stateless action unscaling, which does not depend on training state.

Parameters:: env (GymEnv)

decision_rule(deterministic=True)¶

Return a skagent decision rule; see PPOAgent.decision_rule().

Parameters:: deterministic (bool)
Return type:: dict[str, Callable]

predict_unscaled(obs, deterministic=True)¶

Predict an unscaled action for obs; see PPOAgent.predict_unscaled().

Parameters:: deterministic (bool)
Return type:: ndarray

Loss Functions¶

Objective functions passed to skagent.ann.train_block_nn(). The reward-based losses (StaticRewardLoss, EstimatedDiscountedLifetimeRewardLoss) solve a block directly for the non-recurring case; the equation-residual losses (BellmanEquationLoss, EulerEquationLoss) target the recurring, dynamic case. See Loss Functions for the full reference.

Neural Network Components¶

Net¶

Base neural network class with device management.

class skagent.ann.Net(n_inputs, n_outputs, width=32, n_layers=2, activation='silu', transform=None, init_seed=None, copy_weights_from=None)¶

Bases: Module

A flexible feedforward neural network with configurable architecture.

Parameters:

n_inputs (int) – Number of input features
n_outputs (int) – Number of output features
width (int, optional) – Width of hidden layers. Default is 32.
n_layers (int, optional) – Number of hidden layers (1-10). Default is 2.
activation (str, list, callable, or None, optional) –
Activation function(s) to use. Options: - str: Apply same activation to all layers (‘silu’, ‘relu’, ‘tanh’, ‘sigmoid’) - list: Apply different activations to each layer, e.g., [‘relu’, ‘tanh’, ‘silu’] - callable: Custom activation function - None: No activation (identity function)

Available activations: ‘silu’, ‘relu’, ‘tanh’, ‘sigmoid’, ‘identity’ Default is ‘silu’.
transform (str, list, callable, or None, optional) –
Transformation to apply to outputs. Options: - str: Apply same transform to all outputs (‘sigmoid’, ‘exp’, ‘tanh’, etc.) - list: Apply different transforms to each output, e.g., [‘sigmoid’, ‘exp’] - callable: Custom transformation function - None: No transformation

Available transforms: ‘sigmoid’, ‘exp’, ‘tanh’, ‘relu’, ‘softplus’, ‘softmax’, ‘abs’, ‘square’, ‘identity’ Default is None.

property device¶: Device property for backward compatibility.

forward(x)¶

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

BlockPolicyNet¶

Specialized neural network for policy functions in economic models.

class skagent.ann.BlockPolicyNet(bellman_period, control_sym=None, apply_open_bounds=True, width=32, **kwargs)¶

Bases: BellmanPeriodMixin, Net

A neural network for policy functions in dynamic programming problems.

This network wraps a Net and integrates with the BellmanPeriod interface. It automatically determines input/output dimensions from the model block specification and enforces control variable bounds.

Parameters:

bellman_period (BellmanPeriod) – The model Bellman Period
apply_open_bounds (bool, optional) – If True, then the network forward output is normalized by the upper and/or lower bounds, computed as a function of the input tensor. These bounds are “open” because output can be arbitrarily close to, but not equal to, the bounds. Default is True.
control_sym (string, optional) – The symbol for the control variable.
width (int, optional) – Width of hidden layers. Default is 32.
**kwargs – Additional keyword arguments passed to Net. See Net class documentation for all available options including activation, transform, n_layers, init_seed, copy_weights_from, etc.

decision_function(states_t, shocks_t, parameters)¶

A decision function, from states, shocks, and parameters, to control variable values.

Parameters:

states_t (dict) – symbols : values
shocks_t (dict) – symbols: values
parameters (dict) – symbols : values

Returns:

decisions - dict – symbols : values

forward(x)¶: Note that this uses the same architecture of the superclass but adds on a normalization layer appropriate to the bounds of the decision rule.

get_core_function(length=None)¶

get_decision_rule(length=None)¶: Returns the decision rule corresponding to this neural network.

BlockValueNet¶

A neural network for value functions in dynamic programming problems.

class skagent.ann.BlockValueNet(bellman_period, control_sym=None, width=32, **kwargs)¶

Bases: BellmanPeriodMixin, Net

Standalone value-function network for a Bellman problem.

Maps a control’s information set (the same pre-decision states a policy network sees) to a single unconstrained scalar value. It is the value-only counterpart of BlockPolicyNet, kept for algorithms that approximate a value function separately from the policy. The Maliar/MMW path in this package uses the shared-backbone BlockPolicyValueNet instead, so BlockValueNet is not wired into maliar_training_loop().

Parameters:

bellman_period (BellmanPeriod) – The model Bellman period.
control_sym (str, optional) – Control whose information set defines the value function’s domain. Defaults to the first control.
width (int) – Width of hidden layers. Default 32.
**kwargs – Passed to Net (activation, n_layers, init_seed, etc.).

get_core_function(length=None)¶: Return the value function (the trainable core for this net).

get_value_function()¶: Return a callable (states, shocks, parameters) -> value tensor.

value_function(states_t, shocks_t=None, parameters=None)¶

Evaluate the value function at the control’s information set.

Arrival states_t (with shocks_t and parameters) are mapped to the control’s information set via compute_pre_state(), mirroring BlockPolicyNet.decision_function(), then the network is evaluated.

Returns:: Flattened value estimates, one per input row.
Return type:: torch.Tensor

BlockPolicyValueNet¶

A shared-backbone neural network that jointly represents the policy and value functions.

class skagent.ann.BlockPolicyValueNet(bellman_period, control_sym=None, apply_open_bounds=True, width=32, **kwargs)¶

Bases: BellmanPeriodMixin, Net

Single neural network with shared backbone for both policy and value.

Architecture: shared hidden layers → two output heads: - Policy head — bounded output (sigmoid-scaled to satisfy constraints) - Value head — unconstrained scalar output

Sharing the backbone means one optimizer updates all weights simultaneously, and the value head anchors the control level that first-order-condition-only training (e.g. an Euler residual loss) cannot identify.

Parameters:

bellman_period (BellmanPeriod) – The model Bellman Period.
control_sym (str, optional) – Control variable symbol. Defaults to first control.
apply_open_bounds (bool, optional) – Apply sigmoid/softplus scaling to the policy head. Default True.
width (int, optional) – Width of hidden layers. Default 32.
**kwargs – Passed to Net (activation, n_layers, init_seed, etc.).

decision_function(states_t, shocks_t, parameters)¶

Map states, shocks, and parameters to a controls dict.

Parameters:

states_t (dict) – Arrival state values, symbol -> tensor.
shocks_t (dict or None) – Shock values, symbol -> tensor (None is treated as {}).
parameters (dict) – Model parameters, symbol -> value.

Returns:

{control_sym: tensor} of policy-head outputs. The arrival states are mapped to the control’s information set via compute_pre_state() before the network is evaluated.

Return type:

dict

forward(x)¶

Run shared backbone, then policy head (bounded) + value head.

Returns the (policy, value) pair. The policy tensor is scaled into the control’s open bounds; the value tensor is unconstrained. Both have shape (n, 1).

Return type:: tuple[Tensor, Tensor]

get_core_function(length=None)¶: Return decision rules (policy head) for use with train_block_nn.

get_decision_rule(length=None)¶: Decision rule returning only the policy output.

get_policy_and_value_functions(length=None)¶: Return both policy decision rules and value function.

get_value_function()¶

value_function(states_t, shocks_t=None, parameters=None)¶

Evaluate the value head at the control’s information set.

The input domain mirrors decision_function(): arrival states_t (with shocks_t and parameters) are mapped to the control’s information set via compute_pre_state(), then the shared backbone’s value head is evaluated on that pre-decision representation.

Parameters:

states_t (dict) – Arrival state values, symbol -> tensor.
shocks_t (dict or None, optional) – Shock values (None is treated as {}).
parameters (dict or None, optional) – Model parameters.

Returns:

Flattened value estimates, one per input row.

Return type:

torch.Tensor

Training Functions¶

skagent.ann.train_block_nn(block_policy_nn, inputs, loss_function, epochs=50, lr=0.01, optimizer=None, grad_clip=1.0, verbose=True)¶

Train a policy network by minimizing a loss function over a grid.

This is a generic stochastic-gradient-descent driver, not a solution algorithm in itself. It runs epochs Adam updates that minimize whatever loss_function is supplied, evaluated on a single, fixed grid of inputs; it is agnostic to where that grid came from or which method the loss encodes (Euler residual, Bellman residual, FOC, or a custom loss).

Because it trains on whatever inputs it is given, accuracy depends on the caller re-sampling those states across calls: Maliar, Maliar, and Winant (2021) keep the training data “constantly re-sampled,” and minimizing on a single fixed grid instead lets the solution over-fit those points while drifting elsewhere. Re-draw inputs each call (threading the returned optimizer back in to keep Adam’s momentum), or use maliar_training_loop(), which wraps this driver in the full MMW’21 outer loop: it alternates these inner SGD updates with a forward-simulation step that refreshes the training states toward the model’s ergodic set.

Parameters:

block_policy_nn (BlockPolicyNet or BlockPolicyValueNet) – The network to train. Its get_core_function supplies the decision rule(s) the loss is evaluated against.
inputs (Grid) – Input grid containing states and shocks.
loss_function (Callable) – Loss function (decision_function, input_grid) -> loss_tensor.
epochs (int) – Number of training epochs (default 50).
lr (float) – Learning rate for Adam optimizer (default 0.01).
optimizer (Optional[Optimizer]) – Pre-existing optimizer to reuse (preserves momentum across calls). If None, a new Adam optimizer is created.
grad_clip (Optional[float]) – Maximum gradient norm for clipping (default 1.0). Set to None to disable.
verbose (bool) – Emit a logging.info message with the loss every 100 epochs (default True). Configure the root logger to suppress these.

Returns:

(trained_network, final_loss, optimizer). The optimizer is the one passed in, or the Adam instance created internally when none was supplied; returning it always lets callers warm-start a later call by threading it back in.

Return type:

tuple

skagent.ann.aggregate_net_loss(inputs, df, loss_function)¶

Compute a loss function over a tensor of inputs, given a decision function df. Return the mean.

Parameters:: inputs (Grid)

Equilibrium Schedules and Methods¶

A schedule decides when each decision is solved, and it asks a method to carry out each solve. A method carries one algorithm together with the configuration that algorithm needs. The pairing is free: the same schedule accepts either a policy network or an exact backup, and it returns the same answer in either case.

skagent.solver.solve_in_order(method, order, policies=None)¶

Solve the named decisions, one at a time, in the order given.

This schedule takes its order from the caller rather than deriving one. Each decision is solved against the rules already in hand, so a symbol repeated in order, as in ["c", "d", "c"], is refined after its neighbours have moved. The result is a best-response sweep run by hand.

There is no convergence test here. The iteration stops because order ran out, and not because anything settled, so a repeated symbol buys a fixed number of refinement passes rather than a fixed point. Where a fixed point is wanted, use a schedule that measures one: solve_symmetric_equilibrium() iterates against a residual.

A decision absent from order is returned at its starting rule, which means that it has not been solved. That is the caller’s choice, since the caller writes the order, but the returned profile does not distinguish a solved rule from an unsolved one.

Parameters:

method (object) – A per-decision solver, as NeuralBestResponse, ExactBestResponse and skagent.algos.tabular.TabularBestResponseSolver are. Needs best_response(decision, policies) and, when policies is omitted, initial_policies().
order (sequence of str) – The decisions to solve, in order. Symbols may repeat.
policies (Mapping[str, Callable], optional) – The profile to start from, with a rule for every decision. Defaults to the method’s own starting profile.

Returns:

A decision rule per control of the block.

Return type:

dict

skagent.solver.solve_in_relevance_order(method, policies=None, *, max_iterations=25, tolerance=1e-06)¶

Solve every decision in relevance-component order.

This is a schedule rather than a method: it decides when each decision is solved, and it asks the method to carry out each solve. Acyclic components are solved once, after every decision rule they rely on has been computed. For a cyclic component, every decision’s best response is computed against the same previous profile and installed simultaneously. This repeats until every policy is within tolerance of its own response.

Parameters:

method (object) – A per-decision solver carrying the problem, as NeuralBestResponse and skagent.algos.tabular.TabularBestResponseSolver do. Needs ground, best_response(decision, policies), rule_distance(new, old, iset) and, when policies is omitted, initial_policies().
policies (Mapping[str, Callable], optional) – Starting rules for the decisions, replaced one by one as they are solved. Defaults to the method’s own starting profile.
max_iterations (int, optional) – Maximum number of simultaneous best-response updates for a cyclic relevance component. Must be at least 1. Defaults to 25.
tolerance (float, optional) – Maximum rule distance at convergence. Must be positive. Defaults to 1e-6.

Returns:

A decision rule per control of the block.

Return type:

dict

Raises:

NotImplementedError – If a cyclic component belongs to a recurring block.
RuntimeError – If a cyclic component does not converge within max_iterations.
ValueError – If max_iterations is less than 1 or tolerance is not positive.
TypeError – If max_iterations is not an integer.

skagent.solver.project(ground, actor_suffix='_actor', other_suffix='_other')¶

One instance’s problem, with the rest of its class beside it.

The entity class is split in two – the instance being solved, and the others – and every per-instance equation is copied once per side under a suffixed name. For each symbol that an aggregating equation reads over the class, exactly one equation is synthesized, and it concatenates the two sides back into the original symbol; the aggregating equation is then copied verbatim and reads that symbol. The projection therefore reassembles the entity axis without inspecting the reduction, so a mean, a sum, a maximum and a masked mean all project alike.

The two sides’ rewards are attributed to suffixed agent roles, so a solver told which agent it serves maximizes one instance’s payoff rather than the class’s total.

This first scope has two properties, and both are narrower than the transform has to remain:

The other instances share one broadcast rule. The projected block holds a single control for the whole remainder of the class, so the equilibrium sought is a symmetric one. A class of genuinely distinct rivals is expressible in this shape, but it is not built here.
The projected block declares no entity. The split lives in the shapes – the solved instance is a scalar, the others broadcast to N - 1 – rather than in two declarations, because the solvers refuse a block that declares an entity class at all, and re-keying that refusal is a separate decision.

Parameters:

ground (skagent.ground.GroundedBlock) – The population model and the calibration it is solved at. Its block must declare exactly one entity class of at least two instances, and the calibration must give that class’s size under its own name.
actor_suffix (str, optional) – How the two sides’ symbols are named.
other_suffix (str, optional) – How the two sides’ symbols are named.

Returns:

The projected problem, carrying two controls: the solved instance’s and the others’.

Return type:

skagent.ground.GroundedBlock

Raises:

ValueError – If the block does not declare exactly one entity class, if the calibration does not size it, or if it holds fewer than two instances.

skagent.solver.solve_symmetric_equilibrium(method, *, damping=1.0, tolerance=0.001, max_iterations=20, initial=None)¶

A symmetric equilibrium, by iterated best response over a projection.

Takes a projected problem – one instance’s decision beside the rest of its class, as project() builds – solves the instance’s decision against the others’ current rule, swaps the solved rule in as the others’, and repeats until the rule stops moving. A rule that is its own best response is an equilibrium of the projected game, and because the others play whatever the solved instance plays, it is a symmetric equilibrium of the population.

The method supplies both the per-decision solve and the distance between two rules, since only it knows how a rule is represented. The schedule supplies the damping, the residual test and the swap.

Damping is a correctness requirement rather than a convergence aid. Undamped iteration converges only where the best response is a contraction. Where its slope is -1 the iterates cycle between two points forever, and past that they diverge until the controls’ bounds catch them. Both failures return a plausible number under an iteration cap, which is why the residual here is measured on the rule and never on the iteration count.

Parameters:

method (NeuralBestResponse or ExactBestResponse) – The per-decision solver, carrying its own configuration and the projected problem it solves. Any object with ground, best_response(decision, policies) and rule_distance(new, old, iset) serves.
damping (float, optional) – How far to move toward the best response each round, in (0, 1]. Default 1.0, which is undamped.
tolerance (float, optional) – The rule is converged when it moves less than this.
max_iterations (int, optional) – Rounds before giving up. Reaching this is not convergence and is reported as such.
initial (Callable, optional) – The others’ rule on the first round. Defaults to a constant at the midpoint of the solved control’s declared bounds.

Returns:

rule (Callable) – The equilibrium decision rule, in the solved instance’s symbols.
info (dict) – converged, iterations and distances.

Raises:

ValueError – If the method’s block does not carry a projected pair of controls.

class skagent.solver.NeuralBestResponse(ground, panel, epochs=200, width=32)¶

Best responses by training a policy network, and what that needs.

A method object carries its own construction configuration beside its algorithm, so that a schedule can take any method without carrying every method’s arguments on its own signature. This one needs a training panel and an epoch count; the exact backup needs a state grid and a continuation instead, and neither needs the other’s.

Parameters:

ground (skagent.ground.GroundedBlock) – The problem being solved, already projected if it is a population.
panel (skagent.grid.Grid) – What the network trains on and what two rules are compared over. Must carry every shock of the block, since the loss evaluates the whole period.
epochs (int, optional) – Training epochs per best response.
width (int, optional) – Hidden width of the policy network.

best_response(decision, policies)¶: Train a network for decision, holding the rest of policies fixed.

initial_policies()¶: A starting profile: every decision at a constant, none of them solved.

rule_distance(new_rule, old_rule, iset)¶

Supremum norm between two rules, evaluated on the training panel.

A network has no cells to compare, so the comparison is over a common batch – which is why the distance is the method’s operation and not the schedule’s.

class skagent.solver.ExactBestResponse(ground, state_grid, scope=None, continuation=None, disc_params=None)¶

Best responses by exact backup over a state grid.

The method-object counterpart of NeuralBestResponse: same two operations, entirely different construction configuration.

Parameters:

ground (skagent.ground.GroundedBlock) – The problem being solved, already projected if it is a population.
state_grid (Mapping) – The grid the backup optimizes over, and where two rules are compared. An information-set variable must appear here rather than in scope, even as a single point, since a rule over it needs an axis to vary along.
scope (Mapping, optional) – Shocks pinned to a fixed realization. Defaults to the calibration.
continuation (Callable, optional) – The continuation value. Defaults to a terminal (zero) one, which is what makes the backup a single-period solve.
disc_params (Mapping, optional) – Per-shock discretization arguments for the shocks integrated inside the maximization.

best_response(decision, policies)¶: Back up decision alone, holding the rest of policies fixed.

initial_policies()¶: A starting profile: every decision at a constant, none of them solved.

rule_distance(new_rule, old_rule, iset)¶: Supremum norm between two rules over the grid they were solved on.

Grid and Computational Tools¶

Grid Class¶

class skagent.grid.Grid(labels, values, torched=True)¶

Bases: object

A class representing a labeled grid of numerical values.

Parameters:: (dict) (config) – dictionary with the following keys: “min” (float): The minimum value for the variable.; “max” (float): The maximum value for the variable; “count” (int): The number of points to generate for the variable.

classmethod from_config(config={}, torched=True)¶

classmethod from_dict(kv={}, torched=False)¶

len()¶: Returns the number of columns, similar to a dict.

n()¶: Returns the number of values for each symbol

shape()¶: Returns the shape of the grid values.

to_dict()¶: Returns a data structure, key: column, similar to tensordict or structured array.

torch()¶

update_from_dict(kv)¶

Grid Utility Functions¶

skagent.grid.make_grid(config)¶

Make a ‘grid’ of values based on the provided configuration.

Parameters:

(dict) (config) – dictionary with the following keys: “min” (float): The minimum value for the variable; “max” (float): The maximum value for the variable; “count” (int): The number of points to generate for the variable.

Returns:

numpy.ndarray (A NumPy array of shape (product_of_counts, num_variables), where) – product_of_counts is the product of all count values in the config dictionary, and num_variables is the number of keys in the config.

skagent.grid.cartesian_product(*arrays)¶

Create a Cartesian product of input arrays.

Parameters:

*arrays – Variable length arrays to compute product

Returns:

Array of shape (product_of_lengths, num_arrays)
where product_of_lengths is the product of the lengths of the input arrays,
and num_arrays is the number of input arrays. Each row contains one element
of the Cartesian product.