Algorithms¶

This section contains the API documentation for solution algorithms, neural network components, and grid tools used to solve dynamic stochastic optimization problems.

Value Function Iteration (VFI)¶

The value function iteration (VFI) algorithm derives arrival value functions from a continuation value function and the stage dynamics of model blocks.

Value function iteration (VFI).

Derive a decision rule and a gridded value function for a single BellmanPeriod by value function iteration: at each point of a grid over the period’s arrival states, solve an exact scipy.optimize.minimize() for the control that maximizes the period reward plus the discounted continuation value.

skagent.algos.vfi.AxisSpec¶

One coordinate vector per variable, whose cartesian product is the lattice a value array is tabulated over. Unrelated to skagent.grid.Grid, which is a batch of scattered points rather than a per-axis specification.

alias of Mapping[str, Sequence]

skagent.algos.vfi.ar_from_data(da, clip=None)¶

Build a decision rule from a fitted policy DataArray.

The returned rule follows the library’s decision-rule calling convention: it takes the control’s information-set values as positional arguments, in the order of da.dims (which solve() aligns to control.iset). This matches how block.transition invokes a rule, dr(*[vals[v] for v in iset]), so a VFI-fitted rule is a drop-in for the rest of the stack.

Between the grid’s points the rule interpolates linearly. Past the range the grid covers along a dimension it extends linearly from the outermost segment, so it answers every query rather than returning NaN, and clip is applied to those extended values. The rule’s off_grid attribute counts the queries it has answered that way, which is how a caller tells whether the rule was used where it was not solved.

Interpolation runs in numpy/xarray space. For a torch-tensor interface, wrap the result with tensor_decision_rule().

Parameters:

da (xarray.DataArray) – The fitted policy, with one dimension per information-set variable in control.iset order. A zero-dimensional array encodes a constant rule (empty information set). A dimension of length one encodes a rule constant along that variable: one grid point fixes a level and says nothing about a slope, so the rule takes the value and ignores the argument.
clip (callable, optional) – clip(values, *args), returning values held within the control’s bounds at the information-set values args, all arrays of one length. Applied to queries off the grid only, where the rule is extended rather than fitted. solve_step() supplies one from the control’s declared bounds.

Returns:

A rule ar(*args) taking one positional argument per dimension of da. Scalar arguments return a Python scalar; array-like arguments are interpolated pointwise (not as an outer product) and return a numpy array. Its off_grid attribute is the number of queries answered off the grid.

Return type:

callable

Raises:

TypeError – If the number of positional arguments does not match da.ndim.

skagent.algos.vfi.get_action_rule(action)¶

Build a constant decision rule that ignores its inputs.

Parameters:: action (Any) – The fixed value the rule returns.
Returns:: A zero-argument function ar() returning action. Used to wrap a candidate action as a decision rule during the per-point optimization.
Return type:: callable

skagent.algos.vfi.grid_to_data_array(grid={})¶

Construct a zero-valued DataArray over the coordinates of a grid.

Parameters:: grid (Mapping[str, Sequence]) – A mapping from variable labels to a sequence of numerical values. An empty mapping yields a zero-dimensional array.
Returns:: An array whose dimensions and coordinates are those of grid.
Return type:: xarray.DataArray

skagent.algos.vfi.numpy_decision_rule(torch_rule, dtype=None, device=None)¶

Wrap a torch-space decision rule so it speaks numpy.

The mirror of tensor_decision_rule(). It adapts a rule built in torch – ann.BlockPolicyNet.get_decision_rule, or a Stable-Baselines policy through skagent.algos.sb3 – for the numpy-space machinery: block.transition, TabularBestResponseSolver, and the sampled expectations in skagent.ground. A torch rule stacks its arguments into a tensor, so it cannot be handed numpy or scalars at all; the wrapper is what makes a trained policy scorable by that machinery.

Crossing the boundary severs the autograd graph in this direction too: the returned controls are detached and converted, so the result is valid as a fixed, ground-truth or warm-start policy and not as a trainable one.

Parameters:

torch_rule (callable) – A torch-space decision rule taking positional information-set arguments and returning a torch tensor.
dtype (torch.dtype, optional) – Dtype the arguments are converted to. Defaults to torch.float32, the dtype grid.py builds its tensors with and the networks are trained in. The conversion costs precision, so a numpy computation that is exact in double – a quadrature rule against a polynomial integrand, say – agrees with itself only to single precision once a wrapped rule is in it. Pass torch.float64 where that matters and the rule tolerates it.
device (torch.device, optional) – Device the arguments are placed on. Defaults to the stack’s device (skagent.grid.device).

Returns:

A rule ndr(*args) accepting numpy arrays or scalars (or torch tensors) and returning a numpy array.

Return type:

callable

skagent.algos.vfi.solve_bellman(bp, state_grid, *, continuation_vf=None, agent=None, scope={}, disc_params={}, tol=1e-06, max_iter=100, x0=1.0, raise_on_nonconvergence=False, confine_to_grid=False, search='multistart', policy_evaluations=0, artificial_borrowing_constraint=None)¶

Solve a recurring BellmanPeriod by value-function iteration.

Iterates solve_step() to a fixed point: each backup uses the previous iterate’s value grid as its continuation (rebuilt via value_array_to_function()) and offers the previous iterate’s policy_array as a per-point warm start — one multi-start candidate among the others, so a collapsed iterate cannot entrench itself by seeding the next backup at its own bound. It stops when the sup-norm change in the value grid falls below tol, or after max_iter iterations.

Iteration 1 uses the terminal (zero) continuation, so solve_bellman(..., max_iter=1) reproduces solve_step() under a terminal continuation. For an infinite-horizon problem the loop converges geometrically (modulus the discount factor) to the stationary solution; for a finite horizon of length T set max_iter=T.

The stopping rule assumes an infinite horizon. A finite-horizon problem is structurally indistinguishable from an infinite-horizon one, so a run that reaches max_iter is reported as non-convergence in both cases: at a caller-specified horizon that report is expected rather than a failure, and the residual is a sup-norm change between iterates, not an error against the T-period answer.

A period with no arrival states is not a dynamic problem — nothing carries between periods — and is refused. Such a block is solved statically, by skagent.algos.tabular.TabularBestResponseSolver or skagent.solver.NeuralBestResponse, driven by skagent.solver.solve_in_order(). Every arrival state must be an axis of state_grid, since the continuation is rebuilt from the value grid and cannot represent dependence on a variable the grid has no axis for.

Shocks are discretized internally: disc_params is threaded into every backup (hidden shocks integrated inside the max) and into value_array_to_function() (observed-shock axes integrated into the arrival value between iterations).

Parameters:

bp (BellmanPeriod) – The recurring period providing the model mechanics.
state_grid (Mapping[str, Sequence]) – A grid over the value function’s domain (arrival states and/or observed shocks); see solve_step().
continuation_vf (Optional[Callable]) – Initial continuation guess continuation_vf(states, shocks, parameters). Defaults to the terminal (zero) continuation.
agent (str | None) – If given, the period reward sums only this agent’s reward symbols.
scope (Mapping) – Fixed non-shock exogenous values (and, in this scope, any hidden-shock realization) merged into the model parameters.
disc_params (Mapping) – Per-shock discretization arguments, threaded into each backup (for hidden shocks) and into value_array_to_function() (for observed shocks); see solve_step().
tol (float) – Convergence tolerance on the sup-norm change in the value grid.
max_iter (int) – Maximum number of backups.
x0 (float) – Modest multi-start seed candidate passed to solve_step().
raise_on_nonconvergence (bool) – If True, raise RuntimeError when the loop hits max_iter with the sup-norm change still above tol; otherwise emit a warnings.warn and return the last iterate (the scipy OptimizeResult.success convention, O5). At a finite horizon set through max_iter this condition is expected, so leave it False.
confine_to_grid (bool) – Forwarded to solve_step(): confine next-period arrival states to the state grid’s range, so the rebuilt continuation is never extrapolated off-grid.
search (str) – Forwarded to solve_step(): how each point’s optimum is found. The warm start is one of the seeds either way.
policy_evaluations (int) – How many times to apply the Bellman operator at the maximizing policy after each backup, before the next one (modified policy iteration). Default 0, plain value iteration. Each application costs a fraction of a backup and moves the value grid toward the policy’s own value, so the loop reaches tol in fewer backups. The residual is then the change across a backup, which the backup’s own optimization error bounds below, so tol must not be tighter than that error. For an infinite horizon only: under a horizon set through max_iter the applications lengthen the horizon, and a horizon carried in the state already converges in a backup per period.
artificial_borrowing_constraint (bool | None) – Deprecated alias of confine_to_grid.

Return type:

tuple[dict[str, Callable], DataArray, dict[str, DataArray]]

Returns:

dr_from_data (dict of callable) – One decision rule per control at the fixed point (see solve_step()).
value_array (xarray.DataArray) – The converged value grid. Its attrs carry n_iter, converged (bool), and residual (the final sup-norm change).
policy_array (dict of xarray.DataArray) – The gridded optimal control(s) at the fixed point.

Raises:

ValueError – If max_iter is less than 1, if policy_evaluations is negative, or if the period has no arrival states.
RuntimeError – If raise_on_nonconvergence is True and the loop does not converge.

Warns:

UserWarning – If the period names no discount variable, which solves it undiscounted.
DeprecationWarning – If artificial_borrowing_constraint is given.

skagent.algos.vfi.solve_step(bp, continuation_vf, state_grid, *, agent=None, control=None, scope={}, disc_params={}, decision_rules=None, x0=1.0, x0_policy=None, confine_to_grid=False, search='multistart', artificial_borrowing_constraint=None)¶

One exact value backup over state_grid.

At each grid point the optimal control is found with scipy.optimize.minimize(), maximizing the period reward plus the discounted continuation value of the resulting arrival states. Under a terminal (zero) continuation, continuation_vf = lambda s, sh, p: 0.0, the result is the single-step solution; the value-iteration wrapper solve_bellman() iterates it to a fixed point.

Applies to static and dynamic periods alike, which is why it is not named for Bellman: a period naming no discount variable discounts by 1.0, and a static block has no arrival states for a continuation to read, so the backup reduces to maximizing the period reward.

Speaks the BellmanPeriod protocol the rest of the torch stack uses, with an explicit discount factor and multi-reward summation, and is empty-shock-safe.

Shocks are handled by their information role, which is derived from the block’s own structure (skagent.relevance) rather than inferred from the grid the caller supplies. A shock some control’s information set accounts for becomes a grid axis over its discretization nodes, so its pre-state and bounds are computed per realization; the rest are integrated out inside the per-point max via internal discretization (_discretize_shocks() + expected), so an optimum characterized by an expectation over an unobserved shock is in scope. Any shock may instead be pinned to a fixed realization by supplying it in scope, which takes precedence over both.

Note

Current scope: one or more controls, jointly optimized by scipy.optimize.minimize() over the stacked control vector with per-control bounds — restarted from each seed candidate, keeping the best optimum, or by one bounded scalar search under search="bounded" — each policy then reprojected onto its own information set (_project_to_iset()). A control’s pre-state and bounds are evaluated with each integrated-out shock fixed at its (discretized) mean, since a single value is required there even though the objective integrates the shock. That is exact when the pre-state does not depend on such a shock, and a shock the pre-state does depend on is a grid axis rather than an integrated one, so it does not arise for a well-formed block.

Parameters:

bp (BellmanPeriod) – The recurring period providing the model mechanics.
continuation_vf (Callable) – The continuation value function, called continuation_vf(states, shocks, parameters) on the next-period arrival states (the bp.compute_value convention). Terminal continuation is lambda s, sh, p: 0.0.
state_grid (Mapping[str, Sequence]) – The shared backup grid over arrival states: one axis per variable. Axes for the shocks an information set accounts for are added automatically from their discretization nodes, so supplying one is optional and idempotent. This grid covers the variables the Bellman loop iterates over and is not necessarily equal to any individual control’s information set (a control’s iset may be a strict subset). For an empty grid, pass {} — the empty mapping, not skagent.grid.Grid, whose from_config({}) raises.
agent (str | None) – If given, the period reward sums only this agent’s reward symbols.
scope (Mapping) – Fixed non-shock exogenous values merged into the model parameters. A shock supplied here is pinned to that fixed realization instead of being gridded or integrated. Note this fixes a value for the solve; it does not make the symbol a model parameter, so it does not affect any shock’s information role.
decision_rules (Optional[Mapping[str, Callable]]) – Rules for the controls not being optimized here, which is what reduces a multi-control block to one control’s problem. Supplied per call, since which rules the others are held at is a property of the question rather than of the model; the controls being optimized are pinned to their trial values and ignore any rule given for them.
disc_params (Mapping) – Per-shock discretization arguments, keyed by shock symbol (e.g. {"theta": {"N": 7}}), forwarded to that shock’s Distribution.discretize whether it becomes a grid axis or is integrated out. A shock without an entry uses its distribution’s default discretization (exact for already-discrete shocks).
x0 (float) – A modest optimizer seed, clamped into each control’s bounds. One of the multi-start candidates at every point, not just a fallback: it is what covers a box whose midpoint is far from the optimum, e.g. the natural borrowing limit’s [0, m + H].
x0_policy (Optional[Mapping[str, DataArray]]) – Warm-start seeds keyed by control symbol (e.g. a previous iterate’s policy_array); supplies the first multi-start candidate at each grid point, and wins ties. Supplied by solve_bellman().
confine_to_grid (bool) – When True, restrict each control to the values that keep every next-period arrival state within the state grid’s range. This keeps the continuation interpolated rather than extrapolated past the grid edges, so value iteration cannot ride a control bound by over-crediting off-grid successors. It adds a constraint to the problem solved, which must be slack at the states of interest, or it biases the policy where it binds; in a savings model the grid’s lower edge acts as an artificial borrowing limit. Single control with an affine successor only (raises otherwise).
search (str) – How each point’s optimum is found. "multistart", the default, runs scipy.optimize.minimize() from every seed candidate and keeps the best. "bounded" runs one bounded scalar search over the control’s whole interval (scipy.optimize.minimize_scalar()) and checks both endpoints, then evaluates the objective at each seed candidate, which is what the identification check compares against. It uses no seed, and is exact only for an objective with a single peak on the interval. It applies to a single control whose block declares both its lower and its upper bound.
artificial_borrowing_constraint (bool | None) – Deprecated alias of confine_to_grid.
control (str | None)

Return type:

tuple[dict[str, Callable], DataArray, dict[str, DataArray]]

Returns:

dr_from_data (dict of callable) – One decision rule per control, keyed by control symbol; each takes its information-set values as positional arguments in control.iset order.
value_array (xarray.DataArray) – The gridded optimized decision value over the state grid.
policy_array (dict of xarray.DataArray) – The gridded optimal control(s) over the state grid, keyed by control symbol (a dict for forward-compatibility with multi-control, O1).

Raises:

ValueError – If search is not one of the two methods, or is "bounded" for more than one control or for a control missing a declared bound.

Warns:

DeprecationWarning – If artificial_borrowing_constraint is given.

skagent.algos.vfi.tensor_decision_rule(np_rule, dtype=None, device=None)¶

Wrap a numpy-space decision rule so it speaks torch tensors.

Adapts a rule (e.g. from ar_from_data(), including the rules returned by solve()) for interop with the torch solving stack (BellmanPeriod / loss / solver). Inputs may be torch tensors or numpy/scalars; outputs are torch tensors.

Because interpolation runs in numpy, the autograd graph is severed: the returned controls are detached. This rule is therefore valid as a fixed / ground-truth / warm-start policy (e.g. an other_dr or in a value-residual loss), but not as a trainable policy in a loss that differentiates through the control (FOC-weighted Bellman, Euler).

Parameters:

np_rule (callable) – A numpy-space decision rule taking positional information-set arguments and returning numpy/scalar values.
dtype (torch.dtype, optional) – Output tensor dtype. Defaults to torch.float32 to match grid tensors (grid.py builds them with torch.FloatTensor).
device (torch.device, optional) – Output tensor device. Defaults to the stack’s device (skagent.grid.device).

Returns:

A rule tdr(*args) accepting torch tensors (or numpy/scalars) and returning a detached torch tensor on device with dtype dtype.

Return type:

callable

skagent.algos.vfi.value_array_to_function(value_array, bp, disc_params={})¶

Rebuild a continuation value function from an iterate’s value grid.

solve_bellman() feeds iteration n’s value grid back as iteration (n+1)’s continuation. This wraps the grid as a callable in the bp.compute_value convention wf(states, shocks, parameters), so it drops straight into solve_step()’s continuation_vf slot.

The decision value grid ranges over arrival states and any observed-shock axes. Those shock axes are first integrated out into the arrival value W(s) = E_obs[V(s, obs)], using the weights of the same discretized distribution that produced the axis nodes; a grid built from other node values raises ValueError. wf then interpolates linearly over the remaining arrival-state axes and extrapolates linearly past the grid edges, as scipy.interpolate.RegularGridInterpolator does, so an off-grid next-period state during the backup gets a finite, sloped continuation rather than NaN (which breaks the optimizer) or a flat boundary clamp (which zeroes the marginal value of saving and collapses the policy onto its bound).

When the value grid has no observed-shock axes (the deterministic and hidden-shock-only cases, where the backup already integrated any hidden shocks), the expectation step is a no-op and the value grid over arrival states is W.

wf reads only the axes it has, so it raises ValueError rather than silently discard an arrival state that the value grid has no axis for — with D-3’s survival state liv left off the grid, the mortality discount cancels out of the backup and the loop converges to the no-mortality policy, a plausible value for a different model.

Parameters:

value_array (DataArray) – A gridded decision value function, e.g. the value_array returned by solve_step(); its axes are arrival states and any observed-shock nodes.
bp (BellmanPeriod) – The recurring period; used to identify and discretize the shock axes.
disc_params (Mapping) – Per-shock discretization arguments for the observed-shock axes, keyed by shock symbol. A shock axis without an entry uses its distribution’s default discretization.

Returns:

wf(states, shocks, parameters) returning the interpolated arrival value at states. shocks and parameters are accepted for the bp.compute_value calling convention but unused.

Return type:

Callable

Core VFI Functions¶

One exact value backup over state_grid.

Speaks the BellmanPeriod protocol the rest of the torch stack uses, with an explicit discount factor and multi-reward summation, and is empty-shock-safe.

Note

Parameters:

bp (BellmanPeriod) – The recurring period providing the model mechanics.
continuation_vf (Callable) – The continuation value function, called continuation_vf(states, shocks, parameters) on the next-period arrival states (the bp.compute_value convention). Terminal continuation is lambda s, sh, p: 0.0.
state_grid (Mapping[str, Sequence]) – The shared backup grid over arrival states: one axis per variable. Axes for the shocks an information set accounts for are added automatically from their discretization nodes, so supplying one is optional and idempotent. This grid covers the variables the Bellman loop iterates over and is not necessarily equal to any individual control’s information set (a control’s iset may be a strict subset). For an empty grid, pass {} — the empty mapping, not skagent.grid.Grid, whose from_config({}) raises.
agent (str | None) – If given, the period reward sums only this agent’s reward symbols.
scope (Mapping) – Fixed non-shock exogenous values merged into the model parameters. A shock supplied here is pinned to that fixed realization instead of being gridded or integrated. Note this fixes a value for the solve; it does not make the symbol a model parameter, so it does not affect any shock’s information role.
decision_rules (Optional[Mapping[str, Callable]]) – Rules for the controls not being optimized here, which is what reduces a multi-control block to one control’s problem. Supplied per call, since which rules the others are held at is a property of the question rather than of the model; the controls being optimized are pinned to their trial values and ignore any rule given for them.
disc_params (Mapping) – Per-shock discretization arguments, keyed by shock symbol (e.g. {"theta": {"N": 7}}), forwarded to that shock’s Distribution.discretize whether it becomes a grid axis or is integrated out. A shock without an entry uses its distribution’s default discretization (exact for already-discrete shocks).
x0 (float) – A modest optimizer seed, clamped into each control’s bounds. One of the multi-start candidates at every point, not just a fallback: it is what covers a box whose midpoint is far from the optimum, e.g. the natural borrowing limit’s [0, m + H].
x0_policy (Optional[Mapping[str, DataArray]]) – Warm-start seeds keyed by control symbol (e.g. a previous iterate’s policy_array); supplies the first multi-start candidate at each grid point, and wins ties. Supplied by solve_bellman().
confine_to_grid (bool) – When True, restrict each control to the values that keep every next-period arrival state within the state grid’s range. This keeps the continuation interpolated rather than extrapolated past the grid edges, so value iteration cannot ride a control bound by over-crediting off-grid successors. It adds a constraint to the problem solved, which must be slack at the states of interest, or it biases the policy where it binds; in a savings model the grid’s lower edge acts as an artificial borrowing limit. Single control with an affine successor only (raises otherwise).
search (str) – How each point’s optimum is found. "multistart", the default, runs scipy.optimize.minimize() from every seed candidate and keeps the best. "bounded" runs one bounded scalar search over the control’s whole interval (scipy.optimize.minimize_scalar()) and checks both endpoints, then evaluates the objective at each seed candidate, which is what the identification check compares against. It uses no seed, and is exact only for an objective with a single peak on the interval. It applies to a single control whose block declares both its lower and its upper bound.
artificial_borrowing_constraint (bool | None) – Deprecated alias of confine_to_grid.
control (str | None)

Return type:

tuple[dict[str, Callable], DataArray, dict[str, DataArray]]

Returns:

dr_from_data (dict of callable) – One decision rule per control, keyed by control symbol; each takes its information-set values as positional arguments in control.iset order.
value_array (xarray.DataArray) – The gridded optimized decision value over the state grid.
policy_array (dict of xarray.DataArray) – The gridded optimal control(s) over the state grid, keyed by control symbol (a dict for forward-compatibility with multi-control, O1).

Raises:

ValueError – If search is not one of the two methods, or is "bounded" for more than one control or for a control missing a declared bound.

Warns:

DeprecationWarning – If artificial_borrowing_constraint is given.

Solve a recurring BellmanPeriod by value-function iteration.

Parameters:

bp (BellmanPeriod) – The recurring period providing the model mechanics.
state_grid (Mapping[str, Sequence]) – A grid over the value function’s domain (arrival states and/or observed shocks); see solve_step().
continuation_vf (Optional[Callable]) – Initial continuation guess continuation_vf(states, shocks, parameters). Defaults to the terminal (zero) continuation.
agent (str | None) – If given, the period reward sums only this agent’s reward symbols.
scope (Mapping) – Fixed non-shock exogenous values (and, in this scope, any hidden-shock realization) merged into the model parameters.
disc_params (Mapping) – Per-shock discretization arguments, threaded into each backup (for hidden shocks) and into value_array_to_function() (for observed shocks); see solve_step().
tol (float) – Convergence tolerance on the sup-norm change in the value grid.
max_iter (int) – Maximum number of backups.
x0 (float) – Modest multi-start seed candidate passed to solve_step().
raise_on_nonconvergence (bool) – If True, raise RuntimeError when the loop hits max_iter with the sup-norm change still above tol; otherwise emit a warnings.warn and return the last iterate (the scipy OptimizeResult.success convention, O5). At a finite horizon set through max_iter this condition is expected, so leave it False.
confine_to_grid (bool) – Forwarded to solve_step(): confine next-period arrival states to the state grid’s range, so the rebuilt continuation is never extrapolated off-grid.
search (str) – Forwarded to solve_step(): how each point’s optimum is found. The warm start is one of the seeds either way.
policy_evaluations (int) – How many times to apply the Bellman operator at the maximizing policy after each backup, before the next one (modified policy iteration). Default 0, plain value iteration. Each application costs a fraction of a backup and moves the value grid toward the policy’s own value, so the loop reaches tol in fewer backups. The residual is then the change across a backup, which the backup’s own optimization error bounds below, so tol must not be tighter than that error. For an infinite horizon only: under a horizon set through max_iter the applications lengthen the horizon, and a horizon carried in the state already converges in a backup per period.
artificial_borrowing_constraint (bool | None) – Deprecated alias of confine_to_grid.

Return type:

tuple[dict[str, Callable], DataArray, dict[str, DataArray]]

Returns:

dr_from_data (dict of callable) – One decision rule per control at the fixed point (see solve_step()).
value_array (xarray.DataArray) – The converged value grid. Its attrs carry n_iter, converged (bool), and residual (the final sup-norm change).
policy_array (dict of xarray.DataArray) – The gridded optimal control(s) at the fixed point.

Raises:

ValueError – If max_iter is less than 1, if policy_evaluations is negative, or if the period has no arrival states.
RuntimeError – If raise_on_nonconvergence is True and the loop does not converge.

Warns:

UserWarning – If the period names no discount variable, which solves it undiscounted.
DeprecationWarning – If artificial_borrowing_constraint is given.

skagent.algos.vfi.get_action_rule(action)

Build a constant decision rule that ignores its inputs.

Parameters:: action (Any) – The fixed value the rule returns.
Returns:: A zero-argument function ar() returning action. Used to wrap a candidate action as a decision rule during the per-point optimization.
Return type:: callable

skagent.algos.vfi.ar_from_data(da, clip=None)

Build a decision rule from a fitted policy DataArray.

Interpolation runs in numpy/xarray space. For a torch-tensor interface, wrap the result with tensor_decision_rule().

Parameters:

da (xarray.DataArray) – The fitted policy, with one dimension per information-set variable in control.iset order. A zero-dimensional array encodes a constant rule (empty information set). A dimension of length one encodes a rule constant along that variable: one grid point fixes a level and says nothing about a slope, so the rule takes the value and ignores the argument.
clip (callable, optional) – clip(values, *args), returning values held within the control’s bounds at the information-set values args, all arrays of one length. Applied to queries off the grid only, where the rule is extended rather than fitted. solve_step() supplies one from the control’s declared bounds.

Returns:

Return type:

callable

Raises:

TypeError – If the number of positional arguments does not match da.ndim.

skagent.algos.vfi.grid_to_data_array(grid={})

Construct a zero-valued DataArray over the coordinates of a grid.

Parameters:: grid (Mapping[str, Sequence]) – A mapping from variable labels to a sequence of numerical values. An empty mapping yields a zero-dimensional array.
Returns:: An array whose dimensions and coordinates are those of grid.
Return type:: xarray.DataArray

Tabular Best Response¶

The tabular best-response solver solves one decision at a time from a tabulated payoff table. For every value of what the decision-maker observes, it reports the action that maximizes the payoff of the agent making that decision, conditional on the observation and on a supplied rule for every other decision. The order in which the decisions are solved belongs to a schedule rather than to the method. skagent.solver.solve_in_relevance_order() supplies one such schedule, which takes components from the block’s relevance graph (see Model Analysis and Visualization). Acyclic components are solved once. Cyclic components use simultaneous best-response iteration and either reach a pure-strategy fixed point or report non-convergence. Mixed-strategy equilibria and recurring cyclic games are not supported.

Best responses from a tabulated payoff table.

The solver here handles one decision at a time. For every value of what the decision-maker observes, it reports the action that maximizes their expected payoff, conditional on that observation and on a supplied rule for every other decision. The order in which the decisions are solved, and whether the result is iterated to a fixed point, belong to a schedule rather than to this module; skagent.solver.solve_in_relevance_order() is the sweep this module used to carry.

Expectations are estimated by drawing shock_samples realizations of the block’s shocks once, at construction, and reusing the same draws for every candidate action (common random numbers), so that comparisons between actions carry far less error than their levels do. Conditioning is done by grouping the simulated samples, which makes the beliefs at a decision the posterior induced by the other decision rules in the profile.

The representation is tabular, which sets the limits of this module: a decision’s information set must take finitely many values under the profile being played, since a cell with no repeats has no conditional expectation to estimate, and the actions searched are a finite set. Shocks may be continuous – they are integrated over, never conditioned on. For continuous observations, or where a differentiable policy is wanted, use skagent.solver.NeuralBestResponse in the same schedule.

class skagent.algos.tabular.ConditionalPayoffs(cells, counts, actions, payoff)¶

Expected payoffs by information cell and action.

cellsnumpy.ndarray: Shape (n_cells, len(iset)); the distinct observed values of the decision’s information set. Shape (1, 0) for an empty information set.
countsnumpy.ndarray: Shape (n_cells,); how many samples support each cell.
actionsnumpy.ndarray: Shape (n_actions,); the candidate actions searched.
payoffnumpy.ndarray: Shape (n_cells, n_actions); the estimated expected payoff of the decision’s agent in each cell for each action.

actions¶: Alias for field number 2

cells¶: Alias for field number 0

counts¶: Alias for field number 1

payoff¶: Alias for field number 3

class skagent.algos.tabular.TabularBestResponseSolver(ground, *, actions=None, action_count=21, shock_samples=100000, rng=None, max_cells=1024)¶

Solve a block’s decisions by best response, in relevance-component order.

Parameters:

ground (GroundedBlock) – The block-and-calibration pair to solve. The block’s dynamics must be declared in topological order, as the library requires of any block, and its controls must carry the agent attribution needed to tell whose payoff each maximizes when the block’s utilities are owned by more than one agent. The calibration supplies both the arguments the block’s shocks are constructed from and the values any parameter the dynamics refer to.
actions (array_like, optional) – Candidate actions to search for controls without a declared action_space, shared by those decisions. Defaults to action_count points spanning [0, 1]. A control’s declared action space takes precedence.
action_count (int, optional) – Number of candidate actions when actions is not given.
shock_samples (int, optional) – Number of shock realizations drawn at construction. This is the axis the expectations are estimated over – the integral over the block’s declared shocks that a decision’s objective is defined by, taken by Monte Carlo rather than by quadrature. Grouping splits these across a decision’s information cells, so each cell’s expectation rests on the samples that reached it rather than on all of them.
rng (numpy.random.Generator, optional) – Generator for the shock draws, overriding the one ground carries. When neither supplies a generator, an unseeded one is used.
max_cells (int, optional) – Upper limit on the number of information cells a single decision may have. Exceeding it raises, since grouping samples by observed value only estimates a conditional expectation when observations repeat.

Notes

Constructing the solver draws the block’s shocks from ground. The block itself is left as its author wrote it, so one block may be solved at several calibrations by grounding it against each.

Expectations are Monte Carlo estimates; a solved rule is exact only up to sampling error, which falls as shock_samples rises.

best_response(decision, policies)¶

The payoff-maximizing rule for decision against policies.

Parameters:

decision (str) – The control to solve.
policies (Mapping[str, Callable]) – A decision rule for every control of the block; the profile the best response is computed against.

Returns:

One action per information cell.

Return type:

TabulatedRule

conditional_payoffs(decision, policies)¶

Estimate decision’s payoffs by information cell and action.

Parameters:

decision (str) – The control to evaluate.
policies (Mapping[str, Callable]) – A decision rule for every control of the block, including decision itself: the profile the expectation is taken under.

Return type:

ConditionalPayoffs

initial_policies()¶: A spread rule for every decision in the block.

rule_distance(new_rule, old_rule, iset)¶: Largest action difference over the new tabulated rule’s cells.

spread_rule(weights=None)¶

A rule spreading the candidate actions across the samples.

This rule is held by the decisions that are not yet solved. Conditional payoffs are estimated by grouping simulated samples, so a placeholder playing one constant action would induce only the observations that action produces. Every cell reachable only under another action would then be empty, and its conditional expectation undefined. Spreading the actions over the sample axis is what keeps every cell populated.

This is coverage, not randomization: the assignment is deterministic and the sample axis holds shock draws rather than repeated plays, so the share of samples playing an action is not a probability of playing it.

Parameters:: weights (array_like, optional) – Relative shares of the samples per action, one per candidate action. Defaults to equal shares.
Returns:: A decision rule returning one action per sample.
Return type:: callable

class skagent.algos.tabular.TabulatedRule(iset, cells, actions)¶

A decision rule tabulated over the cells of an information set.

Follows the library’s decision-rule calling convention: the information-set values are passed as positional arguments in iset order. An observation is answered with the action of the nearest tabulated cell, so a rule remains total when queried away from the values it was tabulated on.

For a rule tabulated on a full grid, and where interpolation between grid points is wanted, use skagent.algos.vfi.ar_from_data() instead.

Parameters:

iset (sequence of str) – The information set, in the order the cell columns are given.
cells (array_like) – Shape (n_cells, len(iset)); the tabulated observations.
actions (array_like) – Shape (n_cells,); the action for each cell.

to_dict(decimals=3)¶: The rule as {observed values: action}, rounded for display.

Core Best-Response Classes¶

class skagent.algos.tabular.TabularBestResponseSolver(ground, *, actions=None, action_count=21, shock_samples=100000, rng=None, max_cells=1024)

Solve a block’s decisions by best response, in relevance-component order.

Parameters:

ground (GroundedBlock) – The block-and-calibration pair to solve. The block’s dynamics must be declared in topological order, as the library requires of any block, and its controls must carry the agent attribution needed to tell whose payoff each maximizes when the block’s utilities are owned by more than one agent. The calibration supplies both the arguments the block’s shocks are constructed from and the values any parameter the dynamics refer to.
actions (array_like, optional) – Candidate actions to search for controls without a declared action_space, shared by those decisions. Defaults to action_count points spanning [0, 1]. A control’s declared action space takes precedence.
action_count (int, optional) – Number of candidate actions when actions is not given.
shock_samples (int, optional) – Number of shock realizations drawn at construction. This is the axis the expectations are estimated over – the integral over the block’s declared shocks that a decision’s objective is defined by, taken by Monte Carlo rather than by quadrature. Grouping splits these across a decision’s information cells, so each cell’s expectation rests on the samples that reached it rather than on all of them.
rng (numpy.random.Generator, optional) – Generator for the shock draws, overriding the one ground carries. When neither supplies a generator, an unseeded one is used.
max_cells (int, optional) – Upper limit on the number of information cells a single decision may have. Exceeding it raises, since grouping samples by observed value only estimates a conditional expectation when observations repeat.

Notes

Constructing the solver draws the block’s shocks from ground. The block itself is left as its author wrote it, so one block may be solved at several calibrations by grounding it against each.

Expectations are Monte Carlo estimates; a solved rule is exact only up to sampling error, which falls as shock_samples rises.

best_response(decision, policies)

The payoff-maximizing rule for decision against policies.

Parameters:

decision (str) – The control to solve.
policies (Mapping[str, Callable]) – A decision rule for every control of the block; the profile the best response is computed against.

Returns:

One action per information cell.

Return type:

TabulatedRule

conditional_payoffs(decision, policies)

Estimate decision’s payoffs by information cell and action.

Parameters:

decision (str) – The control to evaluate.
policies (Mapping[str, Callable]) – A decision rule for every control of the block, including decision itself: the profile the expectation is taken under.

Return type:

ConditionalPayoffs

initial_policies(): A spread rule for every decision in the block.

rule_distance(new_rule, old_rule, iset): Largest action difference over the new tabulated rule’s cells.

spread_rule(weights=None)

A rule spreading the candidate actions across the samples.

Parameters:: weights (array_like, optional) – Relative shares of the samples per action, one per candidate action. Defaults to equal shares.
Returns:: A decision rule returning one action per sample.
Return type:: callable

class skagent.algos.tabular.TabulatedRule(iset, cells, actions)

A decision rule tabulated over the cells of an information set.

For a rule tabulated on a full grid, and where interpolation between grid points is wanted, use skagent.algos.vfi.ar_from_data() instead.

Parameters:

iset (sequence of str) – The information set, in the order the cell columns are given.
cells (array_like) – Shape (n_cells, len(iset)); the tabulated observations.
actions (array_like) – Shape (n_cells,); the action for each cell.

to_dict(decimals=3): The rule as {observed values: action}, rounded for display.

Maliar-Style Algorithms¶

This module provides neural network-based solution methods following Maliar et al.

Tools for the implementation of the Maliar, Maliar, and Winant (JME ‘21) method.

This method relies on a simpler problem representation than that elaborated by the skagent Block system.

Note

Shock draws here are Monte Carlo only; structured draws (e.g. exact discretizations) could be supported via BellmanPeriod.

skagent.algos.maliar.generate_givens_from_states(states, bellman_period, shock_copies)¶

Generate omega_i values of the MMW JME ‘21 method.

Parameters:

states (Grid) – A grid of starting state values (exogenous and endogenous).
bellman_period (BellmanPeriod) – The period whose shocks are drawn. It resolves them against its own calibration, so the draws follow its generator.
shock_copies (int) – Number of copies of the shocks to be included. Must be >= 1.

Returns:

Grid containing states augmented with shock copies.

Return type:

Grid

skagent.algos.maliar.maliar_training_loop(bellman_period, loss_function, states_0_n, parameters, shock_copies=2, max_iterations=5, tolerance=1e-06, random_seed=None, simulation_steps=1, network_width=16, epochs_per_iteration=250, lr=0.001)¶

Run the Maliar, Maliar, and Winant (JME ‘21) training loop.

Trains a single neural network policy to minimize empirical risk (loss) on a panel of states drawn forward through the model dynamics. This helper constructs and trains a BlockPolicyNet internally and does not currently accept a pre-built shared-backbone BlockPolicyValueNet. If value-aware training is needed (e.g. for a Bellman residual loss with a value head), call train_block_nn() directly on a BlockPolicyValueNet; a future refactor may add value-network support here.

The loop maps onto the MMW JME’21 algorithm steps as follows: _validate_training_inputs() and the network construction below cover Step 1 (initialize topology and coefficients); the per-iteration train_block_nn() call is Step 2 (minimize the empirical risk \(\Xi^n(\theta)\)); the returned network is the Step 3 trained approximation \(\varphi(\cdot, \theta)\).

Parameters:

bellman_period (BellmanPeriod) – A model definition containing block dynamics and transitions.
loss_function (Callable) – The empirical risk function \(\Xi^n\) from MMW JME’21. This function is passed to the neural network training routine as loss_function(decision_function, input_grid) -> loss_tensor.
states_0_n (Grid) – A panel of starting states for training. Must contain at least one state.
parameters (dict) – Given parameters for the model.
shock_copies (int) – Number of shock copies to include in the training set \(\{\omega_i\}\). Must match the expected number of shock copies in the loss function. Must be >= 1. Default is 2.
max_iterations (int) – Maximum number of training loop iterations before stopping. Must be >= 1. Default is 5.
tolerance (float) – Convergence tolerance. Training stops when either the L2 norm of parameter changes or the absolute difference in loss is below this threshold. Satisfying either criterion alone is sufficient. Must be > 0. Default is 1e-6.
random_seed (Optional[int]) – Random seed for reproducibility. Default is None.
simulation_steps (int) – Number of time steps to simulate forward when determining the next training set \(\{\omega_i\}\). Higher values let the training states explore more of the state space at higher computational cost. Must be >= 1. Default is 1.
network_width (int) – Width of hidden layers in the policy neural network. Must be >= 1. Default is 16.
epochs_per_iteration (int) – Number of training epochs per iteration. Must be >= 1. Default is 250.
lr (float) – Learning rate for the internal Adam optimizer. The optimizer is created once and reused across iterations to preserve momentum. Must be > 0. Default is 0.001.

Returns:

(trained_policy_network, training_states) where trained_policy_network is the trained BlockPolicyNet and training_states is the Grid of states from the final iteration (the convergence point if training converged early, otherwise the states after max_iterations steps).

Return type:

tuple

Raises:

ValueError – If max_iterations < 1, tolerance <= 0, shock_copies < 1, simulation_steps < 1, network_width < 1, epochs_per_iteration < 1, lr <= 0, or states_0_n contains no states.
TypeError – If bellman_period or parameters is None, loss_function is not callable, a count argument is not an integer, or states_0_n is not a Grid.
RuntimeError – If training diverges: the loss becomes non-finite at some iteration.

skagent.algos.maliar.simulate_forward(states_t, bellman_period, decision_function, parameters, big_t)¶

Simulate the model forward for a specified number of periods.

Parameters:

states_t (Grid | dict) – Initial state values.
bellman_period (BellmanPeriod) – The Bellman period containing model dynamics.
decision_function (Callable) – Function mapping (states, shocks, parameters) to controls.
parameters (dict) – Model parameters.
big_t (int) – Number of time periods to simulate forward. If 0, returns the initial states unchanged.

Returns:

Final state values after big_t periods.

Return type:

dict

Raises:

ValueError – If big_t < 0 or if states_t is an empty dict.

Reinforcement Learning (Stable-Baselines3)¶

This module provides Proximal Policy Optimization (PPO) for BellmanPeriod models, via a Stable-Baselines3 backend. The agent wraps a model in a gymnasium environment (see Environments), trains PPO, and emits a standard skagent decision rule.

Stable Baselines3 wrappers for BellmanPeriod models.

Provides PPOAgent, a thin wrapper around SB3’s PPO that:

builds a skagent.env.GymEnv from a BellmanPeriod + initial state distribution,
delegates training to stable_baselines3.PPO.learn,
exposes a PPOAgent.decision_rule() that returns the trained policy as a skagent-style {control_sym: callable} dict — i.e. the same shape consumed by skagent.env.Environment and the rest of the skagent decision-rule API. Actions are unscaled back to real units via GymEnv.unscale_action(), so downstream code does not see the [-1, 1] SB3 representation.

class skagent.algos.sb3.PPOAgent(bp, initial, *, max_episode_steps=200, seed=None, gym_kwargs=None, ppo_kwargs=None, policy='MlpPolicy', device='cpu', verbose=0)¶

Train SB3’s PPO on a BellmanPeriod and emit a skagent decision rule.

Parameters:

bp (BellmanPeriod) – Model definition.
initial (dict) – Maps arrival-state symbols to skagent Distribution objects, used by GymEnv to sample fresh initial states on reset.
max_episode_steps (int) – Episode horizon for the underlying GymEnv. Default 200.
seed (Optional[int]) – Seed for both the environment and the PPO algorithm.
gym_kwargs (Optional[dict]) – Extra keyword arguments forwarded to GymEnv (e.g. default_lower, default_upper, bound_clearance, control_sym).
ppo_kwargs (Optional[dict]) – Extra keyword arguments forwarded to stable_baselines3.PPO (e.g. n_steps, batch_size, learning_rate, n_epochs, policy_kwargs). gamma defaults to bp.calibration[bp.discount_variable] if it is a finite scalar; callers can override by passing gamma here.
policy (str) – SB3 policy class string. Default "MlpPolicy".
device (str) – Torch device for PPO. Default "cpu" (SB3’s recommended default for MlpPolicy).
verbose (int) – Verbosity passed to PPO. Default 0.

env¶

The constructed gymnasium environment.

Type:: GymEnv

model¶

The SB3 model. None until learn() is called the first time (constructed lazily so callers can inspect env without paying PPO’s setup cost).

Type:: stable_baselines3.PPO

decision_rule(deterministic=True)¶

Return a skagent decision rule that uses the trained policy.

The returned dict has the form {control_sym: callable} where the callable accepts positional arguments matching the control’s iset order (i.e. the same signature skagent Environment.step and BellmanPeriod.decision_function call). The callable’s output is a torch.Tensor of unscaled action values; the inputs may be scalars, numpy arrays, or torch tensors of compatible length.

Parameters:: deterministic (bool) – Whether to use a deterministic (mean) policy. Default True — matches typical skagent decision-rule semantics.
Return type:: dict[str, Callable]

learn(total_timesteps, callback=None, **kwargs)¶

Run PPO.learn. Returns self.

Every completed episode’s undiscounted reward is appended to self.episode_rewards via an internal SB3 callback; repeated learn calls accumulate. A user-supplied callback is merged with the internal one via CallbackList. Extra **kwargs forward to model.learn.

Parameters:

total_timesteps (int)
callback (Any)
kwargs (Any)

Return type:

PPOAgent

predict_unscaled(obs, deterministic=True)¶

Predict an unscaled action for obs.

obs may be a single observation (shape (|iset|,)) or a batch ((N, |iset|)). Returns a 1-D array of shape (N,).

Parameters:: deterministic (bool)
Return type:: ndarray

snapshot()¶

Capture the current trained policy as a frozen PolicySnapshot.

The snapshot holds an independent copy of the policy network, so it is unaffected by later learn() calls. This is the supported way to retain the policy at intermediate points during training (e.g. to compare checkpoints) without re-running training or re-implementing the unscaling logic.

Return type:: PolicySnapshot

class skagent.algos.sb3.PolicySnapshot(policy, env)¶

Frozen copy of a trained policy, decoupled from further training.

Returned by PPOAgent.snapshot(). Holds a deep copy of the policy network taken at snapshot time, so subsequent learn calls on the source agent do not change its predictions. Exposes the same predict_unscaled() and decision_rule() interface as PPOAgent.

The GymEnv is shared with the source agent (not copied): it is used only for stateless action unscaling, which does not depend on training state.

Parameters:: env (GymEnv)

decision_rule(deterministic=True)¶

Return a skagent decision rule; see PPOAgent.decision_rule().

Parameters:: deterministic (bool)
Return type:: dict[str, Callable]

predict_unscaled(obs, deterministic=True)¶

Predict an unscaled action for obs; see PPOAgent.predict_unscaled().

Parameters:: deterministic (bool)
Return type:: ndarray

Loss Functions¶

These are the objective functions passed to skagent.ann.train_block_nn(). The reward-based losses (StaticRewardLoss, EstimatedDiscountedLifetimeRewardLoss) solve a block directly for the non-recurring case; the equation-residual losses (BellmanEquationLoss, EulerEquationLoss) target the recurring, dynamic case. See Loss Functions for the full reference.

Neural Network Components¶

Net¶

Net is the base neural network class, with device management.

class skagent.ann.Net(n_inputs, n_outputs, width=32, n_layers=2, activation='silu', transform=None, init_seed=None, copy_weights_from=None)¶

Bases: Module

A flexible feedforward neural network with configurable architecture.

Parameters:

n_inputs (int) – Number of input features
n_outputs (int) – Number of output features
width (int, optional) – Width of hidden layers. Default is 32.
n_layers (int, optional) – Number of hidden layers (1-10). Default is 2.
activation (str, list, callable, or None, optional) –
Activation function(s) to use. Options: - str: Apply same activation to all layers (‘silu’, ‘relu’, ‘tanh’, ‘sigmoid’) - list: Apply different activations to each layer, e.g., [‘relu’, ‘tanh’, ‘silu’] - callable: Custom activation function - None: No activation (identity function)

Available activations: ‘silu’, ‘relu’, ‘tanh’, ‘sigmoid’, ‘identity’ Default is ‘silu’.
transform (str, list, callable, or None, optional) –
Transformation to apply to outputs. Options: - str: Apply same transform to all outputs (‘sigmoid’, ‘exp’, ‘tanh’, etc.) - list: Apply different transforms to each output, e.g., [‘sigmoid’, ‘exp’] - callable: Custom transformation function - None: No transformation

Available transforms: ‘sigmoid’, ‘exp’, ‘tanh’, ‘relu’, ‘softplus’, ‘softmax’, ‘abs’, ‘square’, ‘identity’ Default is None.

property device¶: Device property for backward compatibility.

forward(x)¶

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

BlockPolicyNet¶

BlockPolicyNet is a specialized neural network for policy functions in economic models.

class skagent.ann.BlockPolicyNet(bellman_period, control_sym=None, apply_open_bounds=True, width=32, **kwargs)¶

Bases: BellmanPeriodMixin, Net

A neural network for policy functions in dynamic programming problems.

This network wraps a Net and integrates with the BellmanPeriod interface. It automatically determines input/output dimensions from the model block specification and enforces control variable bounds.

Parameters:

bellman_period (BellmanPeriod) – The model Bellman Period
apply_open_bounds (bool, optional) – If True, then the network forward output is normalized by the upper and/or lower bounds, computed as a function of the input tensor. These bounds are “open” because output can be arbitrarily close to, but not equal to, the bounds. Default is True.
control_sym (string, optional) – The symbol for the control variable.
width (int, optional) – Width of hidden layers. Default is 32.
**kwargs – Additional keyword arguments passed to Net. See Net class documentation for all available options including activation, transform, n_layers, init_seed, copy_weights_from, etc.

decision_function(states_t, shocks_t, parameters)¶

A decision function, from states, shocks, and parameters, to control variable values.

Parameters:

states_t (dict) – symbols : values
shocks_t (dict) – symbols: values
parameters (dict) – symbols : values

Returns:

decisions - dict – symbols : values

forward(x)¶: Note that this uses the same architecture of the superclass but adds on a normalization layer appropriate to the bounds of the decision rule.

get_core_function(length=None)¶

get_decision_rule(length=None)¶

Returns the decision rule corresponding to this neural network.

The rule takes the information-set values as positional arguments and returns the control values. For an empty information set, length is the number of samples to return.

BlockValueNet¶

BlockValueNet is a neural network for value functions in dynamic programming problems.

class skagent.ann.BlockValueNet(bellman_period, control_sym=None, width=32, **kwargs)¶

Bases: BellmanPeriodMixin, Net

Standalone value-function network for a Bellman problem.

Maps a control’s information set (the same pre-decision states a policy network sees) to a single unconstrained scalar value. It is the value-only counterpart of BlockPolicyNet, kept for algorithms that approximate a value function separately from the policy. The Maliar/MMW path in this package uses the shared-backbone BlockPolicyValueNet instead, so BlockValueNet is not wired into maliar_training_loop().

Parameters:

bellman_period (BellmanPeriod) – The model Bellman period.
control_sym (str, optional) – Control whose information set defines the value function’s domain. Defaults to the first control.
width (int) – Width of hidden layers. Default 32.
**kwargs – Passed to Net (activation, n_layers, init_seed, etc.).

get_core_function(length=None)¶: Return the value function (the trainable core for this net).

get_value_function()¶: Return a callable (states, shocks, parameters) -> value tensor.

value_function(states_t, shocks_t=None, parameters=None)¶

Evaluate the value function at the control’s information set.

Arrival states_t (with shocks_t and parameters) are mapped to the control’s information set via compute_pre_state(), mirroring BlockPolicyNet.decision_function(), then the network is evaluated.

Returns:: Flattened value estimates, one per input row.
Return type:: torch.Tensor

BlockPolicyValueNet¶

BlockPolicyValueNet is a shared-backbone neural network that jointly represents the policy and value functions.

class skagent.ann.BlockPolicyValueNet(bellman_period, control_sym=None, apply_open_bounds=True, width=32, **kwargs)¶

Bases: BlockPolicyNet

Single neural network with shared backbone for both policy and value.

A BlockPolicyNet with a second, unconstrained output head; the policy interface (decision function, decision rule, core function) is inherited unchanged and reads the policy head.

Architecture: shared hidden layers → two output heads: - Policy head: bounded output (sigmoid-scaled to satisfy constraints) - Value head: unconstrained scalar output

Sharing the backbone means one optimizer updates all weights simultaneously, and the value head anchors the control level that first-order-condition-only training (e.g. an Euler residual loss) cannot identify.

Parameters:

bellman_period (BellmanPeriod) – The model Bellman Period.
control_sym (str, optional) – Control variable symbol. Defaults to first control.
apply_open_bounds (bool, optional) – Apply sigmoid/softplus scaling to the policy head. Default True.
width (int, optional) – Width of hidden layers. Default 32.
**kwargs – Passed to Net (activation, n_layers, init_seed, etc.).

forward(x)¶

Run shared backbone, then policy head (bounded) + value head.

Returns the (policy, value) pair. The policy tensor is scaled into the control’s open bounds; the value tensor is unconstrained. Both have shape (n, 1).

Return type:: tuple[Tensor, Tensor]

get_policy_and_value_functions(length=None)¶: Return both policy decision rules and value function.

get_value_function()¶: Return a callable (states, shocks, parameters) -> value tensor.

value_function(states_t, shocks_t=None, parameters=None)¶

Evaluate the value head at the control’s information set.

The input domain mirrors decision_function(): arrival states_t (with shocks_t and parameters) are mapped to the control’s information set via compute_pre_state(), then the shared backbone’s value head is evaluated on that pre-decision representation.

Parameters:

states_t (dict) – Arrival state values, symbol -> tensor.
shocks_t (dict or None, optional) – Shock values (None is treated as {}).
parameters (dict or None, optional) – Model parameters.

Returns:

Flattened value estimates, one per input row.

Return type:

torch.Tensor

Training Functions¶

skagent.ann.train_block_nn(block_policy_nn, inputs, loss_function, epochs=50, lr=0.01, optimizer=None, grad_clip=1.0, verbose=True)¶

Train a policy network by minimizing a loss function over a grid.

This is a generic stochastic-gradient-descent driver, not a solution algorithm in itself. It runs epochs Adam updates that minimize whatever loss_function is supplied, evaluated on a single, fixed grid of inputs; it is agnostic to where that grid came from or which method the loss encodes (Euler residual, Bellman residual, FOC, or a custom loss).

Because it trains on whatever inputs it is given, accuracy depends on the caller re-sampling those states across calls: Maliar, Maliar, and Winant (2021) keep the training data “constantly re-sampled,” and minimizing on a single fixed grid instead lets the solution over-fit those points while drifting elsewhere. Re-draw inputs each call (threading the returned optimizer back in to keep Adam’s momentum), or use maliar_training_loop(), which wraps this driver in the full MMW’21 outer loop: it alternates these inner SGD updates with a forward-simulation step that refreshes the training states toward the model’s ergodic set.

Parameters:

block_policy_nn (BlockPolicyNet or BlockPolicyValueNet) – The network to train. Its get_core_function supplies the decision rule(s) the loss is evaluated against.
inputs (Grid) – Input grid containing states and shocks.
loss_function (Callable) – Loss function (decision_function, input_grid) -> loss_tensor.
epochs (int) – Number of training epochs (default 50). Any integer type is accepted, numpy integers included.
lr (float) – Learning rate for Adam optimizer (default 0.01).
optimizer (Optional[Optimizer]) – Pre-existing optimizer to reuse (preserves momentum across calls). If None, a new Adam optimizer is created.
grad_clip (Optional[float]) – Maximum gradient norm for clipping (default 1.0). Set to None to disable.
verbose (bool) – Emit a logging.info message with the loss every 100 epochs (default True). Configure the root logger to suppress these.

Returns:

(trained_network, final_loss, optimizer). The optimizer is the one passed in, or the Adam instance created internally when none was supplied; returning it always lets callers warm-start a later call by threading it back in.

Return type:

tuple

skagent.ann.aggregate_net_loss(inputs, df, loss_function)¶

Compute a loss function over a tensor of inputs, given a decision function df. Return the mean.

Parameters:: inputs (Grid)

Equilibrium Schedules and Methods¶

A schedule decides when each decision is solved, and it asks a method to carry out each solve. A method carries one algorithm together with the configuration that algorithm needs. The pairing is free: the same schedule accepts either a policy network or an exact backup, and it returns the same answer in either case.

skagent.solver.solve_in_order(method, order, policies=None)¶

Solve the named decisions, one at a time, in the order given.

This schedule takes its order from the caller rather than deriving one. Each decision is solved against the rules already in hand, so a symbol repeated in order, as in ["c", "d", "c"], is refined after its neighbours have moved. The result is a best-response sweep run by hand.

There is no convergence test here. The iteration stops because order ran out, and not because anything settled, so a repeated symbol buys a fixed number of refinement passes rather than a fixed point. Where a fixed point is wanted, use a schedule that measures one: solve_symmetric_equilibrium() iterates against a residual.

A decision absent from order is returned at its starting rule, which means that it has not been solved. That is the caller’s choice, since the caller writes the order, but the returned profile does not distinguish a solved rule from an unsolved one.

Parameters:

method (object) – A per-decision solver, as NeuralBestResponse, ExactBestResponse and skagent.algos.tabular.TabularBestResponseSolver are. Needs best_response(decision, policies) and, when policies is omitted, initial_policies().
order (sequence of str) – The decisions to solve, in order. Symbols may repeat.
policies (Mapping[str, Callable], optional) – The profile to start from, with a rule for every decision. Defaults to the method’s own starting profile.

Returns:

A decision rule per control of the block.

Return type:

dict

skagent.solver.solve_in_relevance_order(method, policies=None, *, max_iterations=25, tolerance=1e-06)¶

Solve every decision in relevance-component order.

This is a schedule rather than a method: it decides when each decision is solved, and it asks the method to carry out each solve. Acyclic components are solved once, after every decision rule they rely on has been computed. For a cyclic component, every decision’s best response is computed against the same previous profile and installed simultaneously. This repeats until every policy is within tolerance of its own response.

Parameters:

method (object) – A per-decision solver carrying the problem, as NeuralBestResponse and skagent.algos.tabular.TabularBestResponseSolver do. Needs ground, best_response(decision, policies), rule_distance(new, old, iset) and, when policies is omitted, initial_policies().
policies (Mapping[str, Callable], optional) – Starting rules for the decisions, replaced one by one as they are solved. Defaults to the method’s own starting profile.
max_iterations (int, optional) – Maximum number of simultaneous best-response updates for a cyclic relevance component. Must be at least 1. Defaults to 25.
tolerance (float, optional) – Maximum rule distance at convergence. Must be positive. Defaults to 1e-6.

Returns:

A decision rule per control of the block.

Return type:

dict

Raises:

NotImplementedError – If a cyclic component belongs to a recurring block, or if a decision relies on the other instances of its own entity class.
RuntimeError – If a cyclic component does not converge within max_iterations.
ValueError – If max_iterations is less than 1 or tolerance is not positive.
TypeError – If max_iterations is not an integer.

skagent.solver.project_nash(ground, actor_suffix='_actor', other_suffix='_other')¶

One instance’s problem, with the rest of its class beside it.

The entity class is split in two – the instance being solved, and the others – and every per-instance equation is copied once per side under a suffixed name. For each symbol that an aggregating equation reads over the class, exactly one equation is synthesized, and it concatenates the two sides back into the original symbol; the aggregating equation is then copied verbatim and reads that symbol. The projection therefore reassembles the entity axis without inspecting the reduction, so a mean, a sum, a maximum and a masked mean all project alike.

The two sides’ rewards are attributed to suffixed agent roles, so a solver told which agent it serves maximizes one instance’s payoff rather than the class’s total.

The rest of the class stays a population. The others are declared as an entity class of their own, named for the original and sized one short of it, so their symbols are one value per rival rather than one value shared by all of them. That is what lets rivals with private draws differ from one another: a rule they share is still evaluated once per rival, and the aggregate reads the spread of what they do. A rejoined symbol carries the class the author declared, at the size the author gave it, so the author’s own aggregation reduces over the same class it always did.

The solved instance’s symbols carry no class, because it is one instance rather than a population of one, and that asymmetry is what the two sides are for.

The other instances share one rule, so the equilibrium sought is a symmetric one. A class of genuinely distinct rules is expressible in this shape but is not built here.

Parameters:

ground (skagent.ground.GroundedBlock) – The population model and the calibration it is solved at. Its block must declare exactly one entity class of at least two instances, and the calibration must give that class’s size under its own name.
actor_suffix (str, optional) – How the two sides’ symbols are named.
other_suffix (str, optional) – How the two sides’ symbols are named.

Returns:

The projected problem, carrying two controls – the solved instance’s and the others’ – and a calibration that sizes the rivals’ class.

Return type:

skagent.ground.GroundedBlock

Raises:

ValueError – If the block does not declare exactly one entity class, if the calibration does not size it, or if it holds fewer than two instances.

skagent.solver.project_mean_field(ground)¶

One instance’s problem under price-taking: the aggregate is given.

Every equation that reduces over the entity class is removed, so each aggregate it computed becomes a symbol the remaining equations read rather than one they compute. The equations downstream of an aggregate are kept as the author wrote them. The instance’s symbols carry no class, since they are one decision-maker’s, so the result is a problem a single-agent solver accepts.

Under this concept the instance takes no account of its own effect on the aggregate, which is exact for a continuum of instances and the concept a competitive equilibrium is defined by. Contrast project_nash(), where the instance keeps its own share.

The projection gives the aggregates no value. Supply one before solving, for instance with skagent.ground.GroundedBlock.with_calibration(): a symbol the block reads and neither defines nor calibrates is otherwise an arrival state.

An aggregate is found as skagent.block.Block.crossings() reports it, so a reduction written inside a per-instance equation is not one, and is left in place.

Parameters:: ground (skagent.ground.GroundedBlock) – The population model and its calibration. Its block must declare exactly one entity class and read something out of it.
Returns:: The projected problem, with the calibration and generator of ground.
Return type:: skagent.ground.GroundedBlock
Raises:: ValueError – If the block does not declare exactly one entity class, reads nothing out of it, or reduces over it in a decision, which is a choice rather than an aggregate to take as given.

skagent.solver.solve_symmetric_equilibrium(method, *, damping=1.0, tolerance=0.001, max_iterations=20, initial=None)¶

A symmetric equilibrium, by iterated best response over a projection.

Takes a projected problem – one instance’s decision beside the rest of its class, as project_nash() builds – solves the instance’s decision against the others’ current rule, swaps the solved rule in as the others’, and repeats until the rule stops moving. A rule that is its own best response is an equilibrium of the projected game, and because the others play whatever the solved instance plays, it is a symmetric equilibrium of the population.

The method supplies both the per-decision solve and the distance between two rules, since only it knows how a rule is represented. The schedule supplies the damping, the residual test and the swap.

Damping is a correctness requirement rather than a convergence aid. Undamped iteration converges only where the best response is a contraction. Where its slope is -1 the iterates cycle between two points forever, and past that they diverge until the controls’ bounds catch them. Both failures return a plausible number under an iteration cap, which is why the residual here is measured on the rule and never on the iteration count.

Parameters:

method (NeuralBestResponse, ExactBestResponse, or TabularBestResponseSolver) – The per-decision solver, carrying its own configuration and the projected problem it solves. Any object with ground, best_response(decision, policies) and rule_distance(new, old, iset) serves.
damping (float, optional) – How far to move toward the best response each round, in (0, 1]. Default 1.0, which is undamped.
tolerance (float, optional) – The rule is converged when it moves less than this.
max_iterations (int, optional) – Rounds before giving up. Reaching this is not convergence and is reported as such.
initial (Callable, optional) – The others’ rule on the first round. Defaults to a constant at the midpoint of the solved control’s declared bounds, or to one of its declared actions for a discrete control.

Returns:

rule (Callable) – The equilibrium decision rule, in the solved instance’s symbols.
info (dict) – converged, iterations and distances.

Raises:

ValueError – If the method’s block does not carry a projected pair of controls, or if damping is requested for an unordered discrete action space.

skagent.solver.solve_mean_field(method, population, *, bracket, tol=0.001, periods=500, tail=200, seed=0, warm_start=True)¶

A stationary mean-field equilibrium, by root-finding on the aggregate.

method carries one instance’s problem under price-taking, as project_mean_field() builds it from population, with the aggregate left without a value. For a candidate value of the aggregate, the schedule gives it to the instance’s problem, solves that to its stationary rule, and simulates the full model, population, under the rule. There the aggregate is computed from the instances every period, as the model says, rather than held at the candidate. The residual is the aggregate the simulation settles at, averaged over its last tail periods, less the candidate, and Brent’s method finds its root within bracket.

Every round simulates from the same seed, and from the variables the aggregate reduces each starting at the candidate value, so the residual is a deterministic function of the candidate. Its precision is still that of a finite class: the class size in the calibration sets the simulated aggregate’s noise, and so how sharply the root is pinned.

Every instance plays the one rule, so the equilibrium is symmetric by construction.

Parameters:

method (StationaryBestResponse) – The instance’s solver, carrying the projected problem.
population (skagent.ground.GroundedBlock) – The model the projection was taken from, with its calibration. Its block must reduce over its entity class in exactly one equation.
bracket (tuple of float) – Two values of the aggregate at which the residual has opposite signs.
tol (float, optional) – Tolerance on the aggregate.
periods (int, optional) – Periods simulated per round.
tail (int, optional) – Final periods averaged over to read the aggregate the simulation settles at.
seed (int, optional) – Seed of every round’s simulation.
warm_start (bool, optional) – Start each round’s solve from the previous round’s, through the method’s with_ground. Otherwise each round starts from method.

Returns:

rule (Callable) – The instance’s decision rule at the equilibrium aggregate.
info (dict) – aggregate, the equilibrium value; converged and iterations, as Brent’s method reports them; and rounds, one entry per candidate with its candidate, its simulated aggregate and the rule’s off_grid count where the rule keeps one.

Raises:

TypeError – If method is not a StationaryBestResponse.
ValueError – If population does not reduce over its class in exactly one equation, if the projected problem does not have exactly one decision, or if the residual does not change sign across bracket.

class skagent.solver.NeuralBestResponse(ground, panel, epochs=200, width=32)¶

Best responses by training a policy network, and what that needs.

A method object carries its own construction configuration beside its algorithm, so that a schedule can take any method without carrying every method’s arguments on its own signature. This one needs a training panel and an epoch count; the exact backup needs a state grid and a continuation instead, and neither needs the other’s.

Parameters:

ground (skagent.ground.GroundedBlock) – The problem being solved, already projected if it is a population.
panel (skagent.grid.Grid) – What the network trains on and what two rules are compared over. Must carry every shock of the block, since the loss evaluates the whole period.
epochs (int, optional) – Training epochs per best response.
width (int, optional) – Hidden width of the policy network.

best_response(decision, policies)¶: Train a network for decision, holding the rest of policies fixed.

initial_policies()¶: A starting profile: every decision at a constant, none of them solved.

rule_distance(new_rule, old_rule, iset)¶

Supremum norm between two rules, evaluated on the training panel.

A network has no cells to compare, so the comparison is over a common batch – which is why the distance is the method’s operation and not the schedule’s.

class skagent.solver.ExactBestResponse(ground, state_grid, scope=None, continuation=None, disc_params=None)¶

Best responses by exact backup over a state grid.

The method-object counterpart of NeuralBestResponse: same two operations, entirely different construction configuration.

Parameters:

ground (skagent.ground.GroundedBlock) – The problem being solved, already projected if it is a population.
state_grid (Mapping) – The grid the backup optimizes over, and where two rules are compared. An information-set variable must appear here rather than in scope: scope pins a shock to a realization, which leaves the rule no argument to be a function of. A single point is enough, and gives a rule constant along that variable; give it several to let the backup find whether the optimum varies along it.
scope (Mapping, optional) – Shocks pinned to a fixed realization. Defaults to the calibration.
continuation (Callable, optional) – The continuation value. Defaults to a terminal (zero) one, which is what makes the backup a single-period solve.
disc_params (Mapping, optional) – Per-shock discretization arguments for the shocks integrated inside the maximization.

best_response(decision, policies)¶: Back up decision alone, holding the rest of policies fixed.

initial_policies()¶: A starting profile: every decision at a constant, none of them solved.

rule_distance(new_rule, old_rule, iset)¶: Supremum norm between two rules over the grid they were solved on.

class skagent.solver.StationaryBestResponse¶

A method whose responses solve the infinite-horizon problem.

A contract with no code. A subclass’s best_response(decision, policies) solves the recurring problem on its ground to a fixed point, where ExactBestResponse and NeuralBestResponse solve one period. A schedule that needs the stationary solution, as solve_mean_field() does, checks for this class, so that a one-period method is refused rather than its answer reported as the stationary one.

Beside best_response, rule_distance and initial_policies, a subclass provides with_ground(ground): the same method on another ground, starting from its own last solve when it has made one.

class skagent.solver.ExactStationaryBestResponse(ground, state_grid, discount_variable, *, disc_params=None, tol=1e-06, max_iter=500, search='bounded', policy_evaluations=20, confine_to_grid=False, continuation=None)¶

Stationary best responses by value-function iteration over a state grid.

The stationary counterpart of ExactBestResponse: a response runs skagent.algos.vfi.solve_bellman() to its fixed point rather than one backup. Its rules are gridded as that method’s are, and compared the same way.

The block must have no decision other than the one solved, since skagent.algos.vfi.solve_bellman() optimizes every control jointly and cannot hold one at a given rule.

Parameters:

ground (skagent.ground.GroundedBlock) – The recurring problem being solved, already projected if it is a population. Its calibration is also the solve’s scope.
state_grid (Mapping) – The grid over the value function’s domain; every arrival state must be an axis of it.
discount_variable (str or None) – The symbol holding the discount factor.
disc_params (Mapping, optional) – Per-shock discretization arguments.
tol (float, optional) – Convergence tolerance on the sup-norm change in the value grid.
max_iter (int, optional) – Backups before the solve is reported as unconverged, which raises.
search ({"bounded", "multistart"}, optional) – How each grid point’s optimum is found.
policy_evaluations (int, optional) – Applications of the Bellman operator at the maximizing policy after each backup.
confine_to_grid (bool, optional) – Confine next-period arrival states to the state grid.
continuation (Callable, optional) – The continuation the iteration starts from. Defaults to a terminal (zero) one.

value¶

The value grid of the last response, whose attrs carry n_iter, converged and residual; None before the first.

Type:: xarray.DataArray or None

best_response(decision, policies)¶: Solve decision to its stationary fixed point.

initial_policies()¶: A starting profile: every decision at a constant, none of them solved.

rule_distance(new_rule, old_rule, iset)¶: Supremum norm between two rules over the grid they were solved on.

with_ground(ground)¶: This method on ground, continuing from its last solved value.

Grid and Computational Tools¶

Grid Class¶

class skagent.grid.Grid(labels, values, torched=True)¶

Bases: object

A class representing a labeled grid of numerical values.

Parameters:: (dict) (config) – dictionary with the following keys: “min” (float): The minimum value for the variable.; “max” (float): The maximum value for the variable; “count” (int): The number of points to generate for the variable.

classmethod from_config(config={}, torched=True)¶

classmethod from_dict(kv={}, torched=True)¶

len()¶: Returns the number of columns, similar to a dict.

n()¶: Returns the number of values for each symbol

to_dict()¶: Returns a data structure, key: column, similar to tensordict or structured array.

torch()¶

update_from_dict(kv)¶

Grid Utility Functions¶

skagent.grid.make_grid(config)¶

Make a ‘grid’ of values based on the provided configuration.

Parameters:

(dict) (config) – dictionary with the following keys: “min” (float): The minimum value for the variable; “max” (float): The maximum value for the variable; “count” (int): The number of points to generate for the variable.

Returns:

numpy.ndarray (A NumPy array of shape (product_of_counts, num_variables), where) – product_of_counts is the product of all count values in the config dictionary, and num_variables is the number of keys in the config.

skagent.grid.cartesian_product(*arrays)¶

Create a Cartesian product of input arrays.

Parameters:

*arrays – Variable length arrays to compute product

Returns:

Array of shape (product_of_lengths, num_arrays)
where product_of_lengths is the product of the lengths of the input arrays,
and num_arrays is the number of input arrays. Each row contains one element
of the Cartesian product.