Algorithms¶

This section contains the API documentation for solution algorithms, neural network components, and grid tools used to solve dynamic stochastic optimization problems.

Value Function Iteration (VFI)¶

The value function iteration (VFI) algorithm derives arrival value functions from a continuation value function and the stage dynamics of model blocks.

Value function iteration (VFI).

Derive a decision rule, decision value function, and arrival value function for a single DBlock stage by value function iteration: at each point of a grid over the decision’s information set, solve an exact scipy.optimize.minimize() for the control that maximizes the period reward plus a continuation value.

skagent.algos.vfi.ar_from_data(da)¶

Build a decision rule from a fitted policy DataArray.

The returned rule follows the library’s decision-rule calling convention: it takes the control’s information-set values as positional arguments, in the order of da.dims (which solve() aligns to control.iset). This matches how block.transition invokes a rule, dr(*[vals[v] for v in iset]), so a VFI-fitted rule is a drop-in for the rest of the stack.

Interpolation runs in numpy/xarray space. For a torch-tensor interface, wrap the result with tensor_decision_rule().

Parameters:: da (xarray.DataArray) – The fitted policy, with one dimension per information-set variable in control.iset order. A zero-dimensional array encodes a constant rule (empty information set).
Returns:: A rule ar(*args) taking one positional argument per dimension of da. Scalar arguments return a Python scalar; array-like arguments are interpolated pointwise (not as an outer product) and return a numpy array.
Return type:: callable
Raises:: TypeError – If the number of positional arguments does not match da.ndim.

skagent.algos.vfi.bellman_step(bp, continuation_vf, state_grid, *, agent=None, scope={}, disc_params={}, x0=1.0, x0_policy=None, artificial_borrowing_constraint=False)¶

One exact value backup over state_grid on the BellmanPeriod protocol.

This is the per-iteration update of value-function iteration: at each grid point the optimal control is found with scipy.optimize.minimize(), maximizing the period reward plus the discounted continuation value of the resulting arrival states. Under a terminal (zero) continuation, continuation_vf = lambda s, sh, p: 0.0, the result is the single-step solution; the value-iteration wrapper solve_bellman iterates it to a fixed point.

Unlike legacy solve() (which rides the DBlock continuation API and folds the discount factor into the continuation), this speaks the BellmanPeriod protocol the rest of the torch stack uses, with an explicit discount factor and multi-reward summation, and is empty-shock-safe.

Shocks are handled by their information role, which is derived from the block’s own structure (skagent.relevance) rather than inferred from the grid the caller supplies. A shock some control’s information set accounts for becomes a grid axis over its discretization nodes, so its pre-state and bounds are computed per realization; the rest are integrated out inside the per-point max via internal discretization (_discretize_shocks() + expected), so an optimum characterized by an expectation over an unobserved shock is in scope. Any shock may instead be pinned to a fixed realization by supplying it in scope, which takes precedence over both.

Note

Current scope: one or more controls, jointly optimized by scipy.optimize.minimize() over the stacked control vector with per-control bounds — restarted from each seed candidate, keeping the best optimum — each policy then reprojected onto its own information set (_project_to_iset()). A control’s pre-state and bounds are evaluated with each integrated-out shock fixed at its (discretized) mean, since a single value is required there even though the objective integrates the shock. That is exact when the pre-state does not depend on such a shock, and a shock the pre-state does depend on is a grid axis rather than an integrated one, so it does not arise for a well-formed block.

Parameters:

bp (BellmanPeriod) – The recurring period providing the model mechanics.
continuation_vf (Callable) – The continuation value function, called continuation_vf(states, shocks, parameters) on the next-period arrival states (the bp.compute_value convention). Terminal continuation is lambda s, sh, p: 0.0.
state_grid (Mapping[str, Sequence]) – The shared backup grid over arrival states: one axis per variable. Axes for the shocks an information set accounts for are added automatically from their discretization nodes, so supplying one is optional and idempotent. This grid covers the variables the Bellman loop iterates over and is not necessarily equal to any individual control’s information set (a control’s iset may be a strict subset). For an empty grid, pass {}.
agent (str | None) – If given, the period reward sums only this agent’s reward symbols.
scope (Mapping) – Fixed non-shock exogenous values merged into the model parameters. A shock supplied here is pinned to that fixed realization instead of being gridded or integrated. Note this fixes a value for the solve; it does not make the symbol a model parameter, so it does not affect any shock’s information role.
disc_params (Mapping) – Per-shock discretization arguments, keyed by shock symbol (e.g. {"theta": {"N": 7}}), forwarded to that shock’s Distribution.discretize whether it becomes a grid axis or is integrated out. A shock without an entry uses its distribution’s default discretization (exact for already-discrete shocks).
x0 (float) – A modest optimizer seed, clamped into each control’s bounds. One of the multi-start candidates at every point, not just a fallback: it is what covers a box whose midpoint is far from the optimum, e.g. the natural borrowing limit’s [0, m + H].
x0_policy (Optional[Mapping[str, DataArray]]) – Warm-start seeds keyed by control symbol (e.g. a previous iterate’s policy_array); supplies the first multi-start candidate at each grid point, and wins ties. Supplied by solve_bellman().
artificial_borrowing_constraint (bool) – When True, tighten each control’s bounds so the next-period arrival state stays inside the state grid (_tighten_bounds_to_grid()), an artificial state (borrowing) limit at the grid’s lower edge. This keeps the continuation interpolated rather than extrapolated past the grid edges, so value iteration cannot ride a control bound by over-crediting off-grid successors. Single control with an affine successor only (raises otherwise). The limit must be slack at the states of interest (it is just the grid floor), or it biases the policy where it binds.

Return type:

tuple[dict[str, Callable], DataArray, dict[str, DataArray]]

Returns:

dr_from_data (dict of callable) – One decision rule per control, keyed by control symbol; each takes its information-set values as positional arguments in control.iset order.
value_array (xarray.DataArray) – The gridded optimized decision value over the state grid.
policy_array (dict of xarray.DataArray) – The gridded optimal control(s) over the state grid, keyed by control symbol (a dict for forward-compatibility with multi-control, O1).

skagent.algos.vfi.get_action_rule(action)¶

Build a constant decision rule that ignores its inputs.

Parameters:: action (Any) – The fixed value the rule returns.
Returns:: A zero-argument function ar() returning action. Used to wrap a candidate action as a decision rule during the per-point optimization.
Return type:: callable

skagent.algos.vfi.grid_to_data_array(grid={})¶

Construct a zero-valued DataArray over the coordinates of a grid.

Parameters:: grid (Mapping[str, Sequence]) – A mapping from variable labels to a sequence of numerical values. An empty mapping yields a zero-dimensional array.
Returns:: An array whose dimensions and coordinates are those of grid.
Return type:: xarray.DataArray

skagent.algos.vfi.solve(block, continuation, state_grid, disc_params={}, scope={})¶

Solve a DBlock stage by value function iteration.

At each point of state_grid, the optimal control(s) are found with scipy.optimize.minimize(), maximizing the period reward plus the continuation value of the resulting states. The tabulated optima are then interpolated into a decision rule.

VFI assumes full observation: the decision conditions on its complete information set and the per-point optimization never integrates over unobserved variables. (The only expectation machinery in this module is in block.get_arrival_value_function; the optimization here does not use it.) Hidden-shock problems whose optimum requires an expectation are out of scope.

Parameters:

block (DBlock) – The stage to solve. Must contain at most one control variable; multi-control stages raise Exception.
continuation (callable) – The continuation value function, called with the post-transition values of the variables named in its signature. Fold any discount factor into this function (the backup is reward + continuation).
state_grid (Mapping[str, Sequence]) – A grid over the control’s information set: one axis per variable the decision may condition on. The returned decision rule takes these as positional arguments in control.iset order. Variables the dynamics need but the decision does not (e.g. a shock that only enters the transition) go in scope, not here. For an empty information set, pass {}.
disc_params (Mapping, optional) – Discretization parameters for the shock distribution, forwarded to block.get_arrival_value_function.
scope (Mapping, optional) –
The fixed scope for the per-point optimization: merged with each grid point to form the pre_states under which the dynamics, reward, and continuation are evaluated.

Note

This is broader than calibration elsewhere in the library, which denotes fixed, single-valued parameters only. Here (legacy VFI usage) it is a general scope bag that also holds fixed exogenous values outside the information set, such as a shock realization psi. Read it as “scope,” not “parameters.”

Returns:

dr_from_data (dict of callable) – One decision rule per control, keyed by control symbol; each takes its information-set values as positional arguments in control.iset order.
dec_vf (callable) – The decision value function for the fitted rule.
arr_vf (callable) – The arrival value function for the fitted rule (takes the shock expectation via disc_params).

skagent.algos.vfi.solve_bellman(bp, state_grid, *, continuation_vf=None, agent=None, scope={}, disc_params={}, tol=1e-06, max_iter=100, x0=1.0, raise_on_nonconvergence=False, artificial_borrowing_constraint=False)¶

Solve a recurring BellmanPeriod by value-function iteration.

Iterates bellman_step() to a fixed point: each backup uses the previous iterate’s value grid as its continuation (rebuilt via value_array_to_function()) and offers the previous iterate’s policy_array as a per-point warm start — one multi-start candidate among the others, so a collapsed iterate cannot entrench itself by seeding the next backup at its own bound. It stops when the sup-norm change in the value grid falls below tol, or after max_iter iterations.

Iteration 1 uses the terminal (zero) continuation, so solve_bellman(..., max_iter=1) reproduces bellman_step() under a terminal continuation. For an infinite-horizon problem the loop converges geometrically (modulus the discount factor) to the stationary solution; for a finite horizon of length T set max_iter=T.

Shocks are discretized internally: disc_params is threaded into every backup (hidden shocks integrated inside the max) and into value_array_to_function() (observed-shock axes integrated into the arrival value between iterations).

Parameters:

bp (BellmanPeriod) – The recurring period providing the model mechanics.
state_grid (Mapping[str, Sequence]) – A grid over the value function’s domain (arrival states and/or observed shocks); see bellman_step().
continuation_vf (Optional[Callable]) – Initial continuation guess continuation_vf(states, shocks, parameters). Defaults to the terminal (zero) continuation.
agent (str | None) – If given, the period reward sums only this agent’s reward symbols.
scope (Mapping) – Fixed non-shock exogenous values (and, in this scope, any hidden-shock realization) merged into the model parameters.
disc_params (Mapping) – Per-shock discretization arguments, threaded into each backup (for hidden shocks) and into value_array_to_function() (for observed shocks); see bellman_step().
tol (float) – Convergence tolerance on the sup-norm change in the value grid.
max_iter (int) – Maximum number of backups.
x0 (float) – Modest multi-start seed candidate passed to bellman_step().
raise_on_nonconvergence (bool) – If True, raise RuntimeError when the loop hits max_iter without converging; otherwise emit a warnings.warn and return the last iterate (the scipy OptimizeResult.success convention, O5).
artificial_borrowing_constraint (bool) – Forwarded to bellman_step(): confine next-period arrival states to the state grid (grid edge = slack artificial borrowing limit), so the rebuilt continuation is never extrapolated off-grid.

Return type:

tuple[dict[str, Callable], DataArray, dict[str, DataArray]]

Returns:

dr_from_data (dict of callable) – One decision rule per control at the fixed point (see bellman_step()).
value_array (xarray.DataArray) – The converged value grid. Its attrs carry n_iter, converged (bool), and residual (the final sup-norm change).
policy_array (dict of xarray.DataArray) – The gridded optimal control(s) at the fixed point.

Raises:

RuntimeError – If raise_on_nonconvergence is True and the loop does not converge.

skagent.algos.vfi.tensor_decision_rule(np_rule, dtype=None, device=None)¶

Wrap a numpy-space decision rule so it speaks torch tensors.

Adapts a rule (e.g. from ar_from_data(), including the rules returned by solve()) for interop with the torch solving stack (BellmanPeriod / loss / solver). Inputs may be torch tensors or numpy/scalars; outputs are torch tensors.

Because interpolation runs in numpy, the autograd graph is severed: the returned controls are detached. This rule is therefore valid as a fixed / ground-truth / warm-start policy (e.g. an other_dr or in a value-residual loss), but not as a trainable policy in a loss that differentiates through the control (FOC-weighted Bellman, Euler).

Parameters:

np_rule (callable) – A numpy-space decision rule taking positional information-set arguments and returning numpy/scalar values.
dtype (torch.dtype, optional) – Output tensor dtype. Defaults to torch.float32 to match grid tensors (grid.py builds them with torch.FloatTensor).
device (torch.device, optional) – Output tensor device. Defaults to the stack’s device (skagent.grid.device).

Returns:

A rule tdr(*args) accepting torch tensors (or numpy/scalars) and returning a detached torch tensor on device with dtype dtype.

Return type:

callable

skagent.algos.vfi.value_array_to_function(value_array, bp, disc_params={})¶

Rebuild a continuation value function from an iterate’s value grid.

solve_bellman() feeds iteration n’s value grid back as iteration (n+1)’s continuation. This wraps the grid as a callable in the bp.compute_value convention wf(states, shocks, parameters), so it drops straight into bellman_step()’s continuation_vf slot.

The decision value grid ranges over arrival states and any observed-shock axes. Those shock axes are first integrated out into the arrival value W(s) = E_obs[V(s, obs)], using the weights of the same discretized distribution that produced the axis nodes; a grid built from other node values raises ValueError. wf then interpolates linearly over the remaining arrival-state axes and extrapolates linearly past the grid edges (via scipy.interpolate.RegularGridInterpolator), so an off-grid next-period state during the backup gets a finite, sloped continuation rather than NaN (which breaks the optimizer) or a flat boundary clamp (which zeroes the marginal value of saving and collapses the policy onto its bound).

When the value grid has no observed-shock axes (the deterministic and hidden-shock-only cases, where the backup already integrated any hidden shocks), the expectation step is a no-op and the value grid over arrival states is W.

wf reads only the axes it has, so it raises ValueError rather than silently discard an arrival state that the value grid has no axis for — with D-3’s survival state liv left off the grid, the mortality discount cancels out of the backup and the loop converges to the no-mortality policy, a plausible value for a different model.

Parameters:

value_array (DataArray) – A gridded decision value function, e.g. the value_array returned by bellman_step(); its axes are arrival states and any observed-shock nodes.
bp (BellmanPeriod) – The recurring period; used to identify and discretize the shock axes.
disc_params (Mapping) – Per-shock discretization arguments for the observed-shock axes, keyed by shock symbol. A shock axis without an entry uses its distribution’s default discretization.

Returns:

wf(states, shocks, parameters) returning the interpolated arrival value at states. shocks and parameters are accepted for the bp.compute_value calling convention but unused.

Return type:

Callable

Core VFI Functions¶

skagent.algos.vfi.solve(block, continuation, state_grid, disc_params={}, scope={})

Solve a DBlock stage by value function iteration.

Parameters:

block (DBlock) – The stage to solve. Must contain at most one control variable; multi-control stages raise Exception.
continuation (callable) – The continuation value function, called with the post-transition values of the variables named in its signature. Fold any discount factor into this function (the backup is reward + continuation).
state_grid (Mapping[str, Sequence]) – A grid over the control’s information set: one axis per variable the decision may condition on. The returned decision rule takes these as positional arguments in control.iset order. Variables the dynamics need but the decision does not (e.g. a shock that only enters the transition) go in scope, not here. For an empty information set, pass {}.
disc_params (Mapping, optional) – Discretization parameters for the shock distribution, forwarded to block.get_arrival_value_function.
scope (Mapping, optional) –
The fixed scope for the per-point optimization: merged with each grid point to form the pre_states under which the dynamics, reward, and continuation are evaluated.

Note

This is broader than calibration elsewhere in the library, which denotes fixed, single-valued parameters only. Here (legacy VFI usage) it is a general scope bag that also holds fixed exogenous values outside the information set, such as a shock realization psi. Read it as “scope,” not “parameters.”

Returns:

dr_from_data (dict of callable) – One decision rule per control, keyed by control symbol; each takes its information-set values as positional arguments in control.iset order.
dec_vf (callable) – The decision value function for the fitted rule.
arr_vf (callable) – The arrival value function for the fitted rule (takes the shock expectation via disc_params).

skagent.algos.vfi.get_action_rule(action)

Build a constant decision rule that ignores its inputs.

Parameters:: action (Any) – The fixed value the rule returns.
Returns:: A zero-argument function ar() returning action. Used to wrap a candidate action as a decision rule during the per-point optimization.
Return type:: callable

skagent.algos.vfi.ar_from_data(da)

Build a decision rule from a fitted policy DataArray.

Interpolation runs in numpy/xarray space. For a torch-tensor interface, wrap the result with tensor_decision_rule().

Parameters:: da (xarray.DataArray) – The fitted policy, with one dimension per information-set variable in control.iset order. A zero-dimensional array encodes a constant rule (empty information set).
Returns:: A rule ar(*args) taking one positional argument per dimension of da. Scalar arguments return a Python scalar; array-like arguments are interpolated pointwise (not as an outer product) and return a numpy array.
Return type:: callable
Raises:: TypeError – If the number of positional arguments does not match da.ndim.

skagent.algos.vfi.grid_to_data_array(grid={})

Construct a zero-valued DataArray over the coordinates of a grid.

Parameters:: grid (Mapping[str, Sequence]) – A mapping from variable labels to a sequence of numerical values. An empty mapping yields a zero-dimensional array.
Returns:: An array whose dimensions and coordinates are those of grid.
Return type:: xarray.DataArray

Best Response¶

Solves a block’s decisions one at a time, in the order given by its relevance graph (see Model Analysis and Visualization), each decision maximizing its own agent’s payoff conditional on what that decision observes. For blocks whose relevance graph is acyclic; a cyclic component has to be solved as a simultaneous-move equilibrium and raises instead.

Best-response solving for blocks with several decisions.

Solves the decisions of a DBlock one at a time, in the order given by the block’s relevance graph, so that every decision rule a decision strategically relies on is already computed when its turn comes. This requires the relevance graph to be acyclic; a cyclic component is a set of decisions that have to be solved jointly, which is a simultaneous-move equilibrium problem and is not attempted here.

Each decision is solved per information cell: for every value of what the decision-maker observes, the action maximizing their expected payoff conditional on that observation. Expectations are estimated by drawing samples realizations of the block’s shocks once, at construction, and reusing the same draws for every candidate action (common random numbers), so that comparisons between actions carry far less error than their levels do. Conditioning is done by grouping the simulated samples, which makes the beliefs at a decision the posterior induced by the other decision rules in the profile.

The representation is tabular, which sets the limits of this module: a decision’s information set must take finitely many values under the profile being played, since a cell with no repeats has no conditional expectation to estimate, and the actions searched are a finite set. Shocks may be continuous – they are integrated over, never conditioned on. For continuous observations, or where a differentiable policy is wanted, skagent.solver.solve_multiple_controls() performs the same sweep with a policy network per control in place of a table.

class skagent.algos.best_response.ConditionalPayoffs(cells, counts, actions, payoff)¶

Expected payoffs by information cell and action.

cellsnumpy.ndarray: Shape (n_cells, len(iset)); the distinct observed values of the decision’s information set. Shape (1, 0) for an empty information set.
countsnumpy.ndarray: Shape (n_cells,); how many samples support each cell.
actionsnumpy.ndarray: Shape (n_actions,); the candidate actions searched.
payoffnumpy.ndarray: Shape (n_cells, n_actions); the estimated expected payoff of the decision’s agent in each cell for each action.

actions¶: Alias for field number 2

cells¶: Alias for field number 0

counts¶: Alias for field number 1

payoff¶: Alias for field number 3

class skagent.algos.best_response.TabularBestResponseSolver(block, calibration=None, *, actions=None, action_count=21, samples=100000, rng=None, max_cells=1024)¶

Solve a block’s decisions by best response, in relevance-graph order.

Parameters:

block (skagent.block.DBlock) – The block to solve. Its dynamics must be declared in topological order, as the library requires of any block, and its controls must carry the agent attribution needed to tell whose payoff each maximizes when the block’s utilities are owned by more than one agent.
calibration (dict, optional) – Parameter values, used both to construct the block’s shocks and as values for any parameter the dynamics refer to. Defaults to empty.
actions (array_like, optional) – Candidate actions to search, shared by every decision. Defaults to action_count points spanning [0, 1].
action_count (int, optional) – Number of candidate actions when actions is not given.
samples (int, optional) – Number of shock realizations drawn at construction. Grouping splits these across a decision’s information cells, so each cell’s expectation rests on the samples that reached it rather than on all of them.
rng (numpy.random.Generator, optional) – Generator for the shock draws.
max_cells (int, optional) – Upper limit on the number of information cells a single decision may have. Exceeding it raises, since grouping samples by observed value only estimates a conditional expectation when observations repeat.

Notes

Constructing the solver draws the block’s shocks and mutates the block’s shock distributions, as skagent.block.DBlock.construct_shocks() does.

Expectations are Monte Carlo estimates; a solved rule is exact only up to sampling error, which falls as samples rises.

best_response(decision, policies)¶

The payoff-maximizing rule for decision against policies.

Parameters:

decision (str) – The control to solve.
policies (Mapping[str, Callable]) – A decision rule for every control of the block; the profile the best response is computed against.

Returns:

One action per information cell.

Return type:

TabulatedRule

conditional_payoffs(decision, policies)¶

Estimate decision’s payoffs by information cell and action.

Parameters:

decision (str) – The control to evaluate.
policies (Mapping[str, Callable]) – A decision rule for every control of the block, including decision itself: the profile the expectation is taken under.

Return type:

ConditionalPayoffs

initial_policies()¶: A mixed rule for every decision in the block.

mixed_rule(weights=None)¶

A full-support mixed rule: every action played on some samples.

Held by decisions that are not yet solved, so that every information cell is reached and every conditional expectation is defined.

Parameters:: weights (array_like, optional) – Relative shares of the samples per action, one per candidate action. Defaults to equal shares.
Returns:: A decision rule returning one action per sample.
Return type:: callable

payoff(vals, agent)¶: The sum of agent’s utility nodes, per sample.

solve(policies=None)¶

Solve every decision, in relevance-graph order.

Parameters:: policies (Mapping[str, Callable], optional) – Starting rules for the decisions, replaced one by one as they are solved. Defaults to a mixed rule per decision, which is what makes the conditional expectation at every information cell defined.
Returns:: A decision rule per control of the block.
Return type:: dict
Raises:: NotImplementedError – If the relevance graph has a cyclic component: those decisions rely on each other and admit no one-at-a-time order.

class skagent.algos.best_response.TabulatedRule(iset, cells, actions)¶

A decision rule tabulated over the cells of an information set.

Follows the library’s decision-rule calling convention: the information-set values are passed as positional arguments in iset order. An observation is answered with the action of the nearest tabulated cell, so a rule remains total when queried away from the values it was tabulated on.

For a rule tabulated on a full grid, and where interpolation between grid points is wanted, use skagent.algos.vfi.ar_from_data() instead.

Parameters:

iset (sequence of str) – The information set, in the order the cell columns are given.
cells (array_like) – Shape (n_cells, len(iset)); the tabulated observations.
actions (array_like) – Shape (n_cells,); the action for each cell.

to_dict(decimals=3)¶: The rule as {observed values: action}, rounded for display.

Core Best-Response Classes¶

class skagent.algos.best_response.TabularBestResponseSolver(block, calibration=None, *, actions=None, action_count=21, samples=100000, rng=None, max_cells=1024)

Solve a block’s decisions by best response, in relevance-graph order.

Parameters:

block (skagent.block.DBlock) – The block to solve. Its dynamics must be declared in topological order, as the library requires of any block, and its controls must carry the agent attribution needed to tell whose payoff each maximizes when the block’s utilities are owned by more than one agent.
calibration (dict, optional) – Parameter values, used both to construct the block’s shocks and as values for any parameter the dynamics refer to. Defaults to empty.
actions (array_like, optional) – Candidate actions to search, shared by every decision. Defaults to action_count points spanning [0, 1].
action_count (int, optional) – Number of candidate actions when actions is not given.
samples (int, optional) – Number of shock realizations drawn at construction. Grouping splits these across a decision’s information cells, so each cell’s expectation rests on the samples that reached it rather than on all of them.
rng (numpy.random.Generator, optional) – Generator for the shock draws.
max_cells (int, optional) – Upper limit on the number of information cells a single decision may have. Exceeding it raises, since grouping samples by observed value only estimates a conditional expectation when observations repeat.

Notes

Constructing the solver draws the block’s shocks and mutates the block’s shock distributions, as skagent.block.DBlock.construct_shocks() does.

Expectations are Monte Carlo estimates; a solved rule is exact only up to sampling error, which falls as samples rises.

best_response(decision, policies)

The payoff-maximizing rule for decision against policies.

Parameters:

decision (str) – The control to solve.
policies (Mapping[str, Callable]) – A decision rule for every control of the block; the profile the best response is computed against.

Returns:

One action per information cell.

Return type:

TabulatedRule

conditional_payoffs(decision, policies)

Estimate decision’s payoffs by information cell and action.

Parameters:

decision (str) – The control to evaluate.
policies (Mapping[str, Callable]) – A decision rule for every control of the block, including decision itself: the profile the expectation is taken under.

Return type:

ConditionalPayoffs

initial_policies(): A mixed rule for every decision in the block.

mixed_rule(weights=None)

A full-support mixed rule: every action played on some samples.

Held by decisions that are not yet solved, so that every information cell is reached and every conditional expectation is defined.

Parameters:: weights (array_like, optional) – Relative shares of the samples per action, one per candidate action. Defaults to equal shares.
Returns:: A decision rule returning one action per sample.
Return type:: callable

payoff(vals, agent): The sum of agent’s utility nodes, per sample.

solve(policies=None)

Solve every decision, in relevance-graph order.

Parameters:: policies (Mapping[str, Callable], optional) – Starting rules for the decisions, replaced one by one as they are solved. Defaults to a mixed rule per decision, which is what makes the conditional expectation at every information cell defined.
Returns:: A decision rule per control of the block.
Return type:: dict
Raises:: NotImplementedError – If the relevance graph has a cyclic component: those decisions rely on each other and admit no one-at-a-time order.

class skagent.algos.best_response.TabulatedRule(iset, cells, actions)

A decision rule tabulated over the cells of an information set.

For a rule tabulated on a full grid, and where interpolation between grid points is wanted, use skagent.algos.vfi.ar_from_data() instead.

Parameters:

iset (sequence of str) – The information set, in the order the cell columns are given.
cells (array_like) – Shape (n_cells, len(iset)); the tabulated observations.
actions (array_like) – Shape (n_cells,); the action for each cell.

to_dict(decimals=3): The rule as {observed values: action}, rounded for display.

Maliar-Style Algorithms¶

Neural network-based solution methods following Maliar et al.

Tools for the implementation of the Maliar, Maliar, and Winant (JME ‘21) method.

This method relies on a simpler problem representation than that elaborated by the skagent Block system.

Note

generate_givens_from_states currently accesses bellman_period.block directly rather than working through the BellmanPeriod interface. A future refactoring could route shock generation through BellmanPeriod itself. Similarly, shock draws are currently Monte Carlo only; structured draws (e.g. exact discretizations) could be supported via BellmanPeriod.

skagent.algos.maliar.generate_givens_from_states(states, model_block, shock_copies)¶

Generate omega_i values of the MMW JME ‘21 method.

Parameters:

states (Grid) – A grid of starting state values (exogenous and endogenous).
model_block (Block) – Block information (used to get the shock names).
shock_copies (int) – Number of copies of the shocks to be included. Must be >= 1.

Returns:

Grid containing states augmented with shock copies.

Return type:

Grid

skagent.algos.maliar.maliar_training_loop(bellman_period, loss_function, states_0_n, parameters, shock_copies=2, max_iterations=5, tolerance=1e-06, random_seed=None, simulation_steps=1, network_width=16, epochs_per_iteration=250, lr=0.001)¶

Run the Maliar, Maliar, and Winant (JME ‘21) training loop.

Trains a single neural network policy to minimize empirical risk (loss) on a panel of states drawn forward through the model dynamics. This helper constructs and trains a BlockPolicyNet internally and does not currently accept a pre-built shared-backbone BlockPolicyValueNet. If value-aware training is needed (e.g. for a Bellman residual loss with a value head), call train_block_nn() directly on a BlockPolicyValueNet; a future refactor may add value-network support here.

The loop maps onto the MMW JME’21 algorithm steps as follows: _validate_training_inputs() and the network construction below cover Step 1 (initialize topology and coefficients); the per-iteration train_block_nn() call is Step 2 (minimize the empirical risk \(\Xi^n(\theta)\)); the returned network is the Step 3 trained approximation \(\varphi(\cdot, \theta)\).

Parameters:

bellman_period (BellmanPeriod) – A model definition containing block dynamics and transitions.
loss_function (Callable) – The empirical risk function \(\Xi^n\) from MMW JME’21. This function is passed to the neural network training routine as loss_function(decision_function, input_grid) -> loss_tensor.
states_0_n (Grid) – A panel of starting states for training. Must contain at least one state.
parameters (dict) – Given parameters for the model.
shock_copies (int) – Number of shock copies to include in the training set \(\{\omega_i\}\). Must match the expected number of shock copies in the loss function. Must be >= 1. Default is 2.
max_iterations (int) – Maximum number of training loop iterations before stopping. Must be >= 1. Default is 5.
tolerance (float) – Convergence tolerance. Training stops when either the L2 norm of parameter changes or the absolute difference in loss is below this threshold. Satisfying either criterion alone is sufficient. Must be > 0. Default is 1e-6.
random_seed (Optional[int]) – Random seed for reproducibility. Default is None.
simulation_steps (int) – Number of time steps to simulate forward when determining the next training set \(\{\omega_i\}\). Higher values let the training states explore more of the state space at higher computational cost. Must be >= 1. Default is 1.
network_width (int) – Width of hidden layers in the policy neural network. Must be >= 1. Default is 16.
epochs_per_iteration (int) – Number of training epochs per iteration. Must be >= 1. Default is 250.
lr (float) – Learning rate for the internal Adam optimizer. The optimizer is created once and reused across iterations to preserve momentum. Must be > 0. Default is 0.001.

Returns:

(trained_policy_network, training_states) where trained_policy_network is the trained BlockPolicyNet and training_states is the Grid of states from the final iteration (the convergence point if training converged early, otherwise the states after max_iterations steps).

Return type:

tuple

Raises:

ValueError – If max_iterations < 1, tolerance <= 0, shock_copies < 1, simulation_steps < 1, network_width < 1, epochs_per_iteration < 1, or states_0_n contains no states.
TypeError – If bellman_period is None or loss_function is not callable.

skagent.algos.maliar.simulate_forward(states_t, bellman_period, decision_function, parameters, big_t)¶

Simulate the model forward for a specified number of periods.

Parameters:

states_t (Grid | dict) – Initial state values.
bellman_period (BellmanPeriod) – The Bellman period containing model dynamics.
decision_function (Callable) – Function mapping (states, shocks, parameters) to controls.
parameters (dict) – Model parameters.
big_t (int) – Number of time periods to simulate forward. If 0, returns the initial states unchanged.

Returns:

Final state values after big_t periods.

Return type:

dict

Raises:

ValueError – If big_t < 0 or if states_t is an empty dict.

Reinforcement Learning (Stable-Baselines3)¶

Proximal Policy Optimization (PPO) for BellmanPeriod models, via a Stable-Baselines3 backend. The agent wraps a model in a gymnasium environment (see Environments), trains PPO, and emits a standard skagent decision rule.

Stable Baselines3 wrappers for BellmanPeriod models.

Provides PPOAgent, a thin wrapper around SB3’s PPO that:

builds a skagent.env.GymEnv from a BellmanPeriod + initial state distribution,
delegates training to stable_baselines3.PPO.learn,
exposes a PPOAgent.decision_rule() that returns the trained policy as a skagent-style {control_sym: callable} dict — i.e. the same shape consumed by skagent.env.Environment and the rest of the skagent decision-rule API. Actions are unscaled back to real units via GymEnv.unscale_action(), so downstream code does not see the [-1, 1] SB3 representation.

class skagent.algos.sb3.PPOAgent(bp, initial, *, max_episode_steps=200, seed=None, gym_kwargs=None, ppo_kwargs=None, policy='MlpPolicy', device='cpu', verbose=0)¶

Train SB3’s PPO on a BellmanPeriod and emit a skagent decision rule.

Parameters:

bp (BellmanPeriod) – Model definition.
initial (dict) – Maps arrival-state symbols to skagent Distribution objects, used by GymEnv to sample fresh initial states on reset.
max_episode_steps (int) – Episode horizon for the underlying GymEnv. Default 200.
seed (Optional[int]) – Seed for both the environment and the PPO algorithm.
gym_kwargs (Optional[dict]) – Extra keyword arguments forwarded to GymEnv (e.g. default_lower, default_upper, bound_clearance, control_sym).
ppo_kwargs (Optional[dict]) – Extra keyword arguments forwarded to stable_baselines3.PPO (e.g. n_steps, batch_size, learning_rate, n_epochs, policy_kwargs). gamma defaults to bp.calibration[bp.discount_variable] if it is a finite scalar; callers can override by passing gamma here.
policy (str) – SB3 policy class string. Default "MlpPolicy".
device (str) – Torch device for PPO. Default "cpu" (SB3’s recommended default for MlpPolicy).
verbose (int) – Verbosity passed to PPO. Default 0.

env¶

The constructed gymnasium environment.

Type:: GymEnv

model¶

The SB3 model. None until learn() is called the first time (constructed lazily so callers can inspect env without paying PPO’s setup cost).

Type:: stable_baselines3.PPO

decision_rule(deterministic=True)¶

Return a skagent decision rule that uses the trained policy.

The returned dict has the form {control_sym: callable} where the callable accepts positional arguments matching the control’s iset order (i.e. the same signature skagent Environment.step and BellmanPeriod.decision_function call). The callable’s output is a torch.Tensor of unscaled action values; the inputs may be scalars, numpy arrays, or torch tensors of compatible length.

Parameters:: deterministic (bool) – Whether to use a deterministic (mean) policy. Default True — matches typical skagent decision-rule semantics.
Return type:: dict[str, Callable]

learn(total_timesteps, callback=None, **kwargs)¶

Run PPO.learn. Returns self.

Every completed episode’s undiscounted reward is appended to self.episode_rewards via an internal SB3 callback; repeated learn calls accumulate. A user-supplied callback is merged with the internal one via CallbackList. Extra **kwargs forward to model.learn.

Parameters:

total_timesteps (int)
callback (Any)
kwargs (Any)

Return type:

PPOAgent

predict_unscaled(obs, deterministic=True)¶

Predict an unscaled action for obs.

obs may be a single observation (shape (|iset|,)) or a batch ((N, |iset|)). Returns a 1-D array of shape (N,).

Parameters:: deterministic (bool)
Return type:: ndarray

snapshot()¶

Capture the current trained policy as a frozen PolicySnapshot.

The snapshot holds an independent copy of the policy network, so it is unaffected by later learn() calls. This is the supported way to retain the policy at intermediate points during training (e.g. to compare checkpoints) without re-running training or re-implementing the unscaling logic.

Return type:: PolicySnapshot

class skagent.algos.sb3.PolicySnapshot(policy, env)¶

Frozen copy of a trained policy, decoupled from further training.

Returned by PPOAgent.snapshot(). Holds a deep copy of the policy network taken at snapshot time, so subsequent learn calls on the source agent do not change its predictions. Exposes the same predict_unscaled() and decision_rule() interface as PPOAgent.

The GymEnv is shared with the source agent (not copied): it is used only for stateless action unscaling, which does not depend on training state.

Parameters:: env (GymEnv)

decision_rule(deterministic=True)¶

Return a skagent decision rule; see PPOAgent.decision_rule().

Parameters:: deterministic (bool)
Return type:: dict[str, Callable]

predict_unscaled(obs, deterministic=True)¶

Predict an unscaled action for obs; see PPOAgent.predict_unscaled().

Parameters:: deterministic (bool)
Return type:: ndarray

Loss Functions¶

Objective functions passed to skagent.ann.train_block_nn(). The reward-based losses (StaticRewardLoss, EstimatedDiscountedLifetimeRewardLoss) solve a block directly for the non-recurring case; the equation-residual losses (BellmanEquationLoss, EulerEquationLoss) target the recurring, dynamic case. See Loss Functions for the full reference.

Neural Network Components¶

Net¶

Base neural network class with device management.

class skagent.ann.Net(n_inputs, n_outputs, width=32, n_layers=2, activation='silu', transform=None, init_seed=None, copy_weights_from=None)¶

Bases: Module

A flexible feedforward neural network with configurable architecture.

Parameters:

n_inputs (int) – Number of input features
n_outputs (int) – Number of output features
width (int, optional) – Width of hidden layers. Default is 32.
n_layers (int, optional) – Number of hidden layers (1-10). Default is 2.
activation (str, list, callable, or None, optional) –
Activation function(s) to use. Options: - str: Apply same activation to all layers (‘silu’, ‘relu’, ‘tanh’, ‘sigmoid’) - list: Apply different activations to each layer, e.g., [‘relu’, ‘tanh’, ‘silu’] - callable: Custom activation function - None: No activation (identity function)

Available activations: ‘silu’, ‘relu’, ‘tanh’, ‘sigmoid’, ‘identity’ Default is ‘silu’.
transform (str, list, callable, or None, optional) –
Transformation to apply to outputs. Options: - str: Apply same transform to all outputs (‘sigmoid’, ‘exp’, ‘tanh’, etc.) - list: Apply different transforms to each output, e.g., [‘sigmoid’, ‘exp’] - callable: Custom transformation function - None: No transformation

Available transforms: ‘sigmoid’, ‘exp’, ‘tanh’, ‘relu’, ‘softplus’, ‘softmax’, ‘abs’, ‘square’, ‘identity’ Default is None.

property device¶: Device property for backward compatibility.

forward(x)¶

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

BlockPolicyNet¶

Specialized neural network for policy functions in economic models.

class skagent.ann.BlockPolicyNet(bellman_period, control_sym=None, apply_open_bounds=True, width=32, **kwargs)¶

Bases: BellmanPeriodMixin, Net

A neural network for policy functions in dynamic programming problems.

This network wraps a Net and integrates with the BellmanPeriod interface. It automatically determines input/output dimensions from the model block specification and enforces control variable bounds.

Parameters:

bellman_period (BellmanPeriod) – The model Bellman Period
apply_open_bounds (bool, optional) – If True, then the network forward output is normalized by the upper and/or lower bounds, computed as a function of the input tensor. These bounds are “open” because output can be arbitrarily close to, but not equal to, the bounds. Default is True.
control_sym (string, optional) – The symbol for the control variable.
width (int, optional) – Width of hidden layers. Default is 32.
**kwargs – Additional keyword arguments passed to Net. See Net class documentation for all available options including activation, transform, n_layers, init_seed, copy_weights_from, etc.

decision_function(states_t, shocks_t, parameters)¶

A decision function, from states, shocks, and parameters, to control variable values.

Parameters:

states_t (dict) – symbols : values
shocks_t (dict) – symbols: values
parameters (dict) – symbols : values

Returns:

decisions - dict – symbols : values

forward(x)¶: Note that this uses the same architecture of the superclass but adds on a normalization layer appropriate to the bounds of the decision rule.

get_core_function(length=None)¶

get_decision_rule(length=None)¶: Returns the decision rule corresponding to this neural network.

BlockValueNet¶

A neural network for value functions in dynamic programming problems.

class skagent.ann.BlockValueNet(bellman_period, control_sym=None, width=32, **kwargs)¶

Bases: BellmanPeriodMixin, Net

Standalone value-function network for a Bellman problem.

Maps a control’s information set (the same pre-decision states a policy network sees) to a single unconstrained scalar value. It is the value-only counterpart of BlockPolicyNet, kept for algorithms that approximate a value function separately from the policy. The Maliar/MMW path in this package uses the shared-backbone BlockPolicyValueNet instead, so BlockValueNet is not wired into maliar_training_loop().

Parameters:

bellman_period (BellmanPeriod) – The model Bellman period.
control_sym (str, optional) – Control whose information set defines the value function’s domain. Defaults to the first control.
width (int) – Width of hidden layers. Default 32.
**kwargs – Passed to Net (activation, n_layers, init_seed, etc.).

get_core_function(length=None)¶: Return the value function (the trainable core for this net).

get_value_function()¶: Return a callable (states, shocks, parameters) -> value tensor.

value_function(states_t, shocks_t=None, parameters=None)¶

Evaluate the value function at the control’s information set.

Arrival states_t (with shocks_t and parameters) are mapped to the control’s information set via compute_pre_state(), mirroring BlockPolicyNet.decision_function(), then the network is evaluated.

Returns:: Flattened value estimates, one per input row.
Return type:: torch.Tensor

BlockPolicyValueNet¶

A shared-backbone neural network that jointly represents the policy and value functions.

class skagent.ann.BlockPolicyValueNet(bellman_period, control_sym=None, apply_open_bounds=True, width=32, **kwargs)¶

Bases: BellmanPeriodMixin, Net

Single neural network with shared backbone for both policy and value.

Architecture: shared hidden layers → two output heads: - Policy head — bounded output (sigmoid-scaled to satisfy constraints) - Value head — unconstrained scalar output

Sharing the backbone means one optimizer updates all weights simultaneously, and the value head anchors the control level that first-order-condition-only training (e.g. an Euler residual loss) cannot identify.

Parameters:

bellman_period (BellmanPeriod) – The model Bellman Period.
control_sym (str, optional) – Control variable symbol. Defaults to first control.
apply_open_bounds (bool, optional) – Apply sigmoid/softplus scaling to the policy head. Default True.
width (int, optional) – Width of hidden layers. Default 32.
**kwargs – Passed to Net (activation, n_layers, init_seed, etc.).

decision_function(states_t, shocks_t, parameters)¶

Map states, shocks, and parameters to a controls dict.

Parameters:

states_t (dict) – Arrival state values, symbol -> tensor.
shocks_t (dict or None) – Shock values, symbol -> tensor (None is treated as {}).
parameters (dict) – Model parameters, symbol -> value.

Returns:

{control_sym: tensor} of policy-head outputs. The arrival states are mapped to the control’s information set via compute_pre_state() before the network is evaluated.

Return type:

dict

forward(x)¶

Run shared backbone, then policy head (bounded) + value head.

Returns the (policy, value) pair. The policy tensor is scaled into the control’s open bounds; the value tensor is unconstrained. Both have shape (n, 1).

Return type:: tuple[Tensor, Tensor]

get_core_function(length=None)¶: Return decision rules (policy head) for use with train_block_nn.

get_decision_rule(length=None)¶: Decision rule returning only the policy output.

get_policy_and_value_functions(length=None)¶: Return both policy decision rules and value function.

get_value_function()¶

value_function(states_t, shocks_t=None, parameters=None)¶

Evaluate the value head at the control’s information set.

The input domain mirrors decision_function(): arrival states_t (with shocks_t and parameters) are mapped to the control’s information set via compute_pre_state(), then the shared backbone’s value head is evaluated on that pre-decision representation.

Parameters:

states_t (dict) – Arrival state values, symbol -> tensor.
shocks_t (dict or None, optional) – Shock values (None is treated as {}).
parameters (dict or None, optional) – Model parameters.

Returns:

Flattened value estimates, one per input row.

Return type:

torch.Tensor

Training Functions¶

skagent.ann.train_block_nn(block_policy_nn, inputs, loss_function, epochs=50, lr=0.01, optimizer=None, grad_clip=1.0, verbose=True)¶

Train a policy network by minimizing a loss function over a grid.

This is a generic stochastic-gradient-descent driver, not a solution algorithm in itself. It runs epochs Adam updates that minimize whatever loss_function is supplied, evaluated on a single, fixed grid of inputs; it is agnostic to where that grid came from or which method the loss encodes (Euler residual, Bellman residual, FOC, or a custom loss).

Because it trains on whatever inputs it is given, accuracy depends on the caller re-sampling those states across calls: Maliar, Maliar, and Winant (2021) keep the training data “constantly re-sampled,” and minimizing on a single fixed grid instead lets the solution over-fit those points while drifting elsewhere. Re-draw inputs each call (threading the returned optimizer back in to keep Adam’s momentum), or use maliar_training_loop(), which wraps this driver in the full MMW’21 outer loop: it alternates these inner SGD updates with a forward-simulation step that refreshes the training states toward the model’s ergodic set.

Parameters:

block_policy_nn (BlockPolicyNet or BlockPolicyValueNet) – The network to train. Its get_core_function supplies the decision rule(s) the loss is evaluated against.
inputs (Grid) – Input grid containing states and shocks.
loss_function (Callable) – Loss function (decision_function, input_grid) -> loss_tensor.
epochs (int) – Number of training epochs (default 50).
lr (float) – Learning rate for Adam optimizer (default 0.01).
optimizer (Optional[Optimizer]) – Pre-existing optimizer to reuse (preserves momentum across calls). If None, a new Adam optimizer is created.
grad_clip (Optional[float]) – Maximum gradient norm for clipping (default 1.0). Set to None to disable.
verbose (bool) – Emit a logging.info message with the loss every 100 epochs (default True). Configure the root logger to suppress these.

Returns:

(trained_network, final_loss, optimizer). The optimizer is the one passed in, or the Adam instance created internally when none was supplied; returning it always lets callers warm-start a later call by threading it back in.

Return type:

tuple

skagent.ann.aggregate_net_loss(inputs, df, loss_function)¶

Compute a loss function over a tensor of inputs, given a decision function df. Return the mean.

Parameters:: inputs (Grid)

skagent.solver.solve_multiple_controls(control_order, bellman_period, givens, calibration, epochs=200, loss=None)¶

Solve a block with more than one control by training a policy network for each control in turn.

Each control is given its own skagent.ann.BlockPolicyNet. The networks are trained one at a time, in the order given by control_order, with every network treating the other networks’ current policies as fixed. A control may appear in control_order more than once to refine it after its neighbours have been updated (e.g. ["c", "d", "c"]), which is the multi-control analogue of a best-response sweep.

Currently restricted to single-period (non-recurring) reward objectives; by default the negative immediate reward (skagent.loss.StaticRewardLoss) is maximized.

Parameters:

control_order (list of str) – Control symbols, in the order they should be solved. Symbols may repeat to schedule additional refinement passes.
bellman_period (BellmanPeriod) – The model period whose controls are being solved.
givens (skagent.grid.Grid) – Grid of arrival states and shock realizations to train over.
calibration (dict) – Calibration parameters passed to the loss function.
epochs (int, optional) – Training epochs per pass. Default is 200.
loss (type, optional) – A loss-function class with signature loss(bellman_period, parameters, other_dr). Defaults to skagent.loss.StaticRewardLoss.

Returns:

Mapping from each control symbol to its trained decision rule.

Return type:

dict

Grid and Computational Tools¶

Grid Class¶

class skagent.grid.Grid(labels, values, torched=True)¶

Bases: object

A class representing a labeled grid of numerical values.

Parameters:: (dict) (config) – dictionary with the following keys: “min” (float): The minimum value for the variable.; “max” (float): The maximum value for the variable; “count” (int): The number of points to generate for the variable.

classmethod from_config(config={}, torched=True)¶

classmethod from_dict(kv={}, torched=False)¶

len()¶: Returns the number of columns, similar to a dict.

n()¶: Returns the number of values for each symbol

shape()¶: Returns the shape of the grid values.

to_dict()¶: Returns a data structure, key: column, similar to tensordict or structured array.

torch()¶

update_from_dict(kv)¶

Grid Utility Functions¶

skagent.grid.make_grid(config)¶

Make a ‘grid’ of values based on the provided configuration.

Parameters:

(dict) (config) – dictionary with the following keys: “min” (float): The minimum value for the variable; “max” (float): The maximum value for the variable; “count” (int): The number of points to generate for the variable.

Returns:

numpy.ndarray (A NumPy array of shape (product_of_counts, num_variables), where) – product_of_counts is the product of all count values in the config dictionary, and num_variables is the number of keys in the config.

skagent.grid.cartesian_product(*arrays)¶

Create a Cartesian product of input arrays.

Parameters:

*arrays – Variable length arrays to compute product

Returns:

Array of shape (product_of_lengths, num_arrays)
where product_of_lengths is the product of the lengths of the input arrays,
and num_arrays is the number of input arrays. Each row contains one element
of the Cartesian product.