prysm.x.optym#

Optimizers#

Various optimization algorithms.

prysm.x.optym.optimizers.runN(optimizer, N)#

Perform N iterations of optimization.

Parameters:

optimizer (Any) –

any optimizer from this file, or any type which implements

def step(self): -> (xk, fk, gk)
pass
N (int) – number of iterations to perform

Returns:

yielding (xk, fk, gk) at each iteration

Return type:

generator

prysm.x.optym.optimizers.run_until(optimizer, governor, *, maxiter=None)#

Run an optimizer until a governor decides to stop.

Parameters:

optimizer (Any) – Optimizer exposing a step method returning x, f, g. After each step, optimizer.x is expected to hold the post-step iterate.
governor (Governor) – Stop condition that observes each completed step.
maxiter (int, optional) – Safety cap for runner-owned iterations. This is independent of any MaxIterations governor supplied by the caller.

Returns:

Result with the final optimizer.x, terminal decision, and step records.

Return type:

OptimizationResult

class prysm.x.optym.optimizers.GradientDescent(fg, x0, alpha, lower_bounds=None, upper_bounds=None)#

Bases: object

Gradient Descent optimization routine.

Gradient Descent travels a constant step size alpha along the negative of the gradient on each iteration. The update is:

\[x_{k+1} = x_k - α g_k\]

where g is the gradient vector

The user may anneal alpha over the course of optimization if they wish. The cost function is not used, nor higher order information.

Create a new GradientDescent optimizer.

Parameters:

fg (callable) – a function which returns (f, g) where f is the scalar cost, and g is the vector gradient.
x0 (callable) – the parameter vector immediately prior to optimization
alpha (float) – the step size the user may mutate self.alpha over the course of optimization with no negative effects (except optimization blowing up from a bad choice of new alpha)
lower_bounds (ndarray, optional) – per-variable hard bounds. None means unconstrained on that side.
upper_bounds (ndarray, optional) – per-variable hard bounds. None means unconstrained on that side.

step()#: Perform one iteration of optimization.

class prysm.x.optym.optimizers.AdaGrad(fg, x0, alpha, lower_bounds=None, upper_bounds=None)#

Bases: _Accumulator

Adaptive Gradient Descent optimization routine.

Gradient Descent has the same step size for each parameter. Adagrad self- learns a unique step size for each parameter based on accumulation of the square of the gradient over the course of optimization. The update is:

\[\begin{split}s_k &= s_{k-1} + (g*g) \\ x_{k+1} &= x_k - α g_k / \sqrt{s_k \,}\end{split}\]

The purpose of the square and square root operations is essentially to destroy the sign of g in the denomenator gain. An alternative may be to simply do s_k = s_(k-1) + abs(g), which would have less numerical precision issues.

Ref [1] describes a “”fully connected”” version of AdaGrad that is a cousin of sorts to BFGS, storing an NxN matrix. This is intractible for large N. The implementation here is sister to most implementations in the wild, and is the “Diagonal” implementation, which stores no information about the relationship between “spatial” elements of the gradient vector. Only the temporal relationship between the gradient and its past is stored.

Constructed with AdaGrad(fg, x0, alpha) — see _Accumulator for the shared init signature.

References

[1] Duchi, John, Hazan, Elad and Singer, Yoram. “Adaptive Subgradient: Methods for Online Learning and Stochastic Optimization.” https://doi.org/10.5555/1953048.2021068

step()#: Perform one iteration of optimization.

class prysm.x.optym.optimizers.RMSProp(fg, x0, alpha, gamma=0.9, lower_bounds=None, upper_bounds=None)#

Bases: _Accumulator

RMSProp optimization routine.

RMSProp keeps a moving average of the squared gradient of each parameter.

It is very similar to AdaGrad, except that the decay built into it allows it to forget old gradients. This makes it often superior for non-convex problems, where navigation from one valley into another poisons AdaGrad, but RMSProp will eventually forget about the old valley.

The update is:

\[\begin{split}s_k &= γ * s_{k-1} + (1-γ)*(g*g) \\ x_{k+1} &= x_k - α g_k / \sqrt{s_k \,}\end{split}\]

The decay terms gamma form a “moving average” that is squared, with the square root in the gain it is a “root mean square.”

RMSProp is an unpublished algorithm. Its source is Ref [1]

References

[1] Geoffrey Hinton, Nitish Srivastava, Kevin Swersky: “Neural Networks for Machine Learning Lecture 6a Overview of mini-‐batch gradient descent” U Toronto, CSC 321 https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf

step()#: Perform one iteration of optimization.

class prysm.x.optym.optimizers.Adam(fg, x0, alpha, beta1=0.9, beta2=0.999, lower_bounds=None, upper_bounds=None)#

Bases: _MomentBased

ADAM optimization routine.

ADAM, or “Adaptive moment estimation” uses moving average estimates of the mean of the gradient and of its “uncentered variance”. This causes the algorithm to combine several properties of AdaGrad and RMSPRop, as well as perform a form of self-annealing, where the step size will naturally decay as the optimizer converges. This can cause ADAM to recover itself after diverging, if the divergence is not too extreme.

The update is:

\[\begin{split}m &\equiv \text{mean} \\ v &\equiv \text{variance} \\ m_k &= β_1 m_{k-1} + (1-β_1) * g \\ v_k &= β_2 v_{k-1} + (1-β_2) * (g*g) \\ \hat{m}_k &= m_k / (1 - β_1^k) \\ \hat{v}_k &= v_k / (1 - β_2^k) \\ x_{k+1} &= x_k - α * \hat{m}_k / \sqrt{\hat{v}_k \,} \\\end{split}\]

Constructed with Adam(fg, x0, alpha, beta1=0.9, beta2=0.999) — see _MomentBased for the shared init signature.

References

[1] Kingma, Diederik and Ba, Jimmy. “Adam: A Method for Stochastic Optimization”: http://arxiv.org/abs/1412.6980

step()#: Perform one iteration of optimization.

class prysm.x.optym.optimizers.RAdam(fg, x0, alpha, beta1=0.9, beta2=0.999, lower_bounds=None, upper_bounds=None)#

Bases: _MomentBased

RADAM optimization routine.

RADAM or Rectified ADAM is a modification of ADAM, which seeks to remove any need for warmup time with ADAM, and to stabilize the variance estimate or second moment that ADAM uses. These properties make RADAM more invariant to the choice of hyperparameters, especially alpha, and avoid distorting the trajectory of optimization in early iterations.

The update is:

\[\begin{split}m &\equiv \text{mean} \\ v &\equiv \text{variance} \\ \rho_\infty &= \frac{2}{1-β_2} - 1 \\ m_k &= β_1 m_{k-1} + (1-β_1) * g \\ v_k &= β_2 v_{k-1} + (1-β_2) * (g*g) \\ \hat{m}_k &= m_k / (1 - β_1^k) \\ \rho_k &= \rho_\infty - \frac{2 k β_2^k}{1-β_2^k} \\ \text{if}& \rho_k > 5 \\ \qquad l_k &= \sqrt{\frac{1 - β_2^k}{\sqrt{v_k}}} \\ \qquad r_k &= \sqrt{\frac{(\rho_k - 4)(\rho_k-2)\rho_\infty}{(\rho_\infty-4)(\rho_\infty-2)\rho_t}} \\ \qquad x_{k+1} &= x_k - α r_k \hat{m}_k l_k \\ \text{else}& \\ \qquad x_{k+1} &= x_k - α \hat{m}_k\end{split}\]

References

[1] Liu, Liyuan and Jiang, Haoming and He, Pengcheng and Chen, Weizhu and Liu Xiaodong, and Gao, Jianfeng and Han Jiawei. “On the Variance of the Adaptive Learning Rate and Beyond”: http://arxiv.org/abs/1412.6980

step()#: Perform one iteration of optimization.

class prysm.x.optym.optimizers.AdaMomentum(fg, x0, alpha, beta1=0.9, beta2=0.999, lower_bounds=None, upper_bounds=None)#

Bases: _MomentBased

AdaMomentum optimization routine.

AdaMomentum is an algorithm that is extremely similar to Adam, differing only in the calculation of v. The idea is to reduce teh variance of the correction, which can plausibly improve the generality of found solutions, i.e. enter wider local/global minima.

The update is:

\[\begin{split}m &\equiv \text{mean} \\ v &\equiv \text{variance} \\ m_k &= β_1 m_{k-1} + (1-β_1) * g \\ v_k &= β_2 v_{k-1} + (1-β_2) * m_k^2 \\ \hat{m}_k &= m_k / (1 - β_1^k) \\ \hat{v}_k &= v_k / (1 - β_2^k) \\ x_{k+1} &= x_k - α * \hat{m}_k / \sqrt{\hat{v}_k \,} \\\end{split}\]

Constructed with AdaMomentum(fg, x0, alpha, beta1=0.9, beta2=0.999) — see _MomentBased for the shared init signature.

References

[1] Wang, Yizhou and Kang, Yue and Qin, Can and Wang, Huan and Xu, Yi and Zhang Yulun and Fu, Yun. “Rethinking Adam: A Twofold Exponential Moving Average Approach”: https://arxiv.org/abs/2106.11514

step()#: Perform one iteration of optimization.

class prysm.x.optym.optimizers.Yogi(fg, x0, alpha, beta1=0.9, beta2=0.999, lower_bounds=None, upper_bounds=None)#

Bases: _MomentBased

YOGI optimization routine.

YOGI is a modification of ADAM, which replaces the multiplicative update to the exponentially moving averaged estimate of the variance of the gradient with an additive one. The premise for this is that multiplicative update causes ADAM to forget past gradients too quickly, tailoring it to more localized behavior in the second order momentum term. The additive update in YOGI essentially makes the second order momentum update over a larger space. This allows Yogi to perform better than ADAM for some non-convex or otherwise tumultuous cost functions, which have many changes in local curvature.

The update is:

\[\begin{split}m &\equiv \text{mean} \\ v &\equiv \text{variance} \\ m_k &= β_1 m_{k-1} + (1-β_1) * g \\ v_k &= v_{k-1} - (1-β_2) * \text{sign}(v_{k-1} - (g^2))*(g^2) \\ x_{k+1} &= x_k - α * m_k / \sqrt{v_k \,} \\\end{split}\]

Constructed with Yogi(fg, x0, alpha, beta1=0.9, beta2=0.999) — see _MomentBased for the shared init signature.

References

[1] Zaheer, Manzil and Reddi, Sashank and Sachan, Devendra and Kale, Satyen and Kumar, Sanjiv. “Adaptive Methods for Nonconvex Optimization”: https://papers.nips.cc/paper_files/paper/2018/hash/90365351ccc7437a1309dc64e4db32a3-Abstract.html

step()#: Perform one iteration of optimization.

class prysm.x.optym._prysm_lbfgsb.PrysmLBFGSB(fg, x0, memory=10, lower_bounds=None, upper_bounds=None, c1=0.0001, c2=0.9, maxls=30)#

Bases: object

L-BFGS-B optimizer.

Limited-memory BFGS with box constraints. Implements the algorithm of Byrd, Lu, Nocedal, and Zhu, “A Limited-Memory Algorithm for Bound Constrained Optimization”, SIAM J. Sci. Comput. 16(5), 1995, with the subspace-minimization refinement of Morales and Nocedal, “Remark on Algorithm 778”, ACM Trans. Math. Softw. 38(1), 2011: rather than truncating the path from x_k to the unconstrained subspace minimizer x_hat at the first bound it hits, the iterate is taken to be the componentwise projection of x_hat into the box whenever that direction is descent, falling back to BLNZ truncation only when projection yields an ascent direction.

Choose this over the scipy-backed LBFGSB when any of: (1) the objective lives on a non-numpy backend (cupy, pytorch) where the scipy Fortran wrapper would force a host round-trip per fg call; (2) you want to run at config.precision = 32, which the scipy driver refuses; (3) you want the governor to drive termination without inheriting the scipy task-string state machine. The pure- Python implementation closes the gap with the scipy driver as n grows in the Rosenbrock CPU benchmark, but remains slower for tiny problems due to Python per-call overhead.

Parameters:

fg (callable or Problem) – either fg(x) -> (f, g) or a Problem-shaped object; see as_problem.
x0 (ndarray) – the parameter vector immediately prior to optimization. Its dtype sets the dtype of every internal array.
memory (int) – number of recent (s, y) pairs retained for the L-BFGS approximation (typical 10-30).
lower_bounds (ndarray, optional) – per-variable hard bounds. None means unconstrained on that side; +/-inf entries in an explicit array work the same way.
upper_bounds (ndarray, optional) – per-variable hard bounds. None means unconstrained on that side; +/-inf entries in an explicit array work the same way.
c1 (float) – Wolfe constants for the line search.
c2 (float) – Wolfe constants for the line search.
maxls (int) – bracket-phase iteration cap for the line search.

step()#

Perform one iteration of optimization.

Compute the generalized Cauchy point, refine via subspace minimization, project the unconstrained subspace minimizer into the box (Morales-Nocedal 2011), and take a Wolfe line search along the resulting direction. Falls back to BLNZ-style truncation when the projected direction is not descent.

class prysm.x.optym._lbfgsb.CLBFGSB(fg, x0, memory=10, lower_bounds=None, upper_bounds=None)#

Bases: _LBFGSBBase

L-BFGS-B optimizer using the SciPy >= 1.15 C driver.

Behaves identically to F77LBFGSB from the user’s perspective, but targets the C port of L-BFGS-B that replaced the Fortran 77 driver in SciPy 1.15. The differences vs. the Fortran driver are internal:

csave and iprint are no longer arguments to setulb.
A new two-element integer ln_task workspace array is required.
task is a length-2 integer array; task[0] carries the request code (3=FG, 1=NEW_X, 4=CONVERGENCE, 5=STOP).

See F77LBFGSB for parameter documentation; this class accepts the same arguments and exposes the same step/run_to interface.

Create a new L-BFGS-B optimizer.

Parameters:

fg (callable or Problem) – either fg(x) -> (f, g) or a Problem-shaped object; see as_problem.
x0 (ndarray) – the parameter vector immediately prior to optimization.
memory (int) – number of recent gradient vectors used for the approximate Newton step (typical 10–30).
lower_bounds (ndarray, optional) – hard bounds per variable; None is unconstrained.
upper_bounds (ndarray, optional) – hard bounds per variable; None is unconstrained.

property g#: Current iterate as a snapshot, not the driver’s mutable buffer.

run_to(N)#: Run the optimizer until its iteration count equals N.

step()#

Perform one iteration of optimization.

Drives the underlying setulb call sequence until it signals a completed iteration (NEW_X), termination (CONVERGENCE), or an error (STOP). Each NEW_X corresponds to one outer iteration.

property x#: Current iterate as a snapshot, not the driver’s mutable buffer.

class prysm.x.optym.least_squares.DampedLeastSquares(problem, x0=None, *, equality_constraints=None, inequality_constraints=None, damping=1e-06, damping_mode='identity', damping_floor=1.0, trust_radii=None, adaptive_damping=False, damping_increase=10.0, damping_decrease=0.2, damping_min=0.0, damping_max=inf, max_damping_attempts=6, maxiter=25, xtol=1e-10, ftol=1e-12, constraint_tol=1e-10, active_tol=1e-10, fd_step=1e-06, max_active_iter=20, max_line_search=12)#

Bases: object

Constrained damped least-squares optimizer with a step method.

Parameters:

problem (object) – Object with a residuals(x) method.
x0 (ndarray, optional) – Starting vector. If omitted, problem.x0() is used.
equality_constraints (callable or sequence of callables, optional) – Functions returning values that must equal zero.
inequality_constraints (callable or sequence of callables, optional) – Functions returning values that must be greater than or equal to zero.
damping (float or ndarray, optional) – Scalar or per-variable values added to the normal matrix.
damping_mode ({'identity', 'sensitivity'}, optional) – ‘identity’ adds damping directly to each diagonal term. ‘sensitivity’ multiplies damping by the current per-variable squared sensitivity from the residual and constraint Jacobians.
trust_radii (float or ndarray, optional) – Maximum absolute per-variable step. The proposed step is scaled uniformly so all variables satisfy their radii.
adaptive_damping (bool, optional) – If True, line-search failures increase damping and retry the current step; accepted full steps decrease damping.

Notes

Constraints are imposed on each local linearized DLS subproblem by KKT equations. Linear constraints are exact in the subproblem; nonlinear constraints converge through sequential linearization. Inequality constraints use the convention g(x) >= 0.

Create a new constrained damped least-squares optimizer.

property nfev#: Number of residual function evaluations.

property njev#: Number of residual Jacobian evaluations.

property ncev#: Number of constraint function evaluations.

property constraint_violation#: Current combined equality and inequality constraint violation.

result()#: Return the current result object.

step()#

Perform one iteration of optimization.

Returns:: (x, f, g), where f and g are the cost and least-squares gradient evaluated at x before the update. After a successful step, self.x holds the updated iterate.
Return type:: tuple

run()#: Run until DLS reaches its configured stopping condition.

Governors#

Composable stop conditions for optym optimizers.

class prysm.x.optym.governors.StepRecord(optimizer, iteration, x, f, g, x_next, metadata=None)#

Bases: object

Observation of one completed optimizer step.

Parameters:

optimizer (Any) – Optimizer that produced the step.
iteration (int) – One-based iteration count for the completed step.
x (array_like) – Iterate at which the objective and gradient were evaluated.
f (float) – Objective value at x.
g (array_like) – Gradient at x.
x_next (array_like) – Iterate after the optimizer step.
metadata (Mapping, optional) – Additional optimizer-specific information for the completed step.

optimizer#

Optimizer that produced the step.

Type:: Any

iteration#

One-based iteration count for the completed step.

Type:: int

x#

Snapshot of the pre-step iterate.

Type:: array_like

f#

Objective value at x.

Type:: float

g#

Snapshot of the gradient at x.

Type:: array_like

x_next#

Snapshot of the post-step iterate.

Type:: array_like

metadata#

Additional optimizer-specific information for the step.

Type:: dict

class prysm.x.optym.governors.GovernorDecision(stop=False, success=False, message='')#

Bases: object

Decision returned by a governor after observing a step.

Parameters:

stop (bool, optional) – If True, the optimizer runner should stop.
success (bool, optional) – If True, the stop condition represents successful convergence.
message (str, optional) – Human-readable reason for the decision.

stop#

Whether the optimizer runner should stop.

Type:: bool

success#

Whether the stop condition represents successful convergence.

Type:: bool

message#

Human-readable reason for the decision.

Type:: str

class prysm.x.optym.governors.OptimizationResult(x, decision, records, optimizer=None)#

Bases: object

Result object returned by governed optimizer runners.

Parameters:

x (array_like) – Final optimizer iterate.
decision (GovernorDecision) – Terminal decision that ended the run.
records (Sequence of StepRecord) – Step records observed during the run.
optimizer (Any, optional) – Optimizer used for the run.

x#

Final optimizer iterate.

Type:: array_like

success#

Whether the terminal decision represents successful convergence.

Type:: bool

message#

Human-readable reason for termination.

Type:: str

nit#

Number of completed optimizer steps.

Type:: int

nfev#

Number of function evaluations reported by the optimizer, if present.

Type:: int or None

njev#

Number of Jacobian or gradient evaluations reported by the optimizer, if present.

Type:: int or None

decision#

Terminal decision that ended the run.

Type:: GovernorDecision

records#

Step records observed during the run.

Type:: Sequence of StepRecord

optimizer#

Optimizer used for the run.

Type:: Any

class prysm.x.optym.governors.Governor#

Bases: object

Base class for reusable optimizer stop conditions.

observe(record)#

Observe a step record.

Parameters:: record (StepRecord) – Completed optimizer step.
Returns:: Decision indicating whether optimization should stop.
Return type:: GovernorDecision

class prysm.x.optym.governors.AnyGovernor(governors)#

Bases: Governor

Stop when any child governor stops.

Parameters:: governors (Iterable of Governor) – Child governors to observe each step.

governors#

Child governors to observe each step.

Type:: tuple of Governor

observe(record)#

Observe a step with all child governors.

Parameters:: record (StepRecord) – Completed optimizer step.
Returns:: First stopping decision from a child governor, or a non-stopping decision if no child stops.
Return type:: GovernorDecision

class prysm.x.optym.governors.AllGovernor(governors)#

Bases: Governor

Stop after every child governor has stopped at least once.

Parameters:: governors (Iterable of Governor) – Child governors to observe each step.

governors#

Child governors to observe each step.

Type:: tuple of Governor

observe(record)#

Observe a step with all child governors.

Parameters:: record (StepRecord) – Completed optimizer step.
Returns:: Stopping decision once all child governors have stopped at least once, otherwise a non-stopping decision.
Return type:: GovernorDecision

class prysm.x.optym.governors.MaxIterations(n)#

Bases: Governor

Stop after a fixed number of accepted optimizer steps.

Parameters:: n (int) – Maximum number of completed optimizer steps.

n#

Maximum number of completed optimizer steps.

Type:: int

observe(record)#

Observe a completed optimizer step.

Parameters:: record (StepRecord) – Completed optimizer step.
Returns:: Stopping decision when record.iteration reaches n.
Return type:: GovernorDecision

class prysm.x.optym.governors.MaxEvaluations(n)#

Bases: Governor

Stop when optimizer.nfev reaches a fixed limit.

Parameters:: n (int) – Maximum number of function evaluations.

n#

Maximum number of function evaluations.

Type:: int

observe(record)#

Observe a completed optimizer step.

Parameters:: record (StepRecord) – Completed optimizer step.
Returns:: Stopping decision when the optimizer reports nfev >= n.
Return type:: GovernorDecision

class prysm.x.optym.governors.FunctionTolerance(ftol, relative=True)#

Bases: Governor

Stop when consecutive objective values change by a small amount.

Parameters:

ftol (float) – Function-value tolerance.
relative (bool, optional) – If True, scale the tolerance by the larger of 1 and the magnitudes of the consecutive function values.

ftol#

Function-value tolerance.

Type:: float

relative#

Whether to use relative scaling.

Type:: bool

observe(record)#

Observe a completed optimizer step.

Parameters:: record (StepRecord) – Completed optimizer step. If record.metadata contains ‘f_next’, it is used as the post-step objective value.
Returns:: Stopping decision when the consecutive function values differ by no more than the configured tolerance.
Return type:: GovernorDecision

class prysm.x.optym.governors.GradientTolerance(gtol, norm=inf)#

Bases: Governor

Stop when the gradient norm is below a threshold.

Parameters:

gtol (float) – Gradient norm tolerance.
norm (int, float, or str, optional) – Norm order used to measure the gradient.

gtol#

Gradient norm tolerance.

Type:: float

norm#

Norm order used to measure the gradient.

Type:: int, float, or str

observe(record)#

Observe a completed optimizer step.

Parameters:: record (StepRecord) – Completed optimizer step.
Returns:: Stopping decision when the gradient norm is below gtol.
Return type:: GovernorDecision

class prysm.x.optym.governors.StepTolerance(xtol, relative=True, norm=inf)#

Bases: Governor

Stop when the optimizer step is below a threshold.

Parameters:

xtol (float) – Step norm tolerance.
relative (bool, optional) – If True, scale the tolerance by the larger of 1 and the norm of the pre-step iterate.
norm (int, float, or str, optional) – Norm order used to measure the step.

xtol#

Step norm tolerance.

Type:: float

relative#

Whether to use relative scaling.

Type:: bool

norm#

Norm order used to measure the step.

Type:: int, float, or str

observe(record)#

Observe a completed optimizer step.

Parameters:: record (StepRecord) – Completed optimizer step.
Returns:: Stopping decision when the step norm is below xtol.
Return type:: GovernorDecision

class prysm.x.optym.governors.ConstraintTolerance(tol)#

Bases: Governor

Stop when reported constraint violation is below a threshold.

Parameters:: tol (float) – Constraint violation tolerance.

tol#

Constraint violation tolerance.

Type:: float

observe(record)#

Observe a completed optimizer step.

Parameters:: record (StepRecord) – Completed optimizer step. Constraint violation is read from record.metadata[‘constraint_violation’] when present, otherwise from record.optimizer.constraint_violation when available.
Returns:: Stopping decision when the reported violation is below tol.
Return type:: GovernorDecision

Plotting#

Plotting helpers for optym convergence histories.

prysm.x.optym.plotting.plot_convergence(result_or_records, quantities=('f', 'g_norm'), *, gradient_norm=inf, bounded_atol=1e-12, bounded_rtol=1e-09, fig=None, ax=None, yscale='linear', lw=None, marker=None, colors=None)#

Plot optimizer convergence series against iteration.

Parameters:

result_or_records (OptimizationResult or sequence) – A run_until result, a sequence of StepRecord objects, or a sequence of history dictionaries.
quantities (str or sequence of str, optional) – Series to plot. Supported names are f, g_norm, and bounded. Aliases include cost, objective, g, gnorm, gradient_norm, bounds, n_bounded, and bounded_variables.
gradient_norm (float or str, optional) – Norm order used for g_norm. The default is infinity norm.
bounded_atol (float, optional) – Absolute tolerance for deciding whether a variable is on a box bound.
bounded_rtol (float, optional) – Relative tolerance for deciding whether a variable is on a box bound.
fig (matplotlib.figure.Figure, optional) – Figure containing the plot.
ax (matplotlib.axes.Axis, optional) – Axis or array of axes containing the plot.
yscale (str, optional) – Matplotlib y axis scale applied to all created axes.
lw (float, optional) – Line width.
marker (str, optional) – Matplotlib marker for each point.
colors (sequence, optional) – Colors for each requested quantity.

Returns:

matplotlib.figure.Figure – Figure containing the plot.
matplotlib.axes.Axis or ndarray – Axis object, or an array of axes when multiple quantities are plotted.

Activation Functions#

Activation functions and related nodes.

class prysm.x.optym.activation.Softmax#

Bases: object

Softmax activation function.

Softmax is a soft, differntiable alternative to argmax. It is used as a component of GumbelSoftmaxEncoder to ecourage / softly force variables to take on one of K discrete states.

The arrays passed to forward() and reverse() may take any number of dimensions. The understanding of the inputs should be that the final dimension is what is being softmaxed over, and all preceeding dimensions are independent variables.

For example, to softmax a 256x256 array over 4 activation levels, the input should be 256x256x4.

Create a new Softmax node.

forward(x)#

Perform Softmax activation on logits.

Parameters:: x (ndarray, shape (A,B,C, ... K)) – any number of leading dimensions, required trailing dimension of size K, where K is the number of levels to be used with an encoder
Returns:: same shape as x, activated x, where sum(axis=K) == 1
Return type:: ndarray

backprop(grad)#

Backpropagate grad through Softmax.

Parameters:: grad (ndarray) – gradient of scalar cost function w.r.t. following step in forward problem, of same shape as passed to forward()
Returns:: dcost/dsoftmax-input
Return type:: ndarray

class prysm.x.optym.activation.GumbelSoftmax(tau=1, eps=None)#

Bases: object

GumbelSoftmax combines the softmax activation function with stochastic Gumbel noise to encourage variables to fall into discrete categories.

See: https://arxiv.org/pdf/1611.01144.pdf https://arxiv.org/pdf/1611.00712.pdf

You most likely want to use GumbelSoftmaxEncoder, not this class directly.

Create a new GumbelSoftmax estimator.

tau is the temperature parameter, as tau -> 0, output becomes increasingly discrete; tau should be annealed towards zero, but never exactly zero, over the course of design/optimization.

forward(x)#: Gumbel-softmax process on x.

backprop(protograd)#: Adjoint of forward().

class prysm.x.optym.activation.DiscreteEncoder(estimator, levels)#

Bases: object

An encoder that embds a continuous encoder, which encourages values to cluster at discrete states.

Create a new DiscreteEncoder.

Parameters:

estimator (an initialized estimator) – for example GumbelSoftmax()
levels (int or ndarray) – if int, self-generates arange(levels) else, expected to be K discrete, non-overlapping integer states

forward(x)#

Forward pass through the continuous proxy for optimization.

use discretize() to view the current best fit discrete realization.

backprop(grad)#: Backpropagation through the continuous proxy for optimization.

discretize(x)#

Perform discrete encoding of x.

Note that when the estimator weights are not yet converged or non-sparse the output of this function will not match closely to the continuous proxy that is actually being optimized.

class prysm.x.optym.activation.Tanh(a=1, x0=0, y0=0)#

Bases: _AffineActivation

Affine-scaled hyperbolic tangent: y = tanh(a·(x - x0)) + y0.

class prysm.x.optym.activation.Arctan(a=1, x0=0, y0=0)#

Bases: _AffineActivation

Affine-scaled arctangent: y = arctan(a·(x - x0)) + y0.

class prysm.x.optym.activation.Softplus(a=1, x0=0, y0=0)#

Bases: _AffineActivation

Affine-scaled softplus: y = log(1 + exp(a·(x - x0))) + y0.

class prysm.x.optym.activation.Sigmoid(a=1, x0=0, y0=0)#

Bases: _AffineActivation

Affine-scaled logistic sigmoid: y = σ(a·(x - x0)) + y0.

Operators#

Some differentiable operators.

class prysm.x.optym.operators.SpatialGradient2D#

Bases: object

Spatial parital derivatives and adjoints.

forward_x(x)#: Compute the X spatial gradient of an array.

adjoint_x(xbar)#: Backpropagate through X spatial gradient of an array.

forward_y(x)#: Compute the Y spatial gradient of an array.

adjoint_y(xbar)#: Backpropagate through Y spatial gradient of an array.

Cost (loss) Functions#

Cost functions, aka figures of merit for models.

prysm.x.optym.cost.bias_and_gain_invariant_error(I, D)#

Bias and gain invariant error.

This cost function computes internal least mean squares estimates of the overall bias (DC pedestal) and gain between the signal I and D. This objective is useful when the overall signal level is ambiguous in phase retrieval type problems, and can significantly help stabilize the optimization process.

Sample Problems#

Sample optimization problems.

The functions in this module return objective and gradient pairs, matching the callable interface accepted by prysm.x.optym optimizers. The Problem classes provide analytic objective, gradient, Hessian, and Hessian-vector product hooks for optimizers that can use richer derivative information.

class prysm.x.optym.sample_problems.SphereProblem(fd_method=None, fd_step=None)#

Bases: Problem

Sphere optimization problem.

The global minimum is f(0) = 0.

f(x)#: Evaluate the scalar objective.

fg(x)#: Evaluate objective and gradient.

g(x)#: Evaluate the objective gradient.

h(x)#: Evaluate the dense Hessian.

hvp(x, v)#: Evaluate the Hessian-vector product H(x) @ v.

class prysm.x.optym.sample_problems.RosenbrockProblem(fd_method=None, fd_step=None)#

Bases: Problem

Rosenbrock optimization problem.

The global minimum is f([1, …, 1]) = 0.

f(x)#: Evaluate the scalar objective.

fg(x)#: Evaluate objective and gradient.

g(x)#: Evaluate the objective gradient.

h(x)#: Evaluate the dense Hessian.

hvp(x, v)#: Evaluate the Hessian-vector product H(x) @ v.

class prysm.x.optym.sample_problems.RastriginProblem(fd_method=None, fd_step=None)#

Bases: Problem

Rastrigin optimization problem.

The global minimum is f(0) = 0.

f(x)#: Evaluate the scalar objective.

fg(x)#: Evaluate objective and gradient.

g(x)#: Evaluate the objective gradient.

h(x)#: Evaluate the dense Hessian.

hvp(x, v)#: Evaluate the Hessian-vector product H(x) @ v.

class prysm.x.optym.sample_problems.HimmelblauProblem(fd_method=None, fd_step=None)#

Bases: Problem

Himmelblau optimization problem.

One global minimum is f([3, 2]) = 0.

f(x)#: Evaluate the scalar objective.

fg(x)#: Evaluate objective and gradient.

g(x)#: Evaluate the objective gradient.

h(x)#: Evaluate the dense Hessian.

hvp(x, v)#: Evaluate the Hessian-vector product H(x) @ v.

prysm.x.optym.sample_problems.sphere(x)#

Evaluate the sphere function.

The global minimum is f(0) = 0.

Parameters:: x (ndarray) – optimization variables
Returns:: objective value and gradient
Return type:: float, ndarray

prysm.x.optym.sample_problems.rosenbrock(x)#

Evaluate the Rosenbrock function.

The global minimum is f([1, …, 1]) = 0.

Parameters:: x (ndarray) – optimization variables, at least two elements
Returns:: objective value and gradient
Return type:: float, ndarray

prysm.x.optym.sample_problems.rastrigin(x)#

Evaluate the Rastrigin function.

The global minimum is f(0) = 0.

Parameters:: x (ndarray) – optimization variables
Returns:: objective value and gradient
Return type:: float, ndarray

prysm.x.optym.sample_problems.himmelblau(x)#

Evaluate Himmelblau’s function.

One global minimum is f([3, 2]) = 0.

Parameters:: x (ndarray) – optimization variables, exactly two elements
Returns:: objective value and gradient
Return type:: float, ndarray