IN PROGRESS: Optimal Pension Design: A Naive Approach

In this post we look at why pension plans exist, how they work in Belgium, and how we can naively search for an optimal plan design using a model. The goal is to explain the reasoning behind the model: why you would want such a model in the first place, and how it works under the hood.

Why a Pension Plan?

The honest answer is that most of us are bad at saving for a future self we have never met. This is less a character flaw than a wiring problem: we discount the future steeply, and the person who will actually need the money — retired, decades older — reads to us almost like a stranger we are being asked to sacrifice for. Left to our own devices we under-save, retire into a consumption cliff, and fall back on the state.

And that stranger keeps living longer. Rising life expectancy sounds like unambiguous good news, but for anyone financing their own retirement it introduces a genuinely hard problem: longevity risk, the risk of outliving your money. You cannot simply divide your savings by your remaining years and spend the quotient, because you do not know the denominator. Plan to run dry at 85 and you are destitute at 90; hoard against reaching 100 and you deny your younger retired self a decent life along the way.

The institutional response to both problems — our present bias and our uncertain horizon — is the pension plan: a contractual commitment, made while you are earning, to a future version of you that you cannot be trusted to provide for spontaneously. Concretely, your employer sets aside part of your salary in a fund that is invested until you retire.

But there is a second, less obvious answer, and it is the one that motivates this whole project. A pension plan is not merely a savings vehicle; it is a risk-sharing contract between three parties — employee, employer, and, through guarantees and regulation, the state. Each wants something different. The employee wants a stable, predictable replacement income and as little investment risk as possible. The employer wants costs that are affordable and, above all, foreseeable — a liability it can budget for, not one that balloons the moment markets turn. The state wants neither party to fail: not the employee retiring into poverty and falling back on the first pillar, nor the employer defaulting on a promise it cannot keep.

These wants are not just different; they are in direct, structural tension. The very instrument that protects the employee — the guaranteed minimum return we are about to meet — is precisely the liability that threatens the employer, because a guarantee is a promise to cover the gap whenever the market underdelivers. Dial the guarantee up and you reassure the employee while loading risk onto the employer; dial it down and you do the reverse. No setting is costless for everyone. A pension plan is, at bottom, a particular choice about where on that spectrum to sit.

And notice that the three parties do not enter symmetrically. The employee's and the employer's objectives are the two actually traded off against each other — more for one is, at the margin, less for the other. The state does not sit on that spectrum at all: it draws the boundaries within which any trade is allowed to happen — minimum guarantees, contribution caps, non-discrimination — and rules out the corners where one party walks away holding an empty bag. That distinction — objectives to be balanced against each other, versus constraints that simply must be respected — turns out to be exactly how the model in this project is structured. Which raises the question the post is really about: given that a plan's design is a choice about how to balance these competing claims, can that choice be made optimally — and what would "optimal" even mean?

Belgium Legal Framework: the Three pillars

Belgian retirement income rests on three pillars. The distinction between them is not bureaucratic tidiness — each pillar places the risk of funding your old age on a different party, and the project lives entirely inside the one that shares that risk most explicitly.

The first pillar is the statutory state pension. It is a pay-as-you-go system, and that phrase is worth taking literally: there is no personal pot of money growing in your name. The contributions today's workers pay are handed almost straight back out as today's retirees' pensions — a transfer between generations, not a fund. That design is exactly what makes it demographically sensitive. Its solvency rests on the ratio of contributing workers to drawing retirees, and in Belgium, as across most of Europe, that ratio is deteriorating: people live longer and draw for more years, while relatively fewer workers pay in behind them. The same longevity that complicates individual saving squeezes the collective system too. And for most private-sector careers the first pillar replaces only a modest fraction of final salary — a replacement rate, the share of your working income your pension reproduces, that is low enough to leave a conspicuous gap between the salary you earned and the pension the state provides. That gap is the entire reason the other two pillars exist, and closing it is what a second-pillar plan is created for. If you want more on the first pillar, I refer you to Wikifin.

The second pillar is occupational: pension plans that an employer — or an entire sector — sets up for its employees, funded by contributions paid in across the working career. This is where the project lives. The plan is governed by the WAP (Wet op de Aanvullende Pensioenen), and its defining feature is a guaranteed minimum return on those contributions. That guarantee is what makes the second pillar more than a savings account: whatever the plan's assets actually earn, the employee is promised their contributions grow at least at a legally fixed rate. The risk of falling short doesn't sit with the employee — it sits with the employer, who underwrites the guarantee. If the assets backing the plan return less than the guaranteed rate, the employer makes up the difference out of pocket.

That guarantee used to be a fixed number — historically 3.75% on employee contributions and 3.25% on employer contributions. But a fixed guarantee is a fixed liability, and once safe market yields collapsed well below those levels in the 2010s, employers were underwriting a spread they had no safe way to earn. The 2016 reform resolved this by letting the guarantee float with the market. Since then it has been a single unified rate, recomputed from the yield on the 10-year OLO — the Belgian government bond whose curve we simulated in [previous section]. Concretely: 85% of the trailing 24-month average of the 10-year OLO yield, rounded to the nearest 25 bps, floored at 1.75% and capped at 3.75%, applied separately to the employer and employee legs:

Every term there is a risk-sharing knob. The 85% haircut and the 24-month averaging make the guarantee lag and dampen the market rather than track it tick-for-tick; the floor protects the employee when rates are near zero; the cap protects the employer when they spike.

And this is exactly why the OLO had to come first. Read the formula forward in time: contributions accrue over a whole career, and the rate applying to each future year is set off the OLO yield prevailing in that year. So the guarantee the employer is on the hook for isn't a single number — it's a trajectory, a deterministic function of the entire future path of the 10-year OLO. To know the distribution of shortfalls the employer might eventually have to cover, we first need the distribution of future OLO paths. That is precisely what the Hull-White simulator produces: each simulated short-rate path is mapped back to a future 10-year yield (reconstructFutureYield), the formula above is applied to its trailing average, and out comes the guarantee rate that year's reserve has to clear. The OLO model isn't scene-setting — it's the engine that turns the WAP guarantee from a legal definition into a number the environment can compute at every step.Note that the current implementation makes abstraction of this reasoning and just works with a fixed rate for simplicity reasons.

Sitting alongside the return guarantee are two further hard constraints that bound the plan rather than price it: the 80% rule, capping total pension build-up relative to salary, and the non-discrimination requirements. These enter the model as feasibility constraints the policy must respect, not as terms in the objective it trades off.

The third pillar is individual, tax-favoured saving — a personal pension product you take out and fund yourself, nudged along by a tax break. It completes the arc: where the first pillar pools risk across generations and the second shares it between employer and employee, the third places it squarely and voluntarily on the individual. Because no employer or collective guarantee sits behind it, it falls outside the risk-sharing problem this project is about, and I leave it aside here — relevant to an employee's full retirement picture, but not to the design question that follows.

Different types of Pension Plans

Within the second pillar the classic dichotomy is, at heart, about who carries the risk — which makes it the natural place to pin down exactly where this project sits.

Defined Benefit (DB): the plan promises an outcome — typically a formula on final or career-average salary — and the employer bears the full weight of delivering it. Both the investment risk (the backing assets may underperform) and the longevity risk (retirees may live longer than the plan funded for) land on the employer. That concentration is exactly why DB is increasingly rare: it is an open-ended liability sitting on the sponsor's balance sheet, and few employers still want to hold it.

Defined Contribution (DC): the plan promises an input — a contribution rate — and the retirement outcome is whatever those contributions happen to grow into. In most countries this is the mirror image of DB: fix the input, and every risk on the outcome side passes to the employee, who now carries the market and longevity exposure alone.

Belgium is the interesting exception — and it is the reason a risk-sharing model has anything to chew on here at all. The WAP guarantee bolts a minimum return onto an otherwise DC structure, producing a hybrid sometimes called "DC with a DB flavour": contributions are defined, but the employer still stands behind a floor. Risk here is neither fully concentrated (as in DB) nor fully offloaded (as in pure DC) — it is split. The employee keeps the upside above the guaranteed rate; the employer owns the downside below it. And that split is precisely what makes the central question of this project non-trivial. If all the risk sat with one party, there would be nothing to optimise jointly — you would simply minimise that party's pain. It is the shared middle ground, where a euro of reassurance to the employee is a euro of liability to the employer.

The insurance vehicle matters too, because it sets how the reserve behaves over time — and therefore how often the guarantee actually has to bite. Branch 21 products credit a contractually guaranteed return, plus rretionary profit sharing, directly to the reserve. The crucial modelling consequence is that the reserve is a credited-return process, not a marked-to-market bond portfolio: it grows by a smoothing crediting rule rather than by revaluing an underlying portfolio day to day, so short-term market swings are absorbed rather than passed straight through. That smoothing is itself a cushion between the market and the guarantee. Branch 23 products are unit-linked: the reserve simply tracks fund value, with more upside and no crediting cushion. When markets fall, the fund falls with them — and with nothing to smooth the drop, the WAP floor is reached far more readily, forcing the employer's guarantee to make up the difference. In that sense the guarantee behaves like a put option the employer has written to the employee: dormant while the fund clears the floor, triggered precisely when it does not — and Branch 23 leaves it triggered more often. Comparing how these two vehicles behave under one and the same contribution policy is one of the axes of the wider project.

This post works with the simplest member of the family: a single-employee Belgian DC plan with Branch 21-style deterministic crediting — the guarantee credited at a fixed rate, the put left dormant by construction. That is the same first-rung move as holding the WAP rate flat: strip out the stochastic crediting to get an environment whose behaviour you can check by hand, and switch it back on only once the mechanics are proven. The reasons for starting exactly here are what the next section makes precise.

What is Optimal?

"Optimal" is meaningless until you say for whom. The same contribution policy is a different thing seen from each side of the contract: one that thrills the CFO — low, predictable outlays — can quietly starve the employee's replacement rate, while one generous enough to delight the employee can make plan costs balloon exactly when the firm can least afford them, in a downturn. There is no policy that is optimal in the abstract; there is only optimal-for-a-chosen-balance. So rather than pick a side, the project's central object makes the balance itself explicit — a joint value function:

Read it as a single dial with the two parties at its ends. Each term encodes what that party is actually trying to get out of the plan.

The employer side, \(V_{\text{employer}}\), is the negative of a cost sensitivity index — negative because from the sponsor's chair the plan is a cost, and value means having less of it to worry about. What goes into that index is not merely how large the cost is but how unpleasant its shape: its expected level, its tail risk (the rare, severe funding shortfalls that do real balance-sheet damage), and its procyclicality — whether the cost tends to spike in bad states of the world, when the firm is already under strain. A cost that is high but steady is easier to carry than one that is lower on average but lands its worst blows at the worst possible moment; the index is built to prefer the former.

The employee side, \(V_{\text{employee}}\), is a utility over retirement consumption, anchored to a replacement-rate target — the very quantity introduced back at the first pillar, now doing formal work. Framing it as utility rather than raw expected wealth matters: it builds in that employees are risk-averse and that shortfalls below the target hurt more than equivalent surpluses help, so the objective rewards reliably clearing a decent standard of living over gambling for a larger but less certain pension.

And then the weight \(\lambda\) itself — which, crucially, is not something to be optimised. There is no "correct" \(\lambda\) the model should rover; a higher one does not mean a better plan, only a plan tilted further toward the employee. It is the negotiation dial between the two parties: the numerical form of exactly the question the risk-sharing contract has been posing all along — how much of the risk does each side agree to carry? Fixing \(\lambda\) is choosing a point on the shared-risk spectrum; sweeping it from one end to the other and re-optimising at each setting traces out the whole design frontier — the menu of employee-optimal-given-employer-cost trade-offs, every point on it Pareto-sensible, the choice between them a matter of negotiation rather than mathematics. The model does not tell you where to stand on that frontier. It tells you what the frontier is — which is the genuinely useful thing, because it turns a vague tug-of-war into an explicit, priced menu of options.

That is the full framework. This post uses a radically stripped-down version of it, because the framework only means something if the machinery underneath it can be trusted — and before you optimise anything real, you have to prove that the environment computes what it claims to, on a problem whose answer you already know. That proof is what the rest of this post is about.

The Reward Function

At this first rung the reward is nothing more exotic than a present value: each cash flow discounted at an explicit rate and booked at the moment it occurs. Concretely, we take the two sides of the contract and reduce each to a single number at the retirement date \(T\).

Start with the employer, whose relationship to the plan is a stream of costs. Over the employee's career the employer pays in a contribution \(c_t\) each period; then, at retirement, it faces the guarantee. The plan has accrued a legal reserve \(L_T\) — what the contributions are worth grown at the WAP guaranteed rate — while the insurer holds a mathematical reserve \(R_T\), the actual Branch 21 reserve backing the contract. If the reserve falls short of what the guarantee legally owes, the employer tops up the difference; if it does not, the employer owes nothing. That is exactly the written-put payoff we described earlier — dormant when the reserve clears the floor, triggered only when it does not:

Because from the sponsor's chair this is a cost, its value is the negative of it:

The employee's side is, at this rung, deliberately the simpler of the two: the value is just the present value of the lump-sum benefit \(B_T\) received at retirement —

It is worth being honest that this is a risk-neutral stand-in for the employee objective the framework calls for. The full version, sketched earlier, is a concave utility over retirement consumption anchored to a replacement-rate target — curved precisely so that shortfalls hurt more than surpluses help. Here we strip that curvature out and reward raw discounted wealth. That is not the final employee value; it is the flattest thing that still points in the same direction, chosen so the first rung has as few moving parts as possible.

Combining the two through the negotiation dial \(\lambda\) gives the joint reward the agent actually optimises:

This is a radically simplified version of the reward the framework ultimately reaches for — linear where it should be curved, deterministic where it should be stochastic. But that is the point. Before trusting the agent on a reward that reflects reality, we need to know it behaves correctly on one whose optimum we can work out by hand. Once we understand how the agent responds to this stripped-down environment, each later rung swaps one simplification for its realistic counterpart — curvature into the employee term, stochastic crediting into the reserve — against an agent we already know to be sound.

Reinforcement Learning: A Basic Introduction

Most financial plans are fixed at the outset: a contribution schedule, a rebalancing rule, a glide path decided on day one and followed regardless of what happens next. Reinforcement learning starts from the opposite instinct. Strip the jargon and it is control — steering a system toward a goal by choosing actions as it unfolds — where the strategy is learned from simulated experience rather than fixed in advance. Instead of committing to a plan, you discover a rule by simulating across thousands of possible futures. The basic RL framework is shown below.

A state \(s\) is what is known at the moment the agent makes a decision — here, for our minimal model, deliberately spare: the current year. An action \(a\) is the decision taken in that state — here about as brutal as a decision gets: contribute this year, or don't. A policy \(\pi\) is a rule mapping each state to an action, which is to say it is exactly a contribution strategy — but a dynamic one, reacting to the state the plan has actually reached rather than a fixed schedule laid down at inception. The framing has already bought us something: we are no longer searching for the best contribution number, we are searching for the best contribution rule.

The last piece is the one that should feel most familiar. A value function \(Q(s,a)\) is the expected sum of future rewards from taking action \(a\) in state \(s\) and following the policy thereafter. Given the reward built in the previous section, unfold that definition and it is nothing other than an expected present value, conditional on a state and a first decision. You have computed thousands of these by hand; RL changes only how they are obtained — estimated by simulating many careers rather than read off a formula — and what is done with them once they are.

How is the agent discovering this? The thing to hold onto is that the agent is never shown any of the machinery. It does not see the reward formula, nor the law of motion of the reserve; it only ever sees what the environment hands back at each step — a realised reward and a next state — exactly the loop of the diagram above. Everything it comes to know about value, it infers from those samples alone. Concretely, it keeps a table of estimates, one entry per state–action pair, all initially blank. In each episode it plays a whole career to \(T\), choosing actions by its current best guesses; and only once the career is complete — this is what makes it Monte Carlo rather than a bootstrapping method — does it look back over every state–action pair it actually visited and nudge each estimate a fraction of the way toward the return that followed it. That nudge is the update on the bottom line of the figure below: no single return is trusted outright, each merely tugs the running estimate toward itself, and holding the step size \(\alpha\) constant rather than letting it shrink keeps the junk returns from early training — when the policy was still bad — from anchoring the estimate once the policy has improved.

Learning, then, is the loop between estimating and acting. Better estimates make for better decisions: acting greedily with respect to the table means, in each state, taking whichever action the estimates now rank highest. But an agent that only ever exploits its current best guess will never find out that some neglected action was secretly better — so it explores, taking a random action a small fraction \(\varepsilon\) of the time, just often enough that no promising branch stays permanently untried. Evaluate, improve, repeat: each pass sharpens the estimates, each sharpening shifts the greedy policy, and under the right conditions the policy climbs toward the one that maximises the joint value \(V\). Nobody hands the agent the optimal contribution rule — it discovers it by playing thousands of careers and letting their realised present values vote.

At this first rung that loop is doing something you could, in principle, do without it: the environment is deterministic and the action is a single binary switch each year, so the optimal policy can be found by direct enumeration and checked by hand. That is not a weakness — it is the entire reason to start here. The point of a rung whose answer is independently knowable is precisely that the learner's answer can be checked against it. The reason to reach for RL at all only arrives later, when the reserve stops being deterministic and its stochastic path enters the state: at that point there is no formula to evaluate and no schedule to enumerate, and a policy learned from simulation is the only object that still scales. What we validate now is that the machinery finds the right answer where we can grade it — so that we can trust it where we cannot.

The Environment

The previous section lived on the agent's side of the loop. The environment is the other box — the thing that takes an action and hands back a reward and a next state — and at this first rung it is deliberately tiny: a few numbers evolving under fixed rules. That smallness is the point: the whole validation strategy rests on an environment simple enough to grade by hand.

Each year the plan rule fixes a premium as a fraction of salary, \(c_t = \text{rate} \cdot S_t\), with salary growing deterministically. The size of a contribution is never the agent's to choose — the plan sets it. What the agent chooses is only the binary from the last section: pay this year's premium, or don't.

What makes this more than a savings account is that the environment keeps two ledgers, not one. Every euro contributed is booked into both, but they grow at different rates. The mathematical reserve \(R\) — the actual Branch 21 reserve — is credited at the tariff \(\mu\); the legal reserve \(L\) — what the WAP guarantee owes on those same contributions — compounds at the guaranteed rate \(g\):

The gap between them at retirement is the guarantee. If \(R\) clears the legal floor, the employee takes the upside; if it falls short, the employer tops the difference up to \(L\). That top-up, \(\max(L - R, 0)\), is exactly the written put from the reward section — dormant when the reserve clears the floor, triggered only when it doesn't.

The reward is then just the bookkeeping from before, emitted as each cash flow occurs. A contribution is a cost the instant it is paid, so each year books \(-(1-\lambda)\,c_t\,e^{-\delta t}\) — the burden set by the employer weight \((1-\lambda)\), heavy when the plan tilts toward the employer, near-weightless when it tilts toward the employee. At retirement the reserve funds the payout and the employer settles the put: \(\big(\lambda \max(R_T, L_T) - (1-\lambda)\max(L_T - R_T, 0)\big)\,e^{-\delta T}\).

One choice there is worth dragging into the open. Discounting lives entirely in the reward, through the explicit rate \(\delta\), exactly as an actuary computes a present value. It does not live in the RL machinery. Reinforcement learning carries its own discount factor \(\gamma\), and the reflex is to set it below one — but \(\gamma\) and \(\delta\) discount the same thing, so any \(\gamma < 1\) stacks a second discount on the financial one and silently reprices the objective with a cost of capital you never chose. So \(\gamma = 1\), permanently.

That leaves the state, and at this rung it is just the year \(t\). That suffices only because everything else is a deterministic function of time — salary, both crediting rates, and so the reserve itself at a fixed policy. It will not survive the next rung: once credited return goes stochastic, two careers reaching the same year hold different reserves and face different odds of tripping the guarantee, and the funding position has to enter the state in its own right. That is exactly what will motivate a function approximator over a lookup table. For now, one integer is enough.

And with \(\sigma = 0\) the crediting is frozen, the put is dormant by construction, and the whole episode is a deterministic function of the policy alone — forty-five binary choices with a known payoff each, so the optimum can be enumerated and checked by hand.

Tuning the Learning Behaviour

The RL introduction described the loop as if it simply runs: evaluate, improve, repeat, and the policy climbs toward the optimum. Under the right conditions it does — but "the right conditions" hides the work. The convergence guarantees behind Monte Carlo control are asymptotic and assume every state–action pair keeps being visited; they say nothing about whether a finite run lands on the right policy, or how fast. In practice the same loop can just as easily settle on a confidently wrong answer, and two failure modes drive that. If the agent stops exploring too soon it never learns that an action it tentatively rejected was in fact better, and locks in the mistake. And because it learns on its own behaviour, the returns it averages early — generated while the policy was still bad — are junk, and a naïve average lets them anchor the estimate long after the policy has improved. Left alone, either one produces exactly the trap the whole ladder is built to avoid: a run that converges, plateaus, and looks healthy while optimising something other than what you intended. Tuning is the handful of choices that keep the loop honest between those two failure modes.

Each failure mode gets a lever. Against stopping too soon: explore while you know nothing, commit as the picture sharpens. The random fraction \(\varepsilon\) starts at \(1.0\), decays linearly to \(0.05\) over the first quarter of the million episodes, then holds — the floor kept above zero so every branch stays occasionally re-tested.

Two update rules match those phases. While \(\varepsilon\) is still decaying the policy is a moving target: early returns came from a worse policy, and a plain average would weight that junk as heavily as fresh returns and let it anchor the estimate — the age-plan starvation failure is exactly this. So the moving phase uses a constant step size \(\alpha = 0.02\), letting old returns fade. Once the policy stops moving the target is stationary and we want an honest, variance-shrinking average, so the update switches to a true sample mean \(1/N\).

The handoff is the only subtlety: zeroing the counts would let the first update (\(N=1\)) slam the estimate onto one noisy episode and flip the policy, so the counts restart at a prior weight \(n_0 = 500\) — the estimate in hand treated as \(500\) episodes of evidence, its grip decaying to under a percent by the end. None of this changes what is optimised; it governs only how reliably the agent gets there — a trace that climbs, then settles onto the plateau the next section checks against the benchmark.

Results: the deterministic basic model

The basic model exists to answer one question before any stochasticity is introduced: does the joint value function behave like a negotiation instrument, and does the tabular agent recover the policy that value function implies? The deterministic setting (SIGMA = 0) makes this answerable against a benchmark that can be certified exactly, so that any disagreement between agent and benchmark is a fact about the learner rather than an artifact of sampling noise.

Environment

The episode is a 45-year accumulation horizon (years \(t = 0, \dots, 44\)). At each year the agent chooses a binary action — contribute or don't — and the reward is the marginal change in the joint value function. All runs in this section are deterministic: SIGMA = 0, so a given policy produces a single trajectory with no dispersion.

Each configuration is evaluated against three contribution schedules, which determine how a contribution is sized in a given year:

• fixed — a constant contribution rate across the horizon;
• step — a piecewise-constant rate that increases at fixed break-points;
• age — a rate that varies with the member's age. The three schedules leave the economics of a single contribution unchanged and reshape only its distribution across the horizon; they are included as a robustness axis, not as separate experiments.

Value function and the two levers

The joint objective is

with \(V_{\text{employee}}\) the discounted member benefit and \(V_{\text{employer}}\) the discounted sponsor cost. Two settings govern every result below:

1. Whether surplus exists. A contribution earns the crediting rate MU and must fund the guarantee G, both discounted at DISC. Surplus exists only when MU > G. When MU = G = DISC the two legs cancel and a contribution is a wash.
2. How surplus is split. When MU > G, the weight \(\lambda\) divides the surplus between the two parties. Low \(\lambda\) weights the employer's cost; high \(\lambda\) weights the employee's gain. The four configurations below are chosen to isolate these two levers.

Configurations

All runs share DISC = 0.02 and SIGMA = 0. The indifference control sets MU = G = DISC; the other three fix MU = 0.03 and G = 0.0175 so that surplus exists, and vary only \(\lambda\).

Learner and training

The agent is a tabular Monte Carlo control learner over the discrete \((t, \text{action})\) space, trained with identical hyperparameters across all configurations:

Benchmark and evaluation protocol

The benchmark is the gap policy: for each year the marginal value of contributing is computed numerically, and the optimal action is its sign. In the deterministic setting the value function is certified linear in the contribution decision, so this marginal gap is exact and the benchmark is optimal by construction — derived independently of the learner, satisfying the no-circular-validation requirement.

A year is counted as decided only when the magnitude of its marginal gap exceeds the action resolution RES = 0.01. Below that threshold the two actions are indistinguishable on the evaluation grid and the year is a tie, for which any action is optimal. Agreement is therefore reported as the number of decided years on which the agent's greedy action matches the benchmark; tied years are excluded from the denominator because they carry no target to match.

Results

Joint value and policy agreement

The agent matches the benchmark on every decided year in every configuration and plan. The apparent partial agreement in neut (14/45, 6/45, 0/45) reflects the number of years the benchmark decides, not disagreement on the years it does — see the interpretation below.

Central-path economics of the benchmark policies

The following are computed along the zero-shock central path under the benchmark policy. neg never contributes and neut2 produces no surplus, so both are degenerate; the contributing configurations neut and pos share a central path (the policy contributes in both) and are reported once.

The contributing configurations return an efficiency above 1.2 (payout exceeds contribution, the surplus MU > G realised), while neut2 returns efficiency exactly 1.0 where it contributes at all (a perfect wash) and neg returns nothing.

Interpretation

Read down the \(\lambda\) column of the results table and the value function does what a negotiation parameter should. As \(\lambda\) rises from 0.25 to 0.75 with the economics held fixed, the marginal value of a contribution moves from strictly negative, through a thin positive margin, to strongly positive, and the learned policy tracks that transition on every year the benchmark can rank.

neut2 — the indifference control. With MU = G = DISC the surplus is identically zero, the joint value is flat at zero regardless of \(\lambda\), and no year is decided. The agent is correctly indifferent. This is the indispensable null: it establishes that the framework returns zero signal in the absence of economic content, so the non-zero results elsewhere cannot be an artifact of the reward's construction.

neg — surplus declined. Surplus exists (MU > G), yet at \(\lambda = 0.25\) the employer's funding burden dominates the joint objective, every marginal contribution is priced negative, and the optimal policy is to never contribute. The joint value is zero — but for the opposite reason to neut2: not because no surplus exists, but because the negotiated weighting declines to capture it. That neg and neut2 both report zero value on opposite economics is itself part of the demonstration: the value function distinguishes no surplus from surplus declined.

pos — surplus captured. Nothing changes but \(\lambda\), now 0.75. The employee's gain dominates, every marginal contribution is strongly positive, and the optimal policy is to contribute throughout. The joint value jumps to 1.6–1.7. The neg/pos pair, differing only in \(\lambda\), is the cleanest isolation of the split lever: identical economics, opposite policies, both recovered exactly.

neut — the thin margin, and why the match rate reads as it does. At \(\lambda = 0.5\) the two legs nearly offset, leaving a small positive margin: the optimal policy is still to contribute, but the per-year marginal gap is correspondingly small — peaking around 0.0086 for the fixed plan, beneath the RES = 0.01 threshold for most years. Only the minority of years with a gap above resolution are decidable, and the agent matches the benchmark on all of them; the rest are ties by construction, which is why the decided-year counts are 14/45, 6/45, and 0/45 rather than a sign of error. The age plan reaching 0/45 is the limiting case — a fully-tied schedule — not a failure. Reporting agreement against the decided subset is what keeps this honest, and it is the same overlap between what is verifiable and what the learner finds that will license trusting the learner once exact verification becomes impossible at the stochastic rungs.

Robustness across plan types. The step and age schedules reshape the marginal-gap profile — a staircase and a U-shaped curve respectively, against the fixed plan's smooth decline — but leave the sweep unchanged: the sign of the gap and the direction of the policy flip track \(\lambda\) identically in all three. The negotiation behaviour is therefore a property of the value function, not of any particular schedule.

Convergence is a sanity check, not the proof. The reward curves converge to stable plateaus whose height scales with \(\lambda\) (near zero for neut2, roughly 1.7 for pos), confirming that training completed and did not collapse. This plateau cannot by itself distinguish a stable-but-correct policy from a stable-but-wrong one; the optimality claim rests entirely on the independently derived benchmark. Convergence is necessary evidence, independent benchmarking is the sufficient evidence — the same separation that has to carry the argument once the environment becomes stochastic and exact verification is no longer available.

Summary

Across the four configurations the deterministic basic model shows two things. The joint value function behaves as an economic instrument: it returns zero when no surplus exists, scores a real surplus as fundable or not according to \(\lambda\), and does so monotonically as \(\lambda\) is swept. And on every year the independent benchmark decides, the agent recovers the benchmark's action exactly, across all three plan types; the remaining years are ties, and are reported as such. With framework and learner both certified in the deterministic setting, the environment is ready for the introduction of stochasticity.

Final thoughts

The value of the deterministic basic model is not any single number in the tables above but the fact that every number is checkable. The benchmark is exact, the ties are labelled as ties, and the agent's agreement with the benchmark is established on the years where agreement is even a well-posed question. That discipline is what earns the right to make the same claim later, when the environment stops being exactly solvable and the benchmark can no longer be certified line by line.

The clearest way to see the negotiation lever at work is to turn it. An interactive version of this model is available in the playground at Playground: the reader can move \(\lambda\) between the employer-weighted and employee-weighted extremes and watch the marginal gap change sign and the greedy policy flip in real time, reproducing the sweep from neg through neut to pos without rerunning any training. It is the fastest way to build the intuition that the rest of this chapter formalises.

Next steps: introducing stochasticity

Everything in this chapter rests on SIGMA = 0. The moment stochasticity is switched on, the structure of the problem changes, and it changes in a way the tabular model cannot follow.

The reason is the guarantee itself. The WAP guarantee is path-accumulated: what the sponsor owes at any point depends on the entire history of contributions and returns, not on the current year alone. Under stochastic returns, the year index \(t\) is no longer a sufficient description of the state. The quantity that is sufficient is the funding ratio \(F_t = A_t / L_t\) — accumulated assets over the accumulated guarantee floor — together with the contribution level that has built that floor. These are the stochastic processes the decision actually depends on, and for the problem to remain a well-posed Markov decision process they must enter the state. Omitting them does not simplify the model; it makes it non-Markov and quietly wrong.

Carrying the funding ratio in the state breaks the tabular approach on three fronts at once. The state space becomes continuous, so there is no finite table to fill. Visitation becomes endogenous and starved: the policy itself determines which regions of \(F\) are ever reached, and large parts of the space are visited too rarely to estimate. And the value function acquires a kink at \(F = 1\) — the boundary between funded and underfunded, the moneyness of the guarantee — across which a table cannot generalise. Each of these is fatal on its own; together they close the door on tabular Monte Carlo.

Two moves follow, and they are the substance of the next rung. The benchmark stops being a per-year linearity certificate and becomes dynamic programming over a discretised \((t, F)\) grid — backward induction that sweeps the state space by construction rather than relying on visitation, and so survives one rung further than any sampling-based benchmark could. The learner becomes PPO, a policy-gradient method with a function approximator that can represent a continuous value function and generalise across the kink instead of tabulating around it. The methodology is unchanged — an independent benchmark, then a learner checked against it, then the rung frozen as a regression test — but every component is rebuilt for a state space that can no longer be enumerated. That rung is where the framework stops being a proof of concept and starts being a model of the actual product.