Teaching Machines to Decide
RL Simulation & the Reward-Hacking Trap

You're rushing to catch the last train home. You stop at a vending machine for a drink — every slot flashes "Sold Out." The next machine is a 10-minute walk.

Now imagine you're the vending operator. You run 600 sites across a metro region — transit stations, malls, office towers, housing estates. Every one can sell out. And when it does, you lose the sale, and — because your contract carries a service-level agreement (SLA) capping how often machines may sit empty — repeated stockouts put you in breach.

So the question is deceptively simple: when do you send a van to restock it?

Each restock visit carries a fixed logistics cost. Every hour a site sits effectively empty counts as an "out-of-stock" penalty, and under the SLA it weighs far more heavily than the visit. The goal: minimise both, across all 600 sites, 365 days a year.

The answer seems obvious: restock it when it gets low. But four complications turn a one-liner into a genuinely hard sequential decision.

1 · The van takes time

Decide to send a van at 6am and it arrives sometime the next business day between 8am and 8pm — a 12-hour window, 24+ hours out. The site must survive that whole gap on its current stock.

2 · Weekends multiply the risk

There are no restock visits on weekends or public holidays. That creates an asymmetric urgency:

On Thursday, if you skip today, the earliest restock is Monday — the site must survive Friday, Saturday and Sunday alone. This weekend gap is the defining asymmetry of the problem.

3 · Every site is different

A high-traffic transit-hub site (holding ~1,000 units, selling ~200 a day) has entirely different dynamics from a quiet residential machine (~300 units, ~45 a day). One rule can't serve both.

4 · The forecast is imperfect

A demand model predicts tomorrow's sales — with about ±12% error, growing with horizon. And realised daily sales are themselves noisy (a lognormal spread of ~25% around the mean the model targets), so the true uncertainty is larger still. Every decision is made under it.

Before machine learning, an operations team writes the policy by hand. Three typical approaches:

Calendar Aware is a big improvement — but it was written by a human who already knew about Thursday's risk. The deeper question: what other patterns exist that no one has thought to hard-code?

In RL there are no labelled "correct answers." An agent lives inside an environment, takes actions, and receives a reward. Over millions of interactions it learns which actions raise cumulative reward.

Every RL problem is formalised as a Markov Decision Process — four components.

State — what the agent sees

At 6am each business day, the agent observes an 11-dimensional vector:

# State vector (all values float32, range [-1, 4])
obs[0]  = stock / capacity            # e.g. 0.42 = 42% full
obs[1]  = forecast_sales_day1 / capacity
obs[2]  = forecast_sales_day2 / capacity
obs[3]  = forecast_sales_day3 / capacity
obs[4]  = forecast_uncertainty / capacity  # grows with horizon
obs[5]  = in_transit                  # 1 if van already dispatched
obs[6]  = days_until_arrival / 3      # 0 if no van in flight
obs[7]  = sin(2π × day_of_week / 7)   # cyclical encoding
obs[8]  = cos(2π × day_of_week / 7)
obs[9]  = gap_if_skip / 5             # 0.4 Mon-Wed, 0.8 Thu-Fri
obs[10] = capacity / max_capacity     # site-size signal, 0–1

Feature obs[9] is the critical one — it encodes the whole weekend-gap insight: Thursday/Friday score 0.8 (high urgency if you wait), Monday–Wednesday score 0.4. Day-of-week uses sin/cos so Sunday and Monday sit next to each other cyclically. (Keep this design in mind — §11 shows the trained agent ends up ignoring most of it.)

Action & reward

The action is binary — the simplest possible:

Reward is computed after each business day:

These are penalty points, not dollars — the reward is a score to maximise. The weighting makes one full out-of-stock day (24 × 100 = 2,400 points) roughly 10× a restock visit (250). This encodes the SLA-driven priority: availability matters more than logistics cost. Hold onto the words "but not infinitely more" — the agent will test exactly where we drew the line. Note too that util_waste is the only term penalising over-stocking, weighted far below availability. That imbalance is the seed of §11.

Discount factor γ = 0.99

At γ = 0.99 the agent genuinely cares about stockouts a month away — exactly what the Thursday/weekend problem requires.

There is no single "RL algorithm" — there's a family, each with trade-offs. Knowing the landscape helps you choose the right tool and reason about its behaviour.

Value-based methods

Ask: how good is it to be in state s and take action a? Define the Q-function, the expected total future reward. If you knew Q perfectly, the optimal policy is trivial — pick the highest-Q action. Q-Learning (Watkins, 1989) learns it in a table; DQN (Mnih et al., 2015) replaces the table with a neural network, famously learning 49 Atari games from pixels via a replay buffer and target network. Our 11-dim state and binary action are small enough for DQN — but value-based updates can be unstable when the target you chase keeps moving.

Policy gradient

A different angle: learn the action probabilities directly. REINFORCE (Williams, 1992) nudges the policy toward actions that preceded high return:

The problem is high variance — return depends on luck as well as policy, and REINFORCE credits it all to the policy.

Actor-Critic

The fix: subtract a baseline and measure how much better than average an action was — the advantage A(s,a) = Q(s,a) − V(s). That needs V(s): the Critic. The Actor (policy) uses the Critic's estimate for a much lower-variance gradient. Two networks improving each other.

PPO — Proximal Policy Optimisation

PPO (Schulman et al., 2017) is Actor-Critic with one crucial addition: it stops the policy changing too much in a single update, via a clipped objective.

In plain English: "update, but not too fast." That conservatism made PPO the workhorse behind everything from game-play to RLHF. It also uses GAE (Schulman et al., 2015) to smooth the advantage estimate with a λ = 0.95 weighted average of multi-step errors.

Our exact setup

PPO(
    policy        = "MlpPolicy",   # separate Actor & Critic MLPs, 2×128 each
    n_steps       = 2048,          # rollout length before each update
    batch_size    = 256,
    n_epochs      = 10,
    gamma         = 0.99,          # discount
    gae_lambda    = 0.95,          # GAE smoothing
    clip_range    = 0.2,           # ε in the clip objective
    ent_coef      = 0.01,          # entropy bonus → exploration
    learning_rate = 3e-4,          # Adam
)

We chose PPO for discrete binary actions, a small state space, and — above all — stability: its conservative updates suit a problem where one catastrophic step (say, "never restock") is hard to recover from.

RL learns from experience. Since we can't experiment on live sites, we build a simulator — a faithful replica of the vending world the agent can inhabit cheaply.

Sales follow a lognormal distribution (positive, right-skewed — like real retail demand), with day-of-week multipliers and site-type hourly profiles. The demand model adds proportional noise that grows with horizon. One episode = one simulated year (252 business-day decisions). At 16 parallel environments and ~7,500 steps/second on a laptop CPU, 2 million steps finish in about 4 minutes.

Before training anything, measure the three hand-coded policies. Without baselines you can't know whether RL adds value. Results are for the commercial site, averaged over 50 simulated years.

Fixed Threshold and Forecast Reactive perform almost identically: "restock if stock ≤ 35%" and "restock if stock − forecast ≤ 15%" are nearly the same calculation when sales average ~18–20% of capacity. The forecast is absorbed by the threshold — the demand model is wasted effort. Calendar Aware, just by knowing Thursday is different, cuts stockout hours more than 3× for ~24 extra visits a year.

PPO looks like a clean sweep — one-third the stockouts of Calendar Aware, same visit count, best reward. But look at Avg Idle Stock, the last column we added, and the only one where PPO scores worst: it keeps machines 71% full versus 49%. Hold that thought — §11 is about what that column reveals.

The service pattern is consistent: PPO cuts stockout hours by 6× (transit hub) to nearly 40× (commercial) versus Calendar Aware, while making the same number of visits or fewer. On the metrics we set out to optimise, it's a decisive win.

But Idle Stock tells the other half of the story: PPO holds every site ~20 percentage points fuller. That's real working capital — inventory sitting in machines instead of on the balance sheet — the cost the agent quietly paid for near-perfect availability. §11 explains why.

We expected a clean threshold curve — high restock probability when stock is low, low when high, shifted earlier on Thursday/Friday. That's what all our state features were built to support. Here is what the agent actually learned:

The heatmap is almost uniformly red. High restock probability at every stock level, on every day. Strip away interpretation and the policy collapses to a single reflex: whenever you're allowed to restock, restock. It barely consults stock, forecast, or day-of-week — most of the engineered 11-dim state, and the entire demand model, go unused. Blunter than we designed for — and, on our own scorecard, better.

Why "always restock" wins

Three things conspire to make this blunt policy optimal. Each is a lesson about reward design, not about the cleverness of the agent.

Calendar Aware tried to be selective. PPO asked a sharper question: why be selective, when the in_transit guard already limits how often I can act? Given a lopsided reward and a built-in rate limit, the winning move is simply to keep the pipeline full.

• Reinforcement Learning: An Introduction (2nd ed.) — Sutton & Barto (2018), MIT Press. The definitive textbook; Ch. 3 (MDP), 6 (TD), 9 (function approximation).
  http://incompleteideas.net/book/the-book-2nd.html
• Proximal Policy Optimization Algorithms — Schulman et al. (2017). The PPO paper — introduces the clip objective used here.
  https://arxiv.org/abs/1707.06347
• High-Dimensional Continuous Control Using Generalized Advantage Estimation — Schulman et al. (2015). The GAE paper — λ-smoothed advantage.
  https://arxiv.org/abs/1506.02438
• Human-Level Control Through Deep Reinforcement Learning — Mnih et al. (2015), Nature 518. The DQN paper — replay buffer + target network.
  https://doi.org/10.1038/nature14236
• Asynchronous Methods for Deep Reinforcement Learning — Mnih et al. (2016), ICML. A3C / A2C — actor-critic at scale.
  https://arxiv.org/abs/1602.01783
• Soft Actor-Critic — Haarnoja et al. (2018), ICML. The strongest off-policy alternative to PPO.
  https://arxiv.org/abs/1801.01290
• Simple Statistical Gradient-Following Algorithms for Connectionist RL — Williams (1992), Machine Learning 8. The original REINFORCE.
• Stable-Baselines3: Reliable RL Implementations — Raffin et al. (2021), JMLR 22. The PPO implementation used in this project.
  https://jmlr.org/papers/v22/20-1364.html