I trained an AI to land a small rocket without crashing, then changed the scoring to see how it breaks.
In reinforcement learning you don't tell the AI how to do the task. You give it a score and let it try thousands of times. It keeps whatever earned points and drops whatever lost them.
That makes the scoring rule the real design decision. Change how points are awarded and you change what the AI learns. This is called reward shaping, and it is what this project is about: get a working lander, write my own reward, break it on purpose, then fix it.
Eight numbers every frame (50 frames a second). It never sees the picture.
| Number | Meaning |
|---|---|
| 1, 2 | Left/right and height above the pad |
| 3, 4 | Sideways and vertical speed |
| 5, 6 | Tilt and spin |
| 7, 8 | Left and right leg touching ground |
Four choices per frame: nothing, left engine, main engine, right engine.
A small neural network (two layers of 64 units) trained with PPO, which nudges the network toward choices that scored more, in small steps so one lucky run can't wreck what it has learned.
Each frame the game computes a "how good is this position" number: closer to the pad, slower, more upright and legs down all score higher. The reward is how much that number improved since the last frame, minus a small fuel cost (0.3 for the main engine, 0.03 for a side engine). Landing safely adds +100, crashing subtracts 100.
I scored every model on 50 fresh landings with exploration off and my reward wrapper off, so the score is the plain game score. 200 counts as solved.
The plain time fee was the best result in my runs: 37 of 50 landings scored 200 or more, against 18 of 50 for the baseline. The +1 per frame bonus collapsed the real score to -42 and made 32 of 50 episodes run to the 1,000 frame limit without landing. The improvement-based fix brought hovering back down to 2 of 50 and the score back to 178. It did not beat the plain time fee, and the gap between those two is within the noise of a single training run.
Each version is one training run, and a 50-landing average still moves by about 15 points between evaluations, so read the ordering as evidence, not proof. The collapse caused by the hack is far larger than that noise.
| Model | Mean score | Solved (200+) | Crashes | Timeouts |
|---|---|---|---|---|
| Random (3 episodes) | -260 | 0 / 3 | 3 / 3 | 0 |
| PPO baseline, 1M steps | 142.5 | 18 / 50 | 7 | 2 |
| Baseline + 500k steps (20 episodes) | 165.6 | n/a | 4 / 20 | 1 / 20 |
| Time fee, 0.1 per frame | 195.8 | 37 / 50 | 4 | 2 |
| Hackable bonus, +1 per frame | -41.7 | 0 / 50 | 15 | 32 |
| Improvement-based fix | 178.1 | 29 / 50 | 5 | 2 |
Recorded from the trained model. Put the clips in the videos folder next to
this page and rename them to match the file names below.
The experiment: add a rule of +1 for every frame the rocket stays upright and slow. It sounds reasonable. An attempt is cut off at 1,000 frames, so a rocket that just hangs in the air can collect up to 1,000 points from my rule while paying at most 300 in fuel. Landing ends the attempt and the payments stop, so hovering beats landing, and the AI can learn exactly that.
This is called reward hacking: the AI optimizes the score I wrote, not the goal I meant.
Pay for getting better (the change in position score between frames, like the built-in reward does) instead of for sitting in a good spot, and add a small fee every frame so waiting costs something.
With the bonus, the real game score fell from 143 to -42. None of 50 test landings reached 200, and 32 of 50 episodes ran to the 1,000 frame limit without landing. The rocket had learned to earn my bonus instead of landing. I always scored with my wrapper off, so the bonus could not flatter the result.
With the fix, hovering dropped back to 2 of 50 episodes and the score recovered to 178. The video clips above show both behaviours.
# the fix: pay for improvement, plus a small time fee new_phi = phi(obs) # closer, slower, more upright = higher reward += 5.0 * (0.99 * new_phi - prev_phi) - 0.1 prev_phi = new_phi
swig first (brew install swig).
LunarLander-v2. The current library only has v3.