A group of engineers researching COBOL code migration published a paper proposing to lock AI in a cage.
The paper’s title is unassuming: “Agentic Method for Deterministic Validation of Legacy Code Migration.” They designed a system called Locksmith Loop—a precisely chosen name. It’s a “locksmith,” not a “master.” It doesn’t expect AI to do the job perfectly in one go; instead, it confines the AI within a deterministic framework, letting it do only what it does best.
But the truly interesting part isn’t this conclusion. It’s why they had to do this and what happened after they did.
Three Modules, AI Only Touches One
The Locksmith Loop architecture is divided into three layers.
The first layer is the Migrator. COBOL source code goes in, Java target code comes out. This step does not use an LLM. It uses a deterministic AST translator—hardcoded rules where COMPUTE becomes BigDecimal.add(), and PERFORM VARYING becomes a for loop. Every run yields the exact same result. As TimByte succinctly put it on Hacker News: “They delegate code generation to deterministic tools, and the model only does input fuzzing—letting it wrestle with data, not syntax.”
The second layer is Witness Search. This is where AI makes its sole appearance. The LLM continuously constructs input cases in an attempt to penetrate all branches of the program. After each run, a coverage analyzer checks which branches haven’t been reached, and the LLM then generates targeted new inputs. This process loops until coverage can no longer be pushed further.
The third layer is the Oracle. The original COBOL version and the migrated Java version run the same input simultaneously, and the outputs must be bitwise identical. Any deviation is treated as a migration failure, triggering a rollback and correction.
Why design it this way? Because each step has a clear failure mode.
The Migrator is deterministic, so translation errors are reproducible and traceable. If the precision of a COMPUTE to BigDecimal conversion goes wrong, you know it’s because the conversion rule was written incorrectly, not because the model suffered a probabilistic drift during inference.
The Oracle is deterministic, so the validation result is binary—right is right, wrong is wrong. There is no gray area of “looks close enough.”
Witness Search is the only non-deterministic component. But its failure mode is controllable—the worst case is failing to find a triggering input for a specific branch, resulting in insufficient coverage. It won’t generate erroneous code, won’t introduce hallucinations, and won’t create bugs in the translation process.
This is Locksmith’s design philosophy: put AI where the consequences of its failure are the lightest.
When AI Can’t Push Further
Witness Search isn’t omnipotent. When it hits certain branch boundaries, it can get stuck. The paper calls this situation a “Locked Paragraph”—a condition that prevents deeper exploration, where the AI cannot construct an input capable of penetrating it.
This is the second meaning behind the name “Locksmith”: the system doesn’t expect the AI to pick every lock. When AI gets stuck, the analyzer flags this Locked Paragraph, and the system continues advancing through other branches. Ultimately, total coverage depends on how many “keys” the AI can find.
The paper reported results from three test cases: two open-source COBOL programs and one “internal production-grade” program. Sizes ranged from 430 lines to 4,114 lines. The two open-source programs achieved “near-complete coverage.” The production-grade program reached 91.90% branch coverage.
91.90%. This number itself isn’t worth much discussion because the sample size is too small. What’s truly valuable is what the remaining 8.1% signifies. A Locked Paragraph represents the boundary of the AI’s capabilities—it found 91.90% of the keys, but the remaining 8.1% of the locks it couldn’t open. In real-world systems, these locks likely correspond to the most complex business logic, the deepest nested conditions, and the oldest corner cases.
Bringing Bugs Along is a Design Goal
The architecture has another counterintuitive design: the migrated Java code must behave exactly like the original COBOL, including the bugs.
User aldente0630 clarified this on Hacker News: “Preserving bugs is an explicit design goal.” But this doesn’t mean the paper is encouraging the retention of bugs—it’s simply acknowledging a reality. In the world of legacy migration, this is called bug-for-bug compatibility. Those bugs have been running in production environments for twenty years, and downstream systems depend on their behavior. Fixing a bug is equivalent to fixing a feature that someone else assumed was working normally.
Hyrum’s Law applies perfectly here: “With a sufficiently large number of users of an API, it does not matter what you promised in the contract: all observable behaviors of your system will be depended on by somebody.”
But bug-for-bug has a prerequisite: the Oracle compares behavior, not code. You preserve the behavior of the bug, but you don’t preserve the bug’s understandability. The old COBOL programmers knew why that bug existed—maybe it was left over from an emergency hotfix, or perhaps it was a relic of a business rule change. But when Java programmers look at this code with “intentionally preserved bug behavior,” they only see code that looks like a mistake, with zero context.
4KLOC and 230KLOC
Commenters on Hacker News didn’t discuss the architectural design; they immediately questioned the scale of the study.
pacaro offered a real-world perspective: “The IRS alone has about 160 COBOL programs, averaging 230,000 lines of code each.”
fock’s comment was more specific because he had tried it: “Our code had inline assembly embedded across a dozen files, 50,000 lines. This program used a certain preprocessor, and all the LLMs we tested had absolutely no idea what it was—they would fabricate its functionality. Gemini 2.5 didn’t even notice the preprocessor existed.”
This points to the true boundary of Locksmith Loop. The Migrator is a deterministic AST translator, but AST translation requires the ability to parse the source code. If the source code contains preprocessor macros, inline assembly, CICS transaction commands, or JCL batch job scheduling—these aren’t part of COBOL syntax; they’re part of the environment. The AST translator doesn’t recognize them and won’t fabricate them (which is good), but it will fail outright (which is the problem).
Zenst added concerns about precision: “COBOL is known for fixed-point math with no rounding. In Java, 0.1 + 0.2 equals 0.30000000000000004. Unless you comprehensively use BigDecimal, the migration is broken from the start.” The paper used BigDecimal, which is correct. However, financial system COBOL programs typically use packed-decimal (BCD encoding), whose precision and rounding behavior aren’t entirely equivalent to BigDecimal. The Oracle’s bitwise comparison will catch these discrepancies, but fixing them means writing increasingly complex rules in the Migrator.
In other words, the deterministic architecture direction of Locksmith Loop is right, but its engineering complexity will grow non-linearly with system scale. Going from 4KLOC to 40KLOC isn’t just multiplying the workload by 10; it’s an simultaneous explosion of edge cases, environmental dependencies, precision discrepancies, and non-standard syntax you have to handle.
COBOL-in-Java
dragonwriter on Hacker News pointed out the deepest contradiction of this architecture: “The only realistic low-error path isn't more sophisticated AI usage, but deterministic translation. The problem is, what you get that way is COBOL-in-Java—runs correct[ly], but is a maintainability nightmare.”
The product of deterministic translation is: the syntax is Java, but the soul is COBOL. Variable names, control flow, and data structures all carry a 1960s flavor. PERFORM VARYING becomes a for loop, but the naming conventions for variables, the organizational logic of data, and the branching structures for error handling are all preserved exactly as they were.
When human programmers do migration, they do something the Migrator won’t: re-understand the system’s intent, then re-express it using the idiomatic ways of the new language. This introduces risk (the understanding could be wrong), but it’s the only way to produce maintainable code.
The Migrator chose the zero-risk path, but paid the price of maintainability. You lose the old COBOL programmers only to gain Java programmers—who, looking at this code, are even more confused than the old programmers were.
matsemann’s first sentence on Hacker News hit the nail on the head: “At least previously a few old COBOL programmers knew the system. Now nobody knows it.”
You’ve transformed an old system nobody understands into a new system nobody understands. The problem isn’t solved; it just changed languages.
What the Paper Left Unsaid
Locksmith Loop’s architectural design is sound. Restricting AI to test generation while leaving translation and validation to deterministic code—this might be the only reliable way to use AI in serious software engineering.
But it sidesteps a larger issue: the true bottleneck in migrating COBOL systems has never been “translating a piece of code,” but rather understanding what this system actually does. Code untouched for twenty years, with no documentation, no tests, and original developers retired. If you don’t even know what output it’s supposed to produce, how do you validate that the Oracle’s output is correct?
The paper assumes the Oracle is a given—the original COBOL version can run, produce output, and be compared against the Java version. But in real-world scenarios, the COBOL system might not even run anymore, or it might only run in a specific mainframe environment that can’t be replicated. The Oracle isn’t ready-made; the Oracle itself is something that needs to be rebuilt.
This isn’t Locksmith Loop’s fault. It’s a validation methodology paper, not a migration engineering paper. But if you are seriously considering migrating a COBOL system, what you need to know isn’t “AI can do test generation,” but rather “Can your COBOL program still run?”
References
• Agentic Method for Deterministic Validation of Legacy Code Migration — arxiv (https://arxiv.org/abs/2607.28271)
• Hacker News Discussion (https://news.ycombinator.com/item?id=49150773)