OpenAI releases MultiLangBench and COBOL migration eval on Oct 19; GPT-6 Astra hits 87.4% pass rate across 20 low-resource languages
OpenAI released MultiLangBench and COBOL migration eval: building a SQLite-compatible engine from scratch across 20 languages; GPT-6 Astra averaged 87.4% pass rate; GPT-6 Luna reached 40.3% at 78% lower cost; COBOL migration GPT-6.1 Sol reached 86.6% at low reasoning effort and 97.9% at high
On October 19, OpenAI published research titled "Advancing coding in low-resource programming languages," covering languages such as COBOL, Fortran, and Visual Basic that still underpin critical infrastructure but suffer from scarce training data and documentation. The post releases two complementary evaluations and the latest model numbers.
The first is MultiLangBench, which asks the model to build a SQLite-compatible database engine from scratch in any of 20 languages — C, C#, C++, COBOL, Common Lisp, Fortran, Go, Java, JavaScript, Lean, OCaml, PHP, PowerShell, Python, Ruby, Rust, Scala, Swift, Visual Basic, and Zig — judged by byte-level output parity with a reference SQLite. The test set is adapted from SQLite's existing test suite, emphasizing sustained build capability across long coding sessions.
The second is COBOL migration: migrate COBOL programs to C++, Java, Kotlin, Python, or Rust while preserving exact output against hidden tests, with particular attention to COBOL-specific quirks such as fixed-point arithmetic, rounding, and missing-record handling. Both the eval harness and tests are open-sourced on GitHub.
GPT-6 Astra averages 87.4% pass rate across the 20 languages, up from GPT-5.6 Sol's 26.8% — roughly a 60-percentage-point lift. Standout low-resource lifts include COBOL from 10.9% to 72.1%, OCaml from 25.6% to 85.8%, Fortran from 21.4% to 83.5%, and Visual Basic from 27.5% to 89.4%. GPT-6 Luna hits 40.3% at about $0.77 per sample, roughly 78% cheaper than GPT-5.6 Sol's $3.48, positioning it for cost-sensitive settings.
On COBOL migration, GPT-6.1 Sol reaches 86.6% at low reasoning effort (vs 5.4% for GPT-5.6 Sol) with 39% lower cost, and 97.9% at high reasoning effort, near saturation. COBOL today underpins more than 40% of online banking systems, 80% of in-person credit-card transactions, and 95% of ATM transactions; Fortran remains prevalent in weather forecasting and nuclear-reactor safety analysis. The post frames these gains as helping institutions maintain and migrate critical legacy systems while emphasizing that production deployment still requires expert review and rigorous business-rule testing.