LLMs之Benchmarks：《ProgramBench: Can Language Models Rebuild Programs From Scratch?》翻译与解读

地址

论文地址：https://arxiv.org/abs/2605.03546

时间

2026年05月05日

作者

Meta FAIR、Meta TBD、斯坦福大学、哈佛大学

Turning ideas into full software projects from scratch has become a popular use case for language models. Agents are being deployed to seed, maintain, and grow codebases over extended periods with minimal human oversight. Such settings require models to make high-level software architecture decisions. However, existing benchmarks measure focused, limited tasks such as fixing a single bug or developing a single, specified feature. We therefore introduce \bench to measure the ability of software engineering agents to develop software holisitically. In \bench, given only a program and its documentation, agents must architect and implement a codebase that matches the reference executable’s behavior. End-to-end behavioral tests are generated via agent-driven fuzzing, enabling evaluation without prescribing implementation structure. Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task, with the best model passing 95% of tests on only 3% of tasks. Models favor monolithic, single-file implementations that diverge sharply from human-written code.

将想法从零开始转化为完整的软件项目已成为语言模型的一个热门用例。代理被部署用于在长时间内以最少的人工监督来播种、维护和扩展代码库。这种设置要求模型能够做出高层次的软件架构决策。然而，现有的基准测试衡量的是诸如修复单个错误或开发单个指定功能等集中且有限的任务。因此，我们引入了 \bench来衡量软件工程代理全面开发软件的能力。在 \bench 中，仅给定程序及其文档，代理必须架构并实现一个与参考可执行文件行为相匹配的代码库。通过代理驱动的模糊测试生成端到端的行为测试，从而能够在不规定实现结构的情况下进行评估。我们的200 个任务涵盖了从紧凑的命令行工具到广泛使用的软件，如 FFmpeg、SQLite 和 PHP 解释器。我们评估了 9 个语言模型，发现没有一个能完全解决任何任务，表现最佳的模型仅在 3%的任务中通过了 95%的测试。模型倾向于采用单体式、单文件的实现方式，这与人类编写的代码大相径庭。

Language Models (LMs) are increasingly being used to turn ideas expressed in natural language into full-fledged code repositories (Carlini, 2026; Lin, 2026; Replit, 2026). Unlike smaller scope tasks such as function generation (Hendrycks et al., 2021) or GitHub issue resolution (Jimenez et al., 2024), which typically demand understanding a pre-existing codebase well enough to make localized changes, building a functional application from scratch requires models to engage heavily with software design (Jansen and Bosch, 2005).

To understand what this entails, consider how a human programmer approaches the same task. Before a single line is written, she asks herself a series of important questions: What programming language and build system should be used? How should the codebase be organized? What data structures should represent the program’s core entities? How should errors be detected and communicated? Such requisite questions, which developers constantly revisit throughout the development lifecycle, lead to pivotal design decisions that shape the codebase far more profoundly than any individual code change. Although we are progressively entrusting LMs to similarly build software from the ground up, the ability of LMs to make such architectural decisions, choose abstractions, and decompose a system into coherent modules has not been studied extensively.

To bridge this gap, we introduce \bench, a benchmark that challenges software engineering (SWE) agents to produce code that recovers the functionality of a software program (e.g., executables, .dmg’s, .pkg’s). Given a program and documentation, a SWE-agent, defined as an LM equipped with an agent scaffold to interact with a terminal environment (Yang et al., 2024a), must write source code and a compile script that reproduces the original program’s behavior. Every software design decision is entirely the model’s to make.

语言模型（LMs）正越来越多地被用于将自然语言表达的想法转化为完整的代码库（Carlini，2026；Lin，2026；Replit，2026）。与诸如函数生成（Hendrycks 等人，2021）或 GitHub 问题解决（Jimenez 等人，2024）这类较小范围的任务不同，这些任务通常需要对现有的代码库有足够深入的理解以便进行局部修改，从零开始构建一个功能性的应用程序则要求模型在软件设计方面投入大量精力（Jansen 和 Bosch，2005）。

要理解这涉及哪些内容，不妨考虑一下人类程序员是如何处理这一任务的。在编写任何一行代码之前，她会问自己一系列重要的问题：应该使用哪种编程语言和构建系统？代码库应该如何组织？哪些数据结构应该代表程序的核心实体？如何检测和传达错误？这些开发者在整个开发周期中不断回顾的关键问题，会引导出对代码库产生深远影响的重要设计决策，其影响远超任何单独的代码更改。尽管我们正逐步将构建软件从头开始的任务交由语言模型（LM）来完成，但语言模型在做出此类架构决策、选择抽象概念以及将系统分解为连贯模块方面的能力尚未得到充分研究。

为了弥补这一空白，我们引入了 \bench 基准测试，该测试要求软件工程（SWE）代理生成能够恢复软件程序功能（例如可执行文件、.dmg 文件、.pkg 文件）的代码。给定一个程序及其文档，SWE 代理（定义为配备有代理框架以与终端环境进行交互的语言模型，Yang 等人，2024a）必须编写源代码和编译脚本，以重现原始程序的行为。每个软件设计决策完全由模型自行决定。

We synthesize \bench tasks from open-source GitHub repositories. First, we identify repositories written in compiled languages (e.g., C/C++, Golang, Rust, Java) that build a program. Next, to convert a repository into a task instance, we compile the program, then strip away all source code and tests, leaving only the program and its documentation as the task’s starting point.

To evaluate a model’s solution, we generate behavioral tests by prompting a SWE-agent to systematically probe the original program with varied inputs and codify the observed input-output behavior into assertions that a candidate reconstruction must satisfy. Crucially, these tests are never revealed to the task worker. Since tests target executable behavior rather than source code, evaluation is entirely implementation agnostic; a model may use different algorithms, abstractions, or even programming languages than the original codebase, and still pass as long as the input-output behavior matches. While any test suite necessarily under-approximates an executable’s full specification, we empirically demonstrate that our test generation pipeline creates large suites that reliably capture core functionality.

Using our pipeline, we collect 200 task instances, ranging from compact CLI tools to complex, widely used software including language interpreters (PHP, Lua, tinycc), databases (DuckDB, SQLite), media and compression utilities (FFmpeg, zstd, xz), and developer tools (ripgrep, fzf, jq). We evaluate 9 language models equipped with mini-SWE-agent, a widely adopted coding agent scaffold for open source SWE-agent research. The results resoundingly confirm \bench’s difficulty for today’s models; no task instance is fully resolved. However, test pass rates are significantly different between models. The best model, Opus 4.7, manages to pass 95% of tests for 3% of task instances. Further analysis reveals that model-written codebases diverge significantly from human-written ones, favoring monolithic file structures with longer functions. Our trajectory analyses showcase how models vary in the length and make up of the way they develop software.

We open source \bench to enable the community to reproduce and build upon our investigations.

我们从开源的 GitHub 代码库中综合生成 \bench 任务。首先，我们识别出用编译型语言（例如 C/C++、Golang、Rust、Java）编写的构建程序的代码库。接下来，为了将一个代码库转换为任务实例，我们先编译该程序，然后删除所有源代码和测试，仅保留程序及其文档作为任务的起点。为了评估模型的解决方案，我们通过提示软件工程师代理（SWE-agent）系统地用各种输入对原始程序进行探测来生成行为测试，并将观察到的输入输出行为编码为候选重建必须满足的断言。至关重要的是，这些测试永远不会透露给任务执行者。由于测试针对的是可执行行为而非源代码，因此评估完全不受实现方式的影响；模型可以使用与原始代码库不同的算法、抽象或甚至编程语言，只要输入输出行为匹配即可通过。虽然任何测试套件必然无法完全涵盖可执行文件的完整规范，但我们通过实证表明，我们的测试生成流程能够创建包含大量测试的套件，可靠地捕获核心功能。

我们利用流水线收集了 200 个任务实例，涵盖从紧凑的命令行工具到复杂的、广泛使用的软件，包括语言解释器（PHP、Lua、tinycc）、数据库（DuckDB、SQLite）、媒体和压缩工具（FFmpeg、zstd、xz）以及开发工具（ripgrep、fzf、jq）。我们评估了 9 个配备 mini-SWE-agent（一个广泛采用的开源 SWE-agent 研究的编码代理框架）的语言模型。结果明确证实了 \bench 对当今模型的难度；没有一个任务实例能完全解决。然而，不同模型的测试通过率差异显著。表现最佳的模型 Opus 4.7 在 3%的任务实例中通过了 95%的测试。进一步分析表明，模型编写的代码库与人工编写的代码库差异显著，倾向于采用单体文件结构和更长的函数。我们的轨迹分析展示了模型在软件开发的长度和构成方面的差异。

我们开源 \bench 以使社区能够重现并在此基础上进行研究。

Limitations. \bench relies on a finite set of behavioral tests, which under-approximates each executable’s full specification. Evaluation therefore is a “lower bound” on correctness: solutions that fail are definitively incorrect, while those that pass may still diverge from the original on untested inputs. \bench tests also currently focus exclusively on input-output equivalence. Non-functional properties like execution speed, memory usage, or disk footprint are not captured. Therefore, it is possible a model reproduces behavior with an implementation orders of magnitude slower or more resource intensive than the original. Developing richer test generation strategies to improve coverage and incorporate system constraints is a promising direction.

bench 依赖于一组有限的行为测试，这些测试对每个可执行文件的完整规范进行了下近似。因此，评估是正确性的“下限”：失败的解决方案肯定是不正确的，而通过的解决方案在未测试的输入上仍可能与原始程序存在差异。 \bench 测试目前也仅专注于输入输出等价性。执行速度、内存使用或磁盘占用等非功能性属性未被涵盖。因此，有可能模型重现的行为在执行速度或资源消耗方面比原始程序慢几个数量级或更耗费资源。开发更丰富的测试生成策略以提高覆盖率并纳入系统约束是一个有前景的方向。

Future work. Several technical reports and blogs have suggested the effectiveness of applying multiple SWE-agents towards long horizon coding tasks (Lin, 2026; Carlini, 2026; Geng and Neubig, 2026; Mishra-Sharma, 2026). \bench can serve as a testbed for such works. Our work uses a single SWE-agent as the baseline; this design reflects prior benchmark evidence, notably SWE-bench, where well-tuned single-agent systems have performed competitively, and multi-agent variants have not consistently shown clear advantages. We are excited to use \bench to delineate the benefits of multi-agent approaches. Similarly, \bench could further exploration into human-centered coding agents, where a developer, given the executable, iteratively guides the agent through design decisions (Liu et al., 2025; Baumann et al., 2026; Wang et al., 2026).

未来工作。几份技术报告和博客都表明，将多个 SWE 代理应用于长期编码任务是有效的（林，2026 年；卡林尼，2026 年；耿和纽比格，2026 年；米什拉 - 沙尔马，2026 年）。\bench 可以作为此类工作的试验平台。我们的工作以单个 SWE 代理作为基准；该设计反映了先前基准测试的证据，尤其是 SWE-bench，其中经过良好调优的单体系统表现出了竞争力，而多体变体并未始终展现出明显的优势。我们很高兴能利用 \bench 来阐明多体方法的优势。同样，\bench 还能进一步探索以人类为中心的编码代理，即开发者在给定可执行文件的情况下，通过设计决策来逐步引导代理（Liu 等人，2025 年；Baumann 等人，2026 年；Wang 等人，2026 年）。

Conclusion. We introduce \bench, a benchmark for measuring the ability of software engineering agents to develop, from scratch, programs that match a given executable’s behavior. Existing models struggle substantially, and none fully resolve any task. However, via fine-grained metrics, we find that models achieve meaningful partial progress, with stark differences in how models expend turns and the final form of their codebases. Our analyses reveal meaningful gaps in models’ decision making in architecting, developing and testing software. We hope that \bench could serve as a testbed for efforts focused on end-to-end autonomous software development.

结论。我们引入了 \bench，这是一个用于衡量软件工程代理从零开始开发与给定可执行文件行为相匹配的程序的能力的基准。现有的模型在很大程度上存在困难，且没有一个能完全解决任何任务。然而，通过细粒度的指标，我们发现模型取得了有意义的部分进展，模型在消耗步数和最终代码库的形式方面存在显著差异。我们的分析揭示了模型在软件架构、开发和测试决策方面存在的有意义的差距。我们希望 \bench 能够成为专注于端到端自主软件开发工作的试验场。