Beyond the Leaderboard: Multi-Dimensional Evaluation of Dense and Mixture-Of-Experts Models for Automated Program Repair

Automated Program Repair APR with language models is usually evaluated by whether a generated patch passes the test suite, which can hide differences in maintainability, security, and computational cost. We propose a Weighted Quality Index QI, inspired by the ISO/IEC 25010 software quality model,...