On Evaluating the Efficiency of Source Code Generated by LLMs

Changan Niu, Ting Zhang, Chuanyi Li, Bin Luo, and Vincent Ng.
Proceedings of the 2024 IEEE/ACM First International Conference on AI Foundation Models and Software Engineering, pp. 103-107, 2024.

Click here for the PDF version.

Abstract

Recent years have seen the remarkable capabilities of large language models (LLMs) for code generation. Different from existing work that evaluate the correctness of the code generated by LLMs, we propose to further evaluate its efficiency. More efficient code can lead to higher performance and execution efficiency of programs and software completed by LLM-assisted programming. First, we evaluate the efficiency of the code generated by LLMs on two benchmarks, HumanEval and MBPP. Then, we choose a set of programming problems from the online judge platform LeetCode to conduct a more difficult evaluation. Finally, we explore several prompts that would enable LLMs to generate more efficient code.

BibTeX entry

@InProceedings{Niu+etal:24b,
  author = {Changan Niu and Ting Zhang and Chuanyi Li and Bin Luo and Vincent Ng},
  title = {On Evaluating the Efficiency of Source Code Generated by LLMs},
  booktitle = {Proceedings of the 2024 IEEE/ACM First International Conference on AI Foundation Models and Software Engineering},
  pages = {103--107},
  year = 2024}