Authors: Tim But, Nan Yang, Joseph Reynolds, Laurens Prast, Mauricio Verano Merino and Rosilde Corvino
Venue: IEEE International Conference on Source Code Analysis & Manipulation
Abstract
Large language models (LLMs) have become central to code generation tasks; however, effectively adapting them to specialized and low-data domains remains challenging. Although prior work has explored Text-to-Cypher generation, existing datasets largely target domains outside software engineering. Furthermore, the high computational cost of fine-tuning these models motivates the investigation of alternative adaptation strategies, such as Retrieval-Augmented Generation (RAG).
This paper presents an empirical evaluation of adaptation strategies for converting software developers’ questions about their codebase into Cypher queries for code graph. First, we explore how fine-tuned models with a cross-domain dataset by Neo4j could perform code graph analysis tasks. We then evaluate pre-trained models, their fine-tuned variants, and RAG-based architectures using a custom code graph dataset. To support this study, we expanded our prior benchmark dataset CodeGraphQA from 420 to 790 annotated questions by collecting questions from professional developers, evaluating performance based on executability, schema adherence, and semantic accuracy.
Our contributions are three-fold. First, we extend previous benchmark to examine LLMs’ ability in Text-to-Cypher translation for code graph analysis. Using this benchmark, we demonstrate that fine-tuning on a small and software engineering–specific code graph dataset is more effective than fine-tuning on a much larger cross-domain dataset for code graph analysis tasks. Furthermore, we show that when code graph data is scarce, RAG consistently outperforms fine-tuning in terms of overall performance. This work provides practical suggestions to practitioners and researchers who develop and deploy LLM-based solutions for code graph analysis in practice.