CGFuse: Deep Graph-Language Fusion for Structure-Aware Code Generation

Code for the FORGE '26 paper Deep Graph-Language Fusion for Structure-Aware Code Generation by Mert Tiftikci, Amir Molzam Sharifloo and Mira Mezini (arXiv:2605.03689).

CGFuse pre-trains a graph neural network (GNN) on code graphs (ASTs augmented with data-flow edges) and fuses its node representations into intermediate layers of a pre-trained language model at the token level:

$$h^l_f = h^l_\theta + \lambda , h^l_g$$

where $h^l_\theta$ is the transformer's output at layer $l$, $h^l_g$ the GNN's output for the nodes in the current context, and $\lambda$ the fusion strength.

Setup

conda create -n cgfuse python=3.12 && conda activate cgfuse
pip install torch==2.8.0 --index-url https://download.pytorch.org/whl/cu128   # pick your CUDA
pip install -r requirements.txt

All commands below run from the repository root. Outputs go to .dataset/ (graph cache), .logs/ (checkpoints) and .aim/ (metrics; browse with aim up). Add debug=True to any training command for a quick run on 128 samples without logging (and trainer.accelerator=cpu on a machine without a GPU).

The three stages

1. Build the code graphs

python -m cgfuse.build_graphs --dataset concode --language java          # fusion / evaluation data
python -m cgfuse.build_graphs --dataset codesearchnet --language java    # GNN pre-training data

Each sample's code is parsed with tree-sitter into an AST and augmented with comingFrom / calculatedBy data-flow edges (cgfuse/utils/preprocessors/). The graphs are tokenizer independent; sub-tokens are created at training time with the target model's tokenizer. A handful of samples cannot be parsed and are excluded (see graph_not_generated in cgfuse/utils/datasets/).

2. Pre-train the GNN expert

The GNN is pre-trained on node classification for each target PLM (its embeddings are initialised from that PLM's token embeddings, so an expert belongs to one PLM):

python -m cgfuse task=bs_gnn_jv \
    task/model@model=PLBART \
    task/gnn@gnn=rgcn \
    task/dataset@dataset=codesearchnet dataset.pl_language=java \
    gnn.intermediate=[] gnn.activation=gelu

• task/gnn@gnn: rgcn, graphsage or gin.
• gnn.intermediate: [] is a single GNN layer (used for the main results); [512] and [512,512] give the 2- and 3-layer variants of Table 4.
• gnn.activation: gelu for the BERT/BART families, relu for T5/CodeT5.

The checkpoint is written to .logs/bootstrap/gnn_java/ckpt/<run hash>/last.ckpt.

3. Fuse the GNN into the PLM and fine-tune

python -m cgfuse task=cgfuse \
    task/model@model=PLBART \
    task/trainer/optimizer@trainer.optimizer=PLBART \
    task/trainer/scheduler@trainer.scheduler=PLBART \
    trainer.fusion.expert=.logs/bootstrap/gnn_java/ckpt/<run hash>/last.ckpt \
    "trainer.fusion.target_layers=[11]" \
    trainer.fusion.scale=1.0

Training first adapts only the attached GNN (trainer.fusion.adapting_epoch), then fine-tunes the PLM and GNN jointly (trainer.fusion.epoch), with early stopping on validation CodeBLEU. The best checkpoint is evaluated on the CONCODE test set (EM, BLEU, CodeBLEU).

trainer.fusion.scale is $\lambda$. trainer.fusion.target_layers lists the layers the GNN is attached to. The paper attaches a single-layer GNN to the last layer only: [11] for the encoder models (indexing the decoder they are turned into) and for BART/PLBART (encoder+decoder stack of 12), [23] for T5/CodeT5 (stack of 24).

scripts/main_results.sh runs stages 2 and 3 for every PLM with the settings of Table 2:

PLM	task/model@model	optimizer / scheduler	GNN	λ	target_layers
BERT	bert	bert	R-GCN	1	[11]
RoBERTa	roberta	roberta	GIN	0.5	[11]
BART	bart	bart	GIN	0.5	[11]
T5	t5	t5	GIN	0.5	[23]
CodeBERT	CodeBERT	CodeBERT	GIN	1	[11]
GraphCodeBERT	GraphCodeBERT	GraphCodeBERT	GraphSAGE	1	[11]
UniXcoder	unixcoder	unixcoder	GIN	1	[11]
PLBART	PLBART	PLBART	R-GCN	1	[11]
CodeT5	codeT5_M220	codeT5	GIN	1	[23]

bash scripts/main_results.sh             # all nine PLMs
bash scripts/main_results.sh PLBART      # one PLM (DEVICE=1 to pick another GPU)

Baselines

The same PLMs fine-tuned without a GNN:

bash scripts/baselines.sh                # all nine PLMs, or e.g. `bash scripts/baselines.sh PLBART`

Layout

cgfuse/
  __main__.py          Hydra entry point (python -m cgfuse task=...)
  build_graphs.py      stage 1: code graph cache
  config/              Hydra configs: task/{baseline,bs_gnn_jv,cgfuse}.yaml + model/gnn/dataset/trainer groups
  tasks/               bootstrap.py (GNN pre-training), struct_infuse.py (fusion), baseline.py
  models/              GNN experts (R-GCN, GraphSAGE, GIN) and encoder -> encoder-decoder conversion
  utils/adapter.py     the fusion adapter that injects GNN features into transformer layers
  utils/preprocessors/ tree-sitter AST + data-flow graph extraction per language
  utils/datasets/      CONCODE and CodeSearchNet loaders
  utils/metrics/       EM, BLEU, CodeBLEU
scripts/               main_results.sh (Table 2, CGFuse rows), baselines.sh (Table 2, baseline rows)

Notes

• Rebuilding the graphs with the pinned tree-sitter versions reproduces the caches used for the paper (checked for CONCODE and CodeSearchNet Java), except for a handful of samples (4 of the 100k CONCODE training samples) whose non-ASCII character literals are now tokenized as <unk> instead of ?.
• All results in the paper come from a single run with seed 42.
• cgfuse/utils/metrics/ vendors third-party evaluation code under its original licenses: CodeBLEU (Microsoft / k4black/codebleu, MIT), NLTK's BLEU (Apache 2.0), Google's BLEU (Apache 2.0) and the smoothed BLEU of CodeXGLUE.

License

MIT, see LICENSE. Vendored third-party code keeps its own license (see Notes).

Citation

@inproceedings{tiftikci2026cgfuse,
  title     = {Deep Graph-Language Fusion for Structure-Aware Code Generation},
  author    = {Tiftikci, Mert and Molzam Sharifloo, Amir and Mezini, Mira},
  booktitle = {Proceedings of the 2026 IEEE/ACM Third International Conference on AI Foundation Models and Software Engineering (FORGE '26)},
  year      = {2026},
  address   = {Rio de Janeiro, Brazil},
  publisher = {ACM},
  doi       = {10.1145/3793655.3793724}
}