Shuu12121 / CodeEncoderDecoderModel-Ghost-large

CodeEncoderDecoderModel-Ghost-large👻

A multilingual encoder-decoder model for generating docstrings from code snippets .
It is based on a custom BERT-style encoder pretrained on source code ( CodeModernBERT-Ghost ) and a large-scale decoder model ( GPT2-large ).

🏗️ Model Architecture

• Encoder: Shuu12121/CodeModernBERT-Ghost
• Decoder: openai-community/gpt2-large
• Connected via HuggingFace's EncoderDecoderModel with cross-attention.

🎯 Intended Use

• Generating docstrings (documentation comments) for functions or methods in multiple languages.
• Summarizing code for educational or review purposes.
• Assisting in automated documentation generation pipelines.

Supported languages (code input):

• Python
• Java

📦 How to Use

from transformers import AutoTokenizer, EncoderDecoderModel
import torch

model = EncoderDecoderModel.from_pretrained("Shuu12121/CodeEncoderDecoderModel-Ghost-large").to("cuda")
encoder_tokenizer = AutoTokenizer.from_pretrained("Shuu12121/CodeEncoderDecoderModel-Ghost-large", subfolder="encoder_tokenizer")
decoder_tokenizer = AutoTokenizer.from_pretrained("Shuu12121/CodeEncoderDecoderModel-Ghost-large", subfolder="decoder_tokenizer")

if decoder_tokenizer.pad_token is None:
    decoder_tokenizer.pad_token = decoder_tokenizer.eos_token

code = '''
def greet(name):
    return f"Hello, {name}!"
'''

inputs = encoder_tokenizer(code, return_tensors="pt", truncation=True, padding=True, max_length=2048).to("cuda")
outputs = model.generate(
    input_ids=inputs.input_ids,
    attention_mask=inputs.attention_mask,
    max_length=256,
    num_beams=5,
    early_stopping=True,
    decoder_start_token_id=model.config.decoder_start_token_id,
    eos_token_id=model.config.eos_token_id,
    pad_token_id=model.config.pad_token_id,
    no_repeat_ngram_size=2
)

docstring = decoder_tokenizer.decode(outputs[0], skip_special_tokens=True)
print(docstring)

🧪 Training Details

• Task: Code-to-docstring generation
• Dataset: CodeXGLUE: Code-to-Text – using subsets of Python, Java, JavaScript, Go, Ruby, PHP
• Loss: Cross-entropy loss over tokenized docstrings
• Max input length: 2048 (encoder), max output length: 256 (decoder)
• Decoder modifications: Adapted GPT2-large with padding and cross-attention

⚠️ Limitations & Risks

1. Generated documentation may be inaccurate, incomplete, or misleading . Always review generated docstrings manually.
2. Formatting may not follow specific standards (e.g., Google/Numpy style in Python or full Javadoc).
3. Limited context: Only considers single-function input; lacks broader project-level understanding.
4. Language variance: Performance may differ depending on the programming language due to data distribution.
5. ⚠️ Decoder risks (GPT2-large):
   GPT-2 models are known to sometimes generate inappropriate, offensive, or biased outputs, depending on the prompt.
   Although this model is fine-tuned on technical datasets (code-docstring pairs), due to inherited properties from gpt2-large , similar risks may still be present in edge cases. Please exercise caution, especially when using the model in public or educational settings.

📄 License

Apache-2.0
Model weights and tokenizer artifacts are released under the same license. You are free to use, modify, and redistribute with attribution.