GitHub Models: A Comprehensive Guide to AI Development

Embark on your AI development journey with GitHub Models! This comprehensive guide provides an in-depth look at GitHub's AI model playground. Whether you're a seasoned developer or just starting out, GitHub Models offers a catalog of AI models to help you build cutting-edge products and features. It starts completely free during development, letting you explore and experiment without initial costs. Dive in and discover how to implement any AI model into your application seamlessly.

Navigating to GitHub Models

The first step to harness the power of GitHub Models is to navigate to the correct URL: github.com/marketplace/models. This will take you to the central hub where you can browse and select from a variety of AI models. A regular, free GitHub account is all you need to get started exploring this powerful platform. Once you're logged in, you will see a variety of options to choose from, catering to different AI development needs. The goal here is to create awareness and provide developers the accessibility for AI features and products. You will have model switching and a single API key for all models. Quick personal setup and a Free to start option.

Selecting and Comparing AI Models

Once you're on the GitHub Models page, selecting an AI model is simple. Just click on the 'Model: Select a Model' dropdown menu.

This opens a search bar where you can find specific models like OpenAI GPT-4o or DeepSeek-R1, along with many others. Once you have selected a Model it opens a page for interacting with it. One of the significant advantages of GitHub Models is the ability to directly compare different models. Click the 'Compare' button to see two models side by side. With the models side by side you can ask your test questions and compare. This functionality is especially valuable for determining which model is best suited for your specific use case, optimizing performance and cost. For example, you can compare non-reasoning and reasoning questions with different models. This streamlined approach facilitates informed decision-making in AI selection.

Implementing AI Models in Your Applications

Most tutorials stop at showcasing models, but this guide goes further, detailing how to implement these models directly into your applications. You can click on the “Use this model” option for initial guidance. The specific instructions vary based on the programming language. This guide uses Python for its examples but there are other options.

You’ll need to acquire an API key. A Personal Access Token (PAT) is required to interface with the model programmatically. The PAT allows developers to query models within specified rate limits, which are sufficient for development purposes. For production environments, migrating to Azure AI is recommended, but note that the initial rate limits are quite generous for development. Regardless of the programming method selected, a Personal Access Token is required. Access tokens allow you to query the models with the rate limits that allow you to develop in your own applications. For production needs, you will need to move to Azure AI. But you will see that the rate limits that you get are quite lenient and will allow you to build out your application and even have some test users before you dedicate to an Azure subscription.

Creating a Personal Access Token on GitHub

To create a Personal Access Token (PAT) on GitHub, follow these steps:

1. Go to your GitHub profile and click on 'Settings'.
2. Scroll down and click on 'Developer Settings'.
3. Select 'Personal access tokens', and then 'Fine-grained tokens' and click the “Generate new token” button. Regular tokens do not require specific permissions.

   These tokens are fine-grained and suitable for personal API use with GIT over HTTPS. With free access you can access inference with your Github PAT, learn more about the limits based on your plan.

4. Enter a name for your token (e.g., 'llm_token').
5. Set an expiration date for the token, which will expire upon the date provided.
6. Leave Public Repositories to read-only
7. Generate the token.

Setting Up the Development Environment with Fast API

To set up your development environment, it’s recommended to use Python and utilize Fast API which plugs into GitHub Models directly. This allows for you to easily compare different models and is free to get started with. This application is a micro service next to your regular application to handle all AI workload. To build this application you need one endpoint for all users to query the API and get a streamed response back.

Let's look at an example Fast API code snippet:

app = FastAPI(title="GitHub Streaming API")
@app.post("/chat/completions/stream")
async def stream_chat_completion(request: ChatRequest, client: OpenAI = Depends(get_openai_client)):
    #Prepare messages with system message
    messages = []
    messages.append({"role": "system", "content": request.system_message})
    messages.extend([{"role": m.role, "content": m.content} for m in request.messages])

   #Create async generator for streaming
    async def generate():
        try:
            response = client.chat.completions.create(
                messages=messages,
                temperature=request.temperature,
                top_p=request.top_p,
                max_tokens=request.max_tokens,
                model=request.model,
                stream=True
            )

            async for chunk in response:
                if chunk.choices:
                    yield data: json.dumps({"content": chunk.choices[0].delta.content})
        except Exception as e:
            yield data: json.dumps({"error": str(e)})

    return StreamingResponse(generate(), media_type="text/event-stream")

This is how you have all your users query the API and get a streamed response back to their service. The next step is make sure the models are valid through Open AI, otherwise the server will return an error. You can then pass that model string to get the appropriate response, or you can construct your Open AI environment.

 #This client can also be used for other models that support the OpenAI API spec, such as DeepSeek-R1
def get_openai_client():
    token = os.environ.get("GITHUB_TOKEN")
    if not token:
        raise HTTPException(status_code=500, detail="GITHUB_TOKEN not found in .env file")

    return OpenAI(
        base_url="https://models.inference.ai.azure.com",
        api_key=token,
    )

Load Test for GitHub Models API

One way to test the functionality of your API is to perform load tests.

To Load test and confirm it's working call different methods, like Open AI. Here is an example test file to run various tests:

import asyncio
import httpx
import json
import time
from datetime import datetime

API_URL = "http://localhost:8000/chat/completions/stream"
MODELS = ["gpt-4o", "DeepSeek-R1", "Phi-4"]
TEST_QUESTION = "What is the capital of France and why is it historically significant?"

async def test_individual_models():
   print("
" + "="*50)
   print("Testing Individual Models")
   print("="*50)

    async with httpx.AsyncClient() as client:
        for model in MODELS:
            print(f"
" + "-"*50)
            print(f"Testing model: {model} at {datetime.now().strftime('%H:%M:%S')}")
            print("-"*50)

            payload = {
                "messages": [{
                    "role": "user",
                    "content": TEST_QUESTION,
                }],
                "model": model,
                "temperature": 0.7,
                "max_tokens": 150
            }

            start_time = time.time()
            async with client.stream("POST", API_URL, json=payload, timeout=60.0) as response:
                if response.status_code == 200:
                   print("Response:")
                    async for line in response.aiter_lines():
                        # Skip the `data: ` prefix and handle the [DONE] marker
                        if line.startswith("data: ") and line != "data: [DONE]":
                            try:
                                data = json.loads(line[6:])
                                if "content" in data:
                                    content = data.get("content")
                                    print(content, end="", flush=True)
                                 full_response += content
                            except json.JSONDecodeError:
                                print("Error: Could not parse line")
                else:
                    print(f"Error: {response.status_code} - {await response.text()}")

            elapsed = time.time() - start_time
            print(f"
Completed in {elapsed:.2f}s")

async def load_test(model: str, concurrent_requests: int = 10):
   print("
" + "*"*50)
   print(f"Load Testing Model: {model} with {concurrent_requests} concurrent requests")
   print("
" + "*"*50)

    async with httpx.AsyncClient(timeout=None) as client:
        tasks = [
            make_async_request(client, model, i+1) for i in range(concurrent_requests)
        ]
        results = await asyncio.gather(*tasks)

   success_count = results.count(True)
   print(f"
Load test results: {success_count} successful requests out of {concurrent_requests}")
   if success_count < concurrent_requests:
       print("Some requests failed, likely due to rate limiting.  If you want this many requests at once, consider creating an Azure OpenAI account")

async def make_async_request(client: httpx.AsyncClient, model: str, request_id: int) -> bool:
    start_time = time.time()
    try:
        payload = {
            "messages": [{
                "role": "user",
                "content": TEST_QUESTION
            }],
            "model": model,
            "temperature": 0.7,
            "max_tokens": 150
        }

        async with client.stream("POST", API_URL, json=payload, timeout=60.0) as response:
            response.raise_for_status()
            async for line in response.aiter_lines():
                pass
            return True

    except httpx.HTTPStatusError as e:
       elapsed = time.time() - start_time
        print(f"Request {request_id} failed ({elapsed:.2f}s): {response.status_code} - {await response.text()}")
        return False

async def main():
   print("Testing GitHub Models API")

   await test_individual_models()

    #Choose one model for load testing
   load_test_model = "gpt-4o"  # Using gpt-4o for load testing

    #Run the load test
   await load_test(load_test_model)

if __name__ == "__main__":
    asyncio.run(main())

You can call certain commands to pull the data from different methods, but be careful because you don’t want to go over the rate limits!

Cost Considerations

GitHub Models offers a freemium approach. For initial exploration and small-scale development, the platform is free, utilizing a Personal Access Token (PAT) for authentication. This is to allow a Free start to the platform. However, once your application is ready for a broader audience, and you require higher usage limits, migrating to Azure OpenAI is recommended. Azure OpenAI offers various pricing tiers tailored to different levels of demand and specific model usage, ensuring scalability and cost-effectiveness. You can visit the pricing page for the particular model you've selected at the Github Marketplace.

Key Capabilities and Benefits

GitHub Models simplifies AI integration into your projects with these core features:

• AI Model Catalog: A wide selection of pre-trained AI models ready to be implemented.
• One-Click Model Switching: Streamlined switching between AI models with a single API key.
• Personal Access Tokens (PAT): Quick and secure personal setup for model installation using GitHub PAT.
• Free Development Tier: Cost-free experimentation until you reach specific usage limits.
• Model Comparison: Easy comparison of language models to determine the most suitable option for your application.
• OpenAI and other APIs: A variety of options for all sorts of programming, including python and Javascript

Diverse Application Scenarios

GitHub Models can be applied across various scenarios, including:

• Chatbots and Conversational AI: Enhancing user interaction through intelligent, responsive chatbots.
• Content Generation: Automating the creation of various content formats, including articles, social media posts, and marketing copy.
• Code Generation and Assistance: Aiding developers in generating code snippets and identifying errors.
• Data Analysis and Insights: Extracting valuable insights from complex datasets to inform decision-making.
• Image Recognition and Processing: Implementing vision-based AI to process and analyze images for various applications.