How to Build Your Own Image Describer Function Using Janus Pro 7B

Want to create your own image describer function? Janus Pro 7B is a multimodal AI model that combines image understanding with natural language generation. Here’s a quick overview of what you’ll need and how to get started:

• Why Use Janus Pro 7B?
  • Customize descriptions for your domain.
  • Maintain data privacy and control.
  • Integrate seamlessly into your workflows.
• System Requirements:
  • An NVIDIA GPU with about 16 GB of free VRAM: the 7B checkpoint is roughly 15 GB in bfloat16 and the README loads it with .to(torch.bfloat16).cuda().
  • Python 3.8 or newer; pip install -e . in the repository pulls in PyTorch 2.0.1+ and transformers 4.38.2+.
• Setup Steps:
  1. Clone the Janus repository and install it into a virtual environment.
  2. Load the model once; the weights download from Hugging Face automatically.
  3. Test your setup with a sample image.
• Key Features:
  • Visual reasoning, object detection, and semantic segmentation.
  • Adjustable parameters for description quality (e.g., temperature, max tokens).
• Advanced Options:
  • Fine-tune the model with domain-specific data for better accuracy.
  • Use multi-step workflows for complex image descriptions.

This guide provides everything you need – from setup to customization – to build a robust image description system tailored to your needs.

Complete Crash Course: Installing and Using Deepseek Janus Pro

Required Tools and Setup

To get started with Janus Pro 7B, ensure your system meets the necessary specifications outlined below. This will help you fully utilize its multimodal capabilities.

Software Requirements

The project publishes no minimum-spec table. In practice you need one NVIDIA GPU with about 16 GB of free VRAM (the Janus-Pro-7B weights are about 15 GB and are loaded in bfloat16), roughly 15 GB of disk for the download, Python 3.8 or newer and PyTorch 2.0.1 or newer. Janus is a PyTorch project; it does not use JAX. An earlier version of this article listed an RTX A6000 “with 24 GB” (the A6000 has 48 GB) and a JAX install line; both were wrong.

Create and activate a virtual environment first:

python -m venv janus_env
source janus_env/bin/activate

API Setup Steps

To access the model, follow these steps:

git clone https://github.com/deepseek-ai/Janus.git
cd Janus
pip install -e .

If you also want the browser demo, install the optional Gradio extras and start it; it prints a local URL (Gradio’s default port is 7860) and, because the script launches with share=True, a temporary public one:

pip install -e .[gradio]
python demo/app_januspro.py

There is no separate download step: the first call to from_pretrained fetches the weights from Hugging Face. This is how the README loads the model and its processor:

import torch
from transformers import AutoModelForCausalLM
from janus.models import MultiModalityCausalLM, VLChatProcessor
from janus.utils.io import load_pil_images

model_path = "deepseek-ai/Janus-Pro-7B"
vl_chat_processor: VLChatProcessor = VLChatProcessor.from_pretrained(model_path)
tokenizer = vl_chat_processor.tokenizer

vl_gpt: MultiModalityCausalLM = AutoModelForCausalLM.from_pretrained(
    model_path, trust_remote_code=True
)
vl_gpt = vl_gpt.to(torch.bfloat16).cuda().eval()

Initial System Check

Run the script below to test your setup:

# run after the loading snippet above
free, total = torch.cuda.mem_get_info()
print(f"GPU memory used: {(total - free) / 1024**3:.1f} GB of {total / 1024**3:.1f} GB")

Your setup is ready if:

• The loading snippet runs without errors and the weights finish downloading.
• PyTorch reports roughly 15 GB of GPU memory in use after loading.
• The describe_image function in the next section returns a sensible answer for a test image.
• If you started python demo/app_januspro.py, the Gradio page opens at the local URL it prints.

Optimization Tip

Keep an eye on system resources during your initial tests. If you encounter memory issues, try lowering batch sizes or enabling gradient checkpointing ^[1][2]. This can help manage resource consumption effectively.

Main Function Development

Now that your environment is set up, it’s time to build the core description function. Here’s how you can piece it together:

System Design Overview

The description function integrates Janus Pro 7B’s vision encoder (a SigLIP encoder working at 384×384) with its language decoder. The code below is the README’s multimodal-understanding example, wrapped as a reusable function; it assumes vl_gpt, vl_chat_processor and tokenizer were created by the loading snippet above:

@torch.inference_mode()
def describe_image(image_path, prompt="Describe this image.", **generate_kwargs):
    conversation = [
        {
            "role": "<|User|>",
            "content": f"<image_placeholder>\n{prompt}",
            "images": [image_path],
        },
        {"role": "<|Assistant|>", "content": ""},
    ]

    # load images and prepare for inputs
    pil_images = load_pil_images(conversation)
    prepare_inputs = vl_chat_processor(
        conversations=conversation, images=pil_images, force_batchify=True
    ).to(vl_gpt.device)

    # run image encoder to get the image embeddings
    inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs)

    # run the model to get the response
    gen = dict(max_new_tokens=512, do_sample=False)
    gen.update(generate_kwargs)
    outputs = vl_gpt.language_model.generate(
        inputs_embeds=inputs_embeds,
        attention_mask=prepare_inputs.attention_mask,
        pad_token_id=tokenizer.eos_token_id,
        bos_token_id=tokenizer.bos_token_id,
        eos_token_id=tokenizer.eos_token_id,
        use_cache=True,
        **gen,
    )
    return tokenizer.decode(outputs[0].cpu().tolist(), skip_special_tokens=True)


print(describe_image("test1.png"))

Image Preparation Steps

You do not need to write any preprocessing code. load_pil_images opens each path (or base64 string) listed under "images" and converts it to RGB, and VLChatProcessor resizes and normalises the image to what the vision encoder expects, using the values in the model’s preprocessor_config.json. An earlier version of this article included a hand-written transforms.Normalize step with ImageNet statistics; that would have fed the model the wrong tensors and has been removed.

Description Output Settings

You can adjust the style and quality of the descriptions by tweaking key parameters. Here’s a quick reference:

Parameter	Range	Recommended Value	Purpose
Temperature	0.1 – 1.0	0.8	Balances creativity in responses
Max Tokens	50 – 200	100	Controls the length of output
Top-k	20 – 100	50	Limits token selection options
Top-p	0.1 – 1.0	0.95	Adjusts sampling diversity

These settings allow you to optimize the output for specific use cases, ensuring the descriptions align with your needs.

The README calls generate with do_sample=False (greedy decoding). Sampling parameters are standard Hugging Face generate arguments and only take effect with do_sample=True; describe_image forwards them:

description = describe_image(
    "test1.png",
    "Describe this image.",
    do_sample=True,
    max_new_tokens=100,
    temperature=0.8,
    top_k=50,
    top_p=0.95,
    repetition_penalty=1.1,
)

Handling Complex Images

For more intricate images, you can use a multi-step approach to refine the descriptions:

def complex_image_description(image_path):
    base_desc = describe_image(image_path, "Provide a brief overview:")

    # Context-aware detailed pass
    detailed_desc = describe_image(
        image_path,
        f"Based on this context: {base_desc}, describe specific details:"
    )

    return detailed_desc

This method starts with a general overview and then dives deeper into specific details, improving the clarity and precision of the generated descriptions.

Janus Pro 7B’s dual processing streams ensure flexibility and accuracy when working with different image types. By fine-tuning parameters and preprocessing correctly, you can maintain consistent output quality across various tasks ^[5].

Improving Description Quality

Building on the core function discussed earlier, these updates integrate seamlessly with the description functionality outlined in Section 3.

Custom Dataset Training

The released Janus-Pro checkpoints are general-purpose. If your images come from a narrow domain (medical, industrial, retail), fine-tuning on a few thousand labelled image-description pairs is the usual way to get vocabulary and emphasis right; the Janus repository ships inference code only, so you would use a standard Hugging Face training loop on top of MultiModalityCausalLM. An earlier version of this article quoted a specific accuracy gain for medical images; that figure had no source and has been removed.

Once the model is trained, you can refine context detection using a layered analysis approach:

Context Detection Methods

Using Janus Pro 7B’s visual reasoning features (referenced in Section 1), you can apply several detection techniques:

def enhance_context(image, base_description):
    # Classify the scene using visual reasoning
    scene_context = scene_classifier.predict(image)

    # Analyze spatial relationships between objects
    spatial_context = analyze_spatial_relations(objects)

    # Combine all contextual elements
    enhanced_prompt = f"""
    Scene: {scene_context}
    Objects: {spatial_context}
    Base: {base_description}
    Generate detailed description:
    """

    return describe_image(image, enhanced_prompt)

scene_classifier and analyze_spatial_relations are placeholders for your own components; the pattern is simply to put structured context into the prompt before asking for the final description.

Quality Testing

To ensure the improvements are effective, validate the results with the following process:

def evaluate_description(image, generated_desc, reference_desc):
    # Compute the CLIP score
    clip_score = calculate_clip_score(image, generated_desc)

    # Measure linguistic quality
    bleu_score = calculate_bleu(generated_desc, reference_desc)
    meteor_score = calculate_meteor(generated_desc, reference_desc)

    return {
        'clip_score': clip_score,
        'bleu': bleu_score,
        'meteor': meteor_score
    }

Combine these automated metrics (CLIP, BLEU, METEOR) with human review of a sample: automated scores catch missing content and fluency problems, while people catch hallucinated objects and tone problems that reference-based metrics miss.

Implementation Guide

Industry Examples

Typical uses for a self-hosted describer are accessibility (generating alt text and scene descriptions for screen readers), cataloguing (keywording large photo archives so they become searchable) and content moderation triage. An earlier version of this article cited percentage gains for a gaming studio and a photo agency; those were not backed by any source and have been removed.

System Integration Steps

Here’s how to integrate the image description function into platforms like WordPress:

WordPress is PHP, so the model cannot run inside a plugin. The practical pattern is to expose describe_image behind a small HTTP endpoint (the repository includes a FastAPI example in demo/fastapi_app.py) and have a PHP plugin hook add_attachment to call it and store the result as the attachment’s alt text.

Ethics Guidelines

Expand on the quality testing framework from Section 4.3 by incorporating these ethical checks:

def ethical_check(description):
    return EthicsReport(
        detect_bias(description),
        check_cultural_sensitivity(description),
        verify_privacy_compliance(description)
    )

Two practices matter most here: a written policy for what the descriptions may and may not say about people, and a feedback path so users can report biased or inaccurate descriptions and you can add those cases to your evaluation set.

Using automated bias detection and cultural sensitivity checks ensures that image descriptions are fair and inclusive for all users.

Summary and Next Steps

After completing quality controls and ethical reviews (Sections 4-5), it’s time to focus on refining the system for long-term success. In the Janus-Pro tech report Janus-Pro-7B scores 79.2 on MMBench for multimodal understanding, and 0.80 on GenEval and 84.19 on DPG-Bench for image generation; there is no “92.7% accuracy” figure, and an earlier version of this article that quoted one was wrong. Its 384×384 vision encoder (discussed in Section 3.1) is enough for most description tasks but will miss small text and fine detail in large images.

Key Areas to Prioritize

• Domain-specific fine-tuning: This step (outlined in Section 4.1) is what turns generic captions into descriptions that use your domain’s vocabulary.
• Continuous feedback loops: Implementing these (from Section 5.3) helps maintain accuracy over time ^[10].

These methods build on the strategies detailed in Section 5 to ensure consistent performance and adaptability.

Industry Applications

E-commerce platforms are already using this technology to improve product discovery ^[13]. Companies that have adopted these systems report better accessibility compliance and more efficient content management processes.

Ongoing Maintenance

Use monitoring tools based on Section 4.3’s metrics to track description accuracy and trends in user engagement ^[12]. This ensures the system continues to perform well while adhering to the quality standards set during earlier phases.

The rising use of image description systems in accessibility tools ^[12] highlights the need for strong quality control tailored to your specific goals and requirements.

Related Blog Posts

• Top AI Image Generators: December 2024 Performance Review
• The 6 most impressive video models in 2024
• How AI Transforms Real-Time 3D Scene Building
• Comparing Gemini 2.0 Flash Thinking to OpenAI o1