Computer Vision & Image Models
Computer vision with AI models has evolved from specialized task-specific models (image classifiers, object detectors) to general-purpose vision-language models that understand images in natural language. Modern vision AI can describe scenes, answer questions about images, extract text (OCR), detect objects, segment regions, and generate images from descriptions.
The landscape spans lightweight embedding models (CLIP, SigLIP) for semantic image search, vision-language models (GPT-4V, Claude 3 Vision, Gemini Pro Vision) for visual question answering, and specialized architectures (YOLO for detection, SAM for segmentation). Choosing the right model depends on task requirements, latency, and cost.
Vision-language models (VLMs) like GPT-4V, Claude 3 Opus, and Gemini Pro Vision accept images alongside text prompts. They can describe image content, answer questions, read text, analyze charts, and reason about visual information. These models use cross-attention between image patches and text tokens.
| 1 | # GPT-4V — image understanding via API |
| 2 | from openai import OpenAI |
| 3 | |
| 4 | client = OpenAI() |
| 5 | |
| 6 | response = client.chat.completions.create( |
| 7 | model="gpt-4-vision-preview", |
| 8 | messages=[ |
| 9 | { |
| 10 | "role": "user", |
| 11 | "content": [ |
| 12 | {"type": "text", "text": "Describe this architecture diagram"}, |
| 13 | { |
| 14 | "type": "image_url", |
| 15 | "image_url": { |
| 16 | "url": "https://example.com/diagram.png", |
| 17 | "detail": "high" |
| 18 | }, |
| 19 | }, |
| 20 | ], |
| 21 | } |
| 22 | ], |
| 23 | max_tokens=500, |
| 24 | ) |
| 25 | |
| 26 | # Claude 3 Vision — Anthropic API |
| 27 | from anthropic import Anthropic |
| 28 | |
| 29 | client = Anthropic() |
| 30 | |
| 31 | response = client.messages.create( |
| 32 | model="claude-3-opus-20240229", |
| 33 | max_tokens=500, |
| 34 | messages=[ |
| 35 | { |
| 36 | "role": "user", |
| 37 | "content": [ |
| 38 | { |
| 39 | "type": "image", |
| 40 | "source": { |
| 41 | "type": "base64", |
| 42 | "media_type": "image/png", |
| 43 | "data": base64_image, |
| 44 | }, |
| 45 | }, |
| 46 | {"type": "text", "text": "What's wrong with this UI mockup?"}, |
| 47 | ], |
| 48 | } |
| 49 | ], |
| 50 | ) |
Image embedding models like CLIP (Contrastive Language-Image Pre-training) project images and text into a shared embedding space. This enables semantic image search by text query, zero-shot classification, and image similarity comparisons without task-specific training.
| 1 | # CLIP — image and text embeddings |
| 2 | import clip |
| 3 | import torch |
| 4 | from PIL import Image |
| 5 | |
| 6 | model, preprocess = clip.load("ViT-L/14") |
| 7 | |
| 8 | image = preprocess(Image.open("photo.jpg")).unsqueeze(0) |
| 9 | text = clip.tokenize(["a dog in a park", "a cat on a couch"]) |
| 10 | |
| 11 | with torch.no_grad(): |
| 12 | image_features = model.encode_image(image) |
| 13 | text_features = model.encode_text(text) |
| 14 | |
| 15 | # Normalize for cosine similarity |
| 16 | image_features /= image_features.norm(dim=-1, keepdim=True) |
| 17 | text_features /= text_features.norm(dim=-1, keepdim=True) |
| 18 | |
| 19 | similarity = (image_features @ text_features.T).squeeze() |
| 20 | # Higher score = better match |
| 21 | |
| 22 | # Zero-shot classification |
| 23 | classes = ["cat", "dog", "bird", "fish"] |
| 24 | text_inputs = clip.tokenize([f"a photo of a {c}" for c in classes]) |
| 25 | |
| 26 | with torch.no_grad(): |
| 27 | logits_per_image, _ = model(image_inputs, text_inputs) |
| 28 | probs = logits_per_image.softmax(dim=-1).cpu().numpy() |
| 29 | |
| 30 | # Using CLIP for image search via vector database |
| 31 | # 1. Pre-compute embeddings for all images |
| 32 | # 2. Store in vector DB (Pinecone, Weaviate, Qdrant) |
| 33 | # 3. Query with text embedding |
| 34 | # 4. Return nearest neighbor images |
Object detection identifies and localizes objects within images. YOLO (You Only Look Once) provides real-time detection for production use cases. SAM (Segment Anything Model) from Meta performs zero-shot segmentation — identifying any object by point, box, or text prompt without task-specific training.
| 1 | # YOLOv8 — object detection |
| 2 | from ultralytics import YOLO |
| 3 | |
| 4 | model = YOLO("yolov8x.pt") |
| 5 | results = model("street_scene.jpg") |
| 6 | |
| 7 | for result in results: |
| 8 | boxes = result.boxes |
| 9 | for box in boxes: |
| 10 | x1, y1, x2, y2 = box.xyxy[0].tolist() |
| 11 | conf = box.conf[0].item() |
| 12 | cls = int(box.cls[0].item()) |
| 13 | label = model.names[cls] |
| 14 | print(f"{label}: {conf:.2f} [{x1:.0f},{y1:.0f},{x2:.0f},{y2:.0f}]") |
| 15 | |
| 16 | # SAM — segment anything |
| 17 | from segment_anything import sam_model_registry, SamPredictor |
| 18 | import numpy as np |
| 19 | |
| 20 | sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h.pth") |
| 21 | predictor = SamPredictor(sam) |
| 22 | predictor.set_image(cv2.imread("image.jpg")) |
| 23 | |
| 24 | # Point-based segmentation |
| 25 | input_point = np.array([[500, 375]]) |
| 26 | input_label = np.array([1]) # 1=foreground, 0=background |
| 27 | masks, scores, _ = predictor.predict( |
| 28 | point_coords=input_point, |
| 29 | point_labels=input_label, |
| 30 | multimask_output=True, |
| 31 | ) |
- Vision-language models (GPT-4V, Claude 3) understand images through natural language
- CLIP embeddings enable semantic image search and zero-shot classification
- YOLO provides real-time object detection; SAM offers zero-shot segmentation
- OCR capabilities are built into most vision-language models and dedicated tools (Tesseract, PaddleOCR)
- Choose between general VLMs and specialized models based on latency, cost, and accuracy requirements
Community
Get help on Slack, Discord or VIP
Stuck on a guide? Join the community and ask.