Sending Base64 Images to GPT-4o, Claude & Gemini

All three big vision models — OpenAI's GPT-4o, Anthropic's Claude, and Google's Gemini — accept images the same fundamental way: Base64-encode the bytes and put them in the JSON request. No file uploads, no multipart, just text in a payload. But each provider wants the Base64 wrapped in a differently shaped object, and the traps are subtle — especially the data: URL prefix, which one provider requires and the other two reject.

The single most common error people hit is some flavor of invalid image / could not process image. The reason is almost never the image itself. It's that the payload shape is slightly off. Here's the exact payload each one wants, side by side. If you'd rather skip the script entirely, the Image to Base64 for AI Vision tool encodes an image locally and shows the ready-to-paste block for all three providers.

OpenAI (GPT-4o)

GPT-4o uses a content array of parts. The image is an image_url part, and — this is the trap — the url field takes a full data URL, prefix and all:

import base64
from openai import OpenAI

client = OpenAI()

with open("photo.png", "rb") as f:
    b64 = base64.standard_b64encode(f.read()).decode("utf-8")

resp = client.chat.completions.create(
    model="gpt-4o",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "What's in this image?"},
            {
                "type": "image_url",
                "image_url": {"url": f"data:image/png;base64,{b64}"},
            },
        ],
    }],
)
print(resp.choices[0].message.content)

The literal payload shape:

{ "type": "image_url", "image_url": { "url": "data:image/png;base64,<BASE64>" } }

Note the data:image/png;base64, is part of the value. Send raw Base64 here and it fails.

Anthropic (Claude)

Claude uses an image content block with a source object. Here the MIME type is a separate field (media_type), and the data field wants raw Base64 — no data: prefix:

import base64
import anthropic

client = anthropic.Anthropic()

with open("photo.png", "rb") as f:
    b64 = base64.standard_b64encode(f.read()).decode("utf-8")

msg = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=1024,
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "image",
                "source": {
                    "type": "base64",
                    "media_type": "image/png",
                    "data": b64,   # raw, NO data: prefix
                },
            },
            {"type": "text", "text": "What's in this image?"},
        ],
    }],
)
print(msg.content[0].text)

The literal payload shape:

{ "type": "image", "source": { "type": "base64", "media_type": "image/png", "data": "<BASE64>" } }

If you paste a data:image/png;base64,... string into data here, Claude will reject it — the prefix is not valid Base64. There's a full walkthrough in Claude API Base64 image.

Google (Gemini)

Gemini uses parts with an inline_data object: a mime_type field plus raw Base64 (again, no prefix):

import base64
from google import genai
from google.genai import types

client = genai.Client()

with open("photo.png", "rb") as f:
    image_bytes = f.read()

resp = client.models.generate_content(
    model="gemini-2.5-flash",
    contents=[
        types.Part.from_bytes(data=image_bytes, mime_type="image/png"),
        "What's in this image?",
    ],
)
print(resp.text)

The SDK handles the encoding for you above, but the wire payload it builds is:

{ "inline_data": { "mime_type": "image/png", "data": "<BASE64>" } }

One more Gemini gotcha: the JavaScript SDK camelCases these keys — it's inlineData and mimeType, not inline_data / mime_type. Same structure, different casing. See Gemini API Base64 image for the details.

The three shapes at a glance

Read it top to bottom: OpenAI is the odd one out that wants the whole data: URL; Claude and Gemini both want raw Base64 with the MIME type broken out into its own field.

Why you're getting "invalid image"

When a request fails, it's almost always one of these:

• Stray data: prefix where raw Base64 is expected. Sending data:image/png;base64,iVBOR... to Claude or Gemini's data field. They want just iVBOR....
• Missing data: prefix where OpenAI wants it. Sending raw Base64 to GPT-4o's image_url.url. It wants the full data URL.
• Mismatched MIME type. Declaring image/jpeg for a PNG (or vice versa). The declared type must match the actual bytes. A telltale sign: your Base64 starts with iVBORw0KGgo (PNG) but you labeled it image/jpeg (JPEG bytes start /9j/).
• Whitespace or newlines in the Base64. Some encoders (looking at you, base64 CLI without -w0, and older MIME encoders) wrap output at 76 columns. Strip newlines before sending.
• Unsupported format. Stick to PNG, JPEG, WebP, and non-animated GIF. HEIC, SVG, TIFF, and animated GIFs are frequent rejects.
• Too large. OpenAI and Claude cap around ~5 MB per image; Gemini allows up to ~20 MB of inline data per request, and above that you're expected to use its Files API instead of inline Base64. Remember Base64 inflates size by ~33%, so a 4 MB file is ~5.3 MB on the wire.

If you just need to grab a correct Base64 string and see the exact payload for each provider without wiring up a script, the free Image to Base64 for AI Vision tool encodes an image locally in your browser (nothing gets uploaded) and shows the ready-to-paste block for all three. There are also provider-specific walkthroughs for Claude and Gemini if you only care about one.

TL;DR

• All three vision APIs take images as Base64 in the JSON body — no uploads needed.
• GPT-4o wants a full data: URL in image_url.url.
• Claude wants raw Base64 in source.data, with the type in source.media_type.
• Gemini wants raw Base64 in inline_data.data, with the type in inline_data.mime_type (inlineData/mimeType in the JS SDK).
• Most invalid image errors are a misplaced data: prefix, a mismatched MIME type, whitespace, or a file over the size cap (~5 MB OpenAI/Claude, ~20 MB Gemini inline).

Get the wrapper right and vision "just works" across all three.

Frequently Asked Questions

How do I send an image to GPT-4o, Claude, or Gemini?

Base64-encode the image bytes and put the string in the JSON request body — no file uploads or multipart needed. Each provider wraps the Base64 in a differently shaped object: GPT-4o uses an image_url with a full data URL, Claude uses a source object with raw Base64, and Gemini uses inline_data with raw Base64.

Do I need the data: prefix on the Base64 for vision APIs?

Only for OpenAI. GPT-4o's image_url.url field wants a full data URL like data:image/png;base64,<BASE64>. Claude's source.data and Gemini's inline_data.data both want raw Base64 with no data: prefix — sending the prefix there causes an "invalid image" error.

Why do I keep getting "invalid image" from a vision API?

The most common causes are a misplaced data: prefix (present where raw Base64 is expected, or missing where OpenAI wants it), a mismatched MIME type, whitespace or newlines in the Base64, an unsupported format, or a file over the provider's size cap.

How large can a Base64 image be for GPT-4o, Claude, and Gemini?

OpenAI and Claude cap around 5 MB per image. Gemini allows up to roughly 20 MB of inline data per request, above which you use its Files API instead of inline Base64. Remember Base64 inflates size by about 33%, so a 4 MB file is about 5.3 MB on the wire.

Does the Gemini JavaScript SDK use the same field names as Python?

No. The Gemini JavaScript SDK camelCases the keys: it is inlineData and mimeType, not inline_data and mime_type. The structure is identical — only the casing differs between the Python and JS SDKs.