AI Model Comparisons

Multimodal API calls process image content through the same data handling infrastructure as text — privacy terms, data retention policies, and processing agreements apply equally to image content. For images containing PII (identity documents, medical images, financial statements with account details), enterprise deployments should evaluate whether the cloud provider's data processing agreement covers image-embedded PII under applicable privacy regulations (GDPR, CCPA). Options for sensitive image processing: use providers with enterprise agreements covering image processing (Azure OpenAI, Vertex AI for Gemini provide stronger enterprise data terms than consumer API access); use image preprocessing to redact known sensitive regions before submission; or deploy self-hosted vision models (Llama 3.2 Vision) for sensitive image workloads. The risk model for image-embedded PII is similar to text PII — the question is whether the API provider's data handling meets your regulatory requirements for the specific data type.

For document processing, image quality significantly impacts output accuracy. Recommended specifications: minimum 150 DPI for machine-printed text, 300 DPI for documents with fine print or complex tables; PNG or high-quality JPEG (quality 85+) to avoid compression artefacts that degrade OCR accuracy; ensure consistent lighting and minimal perspective distortion for scanned documents; for multi-page documents, process as individual page images rather than merged PDFs (models handle per-page images more reliably than full-document PDFs in most implementations). Most multimodal APIs automatically resize images to their optimal internal resolution — submitting higher resolution than the model uses internally doesn't improve accuracy but does increase token consumption and API cost. Check each provider's image resizing documentation to understand how submitted image resolution maps to processed image resolution in their model.

For many document processing use cases, multimodal LLMs are replacing traditional OCR + NLP pipelines with a single model inference step. The advantages: multimodal LLMs understand document structure (tables, headers, footnotes) contextually rather than treating documents as flat text; they handle handwriting, unusual fonts, and degraded image quality better than traditional OCR; and they can extract structured data from complex layouts without separate post-processing logic. The limitations: cost per page is higher than traditional OCR for high-volume processing; accuracy on very degraded images or unusual scripts may still lag specialised OCR; and structured output format consistency requires careful prompt engineering. For structured extraction from standard business documents (invoices, receipts, forms) at moderate volume, multimodal LLMs are now the recommended approach. For very high volume (millions of pages/month) or very degraded/unusual documents, hybrid approaches (traditional OCR for text extraction + LLM for semantic understanding) may provide better cost-accuracy balance.

Multimodal evaluation requires domain-specific test sets rather than relying solely on public benchmark scores — public benchmarks test general capability across diverse visual tasks, not performance on your specific document types, image quality characteristics, or extraction requirements. Build an evaluation set of 50–200 representative examples from your actual workload, with ground truth answers for the outputs you care about (extracted fields, classification labels, or analysis content). Evaluate each model candidate on this set, scoring accuracy on the specific outputs required. Include challenging examples (degraded images, unusual layouts, edge cases) alongside typical examples — model performance on the hard cases often differentiates candidates more than average performance. Run cost calculations alongside accuracy — a model that is 5% more accurate but 3× the cost may not be the right choice for your volume and accuracy requirements.

Multimodal API calls have higher latency than equivalent text-only calls due to image preprocessing and the additional tokens required to represent image content. Typical additional latency: 0.5–2 seconds for image preprocessing (encoding, resizing) before the model begins generating; time-to-first-token is typically 1–3 seconds higher for multimodal vs text calls. Total latency for a typical document processing call (one image + prompt → structured JSON output) ranges from 3–8 seconds, compared to 0.5–2 seconds for a pure text extraction call of similar complexity. For latency-sensitive applications, consider: batching multiple documents into a single API call where the context window permits; using faster models (Gemini 2.0 Flash, GPT-4o mini with vision) that trade accuracy for speed; and caching preprocessing results when the same image is submitted multiple times. For most document processing use cases, 3–8 second latency per document is acceptable; for real-time applications (live video analysis, interactive screen agents), optimise specifically for latency using the fastest available models.

Multimodal LLMs can extract information from architectural and engineering drawings with moderate reliability — better than general image understanding tools but not as reliable as domain-specific CAD analysis software. They perform well at: reading text annotations and dimensions labelled on drawings, identifying general layout structure and spatial relationships, extracting component lists and specifications from drawing title blocks, and answering natural language questions about drawing content. They are less reliable at: precise measurement extraction from scaled drawings (they understand dimensions that are explicitly labelled but cannot reliably measure unlabelled distances), interpreting specialised engineering symbols without explicit training on the relevant standard, and handling very complex, dense technical drawings with overlapping elements. For enterprise use cases involving drawing interpretation, evaluate on representative samples from your actual drawing library — performance varies significantly by drawing type, quality, and domain. Augmenting with domain-specific OCR for text extraction before sending to the LLM often improves reliability for text-heavy engineering documents.

Image tokens are priced differently from text tokens across providers, and the effective cost per API call depends on how images are tokenised. GPT-4o prices images based on a base cost per image plus additional tokens for high-resolution tiling (~170 tokens for a 512×512 tile); a typical 1000×1000 document image costs approximately 765–1,105 tokens equivalent at 768-token-tile resolution. Claude 3.5 Sonnet charges approximately 1,600 tokens per image regardless of resolution (for images processed at standard resolution). Gemini 1.5 Pro charges per image at a fixed rate for images up to specific size thresholds. In practice, a typical document processing pipeline (one A4 page image + extraction prompt + structured output) costs $0.005–0.02 per document across leading providers. At scale (1 million documents/month), this represents $5,000–20,000/month in API costs — significant but often justified by the labour savings from manual data entry or traditional OCR + NLP pipeline costs.

All four major multimodal models handle multilingual document content well, reflecting their multilingual training data. GPT-4o, Gemini, and Claude all support 40+ languages in both text and visual inputs — a document with French header text, English body, and a Spanish footnote will be processed correctly with each language section understood in context. The quality of multilingual understanding does vary: European languages (especially French, German, Spanish, Italian, Portuguese) are handled at near-English quality; East Asian languages (Chinese, Japanese, Korean) are well-supported with native character recognition; and less-common languages (Southeast Asian, Central Asian, African languages) may show lower accuracy depending on representation in training data. For enterprises processing multilingual documents, test with representative samples in all target languages — particularly for extracting specific named entities or following language-specific formatting conventions where accuracy may differ from English baseline performance.