Multimodal Models and Applications

References

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

1. FFmpeg Documentation

   A media container can hold separate video, audio, subtitle and other streams. A demuxer separates those streams; media decoders turn compressed packets into pixel arrays or audio samples. Stream selection determines which streams continue through processing. Thus, one uploaded file can contain several kinds of signal, and retaining the file does not mean a downstream pipeline consumes every stream.

2. Multimodal Machine Learning: A Survey and Taxonomy

   The authors distinguish linguistic, visual, and vocal information, including sounds and prosody. Multimodal alignment identifies correspondences between components, such as caption phrases and image regions; some components have no counterpart, and several alignments may be possible. Fusion integrates information for prediction. Their taxonomy calls feature combination early fusion and combination of separately produced decisions late fusion. Early fusion can capture feature interactions; late fusion permits different predictors but omits low-level interactions. Multiple modalities can provide complementary information or alternative observations of the same phenomenon.

3. Multi-model, multimodal, and multi-agent innovations in Azure AI

   A native vision-language model can interpret pixels and text together, allowing text extraction, translation, and semantic reasoning to happen within one model rather than through a separately staged OCR-to-LLM pipeline.

4. Multi-model, multimodal, and multi-agent innovations in Azure AI

   For operational monitoring, multimodal outputs should be constrained to a machine-readable schema so image-derived assessments can feed dashboards and alerting systems.

5. Hello GPT-4o

   OpenAI’s May 13, 2024 announcement describes the preceding Voice Mode as three models: speech transcription, text processing and speech synthesis. It explains that the middle model lacked direct access to tone, background sounds and speaker information discarded by transcription. OpenAI presented GPT-4o as jointly trained across text, vision and audio. The announcement separately stated that public availability initially covered text and image inputs with text outputs, despite describing broader model capabilities. This distinguishes an application pipeline, its information bottlenecks, model capabilities and exposed interfaces.

6. Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action

   Unified-IO 2 converts text, images, and audio into embedding sequences that are concatenated for an encoder-decoder Transformer. Image inputs use projected vision-encoder patch features. Audio inputs use a spectrogram, a time-frequency representation of sound, followed by an Audio Spectrogram Transformer and projection. Output images and audio instead use discrete codes, with appropriate decoders converting predicted codes into media. This provides a concrete architecture in which continuous input features and discrete output tokens coexist; a shared processing model does not imply identical raw representations for every modality.

7. Gemma 4 Deep Dive — Cassidy Hardin, Google DeepMind

   Variable aspect ratios require spatial positional information so identical flattened patch indices can represent different image locations.

8. Building Generative Image & Video Models at Scale

   Learned autoencoder latents reduce memory requirements while preserving grid structure useful to the generative network.

9. Video understanding — Gemini API

   The documentation describes processing both audio and visual streams, supplying video by URI and MIME type alongside text, and generating textual descriptions with timestamps. It documents default visual sampling at one frame per second and warns that rapid motion or quick scene changes may lose detail. Media resolution changes the representation budget, with higher resolution increasing token use and latency. Thus, accepting a video file does not establish that every source frame reaches the model.

10. LLaVA-NeXT: What Else Influences Visual Instruction Tuning Beyond Data?

    The authors distinguish image resolution from the number of visual representations passed to the language model. Their AnyRes implementation divides an image into separately encoded grids and can reduce the resulting features through pooling or interpolation. In the reported 2×2-grid experiment, pooling reduced tokens per grid from 729 to 183; the reported ChartQA score fell from 49.2 to 38.0 and DocVQA from 58.8 to 46.9. Other tasks changed less. Separate experiments found that artificially enlarging source images increased training time without consistently improving results.

11. Deep Learning, Chapter 14: Autoencoders

    An autoencoder combines an encoder h=f(x) with a decoder reconstructing x from h. Training penalizes reconstruction error. Restricting code size or imposing regularization makes the model prioritize information instead of merely copying inputs. With a linear decoder and squared-error loss, an undercomplete autoencoder learns the principal subspace associated with PCA. Overcomplete representations can have more coordinates than their inputs; regularization rather than dimensional reduction can constrain them. Even a small bottleneck does not ensure useful features when encoder and decoder capacity permit memorizing training examples.

12. Relations Between Two Sets of Variates

    Harold Hotelling's 1936 paper studies relationships between two sets of measurements by finding paired linear combinations whose correlations successively characterize their shared linear structure. The formulation treats each set in its own coordinates and seeks relationships invariant to nonsingular linear transformations within either set. It is a statistical antecedent of shared-space methods: correspondence is estimated from paired observations rather than inferred from equal dimensionality.

13. “Put-That-There”: Voice and Gesture at the Graphics Interface

    Bolt’s 1980 paper describes an MIT Architecture Machine Group interface combining speech recognition with tracked pointing to manipulate displayed shapes. Speech specifies an operation, while pointing supplies an object or destination. The system associates the cursor position with the occurrence of a referring word. This lets users indicate an object without knowing its name and specify a location without describing it relative to another object. The contribution illustrates complementary information and temporal reference resolution in an implemented system predating modern multimodal neural networks.

14. Joint Audio-Visual Speech Processing for Recognition and Enhancement

    This 2003 IBM research account distinguishes combining acoustic and visual feature vectors before classification from combining separately computed recognition scores. Feature concatenation provides one classifier with both signals, but increases dimensionality and data requirements. Decision fusion can weight the streams according to their reliability. The paper also describes models allowing audio and visual processing states to advance differently within a speech unit before synchronizing at its boundary. These approaches addressed noisy speech and timing differences before transformer-based fusion.

15. Show and Tell: A Neural Image Caption Generator

    Vinyals, Toshev, Bengio, and Erhan's preprint was submitted on November 17, 2014. Their Neural Image Caption system uses a convolutional network to encode an image and an LSTM to generate a description word by word, maximizing the probability of a reference sentence conditioned on the image. This recast image description as conditional sequence generation rather than only retrieving or ranking existing captions.

16. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

    Xu and colleagues first submitted this work on February 10, 2015. Instead of reducing an image to one vector, its encoder retains feature vectors associated with different image locations. At each generated word, the decoder computes a new weighted visual context from those locations, conditioned on its previous state. This made the visual information consulted by caption generation dynamic and spatially indexed.

17. VQA: Visual Question Answering

    Agrawal, Lu, Antol, Mitchell, Zitnick, Batra, and Parikh submitted the original VQA paper on May 3, 2015 and published it at ICCV 2015. It defines the task as answering an open-ended natural-language question about an image and supplies roughly 250,000 images, 760,000 questions, and 10 million human answers. The authors also collected answers without showing the image; humans still exploited question and commonsense priors, including about 21% correctness on their non-yes/no subset.

18. Learning Transferable Visual Models From Natural Language Supervision

    CLIP trains separate image and text encoders so matching image-caption pairs have higher similarity than mismatched pairs. At inference, descriptions of candidate labels can be embedded and compared with an image, enabling classification without training a new label-specific head. The learned similarity space also supports retrieval. This is alignment from paired data, not a symbolic database of visual facts; candidate descriptions and data coverage affect what comparisons mean. Zero-shot transfer evaluates a different adaptation regime from training on the target dataset.

19. Flamingo: a Visual Language Model for Few-Shot Learning

    Flamingo accepts interleaved text and images or videos and produces text. A Perceiver Resampler converts a variable collection of visual features into a fixed-size set of visual tokens. Newly trained cross-attention blocks between frozen language-model blocks let text generation depend on those visual representations. This illustrates intermediate interaction: visual information enters language processing through internal conditioning blocks, rather than through a separately generated caption or a final combination of independent answer scores.

20. Visual Instruction Tuning

    Original LLaVA extracts CLIP ViT-L/14 grid features Z and applies a learned linear projection H=WZ into the language model's embedding width. Visual embeddings join instruction text in the autoregressive context; generated answer tokens follow p(a|image,instruction)=Π_t p(a_t|image,instruction,a_<t). Training first freezes both vision encoder and language model while learning the projection from image-caption pairs. The second stage keeps the vision encoder frozen but updates the projection and language model on visual instruction-response data. The bridge therefore learns both embedding compatibility and how visual evidence should condition language responses.

21. CoCa: Contrastive Captioners are Image-Text Foundation Models

    CoCa combines a contrastive objective on separate image and text embeddings with a captioning objective that predicts text conditioned on an image. Its initial text-decoder layers omit image cross-attention, producing unimodal text representations for comparison. Later layers cross-attend to visual features and produce multimodal representations for caption prediction. The objectives therefore reward different operations: distinguishing paired examples and predicting a paired description token by token. One model can train both without making retrieval and conditional generation equivalent.

22. Multimodal Deep Learning

    Ngiam and colleagues’ ICML 2011 paper separates having multiple modalities during representation learning from having them during supervised training and testing. Their deep autoencoder learns to reconstruct both audio and visual features from a single supplied modality, encouraging correspondence beyond independently learned features. Paired audio-video training improved visual speech representations in their experiments even when subsequent classification used video alone. Their speech experiments also show that adding visual features can hurt clean-audio classification, while a combination of learned bimodal and acoustic features helps under the tested noisy conditions.

23. 120k players in a week: Lessons from the first viral CLIP app: Joseph Nelson

    Paint.wtf uses CLIP (Contrastive Language-Image Pre-Training) to rank drawings by cosine similarity between prompt and image embeddings.

24. Vision AI in 2025 — Peter Robicheaux, Roboflow

    Image-caption matching can leave models insensitive to details that captions do not distinguish.

25. Audio-Visual Synchronisation in the Wild

    This 2021 study distinguishes related audio and video from temporally synchronized events. A short bark can provide a synchronization cue lasting less than one second, whereas uniform engine noise may provide no identifiable alignment cue. Training first uses mismatched recordings as easier negatives, then different intervals from the same recording as harder negatives. The general-video evaluation samples at an effective five frames per second and accepts offsets within 0.2 seconds; its speech protocol instead uses 25 frames per second and a 0.04-second tolerance.

26. CLIPScore: A Reference-free Evaluation Metric for Image Captioning

    CLIPScore computes 2.5×max(cos(v,c),0), where v and c are CLIP image and candidate-caption embeddings; corpus scores average pairs. Its original task evaluates captions for supplied images without requiring reference captions, rather than certifying images generated from instructions. The paper reports strong agreement with human judgments in several caption datasets but weaker performance when interpretation requires context. In news captioning, CLIPScore matches pairwise judgments with 65% accuracy versus 93% for METEOR; it also fails to reliably rank engaging nonliteral captions. Inference: embedding compatibility can support an alignment signal, but these evaluations do not establish compliance with every count, relation, edit instruction, or protected region.

27. MultiBench: Multiscale Benchmarks for Multimodal Representation Learning

    MultiBench evaluates task performance, computational requirements and robustness separately. Its robustness tests distinguish modality-specific corruption from correlated failures, such as an interval missing across synchronized streams. Its affective-computing protocols split source videos before assigning their shorter segments to training, validation or testing. The implementation includes preprocessing, splits and evaluation procedures, making those choices part of the comparison. The paper also illustrates varying fusion terminology: its late-fusion baseline concatenates separately learned features before classification, rather than combining final decisions.

28. Does My Multimodal Model Learn Cross-Modal Interactions? It’s Harder to Tell Than You Might Think!

    Hessel and Lee distinguish additive predictions, where text and image each contribute a separately computed score, from interactions whose contribution depends on the pairing. Their EMAP diagnostic approximates a trained classifier’s predictions with an additive function. In several tested image-text classification tasks, this removed little predictive performance even when the original expressive model outperformed simpler baselines. Architectural capacity and a higher combined score therefore did not by themselves establish useful cross-modal feature interactions.

29. Attention Is All You Need

    Attention transforms vectors into queries Q, keys K, and values V through learned projections. Each query–key dot product measures compatibility. Dividing by √dk, where dk is key dimension, controls score scale. Row-wise softmax exponentiates and normalizes scores into nonnegative weights summing to one. The output softmax(QKᵀ/√dk)V is a weighted mixture of value vectors, one per query. In self-attention, all three projections originate from the same representation. In cross-attention, queries originate from the representation being updated, while keys and values originate from a separate conditioning representation. Their sequence lengths can differ. Multiple heads learn separate projections, then concatenate and project their outputs.

30. RFC 8834: Media Transport and Use of RTP in WebRTC

    A MediaStreamTrack represents an individual media stream, such as camera or microphone output. One source can feed several tracks with different encoding constraints. The specification distinguishes stream identity from synchronization: received tracks with different CNAME synchronization identifiers have no defined synchronization. Forwarding a received track into another connection requires relating incoming and outgoing timebases and monitoring clock drift. Consequently, collecting two streams or assigning timestamps does not by itself establish their temporal correspondence.

31. Media Fragments URI 1.0 (basic)

    The W3C specification defines temporal selections using a start and end time and spatial selections using rectangular coordinates. Normal Play Time uses seconds or a clock-like notation; the notation itself does not promise frame accuracy. Spatial selectors identify x, y, width and height using pixels or percentages, with the origin at the upper-left corner. Coordinate interpretation depends on the resource’s dimensions and display characteristics. These conventions provide a concrete way to identify an interval or region instead of attaching a claim only to an entire media file.

32. Learning to Answer Questions in Dynamic Audio-Visual Scenarios

    MUSIC-AVQA studies questions about visible instruments, sounds, and their relationships. Its introductory example asks which clarinet sounds first, requiring association of sound with a visible source and ordering in time. The proposed network separates spatial sound-source association from question-dependent temporal selection, then fuses their representations to predict an answer. This supplies a published example of cross-modal reasoning whose target is a relationship between observations, rather than two independent recognitions.

33. TVQA+: Spatio-Temporal Grounding for Video Question Answering

    TVQA+ evaluates answer correctness, temporal localization, and object grounding separately. Temporal overlap measures intersection divided by union of predicted and reference intervals. Its Answer-Span joint Accuracy requires both a correct answer and interval overlap of at least 0.5. Object grounding is evaluated against annotated regions. The paper compares combined video-subtitle models with video-only and subtitle-only variants and a baseline that sums their separately predicted answer scores. These distinguish answer success, evidence localization, and fusion design.

34. OCR-free Document Understanding Transformer

    Donut maps a document image through an encoder and generates a token sequence that can be converted into structured information. The paper's receipt illustration produces item records without consuming an external OCR result. The authors motivate this route partly through error propagation from separately supplied OCR. OCR-free therefore describes the absence of a separate OCR dependency, while the system still learns to read document content.

35. VoiceVision RAG - Integrating Visual Document Intelligence with Voice Response

    Retrieving original multimodal content calls for a multimodal answer model; retrieving only textual summaries permits a text-only answer model.

36. Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering

    VQA v2’s construction holds a question fixed while seeking a similar image that changes its answer. Annotators must preserve the question’s premise: a question about a woman’s action still needs a visible woman. A separate annotation round collects answers for the replacement image. This makes the image necessary to distinguish the paired cases and illustrates why substitutions require reassessing expected answers. The authors also show why balancing overall answer frequencies alone does not remove correlations between question wording and answers.

37. Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality

    Each Winoground case contains two images and two captions whose words or morphemes are the same but ordered differently. Expert annotators select the intended image-caption pairings over the crossed alternatives. Cases vary object assignments, relations, and other compositional distinctions. Correct matching therefore requires more than detecting the presence of the same objects or words: the relationship expressed by their arrangement must agree with the image.

38. Shifting the Baseline: Single Modality Performance on Visual Navigation & QA

    The authors construct input ablations by replacing withheld modalities with zero vectors while preserving model architecture and parameter count, and train and evaluate navigation and question-answering subtasks separately. Single-modality models outperform several originally reported baselines; performance varies substantially by dataset and training regime. This shows why comparison with random or majority-class prediction can overstate the evidence for multimodal benefit.

39. When Audio and Text Disagree: Benchmarking Text Bias in Large Audio-Language Models under Cross-Modal Inconsistencies

    MCR-BENCH holds audio samples fixed while adding faithful, contradictory, or unrelated textual descriptions, with a no-description condition as reference. Tasks include sound question answering, speech-emotion recognition, and vocal-sound classification. The tested models often change correct audio-based answers when supplied contradictory text, despite instructions warning that descriptions may be inaccurate. This operationalizes modality dominance through controlled input changes rather than inferring it from fluent explanations or architecture alone.

40. Generating content — Gemini API

    The published video example uploads a file, waits for its processing state to become ACTIVE, and then supplies its URI and MIME type with a text part. Streaming generation returns successive response instances. The response schema separately exposes candidates, prompt feedback, candidate finish reasons, model version, and response identity. These distinguish uploaded media readiness, generated content, and generation status rather than treating a returned text fragment as the entire operation.

41. Web Annotation Data Model

    The Web Annotation model separates an annotation’s content from its target. A SpecificResource connects a source asset to additional constraints: a Selector identifies the relevant segment, while a State identifies the intended version or representation. The specification also distinguishes canonical identity from the location where a copy was obtained. This supports attaching a comment or correction to a particular media region while retaining which source representation the selection refers to.

42. Citation Needed: Provenance for LLM-Built Knowledge Graphs

    Lineage must survive graph mutation: entity merges retain both source sets, and invalidation records the new evidence responsible for the change.

43. ModDrop: adaptive multi-modal gesture recognition

    ModDrop randomly withholds modality channels during training. The authors compare models under missing channels and noise and report improved robustness in their gesture-recognition experiments. They also identify a separate correspondence limitation: spoken phrases can precede or follow gestures and occupy shorter intervals, while gesture boundaries are annotated from video. Audio-only localization can therefore disagree with visual boundaries even when speech and movement express the same meaning. Their analysis recommends examining individual channels when diagnosing fusion bottlenecks.

44. Google SRE: Handling Overload

    Admission controls and per-customer quotas limit resource consumption so one workload does not exhaust shared capacity. Resource usage can be a better capacity signal than requests per second because requests vary in cost. Graceful degradation reduces work by returning less complete results or using cheaper, potentially stale cached data. Client-side throttling can prevent rejected requests from consuming backend resources. Under extreme overload, even degraded computation may be impossible and explicit errors are necessary.

45. Hearing lips and seeing voices

    McGurk and MacDonald reported on December 23, 1976 that adults shown lip movements for one syllable synchronized with audio for another could report a third percept: dubbed audio [ba] with visible [ga] was commonly heard as [da]. Participants reported the syllables accurately when hearing the soundtrack alone or viewing untreated matching audiovisual speech. The experiment demonstrates that conflicting modalities can alter the resulting interpretation rather than merely contribute independent labels.

46. OWASP LLM01:2025 Prompt Injection

    Indirect injection occurs when external content is interpreted as instructions that redirect model behavior. OWASP also describes cross-modal injection and recommends separating untrusted content, deterministic output validation, least privilege, code-held credentials, and human approval for privileged actions. Voice application inference: an imported recording or background playback can contain spoken instructions that enter through transcription or a multimodal model. Capturing or recognizing those words must not confer action authority. Tool execution should validate identity, permission, action, and arguments outside the model; high-risk approval should come through a trusted interaction rather than from quoted media.

47. Serving Voice AI at Scale — Arjun Desai (Cartesia) & Rohit Talluri (AWS)

    Reduce time to first synthesized audio to leave more of the end-to-end response budget for the language model.

48. Improving Pinterest Search Relevance Using Large Language Models

    Pinterest’s relevance teacher processes a search query together with textual descriptions of a Pin and predicts five relevance levels. Its inputs include BLIP-generated image captions, creator descriptions, linked-page information and other text features. Images therefore influence this component through generated descriptions rather than direct pixel access. Human assessors judged 88% of captions both relevant and high quality in a sample of 10,000 images. The paper separately describes a smaller student for online serving because the teacher’s latency and cost were difficult to scale.

49. EMMA: End-to-End Multimodal Model for Autonomous Driving

    Waymo's EMMA paper, posted October 30, 2024, describes one unified application architecture: surround-view camera video, a routing command, and text-encoded vehicle history condition a model that emits text-encoded future waypoints and other driving outputs. Shared training covers planning, detection, road graphs, and scene tasks. The authors explicitly report boundaries: few input frames, no LiDAR or radar, weaker long-range depth behavior in one comparison, expensive computation, and a need for costly closed-loop simulation.

50. Waymo's EMMA: Teaching Cars to Think - Jyh-Jing Hwang, Waymo

    The presented benchmark results use replay-based open-loop evaluation; the speaker treats simulation and road testing as additional requirements for validating the prototype.

51. The Build-Operate Divide: Bridging Product Vision and AI Operational Reality

    Prioritize human-in-the-loop review at consequential decision points in high-risk, high-trust workflows.

52. Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics

    FRANK distinguishes summary errors involving predicates, entities, circumstances, references between sentences, temporal or causal connections, and unsupported information. Circumstance errors include altered time, location or modality. Its published examples change an approval into a rejection, substitute an entity and change an event's date. The authors annotate generated news summaries using this taxonomy and evaluate factuality metrics against those annotations. This supports inspecting the specific relationships a summary preserves, rather than judging fidelity from fluent wording.

53. Ensure AI Agents Work: Evaluation Frameworks for Scaling Success

    Voice agents need audio-specific evaluations in addition to the evaluations used for their text and control flow.

54. Ensure AI Agents Work: Evaluation Frameworks for Scaling Success

    Attach evaluations at multiple levels of an execution trace so a failed result can be localized to routing, arguments, or skill execution.

55. How Evals and Prompts Shape Agent Behavior — Preetika Bhateja & Daniel Bump, YouTube Ads

    Start with a few core tasks and test prohibited behavior alongside task completion.

56. Evaling Video Slop

    Prompt alignment and frame consistency do not establish that a video tells the intended story; temporal quality needs explicit evaluation axes.

57. Selective Classification for Deep Neural Networks

    Selective prediction combines a predictor with a selection function that either accepts its prediction or abstains. Coverage is the probability of accepting a case; selective risk is expected loss conditional on acceptance. A risk-coverage curve shows how error among accepted cases changes as coverage changes. The paper selects confidence thresholds using labeled examples and derives risk bounds under independent, identically distributed sampling. The ranking score used for selection need not itself be a calibrated probability.

58. Best practices for the human evaluation of automatically generated text

    The authors recommend defining separate assessment criteria, reporting participant selection and demographics, and measuring agreement among multiple annotators. Practice examples spanning good and bad outputs can help reviewers establish a common rating scale. Random ordering or counterbalancing reduces presentation-order effects; fatigue and practice can otherwise confound system comparisons. Removing participants or missing responses can introduce selection bias. The paper distinguishes exploratory analysis from confirmatory hypothesis testing and explains that testing many hypotheses increases the chance of false-positive conclusions.

59. On Calibration of Modern Neural Networks

    Calibration asks whether predictions assigned confidence p are correct about proportion p of the time. Reliability diagrams compare observed accuracy with confidence in bins; expected calibration error averages absolute bin discrepancies weighted by bin population. Calibration differs from prediction accuracy. Selective risk instead measures errors among accepted predictions, and coverage is the accepted fraction. A threshold can alter risk and coverage without demonstrating calibrated probabilities.

60. Ship Real Agents: Hands-On Evals for Agentic Applications

    Inspect actual traces and categorize failures before defining evaluation criteria.

61. Evaling Video Slop

    The described 'Judge Judy' stage uses human annotations to calibrate AI judges and produce training data for subsequent evaluator versions.

62. Robust Speech Recognition via Large-Scale Weak Supervision

    Whisper maps log-Mel audio features to text and task tokens using an encoder-decoder Transformer. Transcription, translation, language identification, and timestamps are distinct outputs. Long-form decoding uses beam search scored by log probability and temperature fallback triggered by low average token log probability or excessive repetition. The authors found the no-speech token probability insufficient alone; combining it with an average-log-probability threshold improved silence handling. These are decoding heuristics, not a demonstrated calibration of transcript correctness. Evaluation across languages and datasets exposes variation, while normalization changes WER without necessarily changing perceived meaning.