Generative Media

References

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

1. The State of Generative Media Today

   The speaker distinguishes falling marginal production cost from the continuing need for storytelling and creativity.

2. High-Resolution Image Synthesis with Latent Diffusion Models

   Latent diffusion first compresses an image with an autoencoder, performs denoising in the smaller latent representation, and decodes the result back to pixels. Compression trades representational detail against computation. Conditioning can enter through concatenation or cross-attention, allowing inputs such as text, layouts, or images to guide generation. The paper studies synthesis and editing tasks including inpainting and super-resolution. These are different constraints on generation, rather than guarantees that every requested detail will be preserved.

3. Adobe: Create a concrete brief to share out in the discovery phase

   A creative brief gathers the project's goals, target audience and success measures so the creator and client can agree on the work. Adobe's tutorial connects scope, deliverables and timeline to budgeting, and recommends agreeing on the number of ideas and revision rounds. This supports defining the intended deliverable and exploration boundaries before generating candidates.

4. C2PA and Content Credentials Explainer, version 2.2

   Content Credentials associate media with signed provenance assertions and a binding to the asset. Validation can reveal whether recorded assertions or bound content were altered and identify the signer used in the trust decision. The C2PA explainer explicitly separates provenance from truth: a valid history does not establish that a depicted event occurred or a claim is factual. Metadata can be removed or history can be incomplete, so absence of a credential is not proof that media is fabricated.

5. DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation

   DreamBooth adapts a pretrained image generator using a few subject images and a unique identifier paired with a class noun. The identifier can then request that subject in new settings, poses and artistic treatments. Subject fidelity means retaining distinguishing details; prompt fidelity concerns the requested variation, and the paper evaluates them separately. Prior-preservation training mitigates reduced diversity and changes to the meaning of the class noun. Documented failures include context changing a subject's appearance, copying reference settings, hallucinated subject features and reduced variation for some subjects.

6. The Digital Computer as a Musical Instrument

   M. V. Mathews's 1963 Bell Laboratories account separates composing music from computing its sound. A composer defines programmable instruments, then supplies a score specifying notes and their parameters. Connected signal-processing units compute the waveform samples; the published example combines oscillators, a random-number generator and a summing unit. This avoids specifying thousands of samples individually while retaining control over sound. More elaborate instruments require more computation and more parameters. The system records computed samples for subsequent playback rather than requiring computation at playback speed.

7. Texture Synthesis by Non-parametric Sampling

   Efros and Leung's 1999 method grows a texture one pixel at a time. For each missing pixel, it searches a supplied example image for neighborhoods resembling the already synthesized surroundings, then samples a compatible value. The goal is to reproduce local visual structure without simply tiling the example. The paper demonstrates texture generation and constrained hole filling, providing an earlier example of synthesis guided by existing image content.

8. Generative Adversarial Nets

   A GAN learns a generator that maps sampled latent noise to data and a discriminator that distinguishes generated from training samples. Alternating updates make the generator improve against the discriminator rather than explicitly predicting every pixel likelihood. The paper formulates a two-player minimax game; its ideal optimum matches the data distribution under strong capacity and optimization assumptions. Generation uses the trained generator, while the discriminator supplies a training signal. This explains both the appeal of direct sampling and the difficulty of balancing two learners.

9. WaveNet: A Generative Model for Raw Audio

   DeepMind's September 8, 2016 account contrasts concatenative speech, assembled from recorded fragments, with parametric systems that drive vocoders from model outputs. WaveNet instead predicts a distribution for the next waveform sample, samples a value and feeds it back for the next prediction. The authors connect this design to their earlier pixel-based image generators. They report improved listener-rated naturalness against their tested speech baselines, while identifying sample-by-sample generation as computationally expensive.

10. Denoising Diffusion Probabilistic Models

    Training draws an example image x0, a uniform timestep t, and Gaussian noise ε. A fixed schedule defines αt=1−βt and ᾱt=∏s≤t αs, giving xt=√ᾱt x0+√(1−ᾱt)ε. A neural network is a parameterized numerical function: its adjustable weights θ map (xt,t) to predicted noise εθ. The simplified loss L=||ε−εθ(xt,t)||² measures prediction error. Backpropagation computes derivatives of L with respect to weights; a gradient update θ←θ−η∇θL moves weights toward lower sampled error, with learning rate η controlling update size. Training repeats across examples and noise levels. Sampling freezes θ, starts with Gaussian noise, and repeatedly computes reverse transitions, adding scheduled randomness except at the final step. It changes the sample, not the learned weights.

11. Video Diffusion Models: Space-time factorized U-Net

    The model jointly denoises a fixed block of frames. Space-only 1×3×3 convolutions and spatial attention operate within each frame; temporal attention then combines representations across the frame axis at each spatial location, with relative position information. Alternating these operations permits spatial and temporal information exchange; it does not certify object identity or physically correct motion. For longer sequences, reconstruction-guided sampling conditions new blocks on previously generated frames, or fills intermediate frames. This differs from attending to an entire unlimited video at once.

12. Pixel Recurrent Neural Networks

    The 2016 paper models an image through successive conditional predictions of pixel values and their color channels. PixelCNN uses masked convolutions to prevent predictions from accessing later values. Known training images allow predictions at many positions to be computed together, but generation remains sequential because each sampled value becomes context for subsequent predictions. The original PixelCNN has a bounded receptive field, unlike the paper's recurrent alternatives.

13. Building Generative Image & Video Models at Scale

    Learned autoencoder latents reduce memory requirements while preserving grid structure useful to the generative network.

14. Taming Transformers for High-Resolution Image Synthesis

    This system first compresses an image into discrete codebook indices using a convolutional VQGAN, then trains a transformer to predict each index conditioned on earlier indices. The joint probability factors into next-index conditional probabilities. Sampling those tokens and decoding the codebook produces an image, while class or spatial conditioning can be represented as additional context. Learned compression shortens the sequence compared with pixel-level modeling, but introduces a reconstruction bottleneck. Autoregression is a generation objective, not a synonym for text generation.

15. High Fidelity Neural Audio Compression

    EnCodec's learned encoder uses temporal convolutions and downsampling to turn waveform samples into a shorter sequence of vectors. Vector quantization replaces each vector with its nearest learned codebook entry; residual quantization repeatedly encodes the remaining error. The decoder reconstructs waveform samples from the summed quantized vectors. Before entropy coding, bitrate is approximately frames/second × codebooks/frame × log2(codebook size), excluding overhead. More codebooks provide more information and generally improve reconstruction at greater bitrate. Training combines waveform, spectral, adversarial, and commitment losses; listening tests evaluate perceptual quality. Compared with spatial image latents, these tokens represent time, with explicit bitrate and streaming-latency constraints; this comparison is an architectural inference, not a claim that all audio or image codecs behave alike.

16. ComfyUI Workshop with ComfyAnonymous and Jedrick Kosinski

    The Stable Diffusion example samples a compressed spatial representation instead of full-resolution pixels.

17. The Hidden Life of Embeddings

    The demonstrated custom T5 denoising autoencoder encodes text into a pooled embedding and decodes that embedding back into approximately reconstructed text.

18. The Hidden Life of Embeddings

    A trained linear adapter let the custom decoder recover approximate text, including some proper nouns and structure, from an OpenAI embedding without source text at decoding time.

19. The Hidden Life of Embeddings

    The image prototype combined CLIP embeddings with an unCLIP diffusion decoder to generate a gradual transition between a photograph and a cartoon avatar.

20. Auto-Encoding Variational Bayes

    A variational autoencoder learns an approximate posterior over latent variables for each observation and a decoder that maps latent samples back to a distribution over data. Training optimizes a lower bound on data log likelihood, balancing reconstruction with a KL term against the latent prior. Reparameterization expresses a latent sample as a differentiable transform of independent noise, allowing gradients through stochastic sampling. After training, sampling the prior and decoding generates new outputs; encoding a reference is a different operation.

21. Deep Unsupervised Learning using Nonequilibrium Thermodynamics

    Sohl-Dickstein and colleagues framed diffusion probabilistic models in 2015 as a way to combine flexible distributions with tractable learning, sampling and probability evaluation. A forward process gradually destroys data structure; a learned reverse chain constructs samples from a simple distribution. Learning many small transitions avoids having to describe the entire complex distribution with one difficult-to-normalize function. Experiments included handwritten digits, natural images and textures.

22. Extracting Training Data from Diffusion Models

    The authors generate many candidates, identify suspiciously repeated outputs, and compare them with training images. Their experiments extract more than a thousand training examples across studied diffusion systems. For Stable Diffusion v1.4, automated matching finds 96 examples, and manual inspection identifies another 13 near-copies, totaling 109. The paper defines extraction through closeness to particular training examples, rather than merely depicting the same person or category. Its Obama example is explicitly excluded because generated and nearest training images differ substantially. Thus reproducing identifiable training compositions is evidence of memorization; generating a new depiction sharing learned concepts is not, by itself, such evidence.

23. Denoising Diffusion Implicit Models

    Jiaming Song, Chenlin Meng and Stefano Ermon introduced DDIM to reduce diffusion sampling cost without retraining the denoising network. Their construction changes dependencies between noisy states while preserving the marginal corruption distributions used for training. Sampling can therefore reuse a DDPM-trained predictor with a different update rule and a shorter sequence of noise levels. Setting the update's added-noise term to zero produces a deterministic trajectory from the initial noise. Experiments demonstrate a computation–quality tradeoff, separating the learned predictor from the procedure that repeatedly applies it.

24. Generative Modeling by Estimating Gradients of the Data Distribution

    Song and Ermon's 2019 approach learns a score: a vector indicating how log probability changes around a candidate. Direct estimation is difficult where examples occupy narrow regions or provide little evidence. Their method adds several levels of Gaussian noise and trains one network to estimate the corresponding scores. Sampling combines estimated directions with random perturbations, progressing from larger to smaller noise levels. This provides a distinct route to generation through learned local changes.

25. Score-Based Generative Modeling through Stochastic Differential Equations

    Yang Song and colleagues unified earlier noise-conditioned score models and diffusion probabilistic models through continuous-time noise processes. The reverse process uses a learned score: the direction in which the noisy data's log probability increases. Their framework supports both stochastic sampling and a deterministic probability-flow ordinary differential equation. With exact scores and continuous integration, these processes share time-indexed marginal distributions, although their individual trajectories differ. This explains how a model trained through noisy examples can support more than one sampling procedure.

26. Flow Matching for Generative Modeling

    Flow matching trains a vector field that moves a noise distribution toward a data distribution along a chosen probability path. Conditional per-example paths make the regression objective tractable without simulating the full flow during training. Generation then follows the learned field using a numerical solver. The framework includes diffusion-related paths but also supports other paths, so generative modeling need not be explained solely as reversing one fixed noise process. Training objective, path choice, and inference solver are separate parts of the design.

27. DeepCache: Accelerating Diffusion Models for Free

    DeepCache observes that adjacent denoising steps often have similar high-level U-Net features. A refresh step executes the full network and caches an intermediate deep feature. Subsequent steps recompute shallow features from the current noisy input, combine them with cached deep features, and still update the sample through the sampler. Periodic refresh limits staleness. This reduces work per denoising step; reducing steps removes sample updates, while distillation trains a model to approximate a more expensive generation process. DeepCache requires no retraining. Its ablations show that increasing cache intervals can change clothing colors or shapes and eventually causes substantial degradation.

28. You Might Not Need 50 Diffusion Steps

    Distillation still requires training compute and expertise, with resource needs depending on model size; it does not inherently require GB200 hardware.

29. IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models

    IP-Adapter turns a supplied reference image into features using a frozen image encoder and a learned projection. Separate cross-attention paths bring image features and text features into the diffusion network, and their outputs are combined. The adapter is trained while the original diffusion model remains frozen. Once trained, it accepts new image prompts without separately adapting the model to each pictured subject. Its image representation carries both content and style rather than exposing an identity-only control.

30. Adding Conditional Control to Text-to-Image Diffusion Models

    ControlNet accepts spatial maps such as depth, segmentation, or rendered pose keypoints. A small convolutional encoder maps the condition image into features aligned with the diffusion latent grid. It freezes the pretrained backbone and trains a copied branch, connecting its outputs through initially zero-valued convolutions into the backbone's skip connections and middle block. The denoising loss trains this branch to use conditions without initially disturbing the pretrained network. A new pose or depth map is an inference input; personalization instead adapts parameters or embeddings to examples of a particular subject. ControlNet itself also requires training, so conditioning and parameter adaptation are not mutually exclusive categories.

31. Classifier-Free Diffusion Guidance

    Classifier-free guidance trains conditional and unconditional denoising behavior by sometimes dropping the condition. During sampling it combines their predictions so the conditional direction is emphasized without a separate classifier. The guidance scale changes the sampled distribution rather than merely making the same sample more accurate. The paper demonstrates a fidelity/diversity tradeoff: stronger guidance can concentrate outputs in narrower high-scoring regions. Conditioning therefore influences a probabilistic generator; it does not enforce a hard scene specification.

32. ComfyUI Workshop with ComfyAnonymous and Jedrick Kosinski

    Classifier-free guidance, or CFG, uses positive and negative conditioning to steer sampling toward desired content and away from undesired content.

33. SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations

    SDEdit perturbs a guide image at a chosen intermediate noise level, then runs the pretrained reverse process to generate an image. In its variance-exploding formulation, the starting point is guide+σ(t0)ε, with Gaussian ε. Increasing t0 adds more noise and removes more guide information, allowing greater regeneration. Smaller t0 retains more input structure but can retain unrealistic strokes or artifacts. Figure 3 measures a realism–faithfulness tradeoff using distributional KID and squared distance to the guide. Unlike inversion, this procedure does not search for a noise state whose forward generation reconstructs the reference; it deliberately corrupts and resynthesizes the guide.

34. Diffusers Reproducibility

    A seed initializes a pseudorandom generator state; it reproduces random draws only when the generator implementation, state, and draw sequence agree. Generator objects are consumed, so passing the same already-used object does not restart its sequence. Diffusers recommends CPU generators for more comparable noise across devices and documents deterministic-algorithm settings, CUDA workspace configuration, and disabling convolution benchmarking. Identical seeds still do not guarantee identical results. Engineering inference: record model weights, prompt processing, scheduler, timesteps, guidance, dimensions, precision, software versions, hardware, and execution options as well as the seed. Changing the prompt or model changes the transformation applied to noise; the seed therefore carries no semantic-preservation guarantee.

35. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding

    Imagen reports that stronger classifier-free guidance can improve image–text alignment while producing oversaturated, unnatural images. In its pixel-space formulation, guided clean-image predictions can leave the numerical range used during training, and repeated sampling compounds this mismatch. Dynamic thresholding clips and rescales predictions using a percentile of their absolute values. The authors report better results than simple fixed clipping at high guidance weights.

36. Scalable Diffusion Models with Transformers

    DiT replaces a diffusion model's U-Net with a transformer that processes patches of a noisy image latent. Its output is rearranged into noise and reverse-process covariance predictions; sampling still repeatedly updates the noisy latent before a VAE decodes it. The transformer is therefore the prediction network, not an instruction to generate media autoregressively. Smaller latent patches create longer token sequences and increase computation without a corresponding increase in parameter count.

37. Cascaded Diffusion Models for High Fidelity Image Generation

    A cascade samples a low-resolution image z, then a super-resolution diffusion model samples a higher-resolution image conditioned on z; additional stages repeat this process. Formally, p(x)=∫p(x|z)p(z)dz. Later stages learn plausible detail rather than recover uniquely determined original pixels. Training super-resolution on clean downsampled examples creates a mismatch when inference supplies imperfect generated images, allowing errors to compound. The paper's conditioning augmentation perturbs training conditions with Gaussian noise or blur to improve robustness, with experiments supporting better cascade quality. Stronger augmentation is not monotonically better: weakening the condition also removes useful information.

38. Null-text Inversion for Editing Real Images using Guided Diffusion Models

    Diffusion inversion seeks a starting noise representation and compatible trajectory that regenerate a reference under a specified model, prompt, and sampler. DDIM inversion approximately traverses the denoising trajectory backward; finite-step approximation errors accumulate, and strong classifier-free guidance amplifies deviations. Null-text inversion first computes a pivot trajectory, then optimizes a separate unconditional text embedding at each timestep to reduce reconstruction error along it, keeping model weights and the conditional embedding fixed. The paper demonstrates that direct DDIM inversion alone is inadequate for faithful guided reconstruction. Deterministic sampling means the same complete inputs follow the same computation; it does not prove that the approximate inverse recovers those inputs.

39. InstructPix2Pix: Learning to Follow Image Editing Instructions

    Tim Brooks, Aleksander Holynski and Alexei Efros train a diffusion editor using generated examples of images paired with editing instructions and edited results. At inference, it accepts an existing image and an instruction describing a change, without per-image fine-tuning or inversion. Separate image and instruction guidance strengths trade preservation against requested modification. The paper demonstrates edits to real images but also documents excessive changes, difficulty isolating objects and failures involving spatial rearrangement. Its examples show that a local semantic edit can require contextual changes, such as modifying reflections when replacing a skyline.

40. RePaint: Inpainting using Denoising Diffusion Probabilistic Models

    RePaint uses a binary mask m, with one marking known pixels. During each reverse step it combines a correctly noised version of the known image with the model's generated unknown region: x=m⊙xknown+(1−m)⊙xgenerated, where ⊙ means elementwise multiplication. Repeated forward-noising and reverse-denoising revisits help reconcile generated content with known context. This modifies sampling without training a separate mask-specific model. An enlarged canvas can use the same editable/protected construction: embed the source in a larger array and mark surrounding pixels unknown. This extension is an application of the mask formulation, not evidence that every outpainting system preserves source pixels.

41. Photoshop: Add Layer Masks

    Photoshop layer masks selectively hide or reveal an entire layer or selected regions; masks can be moved or duplicated separately between layers. An application design follows from this separation: retain the source on its own layer and place generated replacements on another, with a mask controlling visibility instead of overwriting source storage. This preserves access to source content independently of which generated candidate is displayed. It does not establish equality of the final rendered composite.

42. Photoshop: Export Settings and Export Location Preferences

    Photoshop export can change dimensions, resampling, canvas placement, color space, transparency, and encoding. JPEG quality trades detail against file size; PNG options change channel/palette representation; resampling interpolates or replicates pixels; sRGB conversion changes color representation. Engineering inference: a protected-pixel contract must specify source decoding, channel order, bit depth, alpha handling, color conversion, coordinates, compositing, and output decoding. Keep protected source pixels unchanged through rendering, avoid lossy encoding and unintended resampling or quantization, then compare the decoded output's protected region against the specified decoded source array. Masking alone does not verify this contract.

43. Magic Editor Under the Hood: Weaving Generative AI into a Billion-User App

    Magic Eraser combines distractor detection, segmentation, inpainting, and device rendering rather than relying on a single model output.

44. The Sampling Theorem

    Sample rate fs counts waveform measurements per second; frequency counts oscillation cycles per second. Samples x[n] represent times n/fs. For an ideally band-limited signal with highest frequency B, fs > 2B permits unambiguous reconstruction under ideal sampling assumptions. Otherwise, different continuous frequencies can produce identical samples: aliasing. The textbook illustrates a sinusoid at 0.95fs appearing at 0.05fs after sampling. Once frequencies overlap, the samples alone cannot identify their original frequencies.

45. SciPy Signal Processing: Short-Time Fourier Transform

    An STFT multiplies overlapping waveform segments by a window and Fourier-transforms each segment. A spectrogram is |STFT|², displaying energy-like values over time and frequency. With hop H, sample rate fs, and FFT length N, column spacing is H/fs seconds and frequency-bin spacing is fs/N Hz. Longer windows analyze longer intervals and separate nearby frequencies more finely, while smearing rapid changes; shorter windows reverse this tradeoff. Zero-padding densifies the frequency grid without adding observations. Reconstruction inverse-transforms complex coefficients and sums window-weighted overlapping segments; the window and hop must permit a dual window and nonzero coverage. A magnitude-only spectrogram discards phase, so it is not sufficient for direct exact inversion.

46. Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions

    Jonathan Shen and Google's colleagues combined Tacotron-style spectrogram prediction with a modified WaveNet neural vocoder in Tacotron 2. A vocoder here generates waveform samples conditioned on acoustic features. The mel spectrogram connects two separately trained components; it omits information that the waveform generator must supply. On 100 held-out utterances, the system's mean opinion score approached recorded speech, but a separate paired comparison still found a small, statistically significant preference for the recordings. Mispronunciation and unnatural prosody remained observable failures.

47. AudioLDM: Text-to-Audio Generation with Latent Diffusion Models

    AudioLDM generates continuous latents of a mel-spectrogram autoencoder, then decodes the spectrogram and uses a vocoder to produce a waveform. Its diffusion model trains with audio embeddings from a pretrained language–audio alignment model and uses text embeddings at sampling time. This reduces its diffusion-training dependence on paired descriptions. The system also applies the learned generator to audio inpainting and style transfer without task-specific fine-tuning.

48. Software for a Cascade/Parallel Formant Synthesizer

    Klatt's software synthesizer constructs speech from controlled sound sources and digital resonators representing acoustic properties of the vocal tract. An experimenter specifies changing parameters as time–value points, and the program computes a waveform for later playback. Its motivation was a flexible, precisely controllable tool for speech-perception experiments that could run on a general-purpose laboratory computer instead of requiring dedicated synthesis hardware.

49. Tacotron: Towards End-to-End Speech Synthesis

    Yuxuan Wang and Google's collaborators introduced Tacotron in 2017 to reduce the separately engineered components of speech synthesis. An attention-based encoder–decoder learns to map characters to successive mel-spectrogram frames without supplied phoneme-level alignments. A postprocessing network predicts a linear-frequency magnitude spectrogram, and Griffin–Lim reconstructs a waveform. Thus the original system does not directly predict waveform samples or use a neural vocoder. In its listening evaluation, Tacotron scored above the tested parametric baseline but below the concatenative system. The authors identified waveform-reconstruction artifacts as a remaining limitation.

50. Voxtral TTS

    Voxtral TTS combines autoregressive prediction and flow matching within one speech generator. Reference-audio tokens and text condition a decoder that advances through audio frames. At each step, a classification head predicts a semantic code, while a smaller flow-matching transformer generates continuous acoustic values conditioned on the decoder's hidden state. Those values are quantized before being supplied to the next autoregressive step; the codec converts the generated representation into waveform audio. The reported implementation uses eight flow evaluations per frame. This illustrates that sequential token generation and iterative continuous generation can serve different roles inside one model.

51. Speech Synthesis Markup Language (SSML) Version 1.1

    SSML distinguishes converting written text into spoken forms, determining pronunciation, planning prosody and producing a waveform. A phoneme is a speech-sound unit that distinguishes words in a language. Prosody includes pitch patterns, timing, pauses, speaking rate and emphasis, which affect naturalness and meaning. Separate markup supports pronunciation, voice choice, breaks and prosodic guidance. The standard explicitly allows processor-dependent behavior and says most markup values are indications rather than absolute output specifications.

52. Building Conversational AI Agents - Thor Schaeff, ElevenLabs

    Pronunciation dictionaries were offered as the control for how text-to-speech renders acronyms and particular words.

53. FastSpeech 2: Fast and High-Quality End-to-End Text to Speech

    Yi Ren and colleagues address the fact that one text can have multiple valid spoken realizations by explicitly modeling duration, pitch and energy. FastSpeech 2 expands phoneme representations according to predicted durations and generates spectrogram frames in parallel. During training, durations come from forced alignment, while pitch and energy are extracted from recordings. Unlike its predecessor's teacher-generated targets, it learns from the recorded speech's spectrograms. Ablations found that removing pitch or energy conditioning reduced listener-rated quality, connecting these acoustic variables to the generation mechanism rather than treating delivery as an unspecified style instruction.

54. Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale

    Voicebox uses flow matching to fill missing speech representations given text and surrounding audio. A transcribed reference can guide the style of new speech. For local noise removal, the user masks a damaged interval and the model regenerates it using its transcript and clean context. For word replacement, the method retains durations of unchanged phones, predicts durations for new phones and generates their acoustic frames. A separate style-transfer operation retains the original frame-level text alignment, which the authors identify as useful for audio already synchronized with video. Correctness and intelligibility are evaluated separately from other perceptual properties.

55. About MIDI—Part 1: Overview

    MIDI represents performance instructions, including notes, timing, instruments and relative volumes, rather than recorded sound samples. A receiving instrument or synthesizer interprets those instructions to produce sound. Editing symbolic performance data can change notes, tempo or instrumentation without editing an existing waveform, and the audible result depends on the playback system.

56. ASA Standards Terminology: Timbre

    Timbre is the sound character that lets listeners distinguish sounds even when loudness, pitch, location and duration are matched. It depends on both frequency content and changes over time, particularly the sound's onset. This helps explain why changing a generated instrument or voice is not equivalent to changing pitch alone.

57. Simple and Controllable Music Generation

    MusicGen predicts several streams of discrete audio-codec indices with one autoregressive transformer. Codebook interleaving determines which indices are predicted together and which earlier predictions they can use; prediction steps therefore differ from waveform samples or playback frames. Text supplies descriptive conditioning. Melody conditioning uses a chromagram, a time-varying representation of pitch classes, reduced to dominant bins to discourage reconstruction of the reference recording. The authors separately collect human judgments of overall audio quality and relevance to the supplied text.

58. StemGen: A music generation model that listens

    Stems are separately represented musical parts, such as an instrument performance or synthesizer layer, combined into a finished mix. StemGen frames production as adding and refining parts in relation to existing music. Its training pairs a mixture of selected stems with another stem excluded from that mixture, teaching conditional generation of a complementary part. This supports an iterative workflow in which an existing musical context guides a new layer rather than regenerating the entire mix from a text description.

59. FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds

    Foley creates sound effects matched to visible actions. FoleyCrafter conditions an existing text-to-audio generator on silent video, using a semantic adapter for what the sound should represent and a temporal controller for synchronization. Its generation path produces a latent representation, decodes it into a mel-spectrogram and uses a vocoder to synthesize the waveform. The authors evaluate audio quality, semantic correspondence and temporal alignment separately, illustrating why a plausible sound and a correctly timed sound are different requirements.

60. Building Conversational AI Agents - Thor Schaeff, ElevenLabs

    The Text to Bark example demonstrates sound-effects synthesis, not translation into a dog's language.

61. Adobe: What Is a Storyboard?

    A storyboard organizes a planned video into visual panels with supporting information about action, camera direction, timing, dialogue and sequence. Panels can use sketches, existing images or reference visuals. Their purpose is to make shot relationships and pacing inspectable before producing the finished sequence.

62. Black Forest Labs: FLUX, Open Research, and the Future of Visual AI

    Repeated low-latency image edits can provide an interactive visual experience without using a video-generation model.

63. Photorealistic Video Generation with Diffusion Models

    W.A.L.T compresses video in both space and time into continuous latent tensors. Its causal encoder processes the first frame independently, allowing still images and videos to share a representation. Unlike discrete codec indices, these latents are not quantized. Alternating spatial and spatiotemporal attention windows exchange information within frames and across time while limiting attention cost. For continuation, the model is trained to condition generation on preceding latent frames. Thus representation compression, temporal interaction and conditioning on previous content are distinct mechanisms.

64. VideoPoet: A Large Language Model for Zero-Shot Video Generation

    VideoPoet uses MAGVIT-v2 to compress video into discrete integer codes, then flattens them into a sequence. Its published example maps 17 frames at 128×128 pixels into 1,280 tokens; a learned decoder converts codes back into pixels. Text embeddings and optional visual or audio tokens supply conditioning. Prefix inputs use bidirectional attention, while generated tokens use causal attention: each new prediction depends on conditioning and preceding tokens. The tokenizer encodes the first frame independently, allowing an image to start a video. For longer generation, the paper demonstrates predicting successive one-second segments conditioned on the last second of generated video, including a ten-second example. This is bounded recent context, not attention over an unlimited history. A separate non-autoregressive token model performs spatial super-resolution.

65. Veo 3 for developers

    Outpainting and first/last-frame interpolation provide spatial and temporal constraints for generation.

66. Let's go Bananas with GenMedia

    An image prompt can omit the temporal and dialogue information needed for video; derive a video-specific continuation from the actual starting image.

67. ProPainter: Improving Propagation and Transformer for Video Inpainting

    Video inpainting receives a sequence of frames and corresponding region masks. ProPainter completes optical flow—estimated correspondence between positions in successive frames—then moves available image content into missing regions and refines the result using propagated features and a video transformer. Its image-propagation stage accepts pixels only when correspondence passes a consistency check and the source location is unmasked. It updates masks as regions are filled.

68. CameraCtrl: Enabling Camera Control for Video Diffusion Models

    CameraCtrl supplies a video generator with a sequence of camera conditions. Camera parameters describe imaging geometry and the camera's position and rotation; the method converts them into per-pixel ray representations and trains a camera encoder to condition the generator. This gives a concrete alternative to describing camera movement only in prose. The control specifies the viewing trajectory rather than directly specifying how each object moves.

69. Black Forest Labs: FLUX, Open Research, and the Future of Visual AI

    Multiple reference images can specify items to compose into one coherent output, rather than describing every item through text.

70. LoRA: Low-Rank Adaptation of Large Language Models

    LoRA freezes an existing weight matrix and learns its update as the product of two smaller matrices. This reduces the trainable parameters and the storage needed for each adaptation. The update can be stored separately or merged with the base weights for inference. It is a parameter-adaptation method, distinct from supplying a reference as an input to an unchanged model.

71. AudioLM: a Language Modeling Approach to Audio Generation

    AudioLM first predicts compact semantic tokens autoregressively, p(zt|z<t), to model longer-range linguistic or musical structure. A second transformer predicts coarse acoustic codec tokens conditioned on semantic tokens and preceding acoustics, carrying speaker and recording characteristics. A third predicts fine residual-codebook tokens for waveform detail, conditioned on coarse acoustics; it operates on three-second chunks. For continuation, both semantic and coarse acoustic tokens from the real audio prefix become context. The paper reports voice, prosody, and recording continuity from three-second prompts, including unseen speakers. Compact semantic sequences reduce attention's quadratic sequence-length burden, while acoustic tokens preserve information needed for high-quality decoding.

72. VBench: Comprehensive Benchmark Suite for Video Generative Models

    VBench separates whether a video looks good from whether it matches its condition. Temporal assessment distinguishes subject appearance, background consistency, flicker, motion smoothness and amount of motion; frame-level aesthetics and distortions are evaluated separately. Its authors explicitly note that a static video can score well on several consistency measures, motivating a separate dynamics measure. Human raters compare videos generated from the same prompt on specified dimensions rather than giving only one undifferentiated quality judgment.

73. The Next Medium: Why Real-Time Interactive Video Changes Everything — Ahmed Ahres, Reactor

    Insufficient retained context can cause a world to change when a character looks back at a previously observed scene.

74. Video Diffusion Models

    Conditioning frames xa remain fixed clean targets; their noisy counterparts are replaced with forward-process samples during denoising. Unknown frames xb are sampled with an additional gradient encouraging reconstruction of xa. Resampling noisy conditioning states does not regenerate the accepted clean frames. Extension places newly sampled frames after retained context; interpolation instead fills between known frames. Reusing context supports an overlapping-window illustration, but not a claim that overlapping outputs are regenerated and blended. Figure 4 and Table 6 report improved coherence and perceptual scores over replacement-only sampling for 64-frame outputs from a 16-frame model.

75. Helios: Real Real-Time Long Video Generation Model

    Helios addresses long-video drift by treating generated history as an imperfect condition. The authors distinguish positional mismatch, color shifts and accumulating restoration artifacts. Their method retains the first frame as a visual anchor and corrupts historical frames during training with exposure changes, noise or reduced resolution to improve tolerance of imperfect generated context. Ablations report worse drift when the anchor or history-corruption component is removed.

76. Adobe Premiere: Audio Crossfade Transitions

    A crossfade joins audio clips by reducing one clip's level while increasing the other's. Premiere documents different gain curves: constant-gain transitions can produce a noticeable midpoint volume dip, while constant-power transitions use different curves to smooth the handoff. This is a conventional finishing operation on supplied clips rather than regeneration of their content.

77. Evaling Video Slop

    The sound alignment method checks for an audio spike at the timestamp of a visible event, without establishing the sound's semantic identity.

78. A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild

    Wav2Lip generates lip motion for supplied speech and face video. Training uses a pretrained synchronization discriminator that remains frozen, alongside a separate visual-quality discriminator. The synchronization model compares short sequences of lower-face images with corresponding audio segments, learning from aligned and time-shifted pairs. The paper explains that whole-face reconstruction loss can underweight lip errors because the mouth occupies only a small image region. Attractive frames and low reconstruction error therefore need not imply correct speech–lip correspondence.

79. Google AI for Developers: Video generation in the Gemini API

    Google's documentation describes Veo 3.1 as generating video with native audio and supporting video extension, frame-specific generation and image-based direction. This establishes an upstream example of a service producing audiovisual output in one generation workflow, alongside the separately conditioned sound and lip-generation approaches.

80. Adobe Premiere: Sync Lock to Prevent Changes

    Premiere's Sync Lock determines which other tracks move when an insert or ripple operation changes the timeline. Tracks containing clips directly involved in the operation move regardless of their sync-lock state. This makes the consequences of a duration-changing edit explicit: other accepted audio and video can require coordinated repositioning.

81. HTML Is All Agents Need

    The recommended workflow combines iterative decomposition with a narrative, frame-by-frame storyboarding, motion authoring, composition, and manual finishing.

82. Evaling Video Slop

    Check intermediate generation inputs and individual clips before assembly, and expose evaluation tools to the generating agent.

83. Veo 3 for developers

    The demonstrated Veo 2 workflow requires planning, clip segmentation, separate audio generation, and editing; the Veo 3 example produces an audiovisual excerpt from one prompt.

84. Frame.io: Version Stacking

    Frame.io organizes revisions by uploading each new asset and grouping it with earlier versions. The player initially presents the newest version while allowing reviewers to open previous versions. Stacks support video, images, audio and other asset types, and their order can be changed. This is a concrete example of retaining reviewable artifacts instead of replacing an earlier file with an instruction to regenerate it.

85. Remotion: The fundamentals

    Remotion represents a composition as a React component plus width, height, frame count and frame rate. A frame is one image in the timed sequence; frame rate specifies how many frames represent a second. The documented duration calculation is frame count divided by frame rate. Rendering a component from a specified frame number provides a concrete way to author layout and timed graphics separately from generated footage.

86. The Double-Edged Roles of Generative AI in the Creative Process: Experiments on Design Work

    A randomized field experiment assigned 120 working graphic or visual-design professionals to use Midjourney V6.1 during ideation, implementation, both stages or neither while creating an agricultural-product advertisement. Researchers recorded prompts, images, screen activity and stage completion time. AI-assisted ideation improved panel-rated novelty, relevance and complexity but took longer. Using AI only during implementation increased completion time without statistically significant improvement in those creativity measures. That group also made more revisions.

87. Perceptual Evaluations: Evals for Aesthetics — Diego Rodriguez, Krea.ai

    Object counts, colors, and literal realism can miss artistic meaning and may penalize deliberate departures from ordinary appearance.

88. Perceptual Evaluations: Evals for Aesthetics — Diego Rodriguez, Krea.ai

    Aesthetic metrics should account for the intended user's opinion as well as perceptual properties of the medium.

89. Building in the Gemini Era – Kat Kampf & Ammaar Reshi, Google DeepMind

    The presenters describe Nano Banana Pro as supporting text that conforms to pictured surfaces and translated text within a reference image.

90. Adapting Fréchet Audio Distance for Generative Music Evaluation

    Azalea Gui and colleagues show that FAD's interpretation changes with its audio embedding and reference collection. They distinguish acoustic quality—recording noise and artifacts—from musical quality—composition and performance. Their adaptation computes per-song FAD to inspect outliers, rather than only comparing whole generated collections. Different embeddings emphasize different defects, and reference-set composition can favor particular genres. A listening study separately rated acoustic and musical quality for 300 ten-second excerpts from recordings and three music generators; participants were not asked to judge prompt adherence.

91. Evaling Video Slop

    Prompt alignment and frame consistency do not establish that a video tells the intended story; temporal quality needs explicit evaluation axes.

92. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

    FID compares real and generated image collections after an Inception network maps images to feature vectors. It fits a multivariate Gaussian to each collection using feature means μ and covariances C, then computes ||μr−μg||²+Tr(Cr+Cg−2(CrCg)½). Lower values indicate closer fitted feature distributions. The Gaussian approximation retains only first and second moments, rather than testing equality of complete image distributions. The paper evaluates sensitivity to noise, blur, occlusion, distortions, and dataset contamination. Inference from its inputs: FID contains neither prompts nor paired protected-region references, so it cannot certify instruction adherence, correctness of an individual image, or exact preservation.

93. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

    LPIPS compares an image with a paired reference by extracting multiple network-layer features, normalizing feature vectors across channels, weighting channel differences, and spatially averaging their squared norms and summing across layers. Its calibrated variant learns channel weights from human perceptual judgments. The BAPPS experiments ask people which of two distorted patches is closer to a reference and whether sequentially shown patches appear identical. Tests include conventional distortions, neural-network distortions, and outputs from real image-processing algorithms. Learned feature distances track these judgments better than several traditional metrics in the tested settings. Inference: similarity to a reference does not establish that either image is semantically correct, and a small feature distance does not prove equal pixels.

94. CLIPScore: A Reference-free Evaluation Metric for Image Captioning

    CLIPScore computes 2.5×max(cos(v,c),0), where v and c are CLIP image and candidate-caption embeddings; corpus scores average pairs. Its original task evaluates captions for supplied images without requiring reference captions, rather than certifying images generated from instructions. The paper reports strong agreement with human judgments in several caption datasets but weaker performance when interpretation requires context. In news captioning, CLIPScore matches pairwise judgments with 65% accuracy versus 93% for METEOR; it also fails to reliably rank engaging nonliteral captions. Inference: embedding compatibility can support an alignment signal, but these evaluations do not establish compliance with every count, relation, edit instruction, or protected region.

95. Fréchet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms

    FAD compares Gaussian statistics fitted to audio embeddings from an evaluation collection and a background collection of clean music. Reference-free means no paired clean recording is required for each evaluated clip; a background distribution is still required. The original implementation uses VGGish features from short windows. The authors explicitly identify its inability to assess long-distance temporal changes within a song. Human evaluation tests judgments of studio-like recording quality under artificial distortions, rather than compliance with a musical brief.

96. ITU-T P.800: Methods for Subjective Determination of Transmission Quality

    P.800's listening-quality test asks listeners to rate speech using five ordered categories from bad to excellent; numerical ratings are summarized as a mean opinion score. It separately defines listening-effort judgments. The protocol addresses presentation order, fatigue, listener selection, room noise, calibrated reproduction and recorded listening level. Thus a MOS describes judgments under a particular listening procedure rather than an intrinsic property of an audio file.

97. Effectively Unbiased FID and Inception Score and Where to Find Them

    Min Jin Chong and David Forsyth show that finite-sample FID has a bias that depends on the generator being evaluated, not just the number of generated images. Giving two systems the same sample count therefore does not remove this source of ranking distortion. They estimate an infinite-sample score by extrapolating measurements against inverse sample count and test the procedure on image generators. This adds a statistical limitation distinct from whether the embedding space measures the qualities a creator actually cares about.

98. On the Content Bias in Fréchet Video Distance

    Songwei Ge and colleagues test whether Fréchet Video Distance responds to temporal damage independently of frame appearance. They compare consistently applied frame distortions with frame-varying distortions that introduce temporal artifacts while keeping per-frame degradation similar. The conventional I3D-feature implementation responds insufficiently to the added temporal damage in their experiments. Changing to self-supervised video features improves sensitivity under the tested conditions. They also demonstrate that selecting a different generated sample collection can improve FVD without improving its motion.

99. Content Credentials: C2PA Technical Specification, Version 2.2

    C2PA can connect an asset's manifest to ingredient manifests and requires recursive validation of those included histories. Its action vocabulary includes creation, editing, cropping, placement and transcoding. A published generative-AI example records a creation action, software version, digital-source type and hashed ingredient references. The action schema also permits an explicit assertion that all actions are included; an ordinary action list alone need not claim completeness.

100. NIST AI 100-4: Reducing Risks Posed by Synthetic Content

     NIST distinguishes provenance metadata recording origin/history, watermarks embedded in content, and detectors estimating synthetic origin from content signals. Metadata can be stripped or falsified; signatures authenticate an assertion's signer and integrity, not its truth. Watermarks face removal, spoofing, disabled embedding, and incompatible detector coverage. Detectors have false positives, false negatives, and context-dependent performance. Input/output filtering addresses prohibited generation but cannot reliably infer consent; consent to creation may differ from consent to distribution. Application inference: authenticated access, authorization checks, and recorded consent can gate a service's operations, but do not cover uncontrolled copies or other generators. These safeguards answer different questions, so none alone establishes publication authority.

101. NIST Generative AI Profile: Third-Party Risk

     NIST recommends inventorying third parties with access to organizational content, assessing suppliers and specifying security, ownership, usage and provenance expectations in contracts. Examine secondary data use, incident notification, service continuity and responsibility for system changes. These concerns apply to proprietary and open-source components, fine-tuned models and embedded tools. Hosting inference internally does not remove risks from acquired weights, training data, libraries or external integrations. Evaluation, access governance, monitoring and incident ownership remain organizational responsibilities.

102. Creative Commons: Frequently Asked Questions

     Creative Commons explains that its licenses do not clear third parties' publicity, privacy or personality rights. Under version 4.0, a licensor waives or agrees not to assert certain rights they themselves hold only to the extent needed to exercise the licensed rights. Rights held by other people remain unaffected. Attribution and noting modifications therefore do not themselves establish that every intended use is permitted.

103. U.S. Copyright Office: Musical Works, Sound Recordings & Copyright

     A musical work is the underlying composition and accompanying lyrics; a sound recording fixes a particular series of sounds. The Copyright Office explains that these are distinct works, commonly owned and licensed separately. Reproduction, distribution and derivative uses can implicate different rights in each. Authorized use may depend on a negotiated license, an applicable statutory license or an exception.

104. U.S. Copyright Office: Copyright and Artificial Intelligence, Part 2—Copyrightability

     The Copyright Office's January 2025 report treats copyrightability as a case-specific question about human authorship. Its conclusion allows protection where AI is used as a tool and a human determines expressive elements, while stating that prompts alone were unlikely to satisfy that requirement at the technological stage examined. The report distinguishes human contribution to an output from merely accepting a machine's interpretation.

105. ComfyUI Workshop with ComfyAnonymous and Jedrick Kosinski

     Reference-image editing offers a simpler entry point, while LoRAs provide trainable character or style adaptations and more explicit regional control.

106. Building Generative Image & Video Models at Scale

     A diffusion transformer can use fully bidirectional attention rather than a causal mask, while benefiting from transformer scaling experience.

107. ComfyUI Workshop with ComfyAnonymous and Jedrick Kosinski

     Instruction-driven edit models can simplify earlier mask-and-inpaint workflows, but explicit masks remain useful when edits affect unwanted regions.

108. Veo 3 for developers

     The Veo 2 discussion presents reference-powered video as a way to combine subjects and environments, preserve styles, and control character performance.

109. Why TTS Models Now Look Like LLMs — Samuel Humeau, Mistral

     Mistral's described model generates all tokens within an audio frame together using a diffusion-style model.

110. Magic Editor Under the Hood: Weaving Generative AI into a Billion-User App

     A U-Net portrait segmentation model can produce a usable mask while missing fine hair strands; post-model image understanding can refine those boundaries.