Entropy in Generative AI Models: From Imitation to Imagination

Deep generative models have achieved remarkable success by learning to imitate complex data distributions, yet distributional fidelity alone does not explicitly promote properties such as diversity, novelty, or imagination. In this talk, I will explore how entropy can provide a principled way to extend the conventional objective of generative modeling beyond imitation. Utilizing pretrained neural network representations, we introduce a spectral notion of entropy that captures semantic diversity through the eigenvalues of a kernel covariance operator. I will discuss that the concavity of this entropy has two important consequences. First, finite samples systematically underestimate the entropy of the underlying population, motivating our study of whether this diversity loss persists in trained generative models. Our experiments then reveal a consistent entropy gap between standard generative models and their training distributions. Second, concavity makes entropy superlevel sets convex, allowing us to show that projecting a low-diversity model onto a sufficiently high-entropy set can move it closer to the true data distribution. Building on these insights, I will introduce Imaginative Generative AI (IGA), an entropy-regularized framework that provides a path from diversity correction to deliberate extrapolation, with the entropy of the data defining an Entropy Wall that separates faithful imitation from the generation of more diverse worlds beyond the observed distribution.