The remarkable properties of quantum materials originate from a delicate balance among many interacting degrees of freedom. External stimuli that perturb such precarious equilibrium can steer complex solids into entirely new phases, with novel functionalities and collective responses. In this context, the optical excitation of lattice vibrations with ultra-short and ultra-intense terahertz pulses - nonlinear phononics - has proven to be an effective route to material control, capable of stimulating insulator-to-metal transitions and enhancing superconductivity. In this talk, I will first show how nonlinearities of the crystal lattice can be exploited to induce or control broken-symmetry phases in quantum materials such as the quantum paraelectric Strontium Titanate (SrTiO3) and the magnetic insulators Erbium Ferrite (ErFeO3) and Cobalt Fluoride (CoF2). Secondly, I will discuss the ideas at the core of my current research efforts: can light modify materials properties even in the complete absence of an external electromagnetic field? By engineering the electromagnetic environment through terahertz cavities, we aim to exploit quantum vacuum fluctuations to control collective phases.