Extremely sparse random binary matrices tend to be singular, due to the likely presence of "local dependencies" such as all-zero columns or pairs of identical columns. We discuss this phenomenon, and some recent results showing that these kinds of dependencies are in some sense the only causes of singularity. This includes the resolution of a conjecture of Vu concerning the k-core of an Erds-Rényi random graph. This talk covers joint work with Asaf Ferber, Margalit Glasgow, Ashwin Sah and Mehtaab Sawhney.