Hi, I’m Mark

Introduction In machine learning, there are broadly two classes of models: discriminative, and generative. Discriminative models take data (observations), and produce predictions. Generative models work by learning the distribution that produced the data, and using it to generate new samples. To be more precise, given observations $x$ and predictions $y$, a discriminative model seeks to learn $p(y|x)$, while a generative model seeks to learn $p(x)$. In this post I’ll explore one type of generative model, the variational autoencoder (VAE), first introduced by Kingma and Welling. VAEs are particularly suited to problems where the observed data is believed to be generated by an underlying set of unobserved latent variables — structure that is real but not directly measurable. The VAE addresses this by learning two things jointly: an encoder that maps observations to a distribution over latent variables, and a decoder that maps latent variables back to the data space. Together these give us both a generative model and a structured latent representation of the data. ...

Introduction Neural networks have an extraordinary ability to learn highly non-linear solutions from data. However, the amount of data needed for a neural net to learn a good, generalizable solution can be vast. Modern LLMs, for example, are pre-trained on over 15 trillion tokens. When such vast datasets are out of reach, there are still options. Architectures can bake in known structural properties of the problem – convolutional nets exploit translational invariance, for instance – but this only helps when the constraint is something that can be encoded in the architecture itself. ...