Deep Learning

Slides: Distributed Training for ML

Explore distributed training techniques through this interactive presentation. Navigate through the slides using arrow keys or the navigation controls. Topics Covered Back to Basics: Understanding neural network fundamentals Why Distributed Training: Memory constraints and scaling challenges DDP (Data Distributed Parallel): Replicating models across GPUs Pipeline Parallelism: Splitting models across devices FSDP (Fully Sharded Data Parallel): Advanced sharding techniques Slides Use the arrow keys (← →) or click the navigation arrows to move between slides. Some slides include animations that you can step through using the animation controls at the bottom.

Wednesday, December 3, 2025 | 2 minutes Read

Distributed Training Techniques: DDP, Pipeline Parallelism, and FSDP

Modern deep learning models have grown exponentially in size and complexity. GPT-4 has over a trillion parameters, and even “smaller” models like LLaMA-70B require substantial computational resources. Training or fine-tuning such models on a single GPU is often impossible; not just because of time constraints, but because the model itself may not fit in the memory of a single device. This is where distributed training becomes essential. Why Do We Need Distributed Training? The Memory Wall Problem A modern GPU like the NVIDIA A100 has 80GB of memory. Sounds like a lot? Let’s do some math:

Monday, December 1, 2025 | 8 minutes Read

Understanding Variational Autoencoders and Implementation in Keras

Variational Autoencoders (VAEs)[Kingma, et.al (2013)] let us design complex generative models of data that can be trained on large datasets. This post is about understanding the VAE concepts, its loss functions and how we can implement it in keras. Generating data from a latent space VAEs, in terms of probabilistic terms, assume that the data-points in a large dataset are generated from a latent space. For e.g., let us assume we want to generate the image of an animal. First we imagine that it has four legs, a head and a tail. This is analogous to the latent space and from this set of characteristics that are defined in the latent space, the model will learn to generate the image of an animal.

Tuesday, July 2, 2019 | 10 minutes Read

Tracking Multiple Losses with Keras

Often we deal with networks that are optimized for multiple losses (e.g., VAE). In such scenarios, it is useful to keep track of each loss independently, for fine-tuning its contribution to the overall loss. This post details an example on how to do this with keras. Let us look at an example model which needs to trained to minimize the sum of two losses, say mean square error (MSE) and mean absolute error (MAE). Let $\lambda_{mse}$ be the hyperparameter that controls the contribution of MSE to the toal loss. i.e., the total loss is MAE + $\lambda_{mse}$ * MSE. This loss can be implemented using:

Monday, March 4, 2019 | 2 minutes Read