This post is part of my Practical Deep Learning journey.
This is my experience with Lesson 5 and Lesson 6 of the Practical Deep Learning course. If you’re just tuning in, I’m working through this material in a study group while doing a half-batch at The Recurse Center in May/June 2026. See Learning Practical Deep Learning (fastai) for why.
In this lesson we’re exploring tabular data. One of the points of this lesson is to show that deep learning is not the best machine learning method for all kinds of problems. For instance, for tabular data, decision trees often perform pretty well, and you can augment their performance by bagging them into “random forests”. Moreover, decision trees are useful because you can visualize them to see which columns are contributing to the performance of the model.
In the first few lessons of the course, we learned how to:
In this lesson we’ll:
In these lessons I decided to do the course using my mac laptop instead of cloud GPUs. So I’ve also been learning something about parameter fiddling to make the data fit on the machine and make the training loops run in reasonable amounts of time. Along those lines, because I followed the course advice to run the notebooks with the latest versions of the libraries, I also found a few places where object implementations and parameter names drifted. Nothing crazy! Modern LLMs seem to be pretty good at suggesting up-to-date library compatibilities and letting you know when they’re not up to speed. I was sort of shocked to see a: “I’m sorry I can only make suggestions for best practices through 2025.”
Fixing bugs related to library deprecations and transformations is maybe one of the more boring and time-consuming parts of running code. But if you don’t spend the time doing it, you’ll never get anywhere. Fixing things blindly (e.g. copy-pasting in some fix without reading what it is) is also not great. You might get your code run, but it may silently behave differently than you want. So I sucked it up and did it – I can only expect that the world will keep being filled up with more and more aging code and messy data that supports our swiftly tilting planet. Tasks in the real world are not carefully prepared course notebooks.
In Lessons 5 & 6, we apply deep learning methods to tabular data. The technique the authors use for this type of data is called “Random Forests”, which is a method for getting better performance on decision trees. In these lessons there’s an explosion of jupyter notebooks to check out, but ultimately there are two choices of datasets to use for building, training, and evaluating the random forest models:
The Titanic dataset is too gruesome for me, so I went with the bulldozers. The thing is, the bulldozers competition is
from 13-14 years ago (!?). The data is no longer on kaggle, and you can’t click the “accept the rules” button which you
usually need to do to get access to the data. With a little googling around, I found that someone else made an ML course
using the bulldozers competition and linked the data in their course repo.
When I went through this lesson originally, I ended up using that data and everything went mostly ok (biggest thing was
trying to sort out the latest dependencies for a cool tree visualization library, dtreeviz,
but I figured it out and made some cool-looking graphs!).
Predicting “future” auction prices for bulldozers is interesting because in addition to being tabular like the Titanic passenger manifest, the data is also a timeseries. The authors pointed out a few things to keep in mind for timeseries data:
… Some of these timeseries conditions sound sort of similar to the NLP criteria from the last lesson! In particular, the “before” and “after” conditions, and the dependence of the later data points on the earlier ones. There are some differences though. In the NLP case we wanted to hang on to words from much earlier for inform context - here we’re told maybe we should drop earlier data because behavioral patterns may have shifted (but… what about cyclic patterns in the market??).
For this lesson, we start by learning about how to train a decision tree from a table of data. Next, we move from a single decision tree to a “random forest”. Finally, we compare random forest predictions with those from a neural network.
Decision trees and random forests are part of an arsenal of classical (non-neural net) machine learning techniques that haven’t disappeared from use. The goal of this lesson seems to be showing students that there are other model architectures, besides variations on the neural network, that you might reach for to solve a task with a given dataset.
So how do we train a decision tree? According to the authors, “The basic steps to train a decision tree can be written down very easily”:
After reading this I wasn’t totally sure about what “levels” meant, so I spent some time unpacking the details. I turned what I learned into the following example.
Let’s say we have a dataset with 100 rows and four columns. The columns are “id” (unique), “age” [1,7], “type” [“bulldozer”, “tractor”, “wheelbarrow”, ‘rake’], and “price” (dollars). “price” is the dependent variable. To decide how best to partition the “age” column into two groups, we would sort from youngest to oldest. Then, we would create a list of pairs of groups by moving a partition systematically across the values in the column. If age were [1,7], we would make 6 pairs of row-groups:
partition A: group 1: 1 | group 2: 234567
partition B: group 1: 12 | group 2: 34567
partition C: group 1: 123 | group 2: 4567
...
partition F: group 1: 123456 | group 2: 7
For a given partition A, we would predict that the sale price for a given row in group 1 is the average sale price for all the rows in group 1; similarly the sale price for a given row in group 2 is the average sale price for all the rows in group 2. Then we would calculate the RMSE between the row’s actual sale price and the prediction, for all the rows across both groups of partition A. The partition that produces the lowest RMSE is selected for that particular column.
Then the next column would be considered. For a categorical column like “type”, we would assign each category an arbitrary number, e.g. bulldozer = 1, tractor = 2, wheelbarrow = 3, rake = 4, and proceed with the same partitioning scheme. Then this column’s best partition would be compared with the best partition from the other columns. Whichever had the smallest sale price RMSE would be chosen as the node’s first split.
To build the next node, the same procedure would be repeated with just the rows on the left side of the chosen partition. Simultaneously, the same procedure would be repeated on the right side of the chosen partition.
The last step in the decision tree building algorithm is the stopping criteria. If we don’t impose a stopping criteria, we would end up creating a node for each individual row. The RMSE for the final split into individual rows would be 0, because the row’s price would be equal to its group’s average price (each side of the partition is a group with just one element)! This would be pretty unhelpful unless you were trying to make a visualization of overfitting.
The lesson doesn’t talk about this in detail, but the decision tree’s partitioning algorithm that I just described is
part of the scikit.learn library’s implementation. It seems a little strange to consider categorical variables ordered;
e.g. in the example above there would never be a [“bulldozer”, “rake”] | [“tractor”, “wheelbarrow”] partition, but there’s
no reason this partition couldn’t produce the smallest RMSE. Some googling and ChatGPT-conversing unearthed “category boosting”
libraries; these overcome the scikit.learn ordered categorical partitioning and usually produce better results.
What the lesson does talk about is how we don’t need to make embedding vectors for each of the categorical values. Doing the same kind of partitioning of the ordinal values is good enough, meaning that according to this 2019 paper the ordinal partitioning method apparently finds the same best-split as using embeddings.
The jupyter notebook for this lesson, fastbook chapter 9,
was full of methods for interrogating the model and understanding the predictions by the contributions of each of the columns
and column values. There were also more than a few places where updates to scikit-learn, pandas, and some visualization
libraries meant I had to figure out ways to reproduce the authors’ lessons with newer or alternative versions of these libraries.
For this reason, I ended up writing a lot of notes and quotes from the original lesson in my local jupyter notebook where I executed the cells. So, that’s here.
The main things to remember from this lesson are that:
This post is part of my Practical Deep Learning journey.
This is my experience with Lesson 4 of the Practical Deep Learning course. If you’re just tuning in, I’m working through this material in a study group while doing a half-batch at The Recurse Center in May/June 2026. See Learning Practical Deep Learning (fastai) for why.
Lesson 4 is the Natural Language Processing (NLP) lesson. Here we move from vision models (which learn features of images, e.g. for image label prediction) to NLP models (which learn features of text, e.g. for next word prediction). Many novel applied computer vision tasks no longer require models trained from scratch. Instead, researchers can start from freely available models that were pretrained using huge datasets and lots of compute, and fine-tune them with datasets specific to the novel task. This is called “transfer learning”; I practiced it in Lesson 2 when I started from the 2015 ResNet trained on ImageNet (available in the pytorch library), and fine-tuned it on different types of clouds) downloaded from duck duck go.
Text models can take much longer to train than image models (more on this later) – so wouldn’t it make sense to try transfer
learning for text? “Why yes, that would be a great idea!” say the authors of the course in 2018, as Jeremy Howard and Sebastian
Ruder proceeded to create what might have remained a state-of-the-art transfer learning method to use on Recurrent Neural Network
architectures like Long Short-Term Memory (LSTM). But competition was steep! In 2017, the transformer architecture hit
the scene with its catchy academic title, “Attention is all you Need”. Implementations
swiftly followed in 2018: OpenAI’s GPT
(June) and Google’s BERT (October).
This lesson is a great example of the authors sticking to their goal of showing students that deep learning methods are still in active development. The 2020 version of the NLP lesson (fastbook chapter 10) was built on the state-of-the-art RNN at the time, a Long Short-Term Memory (LSTM) model called AWD-LSTM released by Salesforce Research in 2017. The fastai authors had been inspired by the success of transfer learning for image models, and developed a training strategy called ULMFiT (Universal Language Model Fine-Tuning) for transfer learning for language models. The lesson used ULMFiT with the AWD-LSTM model trained on a Wikipedia dataset (WikiText-103, 103 million tokens), and involved fine-tuning with 50k IMDb (Internet Movie Database) movie reviews. The goal was to classify the sentiment of a subset of IMDb reviews, like we had classified images as dogs, cats, etc. in a prior lesson. Fine-tuning with the IMDb reviews improved the model’s ability to classify sentiment, because the Wikipedia vocabulary would be augmented with IMDb-specific words and contexts, as well as style, syntax, and sentence structure.
By 2022, the transformer architecture was outperforming ULMFiT-assisted AWD-LSTMs. The authors responded by building a new NLP lesson using the huggingface transformer library. The lesson’s task was to try to score well on a US Patent kaggle competition (active Mar-June 2022) using a transformer model. The competition task was to compute similarity scores for pairs of phrases found in US patents. Why? Patent officers need to deny patent applications if it turns out the invention is already patented, but often people use different phrases to describe what is actually the same invention. Ideally, they could use the model to find patents similar to a new application submission, even if the exact phrase does not match (e.g. “TV set” vs “Television set”). The authors chose Microsoft’s deberta-v3-small (2021). From the docs, it has: “6 layers and a hidden size of 768,” and “44M backbone parameters with a vocabulary containing 128K tokens”. One of the lessons discusses how to choose a model, but I forgot which one.
The method for preparing a text dataset is largely the same for an LSTM model or a transformer. We need a strategy for
turning text into numbers, since all deep learning models transform tensors of numbers. It will involve tokenization,
numericalization, and embedding.
Embedding research has its own historical trajectory. Originally embedding vectors were trained separately (e.g.
origin of embeddings for NLP, 2003; Word2Vec,
2013). But, it turned out researchers got great results by initializing embedding vectors randomly and training them as
part of the first layer of the model, like w in the image model from the last lesson. This embedding matrix gets updated
with backpropagation at the end of each training loop.
So the embedding matrix is a parameter in the first layer of the model. What about the last layer of the model?
For the image classification task:
A chunk of text could be sorted into classes as well, e.g. “type of document” or “sentiment”. But we can also see a next word prediction task as a classification problem. In this case, the next word is the most likely selection (class) from a vector of the vocabulary:
Besides making the last layer perform “classification”, we could also make it do “regression”. More on this in a sec. In this case:
In the original notebook, the task was to use the RNN AWD-LSTM model to predict sentiment of an IMDb movie review:
In the updated lesson, the task was to use the transformer model to predict a similarity score for phrases (e.g. “TV set” and “Television set”) found in US patents, given particular categories:
Both of these tasks could be either regression or classification tasks. Even though transformer model classes are numbers, they’re not continuous – they look a lot like the MNIST handwritten digit classes we predicted in the last lesson. And even though the sentiments are words (“pos” and “neg”), we could also see them as values between 0 and 1, pinned to 0 if <0.5 and 1 if >0.5.
For the transformer lesson, the authors chose to set num_labels = 1 for the model. Just one label means they chose the regression. I think I missed the explanation, but it’s probably because they want the model to learn the ordinal relationships from 0 to 1. Smaller and larger similarity scores mean something – better and worse similarities; they’re not just re-orderable categories.
So, what does this all look like in code? WHat kind of stuff does the transformer library come with to turn datasets into something the transformer model can use, and then how do we train the model?
The training data is a csv where each row has the following columns:
The basic steps are the following:
training dataset object from the training df with one string of text per row (concat the phrase & category cols together with separator chars)tokenizer object from your choice of pre-trained model, and run it on the training dataset column that has the concatenated stringtrain and test (test is validation)model object for the task at hand; args include the pretrained model’s name (e.g. 'deberta-v3-small') and num_labels (1=regression; >1=classification)training args object; args include: learning rate (and learning rate sched args), batch sizes for train and eval, number of epochs, machine spec args, and morecompute metrics to print for each epoch; patent task asks for pearson coefficient (from scipy.stats import pearsonr)trainer object; args include: training args object, model, training dataset object's 'train' (train), training dataset object's 'test' (validation), tokenizer object, compute metrics functrainer object’s trainer.train() function, which updates the model object and the tokenizer objecttrainer.save_model("./patent_model_dir") and then zip it up for uploadtest dataset object the same way we made the training dataset object in “prepare the dataset” step 1 above, but using the test df (note: this is not the validation dataset, which was just a split of the training data) create a training dataset objecttokenizer object from “prepare the dataset” step 2 above, and run it on the test dataset column that has the concatenated stringtrainer object that we trained before: preds = trainer.predict(test_ds).predictions.astype(float)preds = np.clip(preds, 0, 1)submissions.csv file with the ids and scores (preds) columnsHere’s the jupyter notebook where I did all of this. It also involved a lot of figuring out values that worked on my macbook pro instead of a GPU.
So… submitting the similarity score csv to Kaggle was actually pretty confusing at first. I thought I could just upload
the output submissions.csv file, but this competition makes you upload a kaggle notebook that generates the file. After
googling around a bunch and having a meandering conversation with ChatGPT, I managed to figure it out and submit successfully
on my first try.
Basically the steps in mid-July 2026 are the following.
test.csv and the model folder and paste each into variables in the notebooktest_ds from the test.csv in the competition dataset, using the tokenizer you just importedpreds = trainer.predict(test_ds).predictions.astype(float).reshape(-1)submission = df_test[["id"]].copy(); submission["score"] = predssubmissions.csv file that Kaggle expects to see in the output submission.to_csv("/kaggle/working/submission.csv", index=False)Then, click the clear and run all cells button to make sure the output file gets created. You can download that file to make sure it looks like what the submission expects/matches the output you got when you ran locally.
Last, submit to the competition! There’s a section literally called that in the section on the right. Clicking this might
tell you you’ll need to “turn off the internet” for the notebook. That’s in settings at the top of the page. I made sure
the notebook still ran after I did this, before I clicked submit. I was a little unsure about whether the libraries I was
using were already installed, but most deep learning libraries are and the ones I used were there without doing anything
special. The notebook runs in a docker image; you can choose what kind of “accelerators” (read: GPUs) you want to use. I
didn’t use any for this task; and you can also pick from “environment preferences”: pinned to original, or latest. I haven’t
explored most of these.
Anyway… voila! The submission runs, and you get a “public” and a “private” score – at least for the already-over competitions. You can select your best attempts to submit to the private leaderboard if the competition is still active, I think.
Here’s my kaggle notebook where I did all of this.
For image modeling we care about recognizing features across images, even if they’re not always in the same spot. For instance,
whether a dog appears in the top right corner or the bottom left, it is still a dog. This can be handled by processing
convolutions (sliding windows), over small patches of each image. We didn’t implement the sliding window part in the
model-from-scratch lesson, but is baked into ResNet.
Text has different kinds of “spatial” relationships: the order of the words and the sequence of the sentences matter. Because these relationships are more “before/after” and “earlier/later” than “up/down” and “left/right”, they’re considered “temporal” or “sequential” or “ordered”. To train a model for next word prediction, instead of e.g. inputting a randomly ordered list of sentence fragments from our dataset (if sentence fragment ~ image), we send ordered chunks of the document through the model, word by word. Unlike dirs full of dog pics and cat pics, the sentences in a document and the words in a sentence depend on each other. When LSTMs send tokens through sequentially, the model compares its prediction with the actual next token, and this error is backpropagated. It’s like guessing cloud and then baking in “yes that’s a cloud” into the parameters.
This was the tension for NLP models: needing to “remember” sequential relationships between words, including grammars that control words quite far from each other, and pronouns whose meanings carry between sentences, without forgetting long-range context or overfitting. Moreover, LSTM models were slow because sequential relationships are difficult to parallelize over GPUs!
These sequence challenges don’t exist for image classification (though they would if you were using snapshots from a video recording…). They are what motivated researchers to develop the transformer architecture. The LSTM model uses a recurrent hidden state to encode long-distance relationships so information can be carried forward from earlier tokens, since words are processed one at a time through the document. Instead of a recurrent hidden state, the transformer model includes layers whose parameters encode long-distance relationships by calculating how much attention a word should pay to many other words (e.g. every previous word) in a given context window. That “attention” to other words is called “self-attention” (I guess because it’s the view from itself?). Perhaps obvious to note, this “self-attention” approach means the transformer model is doing mannny manyyyy pairwise token calculations, but luckily these can be parallelized.
For the image classification, there were a few benchmark datasets that were developed (e.g. MNIST and ImageNet that we discussed in the last post) and used to evaluate model performance. What did NLP use?
NLP models were historically built to accomplish particular tasks: next word prediction, sentiment classification, machine translation, “named-entity recognition” (NER), part of speech tagging, question answering, inference, semantic similarity. Each of these tasks is associated with particular curated benchmark datasets to evaluate model performance. For next-token prediction, the typical metric is “perplexity”, or how surprised the model is by the correct next token in e.g. the Penn Treebank - a million words of 1989 Wall Street Journal material, WikiText-2 - 2 million tokens from good/featured articles on wikipedia, and WikiText-103 - 103 million tokens datasets.
In 2018, just before transformer models were being released, researchers developed a benchmark called General Language Understanding Evaluation (GLUE) so models could be evaluated on performance over all these independent tasks. Soon after, BERT smashed all the performance records. Researchers responded by developing SuperGLUE to evaluate model performance on more difficult tasks.
]]>This post is part of my Practical Deep Learning journey.
This is my experience with Lesson 3 of the Practical Deep Learning course. If you’re just tuning in, I’m working through this material in a study group while doing a half-batch at The Recurse Center in May/June 2026. See Learning Practical Deep Learning (fastai) for why.
The lessons for Practical Deep Learning are about learning by doing. Doing means writing code to create, run, and analyze the results of models. The authors thought this would be an effective way for students to build intuition about how models trained on real-world data perform on particular tasks, and how results vary when you change qualities of the model and data (architecture, batch sizes, loss function, learning rate, metric, methods for handling missing data, etc.).
The first two lessons use wrapper functions from the fastai library to abstract away the
details of creating models like Microsoft Research’s 18- and 34-layer models, ResNet-18 and ResNet-34
(published in 2015), which were trained on ImageNet-1k (both originally and
currently available in the pytorch library).
The Practical Deep Learning authors wanted to show just how good image classification models had become with ResNet’s
breakthrough with “residual” or “skip” connections in 2015, and what kind of classification accuracy you could get by fine-tuning
general models like ResNet on particular labeled image datasets suited to your own purposes. You can try out my
cloud classifying model (fine-tuned from ResNet-18 using clouds sourced from duck duck go) here,
see the code here, and read a stream of
consciousness blog post of me getting the model into a hugging face repo and website here.
For lesson 3, the walkthrough task was to create a binary classification model for images of handwritten 3s and 7s… from scratch. That is, instead of starting from the ResNet model and fine-tuning the latest layers to transfer learn parameters for a new dataset, we’ll start from scratch with a particular dataset and build a few-layers-deep neural network model where we learn what kind of choices we have to make when setting up parameters, the model architecture, and the epoch training and validation methods.
The data comes from MNIST, a dataset derived from handwritten forms collected by the US National Institute of Standards (NIST)
to prepare for digitizing the 1990 US Census forms. The original data was a mixed set of characters (A-Z, a-z, 0-9) written
by Census Bureau field representatives and high school students from one high school in Bethesda, MD.
From March to May 1992, the First Census OCR Systems Conference held a competition to read these character sets.
In 1991, AT&T Bell Labs had bought National Cash Register (NCR); OCR bank check reading was just around the corner.
From 1992 through 1998, Yann LeCun and his team at AT&T Bell Labs remixed and reprocessed the digit data into the MNIST dataset.
In 1998, MNIST was released and it became a benchmark for OCR models. Most of the machine learning python libraries include
it in a datasets library.
So, that’s the data. The learning objective was to learn how to assemble and make choices about the parts that make up a
deep learning model, some of which might be obfuscated by machine learning library defaults, e.g. in fastai and pytorch,
etc. By showing off what’s under the hood, the authors could make sure students don’t overfit on library particulars when
the goal is to generally learn how to apply deep learning, especially when the state-of-the-art methods are still in rapid
development. To focus on the model, the data is provided, sorted into labeled folders and separated into training, validation,
and test sets.
To ready the images for the model, we had to load each of the N images as an mxn matrix of pixels, convert the pixels
to grayscale and normalize their values (/255), stack them into a tensor, and then convert the mxn matrix of pixels for each
image into a vector of length m*n, so we have a resulting stack of images in a tensor of shape (N, m*n) – N rows and
m*n columns. The labels are 1 for a 3 and 0 for a 7, stored as a tensor of shape (N,) (one value for each row).
After walking through the lesson, it’s left to the reader to make modifications so that the model can classify all digits, 0-9. I ended up explaining how to do both classification tasks, as you’ll see if you keep reading. Notably, only three things need to change to accommodate more classes:
1 (it’s a 3) & 0 (it’s a 7) to the actual number represented in the image, (0-9))In the last section I wrote human language for how to turn the folders of images into training data, but let me just do it in code here because it involves some interesting tensor manipulations.
Here’s a way to make training data & label tensors for the 3s and 7s binary classification case. For each digit folder’s
files, if it’s a digit we care about (3 or 7), stack its images, then concatenate the two stacks. For the labels, store
a 1 for a handwritten three, and a 0 for a handwritten seven.
x_stacks = []
y_stacks = []
label_dict = {3: 1, 7: 0} # we're labeling 1 if it's a 3, and 0 if it's a 7
for folder in sorted((path/'training').ls(), key=lambda x: int(x.name)):
num = int(folder.name)
if num in (3,7):
# images
tensors = [tensor(Image.open(o)) for o in folder.ls().sorted()]
x_stacks.append(torch.stack(tensors).float()/255)
# labels
y_stacks.extend([label_dict[num]] * len(folder.ls()))
xs = torch.cat(x_stacks).view(-1, 28*28) # (12396, 784) - 12396 images, 6131 threes and 6265 sevens each with 784 pixels
ys = tensor(y_stacks) # (12396,)
Here’s a way to make training data & label tensors for the full dataset. For each digit folder’s files, stack all images,
then concatenate the two stacks. For the labels, store a 0 if it’s a zero, 1 if it’s a one, … , 9 if it’s a nine.
x_stacks = []
y_stacks = []
for folder in sorted((path/'training').ls(), key=lambda x: int(x.name)):
# images
tensors = [tensor(Image.open(o)) for o in folder.ls().sorted()]
x_stacks.append(torch.stack(tensors).float()/255)
# labels
y_stacks.extend([int(folder.name)] * len(folder.ls()))
xs = torch.cat(x_stacks).view(-1, 28*28) # (60000, 784) - 60,000 images, each with 784 pixels
ys = tensor(y_stacks) # (60000,)
The toy “deep” neural net has two linear layers with a ReLU (“rectified linear unit”) sandwiched between them.
def model(xb):
res = xb@w1 + b1
res = res.max(tensor(0.0))
res = res@w2 + b2
return res
Unpacking the variables, we have:
xb => the stack of a batch of training images as a tensorw1, b1; w2, b2 => vectors of parameters that we initialize randomly; in the training loop they will get “trained” into different valuesUnpacking the functions, we have:
res = xb@w1 + b1 => this is just y = mx+b, the equation for a line; more on the parameters w1 & b1 later, but the 1st dimension has to match the number of pixels in one image and the 2nd corresponds to our choice of “activations”res = res.max(tensor(0.0)) => this is the ReLU - any negative numbers are clipped to 0; this is how we get the output of a series of linear transformations to be something else than a different linear function with some other choice of parametersres = res@w2 + b2 => this is just y = mx+b again, but the 1st dimension of w2 has to match the activation number we chose before, and the 2nd and b2’s only dim has to match the number of classes we want at the end, e.g. 1 if we’re just guessing 3s or 7s, because res is now comprised of “logits” which we can convert to a probabilities of belonging to a particular classJust highlighting what’s going on here… we’re kind of using matrix math to smoosh all of our pixels into classes. Pretty weird!!!
Training a model basically involves finding fixed values for the model parameters so that when you pass in an image of a
handwritten number 8, your model will confidently tell you “that’s an 8”. Finding effective parameters means looping through
the data and updating the parameters with better values at each the end of each training loop. So what’s in a training loop?
What’s the procedure for finding better values? One method is “stochastic gradient descent” (SGD). Pytorch (and fastai
wrappings) include an implementation of this method in their Optimizer module, but in this lesson we’re learning what’s
under the hood.
So just to keep in your head as we go through the details, the basic steps for training are, after you initialize the parameters:
At the heart of the training loop is the loss function. We will be happy if we calculate a smaller and smaller loss
at the end of each training loop, because the loss for each image is smaller the closer we are to 100% certain of our
prediction of 7 for an image actually labeled 7.
In each run of the training loop, we execute our model on a batch of training data and parameters, and send the results (the tensor of raw output scores (“logits”)) into a “loss function” of our choosing. We can write many different loss functions, but any loss function will receive logits computed from the input data from the model and labels from the training data. It will turn each image’s computed scalar or vector of logits into probabilities that an image belongs to each class. It then calculates the “loss” for each image based on the input tensor of labels. It does this by computing how “far” the predicted label was from the correct label. We can’t use a function that just says “right!” or “wrong!” for a given labeled image because that doesn’t help us figure out what direction to modify the model parameters to make better, or at least less wrong, predictions in the next loop.
Small loss means that we were confident and correct in our predictions; large loss means we were either not confident in our predictions, or confidently wrong.
The loss for a particular batch is typically one number: the mean loss over all the images.
For the case where we’re just classifying images as 3 and 7, where 3 is labeled 1 and 7 is labeled 0. yb is a tensor
that contains the label/target/class for each image. logits is the model output – each image’s vector or scalar representing
something that can be turned into a probability or confidence in the predicted label as part of the loss function.
def mnist_loss(model_logits, yb):
probs = model_logits.sigmoid()
return torch.where(yb==1, 1-probs, probs).mean()
loss = mnist_loss(logits, yb)
In this case, loss for all images is calculated in the following steps:
sigmoid(x) = 1 / (1 + e^(-x)) confines the one logit for each image from whatever positive or negative number it is into a value between 0 and 1 corresponding to a probability of being a 3 (closer to 1) or a 7 (closer to 0).1, it calculates how far from 1 the prediction is; if labeled 0, it calculates how far from 0 the prediction isIf we want to calculate loss when there are many classes (e.g. if we want to do the full dataset of [0,1,2,3,4,5,6,7,8,9]), we can use the cross-entropy function.
import torch.nn.functional as F
loss = F.cross_entropy(model_logits, yb)
The cross entropy function:
softmax(xᵢ) = e^xᵢ / (e^x₁ + e^x₂ + ... + e^xₙ)) instead of sigmoid to squash each image’s vector of logits into probabilities that all sum to 1.-log(softmax(x))); we only care about that oneIt uses -log(p) instead of a simple 1-p subtraction for loss so that the optimizer has a larger range of values to work with.
When the probability decreases towards zero the loss can grow very large to indicate that a large change must be made.
When the loss is just 1-p, it’s bound between 0 and 1, which doesn’t distinguish well between “slightly wrong” and “very wrong”.
Calculating the loss just tells us how good or bad we were, on average, at predicting labels. We still need some way to learn
from this loss how to change the parameters (the matrix of weights w1, w2 and vectors of biases b1, b2) for the
next training loop.
How do we do that? Instead of using the loss value itself, we evaluate the slope of the loss function at each parameter’s current value. The slope determines how much and in which direction we should modify the parameters to minimize the loss for next time. We get the slope at a given point (parameter value) by taking the derivative of the function and evaluating it at that point. Because our loss function depends on multiple parameters (four in our case), we take partial derivatives with respect to each parameter. All of these partial derivatives taken together are called the gradient. Hence, the “gradient descent” part of “stochastic gradient descent”.
The parameters have to be initialized with random values before we run any training loops, and they have to be updated at the end of each training loop. How do we do this in practice?
The initialization of the parameters is straightforward:
def init_params(size, std=1.0):
return (torch.randn(size)*std).requires_grad_()
The most complicated part is .requires_grad_(). This is a pytorch method responsible for making the gradient calculation
from the loss function possible.
When a tensor is updated with .requires_grad_(), it gains the attributes .is_leaf=True and .grad=None. When this
tensor first participates in a calculation, the result is given two things:
.grad_fn attribute which is the gradient function of the last operation that produced it (e.g. AddBackward0, MulBackward0, PowBackward0, SigmoidBackward0, ReluBackward0, LogSoftmaxBackward0, NllLossBackward0 (negative log likelihood), etc.),.grad_fn.next_functions attribute which points to the leaf node (AccumulateGrad).When the result participates in another calculation, the next result also gets a .grad_fn which is the gradient function
of the last function that produced it, and a .grad_fn.next_functions which points to the gradient functions from the
previous result rather than directly to the leaf node. And so on.
When .backward() is called on a result down the line, the gradients will be calculated and assembled by moving backwards
up the chain of functions, with chain rule, accumulating the results in place into the .grad attribute of the tensors
originally set up with .requires_grad_() (our parameters). As each gradient calculation occurs, the tensor’s .grad_fn
stored computation buffers are released from memory. What’s left is the value in .grad for the parameters, the same shape as the parameter itself,
which we can use to update the weights. We’ll talk about how in a bit. Once we do that we’ll need to manually clear out
the gradient with .grad.zero_() to prepare for the next training loop.
Although the parameter values are initialized randomly, the shape of each parameter is dictated by your data. In particular,
w1’s 1st dimension is the pixels per image in your datasetw2’s 2nd dimension is the number of output classesw1’s 2nd dimension = w2’s 1st dimension so the matrix math works out. This is also the size for b1. This number, sometimes called the number of “activations”, is something you choose.It’s easy to see in examples.
This would be param initialization when labels are either 3 or 7 (e.g. just one output class, “is 3?”):
w1 = init_params((28*28,30))
b1 = init_params(30)
w2 = init_params((30,1))
b2 = init_params(1)
This would be param initialization when images can be labeled any of the 10 digits from 0 to 9:
w1 = init_params((28*28,30))
b1 = init_params(30)
w2 = init_params((30,10))
b2 = init_params(10)
The last thing we haven’t discussed for updating the parameters in the training loop is the “learning rate” (lr). We calculated
each parameter’s gradient and stored it in .grad, but this number just tells us a quantity and direction to change
the old parameter value. That quantity is usually a lot bigger than we actually want to step the size of the parameter.
We’d likely overshoot the correct value, and have to move it all the way back in the other direction, maybe never minimizing
the loss!
Enter: the “learning rate”. This is a number that we can choose, usually pretty small, like 1e-3 (.001) or 1e-4 (.0001). We multiply the gradient by the learning rate, and subtract it from the parameter value. If we choose a learning rate that is too small, each training loop change to our parameter values will be so small that it will take forever to train our model. If we choose a learning rate that is too large, we might step the value of parameters too far and miss the best value.
Here we set the .data attribute of the parameter to update its value without adding the calculation to a gradient function graph,
and then zero out the gradient to prepare it for the next training loop. This stuff would be written into the optimizer
class if we used one instead of doing all of this manually.
w1.data -= w1.grad * lr
w1.grad.zero_()
Ultimately we probably want to start our training loops with a higher learning rate and then systematically reduce it
as we go through more and more training loops, because we’re probably coming up on the optimal values for the parameters.
The machine learning libraries all have standard methods included in their optimizers for doing this. Fastai also includes
a method .lr_find() to choose an initial learning rate by running training loop with a bunch of different learning
rates.
Ok so, we have:
Here’s what we’ve got:
params = [w1, b1, w2, b2] # initialized parameters
def train_epoch(model, lr, params):
# loop through each batch of the training data in dl (DataLoaders)
for xb,yb in dl:
# run the model, calculate the loss
model_logits = model(xb) # model uses the params we initialized
loss = F.cross_entropy(model_logits, yb) # or mnist_loss(preds, yb)
loss.backward() # this sets .grad in our params
# update the parameters
for p in params:
p.data -= p.grad*lr
p.grad.zero_() # this zeros out .grad for the param
print("Training Loss: ", loss)
In the example notebooks for the course, the authors always print out the training loss, validation loss, and accuracy for each epoch. We have calculated the training loss here, but what of the validation loss? And what is the “accuracy”?
Reiterating from the last section, we have now written code to train a model. But our only sense of how well it’s performing is from calculating the loss on the training data with every loop. We expect that it will perform similarly on the validation set, because typically the validation data is just some random subset of the data that we kept aside specifically for validation. E.g. if we’re predicting categorical labels, we expect future images of handwritten digits to have a pretty similar distribution to the present (e.g. people don’t slowly over time start tilting their 8s until they eventually go sideways ∞). That’s not always the case; we might want to manually make a validation set, particularly if things are changing over time.
Either way, we should check validation loss because it’s our first proxy for how well the model will perform on real world data. We’re only updating the model based on the training loss, so if the validation loss is not also going down, or if it’s increasing between epochs, that’s a sure sign that we’re overfitting on the training data. That means the model is less able to generalize on new data that it hasn’t seen before. Usually the point of training a model is so it will do well on data it hasn’t seen before, because we’ve already spent the time labeling the data we’re training it on.
To do this, we can send the validation data through the model whose parameters we’ve just trained, and compute the loss.
This is the same as what we did for the training data, with the caveat that we don’t want to build the computation graph
(save the gradient functions) and we don’t want to call loss.backward() to compute gradients for the parameters. We can
do this with torch.no_grad(). We’re not going to change anything about the model from this loop, we’re just checking on
the loss.
We also don’t have to bother batching up the images into random groups for this epoch because we’re not looking to make
improvements to the parameters, we just want the average loss for the total run anyway. We can still use batches though,
if our validation dataset is too big to send into memory as one big tensor. Here we pretend we made a valid_dl object,
a DataLoaders for the validation data with a bunch of xb image tensors and yb label vectors.
def validate_epoch(model):
# no gradient tracking
with torch.no_grad():
val_losses = [F.cross_entropy(model(xb), yb) for xb, yb in valid_dl] # or mnist_loss()
val_loss_mean = torch.stack(val_losses).mean()
print("Validation Loss: ", val_loss_mean)
Even more important than checking the validation loss is the metric that you care to check how your model performs in the real world. Smaller loss will tell us that we were more confident and correct with our predicted labels, but a metric will tell us something more like we correctly labeled 24 out of 30 images (accuracy). It’s possible to have a loss that implies that we were not confident with most of our labels, e.g. maybe like 51% confidence, but still find that we actually guessed correct labels for like 90% of the validation set.
For Kaggle competitions, the metric is provided. It could be accuracy, F1, AUC,
etc. Your success is based on how well your model performs as evaluated by the metric. For the MNIST problem we’ve been
working on here, we’ll define a batch accuracy metric for each of the classification tasks (3 vs 7 and the full set
of digits, 0-9).
For the binary case, 3 vs 7:
def batch_accuracy_binary(model_logits, yb):
preds = model_logits.sigmoid() # the value is clipped between 0 and 1
correct = (preds>0.5) == yb # If the value is >.5, it's labeled 1, which "is_3"
return correct.float().mean() # fraction correct in this batch
For the multiclass case, full set of digits, 0-9:
def batch_accuracy_multiclass(model_logits, yb):
return (model_logits.argmax(dim=1) == yb).float().mean() # argmax because we guess the label with the highest probability
We can add the batch_accuracy calculation to the validate_epoch function:
def validate_epoch(model):
with torch.no_grad():
val_losses = [F.cross_entropy(model(xb), yb) for xb, yb in valid_dl] # or mnist_loss()
val_loss_mean = torch.stack(val_losses).mean()
val_accs = [batch_accuracy_multiclass(model(xb), yb) for xb, yb in valid_dl] # or the _binary() option
val_accs_mean = torch.stack(val_accs).mean()
print("Validation Loss: ", val_loss_mean,
"Validation Accuracy :", val_accs_mean)
And then we can wrap all of this up in a loop where we choose how many epochs (full run through all the data) we want to train for. The more epochs, the longer the training process will take:
def train_and_validate_model(model, lr, n_epochs):
params = init_all_params() # set the w1, b1, w2, b2 with init_params() for the proper sizes (see above)
for epoch in range(n_epochs):
train_epoch(model, lr, params) # prints training loss
validate_epoch(model) # prints validation loss & validation accuracy
Ok that was way longer and more involved than I intended to get in this post. Do you … want to see how this model performs on some handwritten images now?
Once you train your parameters, you might want to hang onto them! Those parameters, together with the model function, are what you use to guess the 0-9 digit in new images of handwritten digits.
If you want to do this one at a time, you can reshape your mxn-sized image (actually we’ve been working with square 28x28
images this whole time) into a tensor shaped like (1, m*n): an image_tensor with its m*n pixels all in a row:
from fastai.vision.all import *
# assuming our image is 28x28 pixels like all of our training data
# image gymnastics -- we had to do a similar version of this at the beginning to make our huge stack of images
image = Image.open('my_digit.png').convert('L') # L = convert to grayscale
image_tensor = tensor(image).float()/255 # (28, 28); /255 for normalization
image_tensor = image_tensor.flatten().unsqueeze(0) # (28, 28) → (784,) → (1, 784)
# image gymnastics note -- the "tensor" in the lines above (and in other places in this post) is fastai's implementation
# can also use torch's tensor together with numpy: torch.tensor(np.array(image))
# same method for getting the predicted label as in batch_accuracy, but no comparison with a label because we don't have it!
with torch.no_grad():
model_logits = model(image_tensor)
predicted_class = model_logits.argmax(dim=1) # index of the highest probability
print("The digit is: ", predicted_class.item())
This is a stream of consciousness version of my process for getting the cloud model i trained online.
TL;DR try out the model I fine-tuned for classifying clouds at Deep Learning Learning space on huggingface.
According to lesson 2, the latest and greatest way to put models on the web is no longer jupyter with binder, but now hugging face and gradio.
Making a huggingface space with gradio was very straightforward. I followed the steps in the blog that the lesson2 video suggested.
$something into a box that you can submit, and it appears in a second column as “Hello, $something!!”For the sake of time, I will move fast and not break things. Instead of adding the model training code to that repo
and then having to figure out dependencies right now, i’m just going to copy my export.pkl that I created in
the lesson2 jupyter notebook into the huggingface repo that I just made. If you want some example images to show up, you also have to copy-paste those in.
Um. Well that didn’t work.
remote: -------------------------------------------------------------------------
remote: Your push was rejected because it contains files larger than 10 MiB.
remote: Please use https://git-lfs.github.com/ to store large files.
remote: See also: https://hf.co/docs/hub/repositories-getting-started#terminal
remote:
remote: Offending files:
remote: - export.pkl (ref: refs/heads/main)
remote: -------------------------------------------------------------------------
In the past I spent some time with git-lfs and the question of what to do with large files in repos so … I really should have known better. But the tutorial said use the pickle!!!! If the idea is to train models in this repo though, huggingface probably has something to say about it.
Oh yeah, they do: “Do you have files larger than 10MB? Those files should be tracked with git-xet, which you can initialize with:
git xet install”. Never heard of it. Hello 2026 and the rock I’ve been living under for the past six years.
But it looks like they’ve also moved on from pickles to something called safetensors.
It’s not like huggingface is incompatible with fastai.
Just going to try the simplest thing which is to install git-lfs for now and upload the pickle. I trained it one more time after cleaning the data and the error improved by an order of magnitude (12% to 1%). But for some reason when I looked at the new losses:
interp_clean = ClassificationInterpretation.from_learner(learn_clean)
interp_clean.plot_confusion_matrix()
interp_clean.plot_top_losses(5,nrows=5)
…there were still images in there that looked like the ones I got rid of with the “cleaner”:
cleaner = ImageClassifierCleaner(learn)
cleaner
for idx in cleaner.delete(): cleaner.fns[idx].unlink()
for idx,cat in cleaner.change(): shutil.move(str(cleaner.fns[idx]), path/cat)
After I had cleaned I made the datablock and path variables again, and results were better… so why were the wrong images still in there? Did the clean wizard just not find all of the bad images? I mean I could look through the images to verify this but not right now.
No, of course not.
requirements.txt but doesn’t mention what they already have installed? (e.g. they must have already installed gradio). Got this from the tutorial’s imports (fastai, and scikit-image). I put skimage in the requirements.txt file first and yeah I got an error when I git pushed.gr.Interface(). I just copy-pasted the failure log messages from the hf site gui into chatgpt to figure out what the changes should be. I tried to ask Claude first but it was like, “honestly, I might not be up to date with the latest…” blah blah blah ok Claude thanks for expressing your limitations.I should have probably run this locally before pushing to hf but I didn’t set up a .venv (and added a .gitignore) until a few errors in.
Luckily the hf code tree has a great interface for checking the status & the logs!
Anyway to run locally you just have to start the venv and then call python app.py but fyi for the sake of time I didn’t try that yet.
Because now my cloud classifier is live! I tried it and it works! Check it out here!
]]>I’m working through the “fastai” Practical Deep Learning for Coders course with a study group while doing a half-batch at The Recurse Center in May/June 2026. Lesson 2 suggested blogging the learning process. That’s my cue to set up github pages.
Practical Deep Learning for Coders is a free online course built by Jeremy Howard and Rachel Thomas “with the goal of democratizing deep learning”. It was first run in 2016 as a certificate course at the University of San Francisco, and then became a MOOC. The initial course explored the Keras/Theano/TensorFlow libraries, but soon dropped them in favor of pytorch.
When they created the course, democratizing deep learning meant:
In 2018, the authors published a wrapper/utility library around pytorch called “fastai” which handles various data pre-processing among other things. The course materials teach the principles of deep learning through this library and pytorch. They also peel back the layers of the libraries to build deep learning models from scratch.
In 2019, the University of San Francisco launched the Center for Applied Data Ethics (CADE) - rebranded in 2023 as Center for AI & Data Ethics (CAIDE) initially directed by Rachel Thomas.
In 2020, the authors published a paper on the fastai library and released the fastai book, Deep Learning for Coders with Fastai and PyTorch: AI Applications Without a PhD.
In 2022, the authors responded to transformers hitting the scene by incorporating them into the 2022 lectures and course materials - course22, recorded by Jeremy Howard at the University of Queensland in Brisbane, Australia. See lesson 4 on NLP with a link to the patent data kaggle notebook with an intro to hugging face.
In 2023-2024, the authors responded to Stable Diffusion by making a Part 2 to the course with lecture recordings, which goes through building Stable Diffusion from scratch.
It’s 2026 and fastai has had many new blog posts but no new commits to fastbook or the course materials in a bit.
The lessons are available at https://course.fast.ai/.
Each lesson link includes:
There seem to be two paths for a researcher:
On the way:
What does that have to do with choosing this course? I’m a domain-oriented person too curious to sit still in just one domain. This has led me to cultivate a techniques orientation so that I can move around and satisfy my curiosity. The Practical Deep Learning course promises to teach techniques that can be applied by including datasets from different domains to process with each lesson. The intended audience is domain experts wanting to try out deep learning for their domains, and data techniques people collaborating with domain experts. I fall mostly into the latter category.
By following this course, I hope to gain some understanding of the kinds of problems people expect to be able to solve with deep learning, and intuition about the limitations of these kinds of models. Are there domains of research that have been swiftly adopting deep learning to the detriment of other (slower?) approaches to knowledge production that would aim at answering different but also good questions?
Moreover, the full title of the book is “Deep Learning for Coders with Fastai and PyTorch: AI Applications Without a PhD”. I am a coder, I want to learn how to apply AI, and don’t have a PhD.
There are mannnny deep learning and other machine learning courses and materials to follow today. Why did I choose this one?