Predicting Bulldozer Auction Sale Prices

Kaggle Competition: https://www.kaggle.com/competitions/bluebook-for-bulldozers/overview

import pandas as pd
import numpy as np

pd.__version__

'2.3.3'

# requirements.txt
# - pooch # (download data from the internet)

!pip install pooch

Requirement already satisfied: pooch in /Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages (1.9.0)
Requirement already satisfied: platformdirs>=2.5.0 in /Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages (from pooch) (4.10.0)
Requirement already satisfied: packaging>=20.0 in /Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages (from pooch) (26.2)
Requirement already satisfied: requests>=2.19.0 in /Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages (from pooch) (2.34.2)
Requirement already satisfied: charset_normalizer<4,>=2 in /Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages (from requests>=2.19.0->pooch) (3.4.9)
Requirement already satisfied: idna<4,>=2.5 in /Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages (from requests>=2.19.0->pooch) (3.18)
Requirement already satisfied: urllib3<3,>=1.26 in /Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages (from requests>=2.19.0->pooch) (2.7.0)
Requirement already satisfied: certifi>=2023.5.7 in /Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages (from requests>=2.19.0->pooch) (2026.6.17)

Getting the Data - it's not on Kaggle anymore

https://github.com/mrdbourke/zero-to-mastery-ml/blob/master/data/bluebook-for-bulldozers.zip

#.cache/manualdl/datasets

# hilariously long data extraction script using 'pooch' library
# courtesy of ChatGPT

from pathlib import Path
from urllib.parse import urlparse
import shutil
import zipfile

import pooch


def fetch_zipped_dataset(
    url: str,
    project_root: str | Path = ".",
    cache_root: str | Path | None = None,
    known_hash: str | None = None,
) -> Path:
    """
    Download a ZIP file into a shared cache, extract it, collapse redundant
    same-named wrapper directories, and copy the dataset files into a single
    top-level folder in the current project.

    Example archive name:
        bluebook-for-bulldozers.zip

    Cache layout:
        ~/.cache/datasets/
            bluebook-for-bulldozers/
                bluebook-for-bulldozers.zip
                bluebook-for-bulldozers/
                    <dataset files>

    Project layout:
        ./bluebook-for-bulldozers/
            <dataset files>

    Parameters
    ----------
    url:
        Direct URL to the ZIP archive.

    project_root:
        Directory in which the project dataset folder should be created.
        Defaults to the current working directory.

    cache_root:
        Shared cache root. Defaults to ~/.cache/datasets.

    known_hash:
        Optional Pooch hash, such as "sha256:<hash>".
        Use None to skip hash validation.

    Returns
    -------
    Path
        Path to the dataset folder in the current project.
    """

    project_root = Path(project_root).expanduser().resolve()

    if cache_root is None:
        cache_root = Path.home() / ".cache" / "datasets"
    else:
        cache_root = Path(cache_root).expanduser().resolve()

    # Get the archive filename from the URL.
    archive_name = Path(urlparse(url).path).name

    if not archive_name.lower().endswith(".zip"):
        raise ValueError(f"URL does not appear to point to a ZIP file: {url}")

    # "bluebook-for-bulldozers.zip" -> "bluebook-for-bulldozers"
    dataset_name = Path(archive_name).stem

    # Cache everything for this dataset inside one named directory.
    dataset_cache_dir = cache_root / dataset_name
    dataset_cache_dir.mkdir(parents=True, exist_ok=True)

    zip_path = Path(
        pooch.retrieve(
            url=url,
            known_hash=known_hash,
            path=dataset_cache_dir,
            fname=archive_name,
        )
    )

    # Extract into a folder with the archive's stem.
    extracted_dir = dataset_cache_dir / dataset_name

    if not extracted_dir.exists():
        extracted_dir.mkdir(parents=True)

        with zipfile.ZipFile(zip_path) as archive:
            archive.extractall(extracted_dir)

    # Some ZIP files contain another same-named folder:
    #
    # extracted_dir/
    #     bluebook-for-bulldozers/
    #         bluebook-for-bulldozers/
    #             files...
    #
    # Walk downward through repeated same-named wrapper directories.
    source_dir = extracted_dir

    while True:
        children = [
            child
            for child in source_dir.iterdir()
            if child.name != "__MACOSX"
        ]

        same_named_child = source_dir / dataset_name

        if (
            len(children) == 1
            and same_named_child.is_dir()
            and children[0] == same_named_child
        ):
            source_dir = same_named_child
        else:
            break

    # Create exactly one top-level project directory.
    project_dataset_dir = project_root / dataset_name
    project_dataset_dir.mkdir(parents=True, exist_ok=True)

    # Copy the contents of the innermost useful directory—not the directory
    # itself—so no redundant nesting is introduced.
    for item in source_dir.iterdir():
        if item.name == "__MACOSX":
            continue

        destination = project_dataset_dir / item.name

        if item.is_dir():
            shutil.copytree(
                item,
                destination,
                dirs_exist_ok=True,
            )
        else:
            shutil.copy2(item, destination)

    return project_dataset_dir


# ---------------------------------------------------------------------
# Download and prepare the Blue Book for Bulldozers dataset
# ---------------------------------------------------------------------

DATA = fetch_zipped_dataset(
    url=(
        "https://github.com/mrdbourke/zero-to-mastery-ml/"
        "raw/master/data/bluebook-for-bulldozers.zip"
    )
)

print(f"Dataset available at:\n{DATA}")
print("\nFiles:")
for path in sorted(DATA.iterdir()):
    print(f"  {path.name}")

Dataset available at:
/Users/emma/Documents/file_cabinet/people_projects/emma/deeplearning/fastai/fastbook/bluebook-for-bulldozers

Files:
  Data Dictionary.xlsx
  Machine_Appendix.csv
  Test.csv
  Train.7z
  Train.csv
  Train.zip
  TrainAndValid.7z
  TrainAndValid.csv
  TrainAndValid.zip
  Valid.7z
  Valid.csv
  Valid.zip
  ValidSolution.csv
  median_benchmark.csv
  random_forest_benchmark_test.csv
  test_predictions.csv
  to_bulldozers.pkl
  train_tmp.csv
  valid_xs_final.pkl
  xs_final.pkl

What data do we have?

path = Path('./bluebook-for-bulldozers')

df_machine_appendix = pd.read_csv(path/'Machine_Appendix.csv')
df_machine_appendix.head()

	MachineID	ModelID	fiModelDesc	fiBaseModel	fiSecondaryDesc	fiModelSeries	fiModelDescriptor	fiProductClassDesc	ProductGroup	ProductGroupDesc	MfgYear	fiManufacturerID	fiManufacturerDesc	PrimarySizeBasis	PrimaryLower	PrimaryUpper
0	113	1355	350L	350	NaN	NaN	L	Hydraulic Excavator, Track - 50.0 to 66.0 Metr...	TEX	Track Excavators	1994.0	26	Caterpillar	Weight - Metric Tons	50.0	66.0
1	434	3538	416C	416	C	NaN	NaN	Backhoe Loader - 14.0 to 15.0 Ft Standard Digg...	BL	Backhoe Loaders	1997.0	26	Caterpillar	Standard Digging Depth - Ft	14.0	15.0
2	534	3538	416C	416	C	NaN	NaN	Backhoe Loader - 14.0 to 15.0 Ft Standard Digg...	BL	Backhoe Loaders	1998.0	26	Caterpillar	Standard Digging Depth - Ft	14.0	15.0
3	718	3538	416C	416	C	NaN	NaN	Backhoe Loader - 14.0 to 15.0 Ft Standard Digg...	BL	Backhoe Loaders	2000.0	26	Caterpillar	Standard Digging Depth - Ft	14.0	15.0
4	1753	1580	D5GLGP	D5	G	NaN	LGP	Track Type Tractor, Dozer - 85.0 to 105.0 Hors...	TTT	Track Type Tractors	2006.0	26	Caterpillar	Horsepower	85.0	105.0

df_machine_appendix.columns

Index(['MachineID', 'ModelID', 'fiModelDesc', 'fiBaseModel', 'fiSecondaryDesc',
       'fiModelSeries', 'fiModelDescriptor', 'fiProductClassDesc',
       'ProductGroup', 'ProductGroupDesc', 'MfgYear', 'fiManufacturerID',
       'fiManufacturerDesc', 'PrimarySizeBasis', 'PrimaryLower',
       'PrimaryUpper'],
      dtype='object')

df_train = pd.read_csv(path/'Train.csv', low_memory=False)
df_train_and_valid = pd.read_csv(path/'TrainAndValid.csv', low_memory=False)
df_test = pd.read_csv(path/'Test.csv', low_memory=False)

df_train # 401,125 rows × 53 columns

	SalesID	SalePrice	MachineID	ModelID	datasource	auctioneerID	YearMade	MachineHoursCurrentMeter	UsageBand	saledate	...	Undercarriage_Pad_Width	Stick_Length	Thumb	Pattern_Changer	Grouser_Type	Backhoe_Mounting	Blade_Type	Travel_Controls	Differential_Type	Steering_Controls
0	1139246	66000	999089	3157	121	3.0	2004	68.0	Low	11/16/2006 0:00	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	Standard	Conventional
1	1139248	57000	117657	77	121	3.0	1996	4640.0	Low	3/26/2004 0:00	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	Standard	Conventional
2	1139249	10000	434808	7009	121	3.0	2001	2838.0	High	2/26/2004 0:00	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
3	1139251	38500	1026470	332	121	3.0	2001	3486.0	High	5/19/2011 0:00	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
4	1139253	11000	1057373	17311	121	3.0	2007	722.0	Medium	7/23/2009 0:00	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
401120	6333336	10500	1840702	21439	149	1.0	2005	NaN	NaN	11/2/2011 0:00	...	None or Unspecified	None or Unspecified	None or Unspecified	None or Unspecified	Double	NaN	NaN	NaN	NaN	NaN
401121	6333337	11000	1830472	21439	149	1.0	2005	NaN	NaN	11/2/2011 0:00	...	None or Unspecified	None or Unspecified	None or Unspecified	None or Unspecified	Double	NaN	NaN	NaN	NaN	NaN
401122	6333338	11500	1887659	21439	149	1.0	2005	NaN	NaN	11/2/2011 0:00	...	None or Unspecified	None or Unspecified	None or Unspecified	None or Unspecified	Double	NaN	NaN	NaN	NaN	NaN
401123	6333341	9000	1903570	21435	149	2.0	2005	NaN	NaN	10/25/2011 0:00	...	None or Unspecified	None or Unspecified	None or Unspecified	None or Unspecified	Double	NaN	NaN	NaN	NaN	NaN
401124	6333342	7750	1926965	21435	149	2.0	2005	NaN	NaN	10/25/2011 0:00	...	None or Unspecified	None or Unspecified	None or Unspecified	None or Unspecified	Double	NaN	NaN	NaN	NaN	NaN

401125 rows × 53 columns

df_train_and_valid # 412,698 rows × 53 columns

	SalesID	SalePrice	MachineID	ModelID	datasource	auctioneerID	YearMade	MachineHoursCurrentMeter	UsageBand	saledate	...	Undercarriage_Pad_Width	Stick_Length	Thumb	Pattern_Changer	Grouser_Type	Backhoe_Mounting	Blade_Type	Travel_Controls	Differential_Type	Steering_Controls
0	1139246	66000.0	999089	3157	121	3.0	2004	68.0	Low	11/16/2006 0:00	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	Standard	Conventional
1	1139248	57000.0	117657	77	121	3.0	1996	4640.0	Low	3/26/2004 0:00	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	Standard	Conventional
2	1139249	10000.0	434808	7009	121	3.0	2001	2838.0	High	2/26/2004 0:00	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
3	1139251	38500.0	1026470	332	121	3.0	2001	3486.0	High	5/19/2011 0:00	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
4	1139253	11000.0	1057373	17311	121	3.0	2007	722.0	Medium	7/23/2009 0:00	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
412693	6333344	10000.0	1919201	21435	149	2.0	2005	NaN	NaN	3/7/2012 0:00	...	None or Unspecified	None or Unspecified	None or Unspecified	None or Unspecified	Double	NaN	NaN	NaN	NaN	NaN
412694	6333345	10500.0	1882122	21436	149	2.0	2005	NaN	NaN	1/28/2012 0:00	...	None or Unspecified	None or Unspecified	None or Unspecified	None or Unspecified	Double	NaN	NaN	NaN	NaN	NaN
412695	6333347	12500.0	1944213	21435	149	2.0	2005	NaN	NaN	1/28/2012 0:00	...	None or Unspecified	None or Unspecified	None or Unspecified	None or Unspecified	Double	NaN	NaN	NaN	NaN	NaN
412696	6333348	10000.0	1794518	21435	149	2.0	2006	NaN	NaN	3/7/2012 0:00	...	None or Unspecified	None or Unspecified	None or Unspecified	None or Unspecified	Double	NaN	NaN	NaN	NaN	NaN
412697	6333349	13000.0	1944743	21436	149	2.0	2006	NaN	NaN	1/28/2012 0:00	...	None or Unspecified	None or Unspecified	None or Unspecified	None or Unspecified	Double	NaN	NaN	NaN	NaN	NaN

412698 rows × 53 columns

# df_train.columns # df_test is just missing SalePrice
# Index(['SalesID', 'SalePrice', 'MachineID', 'ModelID', 'datasource',
#        'auctioneerID', 'YearMade', 'MachineHoursCurrentMeter', 'UsageBand',
#        'saledate', 'fiModelDesc', 'fiBaseModel', 'fiSecondaryDesc',
#        'fiModelSeries', 'fiModelDescriptor', 'ProductSize',
#        'fiProductClassDesc', 'state', 'ProductGroup', 'ProductGroupDesc',
#        'Drive_System', 'Enclosure', 'Forks', 'Pad_Type', 'Ride_Control',
#        'Stick', 'Transmission', 'Turbocharged', 'Blade_Extension',
#        'Blade_Width', 'Enclosure_Type', 'Engine_Horsepower', 'Hydraulics',
#        'Pushblock', 'Ripper', 'Scarifier', 'Tip_Control', 'Tire_Size',
#        'Coupler', 'Coupler_System', 'Grouser_Tracks', 'Hydraulics_Flow',
#        'Track_Type', 'Undercarriage_Pad_Width', 'Stick_Length', 'Thumb',
#        'Pattern_Changer', 'Grouser_Type', 'Backhoe_Mounting', 'Blade_Type',
#        'Travel_Controls', 'Differential_Type', 'Steering_Controls'],
#       dtype='object')

df_test # 12457 rows × 52 columns

	SalesID	MachineID	ModelID	datasource	auctioneerID	YearMade	MachineHoursCurrentMeter	UsageBand	saledate	fiModelDesc	...	Undercarriage_Pad_Width	Stick_Length	Thumb	Pattern_Changer	Grouser_Type	Backhoe_Mounting	Blade_Type	Travel_Controls	Differential_Type	Steering_Controls
0	1227829	1006309	3168	121	3	1999	3688.0	Low	5/3/2012 0:00	580G	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
1	1227844	1022817	7271	121	3	1000	28555.0	High	5/10/2012 0:00	936	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	Standard	Conventional
2	1227847	1031560	22805	121	3	2004	6038.0	Medium	5/10/2012 0:00	EC210BLC	...	None or Unspecified	9' 6"	Manual	None or Unspecified	Double	NaN	NaN	NaN	NaN	NaN
3	1227848	56204	1269	121	3	2006	8940.0	High	5/10/2012 0:00	330CL	...	None or Unspecified	None or Unspecified	Manual	Yes	Triple	NaN	NaN	NaN	NaN	NaN
4	1227863	1053887	22312	121	3	2005	2286.0	Low	5/10/2012 0:00	650K	...	NaN	NaN	NaN	NaN	NaN	None or Unspecified	PAT	None or Unspecified	NaN	NaN
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
12452	6643171	2558317	21450	149	2	2008	NaN	NaN	10/24/2012 0:00	80NX3	...	None or Unspecified	None or Unspecified	None or Unspecified	None or Unspecified	Double	NaN	NaN	NaN	NaN	NaN
12453	6643173	2558332	21434	149	2	2005	NaN	NaN	10/24/2012 0:00	28N	...	None or Unspecified	None or Unspecified	None or Unspecified	None or Unspecified	Double	NaN	NaN	NaN	NaN	NaN
12454	6643184	2558342	21437	149	2	1000	NaN	NaN	10/24/2012 0:00	35N	...	None or Unspecified	None or Unspecified	None or Unspecified	None or Unspecified	Double	NaN	NaN	NaN	NaN	NaN
12455	6643186	2558343	21437	149	2	2006	NaN	NaN	10/24/2012 0:00	35N	...	None or Unspecified	None or Unspecified	None or Unspecified	None or Unspecified	Double	NaN	NaN	NaN	NaN	NaN
12456	6643196	2558346	21446	149	2	2008	NaN	NaN	9/19/2012 0:00	55N2	...	None or Unspecified	None or Unspecified	None or Unspecified	None or Unspecified	Double	NaN	NaN	NaN	NaN	NaN

12457 rows × 52 columns

df_train # 53 columns, 401125 rows

	SalesID	SalePrice	MachineID	ModelID	datasource	auctioneerID	YearMade	MachineHoursCurrentMeter	UsageBand	saledate	...	Undercarriage_Pad_Width	Stick_Length	Thumb	Pattern_Changer	Grouser_Type	Backhoe_Mounting	Blade_Type	Travel_Controls	Differential_Type	Steering_Controls
0	1139246	66000	999089	3157	121	3.0	2004	68.0	Low	11/16/2006 0:00	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	Standard	Conventional
1	1139248	57000	117657	77	121	3.0	1996	4640.0	Low	3/26/2004 0:00	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	Standard	Conventional
2	1139249	10000	434808	7009	121	3.0	2001	2838.0	High	2/26/2004 0:00	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
3	1139251	38500	1026470	332	121	3.0	2001	3486.0	High	5/19/2011 0:00	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
4	1139253	11000	1057373	17311	121	3.0	2007	722.0	Medium	7/23/2009 0:00	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
401120	6333336	10500	1840702	21439	149	1.0	2005	NaN	NaN	11/2/2011 0:00	...	None or Unspecified	None or Unspecified	None or Unspecified	None or Unspecified	Double	NaN	NaN	NaN	NaN	NaN
401121	6333337	11000	1830472	21439	149	1.0	2005	NaN	NaN	11/2/2011 0:00	...	None or Unspecified	None or Unspecified	None or Unspecified	None or Unspecified	Double	NaN	NaN	NaN	NaN	NaN
401122	6333338	11500	1887659	21439	149	1.0	2005	NaN	NaN	11/2/2011 0:00	...	None or Unspecified	None or Unspecified	None or Unspecified	None or Unspecified	Double	NaN	NaN	NaN	NaN	NaN
401123	6333341	9000	1903570	21435	149	2.0	2005	NaN	NaN	10/25/2011 0:00	...	None or Unspecified	None or Unspecified	None or Unspecified	None or Unspecified	Double	NaN	NaN	NaN	NaN	NaN
401124	6333342	7750	1926965	21435	149	2.0	2005	NaN	NaN	10/25/2011 0:00	...	None or Unspecified	None or Unspecified	None or Unspecified	None or Unspecified	Double	NaN	NaN	NaN	NaN	NaN

401125 rows × 53 columns

df_train_and_valid.columns # 53 columns

Index(['SalesID', 'SalePrice', 'MachineID', 'ModelID', 'datasource',
       'auctioneerID', 'YearMade', 'MachineHoursCurrentMeter', 'UsageBand',
       'saledate', 'fiModelDesc', 'fiBaseModel', 'fiSecondaryDesc',
       'fiModelSeries', 'fiModelDescriptor', 'ProductSize',
       'fiProductClassDesc', 'state', 'ProductGroup', 'ProductGroupDesc',
       'Drive_System', 'Enclosure', 'Forks', 'Pad_Type', 'Ride_Control',
       'Stick', 'Transmission', 'Turbocharged', 'Blade_Extension',
       'Blade_Width', 'Enclosure_Type', 'Engine_Horsepower', 'Hydraulics',
       'Pushblock', 'Ripper', 'Scarifier', 'Tip_Control', 'Tire_Size',
       'Coupler', 'Coupler_System', 'Grouser_Tracks', 'Hydraulics_Flow',
       'Track_Type', 'Undercarriage_Pad_Width', 'Stick_Length', 'Thumb',
       'Pattern_Changer', 'Grouser_Type', 'Backhoe_Mounting', 'Blade_Type',
       'Travel_Controls', 'Differential_Type', 'Steering_Controls'],
      dtype='object')

df_test.columns # 52 columns

Index(['SalesID', 'MachineID', 'ModelID', 'datasource', 'auctioneerID',
       'YearMade', 'MachineHoursCurrentMeter', 'UsageBand', 'saledate',
       'fiModelDesc', 'fiBaseModel', 'fiSecondaryDesc', 'fiModelSeries',
       'fiModelDescriptor', 'ProductSize', 'fiProductClassDesc', 'state',
       'ProductGroup', 'ProductGroupDesc', 'Drive_System', 'Enclosure',
       'Forks', 'Pad_Type', 'Ride_Control', 'Stick', 'Transmission',
       'Turbocharged', 'Blade_Extension', 'Blade_Width', 'Enclosure_Type',
       'Engine_Horsepower', 'Hydraulics', 'Pushblock', 'Ripper', 'Scarifier',
       'Tip_Control', 'Tire_Size', 'Coupler', 'Coupler_System',
       'Grouser_Tracks', 'Hydraulics_Flow', 'Track_Type',
       'Undercarriage_Pad_Width', 'Stick_Length', 'Thumb', 'Pattern_Changer',
       'Grouser_Type', 'Backhoe_Mounting', 'Blade_Type', 'Travel_Controls',
       'Differential_Type', 'Steering_Controls'],
      dtype='object')

df_train.describe()

	SalesID	SalePrice	MachineID	ModelID	datasource	auctioneerID	YearMade	MachineHoursCurrentMeter
count	4.011250e+05	401125.000000	4.011250e+05	401125.000000	401125.000000	380989.000000	401125.000000	1.427650e+05
mean	1.919713e+06	31099.712848	1.217903e+06	6889.702980	134.665810	6.556040	1899.156901	3.457955e+03
std	9.090215e+05	23036.898502	4.409920e+05	6221.777842	8.962237	16.976779	291.797469	2.759026e+04
min	1.139246e+06	4750.000000	0.000000e+00	28.000000	121.000000	0.000000	1000.000000	0.000000e+00
25%	1.418371e+06	14500.000000	1.088697e+06	3259.000000	132.000000	1.000000	1985.000000	0.000000e+00
50%	1.639422e+06	24000.000000	1.279490e+06	4604.000000	132.000000	2.000000	1995.000000	0.000000e+00
75%	2.242707e+06	40000.000000	1.468067e+06	8724.000000	136.000000	4.000000	2000.000000	3.025000e+03
max	6.333342e+06	142000.000000	2.486330e+06	37198.000000	172.000000	99.000000	2013.000000	2.483300e+06

len(df_train_and_valid)

412698

df_train_and_valid.SalePrice.info()

<class 'pandas.core.series.Series'>
RangeIndex: 412698 entries, 0 to 412697
Series name: SalePrice
Non-Null Count   Dtype  
--------------   -----  
412698 non-null  float64
dtypes: float64(1)
memory usage: 3.1 MB

Let's clean up the data

Handle Ordinal Columns

df_train_and_valid.ProductSize.unique()

array([nan, 'Medium', 'Small', 'Large / Medium', 'Mini', 'Large',
       'Compact'], dtype=object)

sizes = 'Large','Large / Medium','Medium','Small','Mini','Compact'

# set ProductSize to "category" type and give ordered categories
df_train_and_valid['ProductSize'] = df_train_and_valid['ProductSize'].astype('category')
df_train_and_valid['ProductSize'] = df_train_and_valid['ProductSize'].cat.set_categories(sizes, ordered=True)

Make SalePrice into log(SalePrice) b/c the metric will be on RMSE of the log(SalePrice)

dep_var = 'SalePrice'
df_train_and_valid[dep_var] = np.log(df_train_and_valid[dep_var])

Enriching the Dates for the train_and_valid and test sets

First, making sure the saledate column is a datetime type rather than a plain object.

df_train_and_valid['saledate'].unique()

array(['11/16/2006 0:00', '3/26/2004 0:00', '2/26/2004 0:00', ...,
       '3/10/2012 0:00', '3/9/2012 0:00', '4/14/2012 0:00'],
      shape=(4013,), dtype=object)

df_train_and_valid['saledate'] = pd.to_datetime(df_train_and_valid['saledate'])

df_train_and_valid['saledate']

0        2006-11-16
1        2004-03-26
2        2004-02-26
3        2011-05-19
4        2009-07-23
            ...    
412693   2012-03-07
412694   2012-01-28
412695   2012-01-28
412696   2012-03-07
412697   2012-01-28
Name: saledate, Length: 412698, dtype: datetime64[ns]

Then, add the extra fields -- note: this removes the original 'saledate' column. Docs here: https://docs.fast.ai/tabular.core.html

# this is the first invocation of the fastai library for this notebook
# need it for the "add_datepart" thing that adds extra date metadata columns
import fastai
from fastai.tabular.core import add_datepart

fastai.__version__ #'2.8.7'

'2.8.7'

df_train_and_valid = add_datepart(df_train_and_valid, 'saledate')

## for the df_test, too
df_test['saledate'] = pd.to_datetime(df_test['saledate'])
df_test = add_datepart(df_test, 'saledate')

df_train_and_valid now has 13 new fields (and one less: 'saledate')

• 'saleYear',
• 'saleMonth',
• 'saleWeek',
• 'saleDay',
• 'saleDayofweek',
• 'saleDayofyear',
• 'saleIs_month_end',
• 'saleIs_month_start',
• 'saleIs_quarter_end',
• 'saleIs_quarter_start',
• 'saleIs_year_end',
• 'saleIs_year_start',
• 'saleElapsed': ": The Unix timestamp, representing the total number of seconds elapsed since January 1, 1970."

Use fastai's TabularPandas and TabularProc

TabularProc has helper functions you can apply to TabularPandas objects -- e.g. for:

• FillMissing: replaces missing values with the median of the column, and creates a new Boolean column that is set to True for any row where the value was missing
• Categorify: replaces a column with a numeric categorical column

First, assign rows to train and validation sets, paying attention to the fact that this is a timeseries

cond = (df_train_and_valid.saleYear<2011) | (df_train_and_valid.saleMonth<10)
train_idx = np.where( cond)[0]
valid_idx = np.where(~cond)[0]

splits = (list(train_idx),list(valid_idx))

Designate columns as 'categorical' or 'continuous'

from fastai.tabular.core import cont_cat_split
cont,cat = cont_cat_split(df_train_and_valid, 1, dep_var=dep_var)

Set up the TabularPandas object and the Procs we're going to use

from fastai.tabular.all import TabularPandas, Categorify, FillMissing

procs = [Categorify, FillMissing]

to = TabularPandas(df_train_and_valid, procs, cat, cont, y_names=dep_var, splits=splits)

/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/fastai/tabular/core.py:314: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.


  to[n].fillna(self.na_dict[n], inplace=True)
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/fastai/tabular/core.py:314: FutureWarning: A value is trying to be set on a copy of a DataFrame or Series through chained assignment using an inplace method.
The behavior will change in pandas 3.0. This inplace method will never work because the intermediate object on which we are setting values always behaves as a copy.

For example, when doing 'df[col].method(value, inplace=True)', try using 'df.method({col: value}, inplace=True)' or df[col] = df[col].method(value) instead, to perform the operation inplace on the original object.


  to[n].fillna(self.na_dict[n], inplace=True)

Check what the procs have done

• .train vs .valid
• continuous column NaNs are filled with the average value, and {column}_na columns have been added to indicate which rows have filled data
• categorical column NaNs are given the string "#nan#" and all categorical strings are now stored as numbers; note: we apparently don't need to bother making embeddings for these, the partition splitting method that we used for the continuous values is good enough. German biostatistics researchers, 2019: https://peerj.com/articles/6339/#aff-1

confusingly, the TabularPandas object has a .show() function that gives the human readable values in the columns. To see what values are actually stored in the df you have to call .items()

len(to.train),len(to.valid)

(404710, 7988)

to.show(3)

	UsageBand	fiModelDesc	fiBaseModel	fiSecondaryDesc	fiModelSeries	fiModelDescriptor	ProductSize	fiProductClassDesc	state	ProductGroup	ProductGroupDesc	Drive_System	Enclosure	Forks	Pad_Type	Ride_Control	Stick	Transmission	Turbocharged	Blade_Extension	Blade_Width	Enclosure_Type	Engine_Horsepower	Hydraulics	Pushblock	Ripper	Scarifier	Tip_Control	Tire_Size	Coupler	Coupler_System	Grouser_Tracks	Hydraulics_Flow	Track_Type	Undercarriage_Pad_Width	Stick_Length	Thumb	Pattern_Changer	Grouser_Type	Backhoe_Mounting	Blade_Type	Travel_Controls	Differential_Type	Steering_Controls	saleIs_month_end	saleIs_month_start	saleIs_quarter_end	saleIs_quarter_start	saleIs_year_end	saleIs_year_start	auctioneerID_na	MachineHoursCurrentMeter_na	SalesID	MachineID	ModelID	datasource	auctioneerID	YearMade	MachineHoursCurrentMeter	saleYear	saleMonth	saleWeek	saleDay	saleDayofweek	saleDayofyear	saleElapsed	SalePrice
0	Low	521D	521	D	#na#	#na#	#na#	Wheel Loader - 110.0 to 120.0 Horsepower	Alabama	WL	Wheel Loader	#na#	EROPS w AC	None or Unspecified	#na#	None or Unspecified	#na#	#na#	#na#	#na#	#na#	#na#	#na#	2 Valve	#na#	#na#	#na#	#na#	None or Unspecified	None or Unspecified	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	Standard	Conventional	False	False	False	False	False	False	False	False	1139246	999089	3157	121	3.0	2004	68.0	2006	11	46	16	3	320	1.163635e+09	11.097410
1	Low	950FII	950	F	II	#na#	Medium	Wheel Loader - 150.0 to 175.0 Horsepower	North Carolina	WL	Wheel Loader	#na#	EROPS w AC	None or Unspecified	#na#	None or Unspecified	#na#	#na#	#na#	#na#	#na#	#na#	#na#	2 Valve	#na#	#na#	#na#	#na#	23.5	None or Unspecified	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	Standard	Conventional	False	False	False	False	False	False	False	False	1139248	117657	77	121	3.0	1996	4640.0	2004	3	13	26	4	86	1.080259e+09	10.950807
2	High	226	226	#na#	#na#	#na#	#na#	Skid Steer Loader - 1351.0 to 1601.0 Lb Operating Capacity	New York	SSL	Skid Steer Loaders	#na#	OROPS	None or Unspecified	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	Auxiliary	#na#	#na#	#na#	#na#	#na#	None or Unspecified	None or Unspecified	None or Unspecified	Standard	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	False	False	False	False	False	False	False	False	1139249	434808	7009	121	3.0	2001	2838.0	2004	2	9	26	3	57	1.077754e+09	9.210340

to1 = TabularPandas(df_train_and_valid, procs, ['state', 'ProductGroup', 'Drive_System', 'Enclosure'], [], y_names=dep_var, splits=splits)
to1.show(3)

	state	ProductGroup	Drive_System	Enclosure	SalePrice
0	Alabama	WL	#na#	EROPS w AC	11.097410
1	North Carolina	WL	#na#	EROPS w AC	10.950807
2	New York	SSL	#na#	OROPS	9.210340

to.items.head(3)

	SalesID	SalePrice	MachineID	ModelID	datasource	auctioneerID	YearMade	MachineHoursCurrentMeter	UsageBand	fiModelDesc	...	saleDayofyear	saleIs_month_end	saleIs_month_start	saleIs_quarter_end	saleIs_quarter_start	saleIs_year_end	saleIs_year_start	saleElapsed	auctioneerID_na	MachineHoursCurrentMeter_na
0	1139246	11.097410	999089	3157	121	3.0	2004	68.0	2	963	...	320	1	1	1	1	1	1	1.163635e+09	1	1
1	1139248	10.950807	117657	77	121	3.0	1996	4640.0	2	1745	...	86	1	1	1	1	1	1	1.080259e+09	1	1
2	1139249	9.210340	434808	7009	121	3.0	2001	2838.0	1	336	...	57	1	1	1	1	1	1	1.077754e+09	1	1

3 rows × 67 columns

to1.items[['state', 'ProductGroup', 'Drive_System', 'Enclosure']].head(3)

	state	ProductGroup	Drive_System	Enclosure
0	1	6	0	3
1	33	6	0	3
2	32	3	0	6

to.items.columns

Index(['SalesID', 'SalePrice', 'MachineID', 'ModelID', 'datasource',
       'auctioneerID', 'YearMade', 'MachineHoursCurrentMeter', 'UsageBand',
       'fiModelDesc', 'fiBaseModel', 'fiSecondaryDesc', 'fiModelSeries',
       'fiModelDescriptor', 'ProductSize', 'fiProductClassDesc', 'state',
       'ProductGroup', 'ProductGroupDesc', 'Drive_System', 'Enclosure',
       'Forks', 'Pad_Type', 'Ride_Control', 'Stick', 'Transmission',
       'Turbocharged', 'Blade_Extension', 'Blade_Width', 'Enclosure_Type',
       'Engine_Horsepower', 'Hydraulics', 'Pushblock', 'Ripper', 'Scarifier',
       'Tip_Control', 'Tire_Size', 'Coupler', 'Coupler_System',
       'Grouser_Tracks', 'Hydraulics_Flow', 'Track_Type',
       'Undercarriage_Pad_Width', 'Stick_Length', 'Thumb', 'Pattern_Changer',
       'Grouser_Type', 'Backhoe_Mounting', 'Blade_Type', 'Travel_Controls',
       'Differential_Type', 'Steering_Controls', 'saleYear', 'saleMonth',
       'saleWeek', 'saleDay', 'saleDayofweek', 'saleDayofyear',
       'saleIs_month_end', 'saleIs_month_start', 'saleIs_quarter_end',
       'saleIs_quarter_start', 'saleIs_year_end', 'saleIs_year_start',
       'saleElapsed', 'auctioneerID_na', 'MachineHoursCurrentMeter_na'],
      dtype='object')

to.classes['ProductSize']

['#na#', 'Large', 'Large / Medium', 'Medium', 'Small', 'Mini', 'Compact']

Because this processing took time, suggested to save a pickle

from fastai.tabular.core import save_pickle, load_pickle
save_pickle(path/'to_bulldozers.pkl',to)

We can read the pickle back, too

# ok so this is the safe way?
#import pickle
#with open(path/'to_bulldozers.pkl', "rb") as f:
#    to = pickle.load(f)

# this is me mimicking the save_pickle import from the fastai.tabular.core library
to = load_pickle(path/'to_bulldozers.pkl')
# this is what they suggested we would use, but doesn't seem to work after I updated
#to = (path/'to_bulldozers.pkl').load()
# from scikit-learn 1.8 to 1.9 (?)

to.show()

	UsageBand	fiModelDesc	fiBaseModel	fiSecondaryDesc	fiModelSeries	fiModelDescriptor	ProductSize	fiProductClassDesc	state	ProductGroup	ProductGroupDesc	Drive_System	Enclosure	Forks	Pad_Type	Ride_Control	Stick	Transmission	Turbocharged	Blade_Extension	Blade_Width	Enclosure_Type	Engine_Horsepower	Hydraulics	Pushblock	Ripper	Scarifier	Tip_Control	Tire_Size	Coupler	Coupler_System	Grouser_Tracks	Hydraulics_Flow	Track_Type	Undercarriage_Pad_Width	Stick_Length	Thumb	Pattern_Changer	Grouser_Type	Backhoe_Mounting	Blade_Type	Travel_Controls	Differential_Type	Steering_Controls	saleIs_month_end	saleIs_month_start	saleIs_quarter_end	saleIs_quarter_start	saleIs_year_end	saleIs_year_start	auctioneerID_na	MachineHoursCurrentMeter_na	SalesID	MachineID	ModelID	datasource	auctioneerID	YearMade	MachineHoursCurrentMeter	saleYear	saleMonth	saleWeek	saleDay	saleDayofweek	saleDayofyear	saleElapsed	SalePrice
0	Low	521D	521	D	#na#	#na#	#na#	Wheel Loader - 110.0 to 120.0 Horsepower	Alabama	WL	Wheel Loader	#na#	EROPS w AC	None or Unspecified	#na#	None or Unspecified	#na#	#na#	#na#	#na#	#na#	#na#	#na#	2 Valve	#na#	#na#	#na#	#na#	None or Unspecified	None or Unspecified	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	Standard	Conventional	False	False	False	False	False	False	False	False	1139246	999089	3157	121	3.0	2004	68.0	2006	11	46	16	3	320	1.163635e+09	11.097410
1	Low	950FII	950	F	II	#na#	Medium	Wheel Loader - 150.0 to 175.0 Horsepower	North Carolina	WL	Wheel Loader	#na#	EROPS w AC	None or Unspecified	#na#	None or Unspecified	#na#	#na#	#na#	#na#	#na#	#na#	#na#	2 Valve	#na#	#na#	#na#	#na#	23.5	None or Unspecified	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	Standard	Conventional	False	False	False	False	False	False	False	False	1139248	117657	77	121	3.0	1996	4640.0	2004	3	13	26	4	86	1.080259e+09	10.950807
2	High	226	226	#na#	#na#	#na#	#na#	Skid Steer Loader - 1351.0 to 1601.0 Lb Operating Capacity	New York	SSL	Skid Steer Loaders	#na#	OROPS	None or Unspecified	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	Auxiliary	#na#	#na#	#na#	#na#	#na#	None or Unspecified	None or Unspecified	None or Unspecified	Standard	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	False	False	False	False	False	False	False	False	1139249	434808	7009	121	3.0	2001	2838.0	2004	2	9	26	3	57	1.077754e+09	9.210340
3	High	PC120-6E	PC120	#na#	-6E	#na#	Small	Hydraulic Excavator, Track - 12.0 to 14.0 Metric Tons	Texas	TEX	Track Excavators	#na#	EROPS w AC	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	2 Valve	#na#	#na#	#na#	#na#	#na#	None or Unspecified	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	False	False	False	False	False	False	False	False	1139251	1026470	332	121	3.0	2001	3486.0	2011	5	20	19	3	139	1.305763e+09	10.558414
4	Medium	S175	S175	#na#	#na#	#na#	#na#	Skid Steer Loader - 1601.0 to 1751.0 Lb Operating Capacity	New York	SSL	Skid Steer Loaders	#na#	EROPS	None or Unspecified	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	Auxiliary	#na#	#na#	#na#	#na#	#na#	None or Unspecified	None or Unspecified	None or Unspecified	Standard	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	False	False	False	False	False	False	False	False	1139253	1057373	17311	121	3.0	2007	722.0	2009	7	30	23	3	204	1.248307e+09	9.305651
5	Low	310G	310	G	#na#	#na#	#na#	Backhoe Loader - 14.0 to 15.0 Ft Standard Digging Depth	Arizona	BL	Backhoe Loaders	Four Wheel Drive	OROPS	None or Unspecified	None or Unspecified	No	Extended	Powershuttle	None or Unspecified	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	False	False	False	False	False	False	False	False	1139255	1001274	4605	121	3.0	2004	508.0	2008	12	51	18	3	353	1.229558e+09	10.184900
6	High	790ELC	790	E	#na#	LC	Large / Medium	Hydraulic Excavator, Track - 21.0 to 24.0 Metric Tons	Florida	TEX	Track Excavators	#na#	EROPS	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	Standard	#na#	#na#	#na#	#na#	#na#	None or Unspecified	#na#	#na#	#na#	Steel	None or Unspecified	None or Unspecified	None or Unspecified	None or Unspecified	Double	#na#	#na#	#na#	#na#	#na#	False	False	False	False	False	False	False	False	1139256	772701	1937	121	3.0	1993	11540.0	2004	8	35	26	3	239	1.093478e+09	9.952278
7	High	416D	416	D	#na#	#na#	#na#	Backhoe Loader - 14.0 to 15.0 Ft Standard Digging Depth	Illinois	BL	Backhoe Loaders	Four Wheel Drive	OROPS	None or Unspecified	Reversible	No	Standard	Standard	Yes	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	False	False	False	False	False	False	False	False	1139261	902002	3539	121	3.0	2001	4883.0	2005	11	46	17	3	321	1.132186e+09	10.203592
8	Low	430HAG	430	HAG	#na#	#na#	Mini	Hydraulic Excavator, Track - 3.0 to 4.0 Metric Tons	Texas	TEX	Track Excavators	#na#	EROPS	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	Auxiliary	#na#	#na#	#na#	#na#	#na#	Manual	#na#	#na#	#na#	Rubber	None or Unspecified	None or Unspecified	None or Unspecified	None or Unspecified	Double	#na#	#na#	#na#	#na#	#na#	False	False	False	False	False	False	False	False	1139272	1036251	36003	121	3.0	2008	302.0	2009	8	35	27	3	239	1.251331e+09	9.975808
9	Medium	988B	988	B	#na#	#na#	Large	Wheel Loader - 350.0 to 500.0 Horsepower	Florida	WL	Wheel Loader	#na#	EROPS w AC	None or Unspecified	#na#	None or Unspecified	#na#	#na#	#na#	#na#	#na#	#na#	#na#	2 Valve	#na#	#na#	#na#	#na#	None or Unspecified	None or Unspecified	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	#na#	Standard	Conventional	False	False	False	False	False	False	False	False	1139275	1016474	3883	121	3.0	1000	20700.0	2007	8	32	9	3	221	1.186618e+09	11.082143

Creating the Decision Tree

According to the authors, "The basic steps to train a decision tree can be written down very easily":

• Loop through each column of the dataset in turn.
• For each column, loop through each possible level of that column in turn.
• Try splitting the data into two groups, based on whether they are greater than or less than that value (or if it is a categorical variable, based on whether they are equal to or not equal to that level of that categorical variable).
• Find the average sale price for each of those two groups, and see how close that is to the actual sale price of each of the items of equipment in that group. That is, treat this as a very simple "model" where our predictions are simply the average sale price of the item's group.
• After looping through all of the columns and all the possible levels for each, pick the split point that gave the best predictions using that simple model.
• We now have two different groups for our data, based on this selected split. Treat each of these as separate datasets, and find the best split for each by going back to step 1 for each group.
• Continue this process recursively, until you have reached some stopping criterion for each group—for instance, stop splitting a group further when it has only 20 items in it.

import pickle
with open(path/'to_bulldozers.pkl', "rb") as f:
    to = pickle.load(f)

Set up our independent and dependent variables for the train & validation sets

xs,y = to.train.xs,to.train.y
valid_xs,valid_y = to.valid.xs,to.valid.y

1. Let's make the tree!

from sklearn.tree import DecisionTreeRegressor
m = DecisionTreeRegressor(max_leaf_nodes=4)
m.fit(xs, y); # btw the semicolon keeps an interesting visualization of the tree from coming out

import sklearn
print(sklearn.__version__)

1.9.0

Draw the Tree

this uses a helper function provided in utils.py in fastbook; i'm adding it here

from sklearn.tree import export_graphviz
import graphviz, re

def draw_tree(t, df, size=10, ratio=0.6, precision=0, **kwargs):
    s=export_graphviz(t, out_file=None, feature_names=df.columns, filled=True, rounded=True,
                      special_characters=True, rotate=False, precision=precision, **kwargs)
    return graphviz.Source(re.sub('Tree {', f'Tree {{ size={size}; ratio={ratio}', s))

draw_tree(m, xs, size=10, leaves_parallel=True, precision=2)

We can also draw the tree using another library: Terence Parr's "powerful dtreeviz library"

import dtreeviz
print(dtreeviz.__version__)

2.3.2

import numpy as np
import dtreeviz

samp_idx = np.random.permutation(len(y))[:500]

viz_model = dtreeviz.model(
    m,
    X_train=xs.iloc[samp_idx],
    y_train=y.iloc[samp_idx],
    feature_names=[str(c) for c in xs.columns],
    target_name=dep_var,
)

viz_model.view(
    fontname="DejaVu Sans",
    scale=1.6,
    label_fontsize=10,
    orientation="LR",
)

/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2827: UserWarning: X does not have valid feature names, but DecisionTreeRegressor was fitted with feature names

The YearMade shows that there are years all the way down at 1000... let's just fix those values

xs.loc[xs['YearMade']<1900, 'YearMade'] = 1950
valid_xs.loc[valid_xs['YearMade']<1900, 'YearMade'] = 1950

2. And rebuild/refit the tree

m = DecisionTreeRegressor(max_leaf_nodes=4).fit(xs, y)

And remake the dtreeviz

samp_idx = np.random.permutation(len(y))[:500]

viz_model = dtreeviz.model(
    m,
    X_train=xs.iloc[samp_idx],
    y_train=y.iloc[samp_idx],
    feature_names=[str(c) for c in xs.columns],
    target_name=dep_var,
)

viz_model.view(
    fontname="DejaVu Sans",
    scale=1.6,
    label_fontsize=10,
    orientation="LR",
)

/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2827: UserWarning: X does not have valid feature names, but DecisionTreeRegressor was fitted with feature names

3. Let's make a bigger tree!

m = DecisionTreeRegressor() # we won't pass in any criteria for the leaves
m.fit(xs, y);

And let's write a function for our metric, the root mean squared (log) error (recall we already did log(SalePrice))

import math
def r_mse(pred,y): return round(math.sqrt(((pred-y)**2).mean()), 6)
def m_rmse(m, xs, y): return r_mse(m.predict(xs), y)

For the full tree... we have a root mean square error of 0 for the training data!

m_rmse(m, xs, y)

0.0

What's the rmse for the validation data?

m_rmse(m, valid_xs, valid_y)

0.333403

# what are the number of leaves that we have... and what are the number of datapoints?
m.get_n_leaves(), len(xs)
# practically the same number of leaves as data points.

(np.int64(324559), 404710)

4. Let's make a tree that has at least 25 rows per leaf

m = DecisionTreeRegressor(min_samples_leaf=25)
m.fit(to.train.xs, to.train.y);

Now we can check the rmse for the train and validation data... hopefully they're closer to each other

m_rmse(m, xs, y), m_rmse(m, valid_xs, valid_y)

(0.248564, 0.323411)

m.get_n_leaves(), len(xs) # many fewer leaves than data points, much better

(np.int64(12397), 404710)

Random Forests: Bagging Decision Trees

We can write a procedure for generating random forests just like we had for the decision trees:

• Randomly choose a subset of the rows of your data (i.e., "bootstrap replicates of your learning set").
• Train a model using this subset.
• Save that model, and then return to step 1 a few times.
• This will give you a number of trained models.
• To make a prediction, predict using all of the models, and then take the average of each of those model's predictions.

What is "bagging"? "It is based on a deep and important insight: although each of the models trained on a subset of data will make more errors than a model trained on the full dataset, those errors will not be correlated with each other. Different models will make different errors. The average of those errors, therefore, is: zero! So if we take the average of all of the models' predictions, then we should end up with a prediction that gets closer and closer to the correct answer, the more models we have."

Stats deparment at Berkeley, 1994, https://www.stat.berkeley.edu/~breiman/bagging.pdf

# function to define the random forest AND run .fit on the data
from sklearn.ensemble import RandomForestRegressor
def rf(xs, y, n_estimators=40, max_samples=200_000, max_features=0.5, min_samples_leaf=5, **kwargs):
    return RandomForestRegressor(n_jobs=-1,                 # compute decision trees in parallel
                                 n_estimators=n_estimators, # number of trees we want
                                 max_samples=max_samples,   # how many rows to sample for training each tree
                                 max_features=max_features, # how many columns to sample at each split point (where 0.5 means "take half the total number of columns")
                                 min_samples_leaf=min_samples_leaf, oob_score=True).fit(xs, y)

1. Create and fit our first random forest

# create & fit the random forest
m = rf(xs, y);

Calculate the model's rmse for the training and validation data

m_rmse(m, xs, y), m_rmse(m, valid_xs, valid_y)
# we had  (0.248562, 0.323441) for just a single tree
# we have (0.171012, 0.232723) for the average of the 40 tree estimators

(0.171299, 0.233493)

Choosing values for the parameters for random forests

"One of the most important properties of random forests is that they aren't very sensitive to the hyperparameter choices, such as max_features.

• You can set n_estimators to as high a number as you have time to train—the more trees you have, the more accurate the model will be.
• max_samples can often be left at its default, unless you have over 200,000 data points, in which case setting it to 200,000 will make it train faster with little impact on accuracy.
• max_features=0.5 and min_samples_leaf=4 both tend to work well, although sklearn's defaults work well too."

Choosing n_estimators, the number of trees

To see the impact of n_estimators, let's get the predictions from each individual tree in our forest

• these are in the sklearn random forest's estimators_ attribute)

preds = np.stack([t.predict(valid_xs) for t in m.estimators_])

/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names

# this is just showing that the m_rmse for the random forest is equivalent to the average rmse of all predictions
r_mse(preds.mean(0), valid_y)

0.233493

We can make a plot of number of trees vs rmse to see how rmse goes down the more trees we average in

import matplotlib.pyplot as plt
plt.plot([r_mse(preds[:i+1].mean(0), valid_y) for i in range(40)]);

Interpreting the rmse

How do we figure out why our validation data has worse rmse than our training data?

e.g. some reasons could be overfitting, or because the validation set covers a different time period

Enter: Out of Bag error calculation

In addition to checking rmse for the train & validation datasets, we can also check the rmse for what we would predict for the rows not part of our particular estimator's sample.

• Those predictions are in the sklearn random forest's oob_prediction_ attribute: compare with training labels.

print("train   : ", m_rmse(m, xs, y), 
      "\nOutOfBag: ", r_mse(m.oob_prediction_, y),
      "\nvalid   : ", m_rmse(m, valid_xs, valid_y))

train   :  0.171299 
OutOfBag:  0.211303 
valid   :  0.233493

According to the authors, the 0.21 for the "out of bag" error is "much lower" than 0.23 the validation set error. That apparently means that in addition to normal "generalization" error (e.g. overfitting?), there's something else going on.

Interpreting more things about the model

• How confident are we in our predictions using a particular row of data?
• For predicting with a particular row of data, what were the most important factors, and how did they influence that prediction?
• Which columns are the strongest predictors, which can we ignore?
• Which columns are effectively redundant with each other, for purposes of prediction?
• How do predictions vary, as we vary these columns?

Prediction Values vs Prediction Confidence with Variance between Trees

The model gives us log(saleprice) predictions for rows of auction sale data. But what if want to know how confident the model was at making this prediction?

We can do this by looking at the standard deviation of the predictions across all the trees, instead of taking the mean. If they mostly agree, then we could say we're more confident; if they vary a lot, then we're less confident.

Btw what's in each prediction? 40 trees, each trained on a subset of rows, making a prediction for every row

preds = np.stack([t.predict(valid_xs) for t in m.estimators_])

/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/sklearn/utils/validation.py:2820: UserWarning: X has feature names, but DecisionTreeRegressor was fitted without feature names

preds.shape # we have 40 trees and 7988 total auctions (rows)

(40, 7988)

preds_std = preds.std(0)

# first five standard deviations -> standard deviation across all trees, for the first five log(saleprice) predictions
preds_std[:5]

array([0.29928914, 0.11161356, 0.12067076, 0.25767173, 0.11550318])

Figuring out which features are important

Making predictions and determining confidence in the predictions is one part of the story. But it would also be good to understand what features were most important for the prediction and its confidence. Shrinking the number of columns we have to study is apparently good because a "simpler, more interpretable model is easier to roll out and maintain".

• the sklearn random forest has a feature_importances_ attribute

The feature_importances_ are calculated like this:

• The feature importance algorithm loops through each tree, and then recursively explores each branch.
• At each branch, it looks to see what feature was used for that split, and how much the model improves as a result of that split.
• The improvement (weighted by the number of rows in that group) is added to the importance score for that feature.
• This is summed across all branches of all trees, and finally the scores are normalized such that they add to 1.

# we can get the feature_importances_ into a dataframe
def rf_feat_importance(m, df):
    return pd.DataFrame({'cols':df.columns, 
                         'imp':m.feature_importances_}
                       ).sort_values('imp', ascending=False)

# fi: "feature importance"
fi = rf_feat_importance(m, xs)
fi[:10]

	cols	imp
57	YearMade	0.183548
6	ProductSize	0.117152
30	Coupler_System	0.099543
31	Grouser_Tracks	0.073407
7	fiProductClassDesc	0.071620
54	ModelID	0.054812
65	saleElapsed	0.051461
3	fiSecondaryDesc	0.041389
12	Enclosure	0.036508
1	fiModelDesc	0.030147

So it looks like YearMade, ProductSize, and Coupler_System are the top three.

Plotting the Feature Importances

def plot_fi(fi):
    return fi.plot('cols', 'imp', 'barh', figsize=(12,7), legend=False)

plot_fi(fi[:30]);

Reducing the data's columns by only keeping "important" ones

We can get a less noisy understanding of the columns but understanding their contributions to the performance of the tree, and then taking out ones that seem to be much less important.

# only keep columns whose feature importance, fi, is greater than .005
to_keep = fi[fi.imp>0.005].cols
len(to_keep)

21

2. Reduce the number of columns in the dataset based on importance, and retrain the model with this reduced data

xs_imp = xs[to_keep]
valid_xs_imp = valid_xs[to_keep]

m = rf(xs_imp, y)

# new rmses
#m_rmse(m, xs_imp, y), r_mse(m.oob_prediction_, y), m_rmse(m, valid_xs_imp, valid_y)

print("train   : ", m_rmse(m, xs_imp, y), 
      "\nOutOfBag: ", r_mse(m.oob_prediction_, y),
      "\nvalid   : ", m_rmse(m, valid_xs_imp, valid_y))

train   :  0.181599 
OutOfBag:  0.214337 
valid   :  0.231258

EJ: It looks lke train & Out of Bag errors are slightly higher, but validation error is slightly lower

The point is that they're about the same; and there are fewer columns to study -- 66 vs 21:

len(xs.columns), len(xs_imp.columns)

(66, 21)

Make the plot of feature importances again

Because we retrained the model with fewer columns, we don't necessarily expect the values or the order of the feature importances to remain the same.

plot_fi(rf_feat_importance(m, xs_imp));

Reducing the data's columns by keeping the non-redundant ones

Some of the columns might be performing the same role. For instance, two columns might have values that travel together; e.g. if one column is the "short name" and the other, a longer description. In our dataset, that would be something like ProductGroup and ProductGroupDesc. We can remove one of the two columns in each pair like this.

# there's a helper function in fastbook's utils for building a cluster map of the columns using spearman correlation
# i'm copying it here
# import scipy
# from scipy.cluster import hierarchy as hc
# def cluster_columns(df, figsize=(10,6), font_size=12):
#     corr = np.round(scipy.stats.spearmanr(df).correlation, 4)
#     corr_condensed = hc.distance.squareform(1-corr)
#     z = hc.linkage(corr_condensed, method='average')
#     fig = plt.figure(figsize=figsize)
#     hc.dendrogram(z, labels=df.columns, orientation='left', leaf_font_size=font_size)
#     plt.show()

# actually it turns out it's outdated - module 'scipy.cluster.hierarchy' has no attribute 'distance'
# re-written here by ChatGPT
import numpy as np
import matplotlib.pyplot as plt

from scipy.stats import spearmanr
from scipy.cluster.hierarchy import linkage, dendrogram
from scipy.spatial.distance import squareform

def cluster_columns(df, figsize=(10, 6), font_size=12, method="average"):
    # Spearman correlation between columns
    corr = spearmanr(df).correlation

    # Numerical precision can occasionally make the matrix slightly asymmetric
    corr = (corr + corr.T) / 2
    np.fill_diagonal(corr, 1)

    # Convert correlation to a distance matrix
    dist = 1 - corr

    # linkage expects the condensed distance matrix
    condensed = squareform(dist)

    # Hierarchical clustering
    Z = linkage(condensed, method=method)

    # Plot
    plt.figure(figsize=figsize)
    dendrogram(
        Z,
        labels=df.columns,
        orientation="left",
        leaf_font_size=font_size,
    )
    plt.tight_layout()
    plt.show()

    return Z

# we can cluster the smaller set of columns that we already reduced by looking at importance
cluster_columns(xs_imp)

array([[1.30000000e+01, 1.60000000e+01, 4.67248462e-12, 2.00000000e+00],
       [9.00000000e+00, 1.20000000e+01, 3.03357261e-04, 2.00000000e+00],
       [3.00000000e+00, 1.00000000e+01, 5.12781818e-04, 2.00000000e+00],
       [2.00000000e+00, 2.30000000e+01, 1.87632712e-03, 3.00000000e+00],
       [6.00000000e+00, 1.40000000e+01, 2.21485321e-03, 2.00000000e+00],
       [4.00000000e+00, 2.10000000e+01, 1.30702784e-01, 3.00000000e+00],
       [1.10000000e+01, 2.50000000e+01, 4.88140524e-01, 3.00000000e+00],
       [1.90000000e+01, 2.60000000e+01, 5.00935571e-01, 4.00000000e+00],
       [0.00000000e+00, 2.70000000e+01, 5.83405730e-01, 4.00000000e+00],
       [1.00000000e+00, 1.70000000e+01, 6.32270562e-01, 2.00000000e+00],
       [8.00000000e+00, 2.40000000e+01, 6.86774135e-01, 4.00000000e+00],
       [2.20000000e+01, 2.80000000e+01, 7.14296064e-01, 6.00000000e+00],
       [1.80000000e+01, 3.00000000e+01, 8.00971882e-01, 3.00000000e+00],
       [5.00000000e+00, 1.50000000e+01, 8.08047270e-01, 2.00000000e+00],
       [7.00000000e+00, 2.00000000e+01, 8.23528134e-01, 2.00000000e+00],
       [3.20000000e+01, 3.30000000e+01, 8.73496728e-01, 9.00000000e+00],
       [2.90000000e+01, 3.40000000e+01, 8.79297441e-01, 6.00000000e+00],
       [3.10000000e+01, 3.70000000e+01, 9.00306831e-01, 1.00000000e+01],
       [3.60000000e+01, 3.80000000e+01, 1.03857113e+00, 1.90000000e+01],
       [3.50000000e+01, 3.90000000e+01, 1.15655651e+00, 2.10000000e+01]])

You can see:

• SaleYear and SaleElapsed are similar;
• ProductGroup and ProductGroupDesc;
• fiBaseModel and fiModelDesc

A strategy for checking which columns are redundant: use the OOB score

Basically just write a loop where you take out the column and check what your accuracy is! You want to be able to remove these columns without impacting the accuracy.

Write a function that quickly trains a random forest and returns the OOB score:

• quickly: use a lower max_samples and higher min_samples_leaf
• oob_score_: an sklearn trained random forest attribute that ranges between 1.0 for a perfect model and 0.0 for a random model.

(note: before we were looking at oob_prediction_ numbers to calculate the rmse)

def get_oob(df):
    m = RandomForestRegressor(n_estimators=40, min_samples_leaf=15,
        max_samples=50000, max_features=0.5, n_jobs=-1, oob_score=True)
    m.fit(df, y)
    return m.oob_score_

Check the oob_score_ for the model before dropping out any of the potential redundancies

get_oob(xs_imp)

0.8781871105555898

Check what the oob_score_ would be for each model that leaves out one potential redundancy at a time

{c:get_oob(xs_imp.drop(c, axis=1)) for c in (
    'saleYear', 'saleElapsed', 'ProductGroupDesc','ProductGroup',
    'fiModelDesc', 'fiBaseModel',
    'Hydraulics_Flow','Grouser_Tracks', 'Coupler_System')}

{'saleYear': 0.8766551991821786,
 'saleElapsed': 0.8728956088478752,
 'ProductGroupDesc': 0.8776062671534415,
 'ProductGroup': 0.8778658840110456,
 'fiModelDesc': 0.8760580964700083,
 'fiBaseModel': 0.8761182434840864,
 'Hydraulics_Flow': 0.8777899505517228,
 'Grouser_Tracks': 0.8775854930886188,
 'Coupler_System': 0.8771767764627115}

The authors don't say but I think we're supposed to see these numbers as about the same as the original oob_score_, because then they move on to see what happens if we drop one of each pair.

to_drop = ['saleYear', 'ProductGroupDesc', 'fiBaseModel', 'Grouser_Tracks']
get_oob(xs_imp.drop(to_drop, axis=1))

0.8738890620563834

This, too, is in the ballpark of the original oob_score_, so the authors suggest we go for it.

The final set of non-redundant columns - drop in both the train & valid sets

xs_final = xs_imp.drop(to_drop, axis=1)
valid_xs_final = valid_xs_imp.drop(to_drop, axis=1)

And save the data to pickles, apparently

save_pickle(path/'xs_final.pkl', xs_final)
save_pickle(path/'valid_xs_final.pkl', valid_xs_final)

If we wanted to start from our reduced-column dataset here, we can load the train and valid data pickles

xs_final = load_pickle(path/'xs_final.pkl')
valid_xs_final = load_pickle(path/'valid_xs_final.pkl')

3. Let's check out how the model performs on the reduced data

Seems to do basically the same.

m = rf(xs_final, y)
m_rmse(m, xs_final, y), m_rmse(m, valid_xs_final, valid_y)

(0.183506, 0.232802)

Using partial dependence plots to learn how the most important columns impact all the predictions

We saw that the two columns that are the most important are the:

• YearMade
• ProductSize

Well, after we re-ran, the Coupler_System squeezed into second place, but ok. After I cleared everything and ran again the YearMade and ProductSize were once again in the top 2. Not sure what changed, I guess the random forests were different random groups.

Partial Dependence Plot: "if a row varied on nothing other than the feature in question, how would it impact the dependent variable?" E.g. "how does YearMade impact sale price, all other things being equal?"

For each unique YearMade in the valid dataset (1950 to 2011):

• make a copy of the valid dataset but replace all YearMade with 1950. Predict SalePrice for each row of this dataset. Take the average over all the rows of the SalePrice.
• make a copy of the valid dataset but replace all YearMade with 1951. Predict SalePrice for each row of this dataset. Take the average over all the rows of the SalePrice.
• ...
• make a copy of the valid dataset but replace all YearMade with 2011. Predict SalePrice for each row of this dataset. Take the average over all the rows of the SalePrice.

Then, make a the Partial Dependence Plot plot.

• The x-axis is the YearMade
• the y-axis is the average predicted SalePrice

BTW we have been using the "final" set of columns in both the train and valid xs.

Check the count of values per category for ProductSize

# these lines were plotting but mis-labeling the bars
#p = valid_xs_final['ProductSize'].value_counts(sort=False).plot.barh()
#c = to.classes['ProductSize']
#plt.yticks(range(len(c)), c);

# ChatGPT fixing the code so the labels come out right
# but the bars still don't 100% match the fastbook bars... not sure who's wrong
classes = list(to.classes['ProductSize'])

counts = (
    valid_xs_final['ProductSize']
    .value_counts()
    .reindex(range(len(classes)), fill_value=0)
)

counts.index = classes
counts.plot.barh()

<Axes: >

Do the same thing for the YearMade column, but it's continuous so make a histagram

ax = valid_xs_final['YearMade'].hist()

Main thing to notice: "Other than the special value 1950 which we used for coding missing year values, most of the data is from after 1990."

Make the Partial Dependence plot for YearMade

from sklearn.inspection import PartialDependenceDisplay

fig, ax = plt.subplots(figsize=(12, 4))
PartialDependenceDisplay.from_estimator(
  m,    
  valid_xs_final.astype(float), # needed to add the .astype(float) or it wasn't working/doing weird things
  ['YearMade', 'ProductSize'],
  grid_resolution=20,
  ax=ax,
)
plt.tight_layout()
plt.show()

According to the authors, the things to look at in these plots are:

• YearMade look kind of linear from 1990+ where most of the data is; this basically says that price is increasing year over year (what you expect)
• ProductSize seems to have something wrong with it --> the 6 index is where the #na#s are; we wouldn't expect that just because the col has #na# that the price should be so much lower. This sets the scene for a discussion of "data leakage"

Gauging Data Leakage

Data leakage is basically when information from the test set, which you should not be able to use to train the data, leaks into the training set. Things to look out for:

• improbably high accuracies
• strange predictors (see: "Figuring out which features are important" earlier in this notebook)
• partial dependence plots (see: "Using partial dependence plots to learn how the most important columns impact all the predictions" just above, in this notebook)
• tree interpreters (see: "How the important factors actually contribute to specific predictions", just following, in this notebook)

E.g. let's say sales always take place during weekdays, but if a sale price is above a threshhold, the database registers that the sale took place on a Sunday because of some kind of batch process that runs over the weekend. The model might learn that high prices only occur on Sundays. If the company is trying to use the model during the week to predict their auction sale prices, it probably wouldn't predict any sale prices above the threshold because none of the rows would have Sunday.

How the important factors actually contribute to specific predictions

• For the partial dependences, we checked the impact on the predicted average saleprice from setting every row in a dataset to a particular YearMade, for each YearMade from the dataset.
• With tree interpreters, we're going to follow the path of just one row through the decision tree to predict one saleprice.

How? "Let's say we are looking at some particular item at auction. Our model might predict that this item will be very expensive, and we want to know why."

• So, we take that one row of data and put it through the first decision tree, looking to see what split is used at each point throughout the tree.
• For each split, we see what the increase or decrease in the addition is, compared to the parent node of the tree.
• We do this for every tree, and add up the total change in importance by split variable.

#from waterfall_chart import plot as waterfall

NOTE: the treeinterpreter library hasn't been updated since 2021 and doesn't seem to be compatible with sklearn 1.9

So I'm abandoning this approach in favor of something called "SHAP" recommended by Claude

Original:

from treeinterpreter import treeinterpreter
prediction,bias,contributions = treeinterpreter.predict(m, row.values)

treeinterpreter:

• prediction = the model's predicted value for that row
• bias = the average prediction across all training data (baseline) "based on taking the mean of the dependent variable (i.e., the model that is the root of every tree)"
• contributions = how much each feature pushed the prediction up or down from the bias, "the most interesting bit— it tells us the total change in prediction due to each of the independent variables."

The relationship is exact: bias + sum(contributions) = prediction

With SHAP:

import shap
explainer = shap.TreeExplainer(m)
shap_values = explainer.shap_values(row)

SHAP:

• prediction = explainer.expected_value + sum(shap_values[0])
• bias = explainer.expected_value
• contributions = shap_values

# first 5 rows of validation set
row = valid_xs_final.iloc[:5]

# the tree interpreter approach

# from treeinterpreter import treeinterpreter
# prediction,bias,contributions = treeinterpreter.predict(m, row.values)

# treeinterpreter: Looking at the first row

# prediction[0], bias[0], contributions[0].sum()
# (array([10.01216396]), 10.104746057831765, -0.0925820990266335)

# using shap instead tree interpretation
#!pip install --upgrade shap
import shap
explainer = shap.TreeExplainer(m)
shap_values = explainer.shap_values(row)

# shap: Looking at the first row
# prediction[0] = explainer.expected_value + shap_values[0].sum()
# bias[0] = explainer.expected_value
# contributions[0].sum() = shap_values[0].sum()

# prediction[0], bias[0], contributions[0].sum()
explainer.expected_value + shap_values[0].sum(), explainer.expected_value, shap_values[0].sum()
# (array([9.94610828]), array([10.10412849]), np.float64(-0.15802020657617752))

(array([9.94610828]), array([10.10412849]), np.float64(-0.15802020657617752))

The TreeInterpreter vs Shap are not exactly equivalent; they use slightly different algorithms

...and thus produce slightly different values

Claude's interpretation of their differences: "The bias values are very close (~10.104) which makes sense — both compute the average prediction over the training data, so they should nearly match. Small difference is likely just a different model (different random seed between runs).

The prediction and contributions differ more noticeably (-0.09 vs -0.16), which is expected — SHAP and treeinterpreter use genuinely different algorithms to allocate credit among features. treeinterpreter follows the exact decision path; SHAP computes Shapley values which average over all possible feature orderings. Same direction (both negative, both predicting below baseline), different magnitude.

The conceptual interpretation is the same:

• this bulldozer is predicted to sell for less than the average, and
• SHAP is telling you which features are responsible for pulling it down

We can see which features are pulling in which direction with the waterfall plot.

Waterfall Plot to check what each feature does to the final predicted saleprice, "useful in production"

#valid_xs_final.columns

# had to swap out contributions[0] for shap_values[0] (these are the same thing)
from waterfall_chart import plot as waterfall
waterfall(valid_xs_final.columns, shap_values[0], threshold=0.08, 
          rotation_value=45,formatting='{:,.3f}');

/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/waterfall_chart.py:141: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`
/Users/emma/miniforge3/envs/fastbook/lib/python3.13/site-packages/waterfall_chart.py:139: FutureWarning: Series.__getitem__ treating keys as positions is deprecated. In a future version, integer keys will always be treated as labels (consistent with DataFrame behavior). To access a value by position, use `ser.iloc[pos]`

I think the way to interpret this is:

• YearMade drags the price down
• Product Size brings it up
• Coupler_System brings it up
• Model ID brings it up
• saleElapsed brings it down
• fiSecondaryDesc brings it down
• Hydraulics Flow brings it up
• Sales ID brings it down
• ProductGroup brings it up
• other brings it slightly down
• for a slightly less than the average sale price for all auctioned machines?

EJ: My questions would be:

• Why would SaleID be relevant? (maybe this is basically just a proxy for year...? Larger numbers for later sales?
• Same question for saleElapsed? (it's also greater for later sales... so probably also a proxy for year?)

Generalizing Random Forests for New Data

Problems with Random Forests for New Data

a. Timeseries with random forests are problematic

• "Remember, a random forest just averages the predictions of a number of trees.
• And a tree simply predicts the average value of the rows in a leaf.
• Therefore, a tree and a random forest can never predict values outside of the range of the training data.

This is particularly problematic for data where there is a trend over time, such as inflation, and you wish to make predictions for a future time. Your predictions will be systematically too low.

b. Column values not present in the training set ("out of domain") will never be predicted

Solving the Problems with Random Forests for New Data

Option 1: Check whether the distribution of data is different between train and valid/test and remove bad columns

You can actually use a Random Forest to check whether the distributions of data in your test set match your training set! And, if it's different, you can find out which columns reflect the difference.

Use the random forest to: predict whether a row is in the validation set or the training set.

How?

1. Combine our training and validation sets together
2. create a dependent variable that represents which dataset each row comes from
3. build a random forest using that data
4. and get its feature importance

Feature Importance for train/valid prediction

df_dom = pd.concat([xs_final, valid_xs_final])
is_valid = np.array([0]*len(xs_final) + [1]*len(valid_xs_final))

m = rf(df_dom, is_valid)
rf_feat_importance(m, df_dom)[:6]

	cols	imp
5	saleElapsed	0.891161
10	SalesID	0.069815
12	MachineID	0.033834
0	YearMade	0.001726
4	ModelID	0.000681
15	Tire_Size	0.000516

Result: "three columns that differ significantly between the training and validation sets: saleElapsed, SalesID, and MachineID"

• saleElapsed: basically encodes the sale date - the train and valid sets were "earlier" and "later"
• sales id: probably ticks up over time as more sales are made
• machine id: maybe it also ticks up over time in these auctions

RMSE for final model after removing each problematic columns one at a time

m = rf(xs_final, y)
print('orig', m_rmse(m, valid_xs_final, valid_y))

for c in ('SalesID','saleElapsed','MachineID'):
    m = rf(xs_final.drop(c,axis=1), y)
    print(c, m_rmse(m, valid_xs_final.drop(c,axis=1), valid_y))

orig 0.233494
SalesID 0.229908
saleElapsed 0.236235
MachineID 0.230896

Based on these accuracies (which look mostly similar to the original -- accuracy looks .003 worse for saleElapsed), "it looks like we should be able to remove SalesID and MachineID without losing any accuracy."

time_vars = ['SalesID','MachineID']
xs_final_time = xs_final.drop(time_vars, axis=1)
valid_xs_time = valid_xs_final.drop(time_vars, axis=1)

m = rf(xs_final_time, y)
m_rmse(m, valid_xs_time, valid_y)

0.229757

Takeaway: Check for domain shifts. Make a random forest trained on the train/valid dataset (make an is_valid column) and use the feature importance function to figure out columns to try removing from the model to improve generalization ability

-> "Slightly improved the model's accuracy; but more importantly, it should make it more resilient over time, and easier to maintain and understand. We recommend that for all datasets you try building a model where your dependent variable is is_valid, like we did here."

Option 2: Try removing old data

"old data shows relationships that just aren't valid any more"

xs['saleYear'].hist();

Try training on just this subset:

filt = xs['saleYear']>2004
xs_filt = xs_final_time[filt]
y_filt = y[filt]

m = rf(xs_filt, y_filt)
m_rmse(m, xs_filt, y_filt), m_rmse(m, valid_xs_time, valid_y)

(0.177621, 0.229264)

The authors look at these numbers and call them "a tiny bit better": "It's a tiny bit better, which shows that you shouldn't always just use your entire dataset; sometimes a subset can be better."

Option 3: Use a Neural Net

df_nn = pd.read_csv(path/'TrainAndValid.csv', low_memory=False)
df_nn['ProductSize'] = df_nn['ProductSize'].astype('category')
df_nn['ProductSize'] = df_nn['ProductSize'].cat.set_categories(sizes, ordered=True)
df_nn[dep_var] = np.log(df_nn[dep_var])
df_nn['saledate'] = pd.to_datetime(df_nn['saledate'])
df_nn = add_datepart(df_nn, 'saledate')

# just use the columns that we've been working with after all the cleanup before
df_nn_final = df_nn[list(xs_final_time.columns) + [dep_var]]

For the neural net, we need to create embeddings for the categorical columns

• To create embeddings, fastai needs to determine which columns should be treated as categorical variables.
• It does this by comparing the number of distinct levels in the variable to the value of the max_card parameter.
• If it's lower, fastai will treat the variable as categorical.

Embedding sizes larger than 10,000 should generally only be used after you've tested whether there are better ways to group the variable, so we'll use 9,000 as our max_card

cont_nn,cat_nn = cont_cat_split(df_nn_final, max_card=9000, dep_var=dep_var)
# EJ: just noting that we did use this function earlier: cont,cat = cont_cat_split(df_train_and_valid, 1, dep_var=dep_var)
#     the 1: "The 1 is just a conservative threshold saying "don't try to treat numeric columns with few unique values as categorical."

Apparently we're doing this to check that saleElapsed is treated as continuous: "there's one variable that we absolutely do not want to treat as categorical: the saleElapsed variable. A categorical variable cannot, by definition, extrapolate outside the range of values that it has seen, but we want to be able to predict auction sale prices in the future. Let's verify that cont_cat_split did the correct thing."

cont_nn

['saleElapsed']

For the categorical columns, how many categories do we have for each?

df_nn_final[cat_nn].nunique()

YearMade                73
ProductSize              6
Coupler_System           2
fiProductClassDesc      74
ModelID               5281
fiSecondaryDesc        177
Enclosure                6
fiModelDesc           5059
Hydraulics_Flow          3
ProductGroup             6
Hydraulics              12
fiModelDescriptor      140
Tire_Size               17
Drive_System             4
dtype: int64

The size of ModelID and fiModelDesc is sort of high (~5000 unique values where the next highest is in the hundreds) and that's annoying because we'd need a large embedding vector

... So we'll try to see if the model accuracy remains the same if we just got rid of them!

# recall our recent accuracies
m_rmse(m, xs_filt, y_filt), m_rmse(m, valid_xs_time, valid_y)

(0.177621, 0.229264)

# remove from train & validation inputs
xs_filt2 = xs_filt.drop('ModelID', axis=1)
valid_xs_time2 = valid_xs_time.drop('ModelID', axis=1)
# recreate the random forest on the new xs_filt2
m2 = rf(xs_filt2, y_filt)
# check the m_rmse for this model on the train and valid data (without the column we dropped)
m_rmse(m2, xs_filt2, y_filt), m_rmse(m2, valid_xs_time2, valid_y)

(0.17895, 0.232571)

# remove from train & validation inputs
xs_filt2 = xs_filt.drop('fiModelDesc', axis=1)
valid_xs_time2 = valid_xs_time.drop('fiModelDesc', axis=1)
# recreate the random forest on the new xs_filt2
m2 = rf(xs_filt2, y_filt)
# check the m_rmse for this model on the train and valid data (without the column we dropped)
m_rmse(m2, xs_filt2, y_filt), m_rmse(m2, valid_xs_time2, valid_y)

(0.180477, 0.232569)

# remove from train & validation inputs
xs_filt2 = xs_filt.drop('fiModelDescriptor', axis=1)
valid_xs_time2 = valid_xs_time.drop('fiModelDescriptor', axis=1)
# recreate the random forest on the new xs_filt2
m2 = rf(xs_filt2, y_filt)
# check the m_rmse for this model on the train and valid data (without the column we dropped)
m_rmse(m2, xs_filt2, y_filt), m_rmse(m2, valid_xs_time2, valid_y)

(0.176707, 0.229083)

We can't actually remove ModelID and fiModelDesc because the accuracy worsens, but we can remove another seemingly redundant field, fiModelDescriptor

cat_nn.remove('fiModelDescriptor')

Create the TabularPandas object like before, but now with Normalization

"We can create our TabularPandas object in the same way as when we created our random forest, with one very important addition: normalization. A random forest does not need any normalization—the tree building procedure cares only about the order of values in a variable, not at all about how they are scaled. But as we have seen, a neural network definitely does care about this. Therefore, we add the Normalize processor when we build our TabularPandas object"

from fastai.tabular.all import TabularPandas, Categorify, FillMissing, Normalize

procs_nn = [Categorify, FillMissing, Normalize]
to_nn = TabularPandas(df_nn_final, procs_nn, cat_nn, cont_nn,
                      splits=splits, y_names=dep_var)

"Tabular models and data don't generally require much GPU RAM, so we can use larger batch sizes"

dls = to_nn.dataloaders(1024)

We're making a regression model, best practices to set y_range

"so let's find the min and max of our dependent variable"

y = to_nn.train.y
y.min(),y.max()

(np.float32(8.465899), np.float32(11.863583))

Create the Learner

"By default, for tabular data fastai creates a neural network with two hidden layers, with 200 and 100 activations, respectively. This works quite well for small datasets, but here we've got quite a large dataset, so we increase the layer sizes to 500 and 250"

from fastai.tabular.all import tabular_learner, F
learn = tabular_learner(dls, y_range=(8,12), layers=[500,250],
                        n_out=1, loss_func=F.mse_loss)

Use the learning rate finder

learn.lr_find()

SuggestedLRs(valley=0.0003311311302240938)

Train the learner

We're not fine tuning, so "train with fit_one_cycle for a few epochs and see how it looks"

learn.fit_one_cycle(5, 1e-2)

epoch	train_loss	valid_loss	time
0	0.063897	0.068394	00:04
1	0.052258	0.055638	00:03
2	0.047633	0.056667	00:03
3	0.042598	0.051853	00:03
4	0.039863	0.050154	00:03

preds,targs = learn.get_preds()
r_mse(preds,targs)

0.22395

Apparently .223 is 'quite a bit better' than the random forest (...at .229...) "although it took longer to train, and it's fussier about hyperparameter tuning"

learn.save('nn')

Path('models/nn.pth')

Fastai's Tabular learner and Tabular model

• Tabular learner: https://docs.fast.ai/tabular.learner.html - regular learner with customized predict method
  1. get_emb_sz to calculate appropriate embedding sizes
  2. create a Tabular model
• Tabular model: https://docs.fast.ai/tabular.model.html
  • takes columns of continuous (continuous variables are concatenated as well) or categorical data (Categorical independent variables are passed through an embedding and concatenated)
  • predicts a category (a classification model) or a continuous value (a regression model)

Ensembling

Bagging different kinds of models

Bagging: Just like how we can ensemble a bunch of decision trees together into a random forest by averaging the predictions, we can ensemble the random forest with the neural network.

PyTorch model and our sklearn model create data of different types:

• PyTorch gives us a rank-2 tensor (i.e, a column matrix), whereas
• NumPy gives us a rank-1 array (a vector).

Easily fixable: "squeeze removes any unit axes from a tensor, and to_np converts it into a NumPy array"

from fastai.torch_core import to_np
rf_preds = m.predict(valid_xs_time)
ens_preds = (to_np(preds.squeeze()) + rf_preds) /2

r_mse(ens_preds,valid_y)

0.221364

^ better than either by itself (.... EJ: again, by just a little bit...)

Boosting

Boosting: instead of averaging the predictions, add them

Procedure:

• Train a small model that underfits your dataset.
• Calculate the predictions in the training set for this model.
• Subtract the predictions from the targets; these are called the "residuals" and represent the error for each point in the training set.
• Go back to step 1, but instead of using the original targets, use the residuals as the targets for the training.
• Continue doing this until you reach some stopping criterion, such as a maximum number of trees, or you observe your validation set error getting worse.

Effect:

• Each new tree will be attempting to fit the error of all of the previous trees combined.
• Because we are continually creating new residuals, by subtracting the predictions of each new tree from the residuals from the previous tree, the residuals will get smaller and smaller.

Predictions with an ensemble of boosted trees:

• calculate the predictions from each tree, and then
• add them all together

Boosting Strategies

• Gradient boosting machines (GBMs) and
• gradient boosted decision trees (GBDTs)

Library that implements this circa 2022: XGBoost

Boosting Downfalls: overfitting

"Using more trees in a random forest does not lead to overfitting, because each tree is independent of the others. But in a boosted ensemble, the more trees you have, the better the training error becomes, and eventually you will see overfitting on the validation set."

Gradient boosting was in rapid development when fastbook was being published, so the authors didn't want to spend too much time showing how to "train a gradient boosted tree ensemble".

"As we write this, sklearn has just added a HistGradientBoostingRegressor class that provides excellent performance. There are many hyperparameters to tweak for this class, and for all gradient boosted tree methods we have seen. Unlike random forests, gradient boosted trees are extremely sensitive to the choices of these hyperparameters; in practice, most people use a loop that tries a range of different hyperparameters to find the ones that work best."

Gradient Boosting: 2026 (according to ChatGPT)

Instead of one implementation, there are now three dominant gradient boosting libraries.

Library	Strengths	Typical use
XGBoost	Extremely robust, mature, excellent defaults	General-purpose baseline
LightGBM	Very fast on large datasets, low memory usage	Large tabular datasets
CatBoost	Handles categorical variables exceptionally well with minimal preprocessing	Mixed numerical/categorical datasets

Hyperparameters for Gradient Boosting:

• learning_rate
• max_leaf_nodes (or max_depth)
• min_samples_leaf
• max_bins
• l2_regularization
• max_features
• n_estimators

Methods for searching parameter space:

• Optuna
• Bayesian optimization
• RandomizedSearchCV
• HalvingGridSearchCV
• AutoML systems

special note: "One thing many people expected in 2020 was that tabular neural networks would replace boosting. That largely didn't happen. Deep learning has become dominant for images, language, audio, and many multimodal tasks, but for ordinary structured tables, gradient-boosted trees remain extremely competitive. Recent competition analyses still show XGBoost, LightGBM, and CatBoost as the most common winning approaches, though AutoML systems and newer tabular foundation models are starting to appear in some competitions."

Embeddings + small decision tree might be a speedy replacement for inference time predictions

"you can get much of the performance improvement of a neural network without actually having to use a neural network at inference time. You could just use an embedding, which is literally just an array lookup, along with a small decision tree ensemble.

These embeddings need not even be necessarily learned separately for each model or task in an organization. Instead, once a set of embeddings are learned for some column for some task, they could be stored in a central place, and reused across multiple models. In fact, we know from private communication with other practitioners at large companies that this is already happening in many places."

Takeaways from this lesson on three methods for tabular data

• Random forests are the easiest to train, because they are extremely resilient to hyperparameter choices and require very little preprocessing. They are very fast to train, and should not overfit if you have enough trees. But they can be a little less accurate, especially if extrapolation is required, such as predicting future time periods.

• Gradient boosting machines in theory are just as fast to train as random forests, but in practice you will have to try lots of different hyperparameters. They can overfit, but they are often a little more accurate than random forests.

• Neural networks take the longest time to train, and require extra preprocessing, such as normalization; this normalization needs to be used at inference time as well. They can provide great results and extrapolate well, but only if you are careful with your hyperparameters and take care to avoid overfitting.

How to choose a strategy for tabular data

1. "We suggest starting your analysis with a random forest. This will give you a strong baseline, and you can be confident that it's a reasonable starting point. You can then use that model for feature selection and partial dependence analysis, to get a better understanding of your data."

2. "From that foundation, you can try neural nets and GBMs, and if they give you significantly better results on your validation set in a reasonable amount of time, you can use them. If decision tree ensembles are working well for you, try adding the embeddings for the categorical variables to the data, and see if that helps your decision trees learn better."