Training Transformers to Produce Patent Phrase Similarity Scores¶

In [1]:
import fastai
from pathlib import Path
In [2]:
import os
import shutil
import kagglehub

Step 1: Grab the data from Kaggle¶

Actually there was a Step 0: read about the data: what files are included, what do the columns mean and what kinds of values can we find in them, and what kind of metric kaggle is using for scores.

In [3]:
# grab the data from kaggle - works because I have ~/.kaggle/access_token file
cache_path = kagglehub.competition_download('us-patent-phrase-to-phrase-matching')
In [4]:
print("Path to competition files:", cache_path)
# Path to competition files: /Users/emma/.cache/kagglehub/competitions/us-patent-phrase-to-phrase-matching

Path to competition files: /Users/emma/.cache/kagglehub/competitions/us-patent-phrase-to-phrase-matching
In [5]:
# bring the dataset from local default kaggle cache to this notebook's dir
local_dest = Path("./us-patent-phrase-to-phrase-matching")

if not os.path.exists(local_dest):
    shutil.copytree(cache_path, local_dest)
    print(f"Files copied from cache to local folder: {local_dest}")
else:
    print("Folder already exists.")

print("Files:", os.listdir(local_dest))

Folder already exists.
Files: ['test.csv', 'train.csv', 'sample_submission.csv']
In [6]:
import pandas as pd
In [7]:
df_train = pd.read_csv(local_dest/'train.csv')

What have we got?¶

In [8]:
df_train.head()
Out[8]:
id anchor target context score
0 37d61fd2272659b1 abatement abatement of pollution A47 0.50
1 7b9652b17b68b7a4 abatement act of abating A47 0.75
2 36d72442aefd8232 abatement active catalyst A47 0.25
3 5296b0c19e1ce60e abatement eliminating process A47 0.50
4 54c1e3b9184cb5b6 abatement forest region A47 0.00
In [9]:
type(df_train['score'][0])
Out[9]:
numpy.float64
In [10]:
df_train.dtypes
Out[10]:
id          object
anchor      object
target      object
context     object
score      float64
dtype: object
In [11]:
df_test = pd.read_csv(local_dest/'test.csv')
In [12]:
df_test.head()
Out[12]:
id anchor target context
0 4112d61851461f60 opc drum inorganic photoconductor drum G02
1 09e418c93a776564 adjust gas flow altering gas flow F23
2 36baf228038e314b lower trunnion lower locating B60
3 1f37ead645e7f0c8 cap component upper portion D06
4 71a5b6ad068d531f neural stimulation artificial neural network H04
In [13]:
df_sample_submission = pd.read_csv(local_dest/'sample_submission.csv')
In [14]:
df_sample_submission.head()
Out[14]:
id score
0 4112d61851461f60 0
1 09e418c93a776564 0
2 36baf228038e314b 0
3 1f37ead645e7f0c8 0
4 71a5b6ad068d531f 0
In [15]:
# i saw one instance of anchor='neural stimulation' in test.csv and was curious if/how it was represented in train.csv
# and what targets it mapped to; looks like there are 55 of them (from adding .count() to the end)
df_train[df_train['anchor'].str.contains('neural stimulation', na=False)]
Out[15]:
id anchor target context score
21037 baba9c4f0962c4b2 neural stimulation artificial neural network A61 0.00
21038 abc28aee01d27890 neural stimulation axons stimulation A61 0.50
21039 2dc1b13a460ca64c neural stimulation cable like bundle A61 0.25
21040 cb727a72c1a2700b neural stimulation deliver neural stimulation A61 0.50
21041 0df7dc5df17955ea neural stimulation electrical stimulation A61 0.50
21042 14b8d6154c6ca5a4 neural stimulation nerve stimulation A61 0.50
21043 6bd2e81d1aed66de neural stimulation nervous systemstimulation A61 0.50
21044 11e4b59194412e3e neural stimulation neural communication A61 0.25
21045 a43e849b746505ea neural stimulation neural control A61 0.00
21046 56d138ca9233d484 neural stimulation neural nerve A61 0.25
21047 7473080515892c85 neural stimulation neural network A61 0.00
21048 d7d5de1e9905c03e neural stimulation neural stimulation device A61 0.50
21049 d5f0884fbc5b5d43 neural stimulation neural stimulation signal A61 0.50
21050 95a09d3501a21a17 neural stimulation neural stimulation therapy A61 0.50
21051 39adff29cb9327fe neural stimulation neural tube A61 0.00
21052 e0c998653413fb1d neural stimulation neuromodulation A61 0.75
21053 481b8dc03287bea8 neural stimulation neuromuscular electrical stimulation A61 0.75
21054 7a326eba6e9f74ea neural stimulation neurostimulator A61 0.50
21055 5b30ced7770eecc4 neural stimulation parameter A61 0.25
21056 6f6d9f597b737713 neural stimulation peripheral nervous system A61 0.50
21057 570d31a957c3c421 neural stimulation pulses A61 0.50
21058 4e05bed83a8b9434 neural stimulation simulated neural networks A61 0.00
21059 8751d0501c35d80b neural stimulation stimulation machine A61 0.00
21060 8add2955dc0958c7 neural stimulation transmit signal A61 0.25
21061 f8724f396b800f6f neural stimulation anti stimulant composition H04 0.25
21062 71a5b6ad068d531f neural stimulation artificial neural network H04 0.00
21063 466eccc3c7bac37f neural stimulation convolutional neural network H04 0.00
21064 1c32c055a8d4c9c0 neural stimulation de energize electronic components H04 0.25
21065 625e0202be8790f5 neural stimulation de energize the laser H04 0.25
21066 8c7d208c7602ff4f neural stimulation deliver neural stimulation H04 0.75
21067 a6c2ab570f800398 neural stimulation electrical nerve stimulation H04 0.75
21068 87664d83b5eb5961 neural stimulation electrical stimulation H04 0.75
21069 77f55e0ca2f985c2 neural stimulation electrical therapy H04 0.75
21070 d0d3e8a06e00b90a neural stimulation implant H04 0.25
21071 cebe5aeacde29cb6 neural stimulation implant control H04 0.50
21072 e0a8e1e950106bab neural stimulation implantable control H04 0.50
21073 9f0833059e08e87d neural stimulation medical H04 0.25
21074 dddb4b80bf87a534 neural stimulation medical stimulation H04 0.50
21075 2034a6557e5282e0 neural stimulation nerve stimulation H04 1.00
21076 b47f34efc77c0064 neural stimulation neural signal H04 0.25
21077 85cf3a591c40b071 neural stimulation neural stimulation device H04 0.50
21078 533b2d385f9f4142 neural stimulation neural stimulation parameters H04 0.75
21079 86a6c2ead9756d9a neural stimulation neural stimulation signal H04 0.50
21080 843784b10781b1af neural stimulation neural stimulation therapy H04 0.50
21081 2e9a5d79196fd5a6 neural stimulation neural stimulator H04 0.50
21082 41ae9c67de2e8165 neural stimulation neural stimuli H04 0.75
21083 e9d89aba0f739cde neural stimulation neurological stimulation H04 1.00
21084 46d1e96f5505ac52 neural stimulation neuroscience H04 0.00
21085 d60cc886efad9890 neural stimulation quantum simulation H04 0.00
21086 28501c0820eba8fb neural stimulation reinforced neural network H04 0.00
21087 4aa804d58c6deae1 neural stimulation simulation hypothesis H04 0.00
21088 677afc6fe4e08053 neural stimulation stimulation H04 0.25
21089 7e897f9e9d27ba46 neural stimulation stimulation signal H04 0.25
21090 77413f297bfcf041 neural stimulation stimulation sites H04 0.25
21091 b30688496db8abb9 neural stimulation tissue stimulation H04 0.50
In [16]:
# how many unique contexts do we have? 106 for training data.
df_train['context'].sort_values().unique()
Out[16]:
array(['A01', 'A21', 'A22', 'A23', 'A24', 'A41', 'A43', 'A44', 'A45',
       'A46', 'A47', 'A61', 'A62', 'A63', 'B01', 'B02', 'B03', 'B05',
       'B07', 'B08', 'B21', 'B22', 'B23', 'B24', 'B25', 'B27', 'B28',
       'B29', 'B31', 'B32', 'B41', 'B44', 'B60', 'B61', 'B62', 'B63',
       'B64', 'B65', 'B66', 'B67', 'B81', 'C01', 'C02', 'C03', 'C04',
       'C06', 'C07', 'C08', 'C09', 'C10', 'C11', 'C12', 'C13', 'C14',
       'C21', 'C22', 'C23', 'C25', 'D01', 'D03', 'D04', 'D05', 'D06',
       'D21', 'E01', 'E02', 'E03', 'E04', 'E05', 'E06', 'E21', 'F01',
       'F02', 'F03', 'F04', 'F15', 'F16', 'F17', 'F21', 'F22', 'F23',
       'F24', 'F25', 'F26', 'F27', 'F28', 'F41', 'F42', 'G01', 'G02',
       'G03', 'G04', 'G05', 'G06', 'G07', 'G08', 'G09', 'G10', 'G11',
       'G16', 'G21', 'H01', 'H02', 'H03', 'H04', 'H05'], dtype=object)
In [17]:
# 29 for test
df_test['context'].sort_values().unique()
Out[17]:
array(['A44', 'A61', 'A63', 'B01', 'B21', 'B22', 'B23', 'B60', 'B61',
       'B63', 'C07', 'C10', 'C12', 'C23', 'D06', 'E04', 'F01', 'F02',
       'F04', 'F16', 'F23', 'G01', 'G02', 'G05', 'G11', 'H02', 'H03',
       'H04', 'H05'], dtype=object)
In [18]:
import numpy as np
In [19]:
contexts = set(df_test['context'].unique().tolist())
contexts.update(df_train['context'].unique().tolist())
print(len(contexts))
np.array(sorted(list(contexts)))

106
Out[19]:
array(['A01', 'A21', 'A22', 'A23', 'A24', 'A41', 'A43', 'A44', 'A45',
       'A46', 'A47', 'A61', 'A62', 'A63', 'B01', 'B02', 'B03', 'B05',
       'B07', 'B08', 'B21', 'B22', 'B23', 'B24', 'B25', 'B27', 'B28',
       'B29', 'B31', 'B32', 'B41', 'B44', 'B60', 'B61', 'B62', 'B63',
       'B64', 'B65', 'B66', 'B67', 'B81', 'C01', 'C02', 'C03', 'C04',
       'C06', 'C07', 'C08', 'C09', 'C10', 'C11', 'C12', 'C13', 'C14',
       'C21', 'C22', 'C23', 'C25', 'D01', 'D03', 'D04', 'D05', 'D06',
       'D21', 'E01', 'E02', 'E03', 'E04', 'E05', 'E06', 'E21', 'F01',
       'F02', 'F03', 'F04', 'F15', 'F16', 'F17', 'F21', 'F22', 'F23',
       'F24', 'F25', 'F26', 'F27', 'F28', 'F41', 'F42', 'G01', 'G02',
       'G03', 'G04', 'G05', 'G06', 'G07', 'G08', 'G09', 'G10', 'G11',
       'G16', 'G21', 'H01', 'H02', 'H03', 'H04', 'H05'], dtype='<U3')
In [20]:
# total rows: 36473
len(df_train)
Out[20]:
36473
In [21]:
# how many unique anchors? 733 // and how many unique targets? 29,340
# in 106 unique contexts
print(len(df_train['anchor'].unique()))
print(len(df_train['target'].unique()))
print(len(df_train['context'].unique()))

733
29340
106
In [22]:
print(len(df_test['anchor'].unique()))
print(len(df_test['target'].unique()))
print(len(df_test['context'].unique()))

34
36
29
In [23]:
print(len(df_test))
print(len(df_sample_submission))

36
36
In [24]:
# ok so the idea is to submit the scores for the ids in test.csv in the sample_submission.csv?
# all of the same ids are in test as in sample_submission
df_sample_submission[df_sample_submission['id'].str.contains('71a5b6ad068d531f', na=False)]
Out[24]:
id score
4 71a5b6ad068d531f 0

The scores are float64, which are apparentlly overkill¶

In [25]:
print(df_train['score'].dtypes)
df_train['score'] = df_train['score'].astype('float32')
print(df_train['score'].dtypes)

float64
float32

Step 2: Prepare the dds = ('train' as subset of train.csv, 'test' as validation as subset of train.csv), and test_ds = (test.csv) for the transformer model¶

In [26]:
df_train
Out[26]:
id anchor target context score
0 37d61fd2272659b1 abatement abatement of pollution A47 0.50
1 7b9652b17b68b7a4 abatement act of abating A47 0.75
2 36d72442aefd8232 abatement active catalyst A47 0.25
3 5296b0c19e1ce60e abatement eliminating process A47 0.50
4 54c1e3b9184cb5b6 abatement forest region A47 0.00
... ... ... ... ... ...
36468 8e1386cbefd7f245 wood article wooden article B44 1.00
36469 42d9e032d1cd3242 wood article wooden box B44 0.50
36470 208654ccb9e14fa3 wood article wooden handle B44 0.50
36471 756ec035e694722b wood article wooden material B44 0.75
36472 8d135da0b55b8c88 wood article wooden substrate B44 0.50

36473 rows × 5 columns

In [27]:
df_train.describe(include='object')
Out[27]:
id anchor target context
count 36473 36473 36473 36473
unique 36473 733 29340 106
top 37d61fd2272659b1 component composite coating composition H01
freq 1 152 24 2186

Add a new column called 'input' which concatenates the anchor, target, and context¶

In [28]:
# putting "context" first in the text order is apparently useful for cross validation error
df_train['input'] = 'TEXT1: ' + df_train.context + '; TEXT2: ' + df_train.target + '; ANC1: ' + df_train.anchor
In [29]:
### Create a Dataset object
from datasets import Dataset,DatasetDict

ds = Dataset.from_pandas(df_train)
In [30]:
ds
Out[30]:
Dataset({
    features: ['id', 'anchor', 'target', 'context', 'score', 'input'],
    num_rows: 36473
})

Choose the model, and then set up the same tokenizer used for text that pretrained the model¶

In [31]:
model_nm = 'microsoft/deberta-v3-small'
In [32]:
from transformers import AutoModelForSequenceClassification,AutoTokenizer
tokz = AutoTokenizer.from_pretrained(model_nm)
In [33]:
print(list(tokz.vocab.keys())[:10])
print(list(tokz.vocab.values())[:10])

['▁Thankfully', '▁919', '▁sh', '▁docks', '▁mechanism', 'evil', '▁Rubio', '▁preparedness', '▁Exchange', '▁cafeteria']
[9651, 92279, 111719, 85442, 12501, 82185, 8349, 54165, 98976, 121420]
In [34]:
# set up a tokenization function to 
def tok_func(x): return tokz(x["input"])
In [35]:
tok_ds = ds.map(tok_func, batched=True)

Map:   0%|          | 0/36473 [00:00<?, ? examples/s]
In [36]:
tok_ds
Out[36]:
Dataset({
    features: ['id', 'anchor', 'target', 'context', 'score', 'input', 'input_ids', 'token_type_ids', 'attention_mask'],
    num_rows: 36473
})

Rename the 'score' to 'labels' because the transformers expects the learned column to be called that¶

In [37]:
tok_ds = tok_ds.rename_columns({'score':'labels'})

Build a validation set from the training data -- which happens to be called 'test'¶

In [38]:
dds = tok_ds.train_test_split(0.25, seed=42)
dds
Out[38]:
DatasetDict({
    train: Dataset({
        features: ['id', 'anchor', 'target', 'context', 'labels', 'input', 'input_ids', 'token_type_ids', 'attention_mask'],
        num_rows: 27354
    })
    test: Dataset({
        features: ['id', 'anchor', 'target', 'context', 'labels', 'input', 'input_ids', 'token_type_ids', 'attention_mask'],
        num_rows: 9119
    })
})

Create a dataset from the test.csv data for the actual test (not validation) data¶

In [39]:
df_test['input'] = 'TEXT1: ' + df_test.context + '; TEXT2: ' + df_test.target + '; ANC1: ' + df_test.anchor
test_ds = Dataset.from_pandas(df_test).map(tok_func, batched=True)

Map:   0%|          | 0/36 [00:00<?, ? examples/s]

Step 3: Training the Model¶

In [40]:
from transformers import TrainingArguments,Trainer
In [41]:
# set batch size and number of epochs
# Gemini advice: "For fine-tuning or inference with microsoft/deberta-v3-small on a unified memory Mac, 
# use a per-device batch size of 8 to 16 for training, or 16 to 64 for inference. 
# Leverage Apple's Metal Performance Shaders (MPS) via the Hugging Face Transformers backend to optimize memory.
# If you encounter memory errors with longer sequences, use Gradient Accumulation 
# (e.g., gradient_accumulation_steps=4 with a batch size of 8 gives an effective batch size of 32) 
# to hit standard target batch sizes without pushing memory pressure too high."

# GPU defaults
#bs = 128
#epochs = 4
#lr = 8e-5

# ---- above the line are the values in the notebook by Jeremy Howard in the fastai course
# ---- below the line are values that gemini suggested for the TrainingArguments object

# Pass the per-device batch sizes into TrainingArguments:
# per_device_train_batch_size=8,   # Batch size per device during training
# per_device_eval_batch_size=16,   # Batch size per device during evaluation
# & 
# Optional: Combine with gradient accumulation to mimic a larger batch size
# gradient_accumulation_steps=4,   # Total effective train batch size = 8 * 4 = 32
# use_mps_device=True,             # Ensures it targets your Mac M4 GPU
In [42]:
# These are the values in the notebook by Jeremy Howard in the fastai course
# args = TrainingArguments('outputs', 
#                          learning_rate=lr, 
#                          warmup_ratio=0.1, 
#                          lr_scheduler_type='cosine', 
#                          fp16=True,
#                          evaluation_strategy="epoch", 
#                          per_device_train_batch_size=bs, 
#                          per_device_eval_batch_size=bs*2,
#                          num_train_epochs=epochs, 
#                          weight_decay=0.01, 
#                          report_to='none')

# These are arguments that Gemini suggested for later libraries and Macbook Pro M4
#bs = 8       # Safe training batch size for deberta-v3-small on 24GB RAM
lr = 3e-5     # Recommended learning rate
epochs = 3    # Balanced epoch count to prevent overfitting

# ChatGPT recommendations after the kernel died on one run
# per_device_train_batch_size=2
# per_device_eval_batch_size=4
# gradient_accumulation_steps=8
# dataloader_pin_memory=False
# dataloader_num_workers=0
# fp16=False
# bf16=False

bs = 2

args = TrainingArguments('outputs', 
                         learning_rate=lr, 
                         warmup_steps=0.1, # new term; old was warmup_ratio=0.1, 
                         lr_scheduler_type='cosine', 

                         # --- MAC M4 SPECIFIC OPTIMIZATIONS ---
                         bf16=False,                         # Use Bfloat16 mixed precision (M4 native & stable) #was true
                         fp16=False,                         # # was unset, adding after kernel died on one run
                         #use_mps_device=True,               # default now to force Hugging Face to use Apple's Metal backend
                         dataloader_num_workers=0,           # Utilizes Mac's high-efficiency CPU cores for data loading (if =2)
                         dataloader_pin_memory=False,        # something for GPUs
                         # -------------------------------------

                         logging_strategy="steps",
                         logging_steps=20,
                         
                         eval_strategy="epoch", 
                         per_device_train_batch_size=bs, 
                         per_device_eval_batch_size=bs*2,
                         gradient_accumulation_steps=8, # adding after kernel died on one run
                         num_train_epochs=epochs, 
                         weight_decay=0.01, 
                         report_to='none')
In [43]:
# args for "loading the best model at the end", e.g. if you have too many training steps and you start to overfit, and you want to
# submit your test data labels from the best model
    # eval_strategy="epoch",
    # save_strategy="epoch",
    # load_best_model_at_end=True,
    # metric_for_best_model="pearson",
    # greater_is_better=True,

Set up compute_metrics for each run¶

from ChatGPT:

  • Pearson is the metric that determines performance on this task.
  • Spearman tells you whether the model is getting the ranking of similarities right, even if the absolute calibration is a bit off.
  • RMSE helps diagnose whether two models with similar Pearson scores differ in how close their predicted values are to the targets.
In [44]:
# setting up compute_metrics
import numpy as np
from scipy.stats import pearsonr
from sklearn.metrics import mean_squared_error

def compute_metrics(eval_pred):
    predictions, labels = eval_pred

    predictions = np.asarray(predictions).reshape(-1)
    labels = np.asarray(labels).reshape(-1)

    return {
        "pearson": pearsonr(labels, predictions).statistic,
        "rmse": np.sqrt(mean_squared_error(labels, predictions)),
    }

# ChatGPT: the main red flags would be NaN, Pearson staying near zero, or predictions collapsing to nearly one constant value.
In [46]:
import torch
# set up the model with pretrained values
# if num_labels = 1, Regression
#                    outputs single continuous number and uses MSELoss (Mean Squared Error) 
#                    to measure how close your model's prediction is to a target number.
# if num_labels > 1, Classificaton
#                    outputs multiple prediction scores (logits), uses CrossEntropyLoss 
#                    to handle discrete categories (like predicting 0, 1, or 2)

model = AutoModelForSequenceClassification.from_pretrained(model_nm, 
                                                           num_labels=1,
                                                           problem_type="regression", # this is implied by num_labels=1
                                                           dtype=torch.float32,)      # adding after kernel failure

Loading weights:   0%|          | 0/102 [00:00<?, ?it/s]
[transformers] DebertaV2ForSequenceClassification LOAD REPORT from: microsoft/deberta-v3-small
Key                                     | Status     | 
----------------------------------------+------------+-
mask_predictions.dense.bias             | UNEXPECTED | 
mask_predictions.LayerNorm.weight       | UNEXPECTED | 
mask_predictions.LayerNorm.bias         | UNEXPECTED | 
mask_predictions.classifier.bias        | UNEXPECTED | 
lm_predictions.lm_head.dense.weight     | UNEXPECTED | 
lm_predictions.lm_head.LayerNorm.bias   | UNEXPECTED | 
lm_predictions.lm_head.bias             | UNEXPECTED | 
lm_predictions.lm_head.LayerNorm.weight | UNEXPECTED | 
lm_predictions.lm_head.dense.bias       | UNEXPECTED | 
mask_predictions.dense.weight           | UNEXPECTED | 
mask_predictions.classifier.weight      | UNEXPECTED | 
classifier.weight                       | MISSING    | 
pooler.dense.weight                     | MISSING    | 
pooler.dense.bias                       | MISSING    | 
classifier.bias                         | MISSING    | 

Notes:
- UNEXPECTED:	can be ignored when loading from different task/architecture; not ok if you expect identical arch.
- MISSING:	those params were newly initialized because missing from the checkpoint. Consider training on your downstream task.
In [47]:
# set up the Trainer (this is called a Learner in the fastai library)
trainer = Trainer(model, 
                  args, 
                  train_dataset=dds['train'], 
                  eval_dataset=dds['test'],
                  processing_class=tokz, # tokenizer is now called processing_class
                  compute_metrics=compute_metrics)
In [48]:
# for debugging
# import torch

# batch = next(iter(trainer.get_train_dataloader()))
# batch = {
#     key: value.to(model.device)
#     for key, value in batch.items()
# }

# model.eval()
# with torch.no_grad():
#     output = model(**batch)

# print(output.logits.shape)
# print(output.loss)
# # results in
# # torch.Size([2, 1])
# # tensor(0.6705, device='mps:0')
In [49]:
print(tokz.name_or_path)

print(tokz.special_tokens_map)

print(model.config)

microsoft/deberta-v3-small
{'bos_token': '[CLS]', 'eos_token': '[SEP]', 'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'mask_token': '[MASK]'}
DebertaV2Config {
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": null,
  "dtype": "float32",
  "eos_token_id": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "id2label": {
    "0": "LABEL_0"
  },
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "label2id": {
    "LABEL_0": 0
  },
  "layer_norm_eps": 1e-07,
  "legacy": true,
  "max_position_embeddings": 512,
  "max_relative_positions": -1,
  "model_type": "deberta-v2",
  "norm_rel_ebd": "layer_norm",
  "num_attention_heads": 12,
  "num_hidden_layers": 6,
  "pad_token_id": 0,
  "pooler_dropout": 0.0,
  "pooler_hidden_act": "gelu",
  "pooler_hidden_size": 768,
  "pos_att_type": [
    "p2c",
    "c2p"
  ],
  "position_biased_input": false,
  "position_buckets": 256,
  "problem_type": "regression",
  "relative_attention": true,
  "share_att_key": true,
  "tie_word_embeddings": true,
  "transformers_version": "5.9.0",
  "type_vocab_size": 0,
  "use_cache": false,
  "vocab_size": 128100
}
In [50]:
print(model.config.bos_token_id)
print(model.config.eos_token_id)

print(tokz.bos_token_id)
print(tokz.eos_token_id)

None
None
1
2
In [51]:
trainer.train()

[transformers] The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'eos_token_id': 2, 'bos_token_id': 1}.
[5130/5130 1:05:34, Epoch 3/3]
Epoch Training Loss Validation Loss Pearson Rmse
1 0.218448 0.027939 0.795433 0.167150
2 0.159067 0.023375 0.823309 0.152890
3 0.117297 0.022655 0.829435 0.150516 

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
Out[51]:
TrainOutput(global_step=5130, training_loss=0.22436229297292162, metrics={'train_runtime': 3935.5088, 'train_samples_per_second': 20.852, 'train_steps_per_second': 1.304, 'total_flos': 431993702403300.0, 'train_loss': 0.22436229297292162, 'epoch': 3.0})

The trained model, 'trainer' is above!¶

In [52]:
preds = trainer.predict(test_ds).predictions.astype(float)
preds
Out[52]:
array([[ 0.50793177],
       [ 0.69482702],
       [ 0.53050959],
       [ 0.3323034 ],
       [-0.01405059],
       [ 0.54094654],
       [ 0.49185976],
       [-0.06247338],
       [ 0.286567  ],
       [ 1.07577682],
       [ 0.24871875],
       [ 0.25795841],
       [ 0.7375955 ],
       [ 0.71554035],
       [ 0.75742549],
       [ 0.43514419],
       [ 0.28614819],
       [-0.04117407],
       [ 0.58906227],
       [ 0.27050924],
       [ 0.38998127],
       [ 0.26151669],
       [ 0.05747152],
       [ 0.23423766],
       [ 0.57260197],
       [-0.04425101],
       [-0.04549832],
       [-0.04818187],
       [-0.04342794],
       [ 0.65266436],
       [ 0.35751534],
       [ 0.02258844],
       [ 0.69498122],
       [ 0.49264497],
       [ 0.39684594],
       [ 0.23492719]])
In [53]:
### The values have to be between 0 and 1!!!
preds = np.clip(preds, 0, 1)
In [54]:
preds
Out[54]:
array([[0.50793177],
       [0.69482702],
       [0.53050959],
       [0.3323034 ],
       [0.        ],
       [0.54094654],
       [0.49185976],
       [0.        ],
       [0.286567  ],
       [1.        ],
       [0.24871875],
       [0.25795841],
       [0.7375955 ],
       [0.71554035],
       [0.75742549],
       [0.43514419],
       [0.28614819],
       [0.        ],
       [0.58906227],
       [0.27050924],
       [0.38998127],
       [0.26151669],
       [0.05747152],
       [0.23423766],
       [0.57260197],
       [0.        ],
       [0.        ],
       [0.        ],
       [0.        ],
       [0.65266436],
       [0.35751534],
       [0.02258844],
       [0.69498122],
       [0.49264497],
       [0.39684594],
       [0.23492719]])
In [61]:
preds.reshape(-1)
Out[61]:
array([0.50793177, 0.69482702, 0.53050959, 0.3323034 , 0.        ,
       0.54094654, 0.49185976, 0.        , 0.286567  , 1.        ,
       0.24871875, 0.25795841, 0.7375955 , 0.71554035, 0.75742549,
       0.43514419, 0.28614819, 0.        , 0.58906227, 0.27050924,
       0.38998127, 0.26151669, 0.05747152, 0.23423766, 0.57260197,
       0.        , 0.        , 0.        , 0.        , 0.65266436,
       0.35751534, 0.02258844, 0.69498122, 0.49264497, 0.39684594,
       0.23492719])
In [63]:
submission = df_test[["id"]].copy()
submission["score"] = preds.reshape(-1)
submission.to_csv("submission_july13.csv", index=False)
In [60]:
import datasets

submission = datasets.Dataset.from_dict({
    'id': test_ds['id'],
    'score': preds.reshape(-1)
})

submission.to_csv('submission_july13.csv', index=False)

Creating CSV from Arrow format:   0%|          | 0/1 [00:00<?, ?ba/s]
Out[60]:
1192

Trying to figure out how to submit the trained model and its predictions to kaggle competition...¶

In [57]:
# ok i think this torch method isn't right for this notebook; chatgpt says you need to save tokens too
#torch.save(model.state_dict(), 'model.pth')

#the trainer has a save_model function
trainer.save_model("./patent_model.pth")

Writing model shards:   0%|          | 0/1 [00:00<?, ?it/s]
In [59]:
import shutil
shutil.make_archive("./patent_model", "zip", "./patent_model.pth")
Out[59]:
'/Users/emma/Documents/file_cabinet/people_projects/emma/deeplearning/nlp/patent_model.zip'

Apparently we have to upload this on kaggle to my private model¶

https://www.kaggle.com/work/models

Used to be called "datasets", I think

The upload worked, it's here: https://www.kaggle.com/models/emmjab/trained-patent-similarity-transformer-model

We can make a notebook for the kaggle competition¶

If we do this from the competition page, it automatically associates the datasets for the competition with the notebook, and a little "copy" button next to the files to get the correct path.

We have to add the model to the notebook -- i think we have to first associate the model with the competition from the private workspace page (or at least my model didn't seem to show up in the search for adding data to the notebook until i did this). When the model is added, it also has the copy path button.

We need to then make a whole notebook with all of our import library to

  • load in our trained model & tokens
  • load in the test data from the competition
  • run the prediction function on text.csv data from the competition (which we have just done locally)
  • generate the submissions.csv (which we had done locally to, but we weren't allowed to just upload this data by itself)