AndreaTacchella commited on
Commit
63e6768
·
verified ·
1 Parent(s): d101cc2

Trim README explanation to a brief code comment

Browse files
Files changed (1) hide show
  1. README.md +5 -12
README.md CHANGED
@@ -73,8 +73,6 @@ For details see the full paper.
73
 
74
  ## Usage
75
 
76
- > **Note on loading the model:** the checkpoint is saved in `BertForMaskedLM` format (used during fine-tuning), but for embedding extraction you should load it with `BertModel` instead, as shown below. `BertModel` loads only the encoder and skips the masked-language-modeling head, which projects every token's hidden state to a logit over the full ~48k vocabulary -- an expensive computation that fine-tuning needs but embedding extraction never uses. This has no effect on the resulting embeddings (the MLM head sits downstream of the hidden states and doesn't feed back into them) but is substantially faster. Only load `BertForMaskedLM` if you actually need masked-token prediction.
77
-
78
  > **Minimal example:** build a batch where only the abstract is truncated (on the right), while the title, tech tokens, and all special tokens are preserved. Then run the model and extract the `[CLS]` embedding for each example.
79
 
80
  ```python
@@ -86,11 +84,12 @@ from transformers import BertTokenizer, BertModel
86
  # ---------------------------------------------------------------------------
87
  MODEL_NAME = "AndreaTacchella/TechTokenBert"
88
  tokenizer = BertTokenizer.from_pretrained(MODEL_NAME)
89
- model = BertModel.from_pretrained(MODEL_NAME) # encoder only, no MLM head -- see note above
 
 
 
 
90
  model.eval()
91
- # You will see a warning about "pooler.dense.weight/bias" being newly initialized: this is
92
- # expected and harmless. The checkpoint has no pooler (BertForMaskedLM doesn't use one), so
93
- # BertModel creates a random one; we only ever read `last_hidden_state`, never `pooler_output`.
94
 
95
  # ---------------------------------------------------------------------------
96
  # 2. Toy data (3 rows)
@@ -202,12 +201,6 @@ for i, ids in enumerate(input_ids):
202
  # a tensor of shape (num_tech_tokens, hidden_size)
203
  ```
204
 
205
- ### Performance tips for embedding large collections
206
-
207
- - **Sort by length before batching.** Grouping patents of similar title+abstract length into the same batch minimizes padding, which can nearly double throughput compared to processing patents in arbitrary order. Since embeddings are typically stored keyed by patent ID, results don't need to be restored to the original order afterwards.
208
- - **Use fp16 autocast on GPU** (`torch.autocast(device_type="cuda", dtype=torch.float16)` around the forward pass) for a further speedup with no measurable loss in embedding quality.
209
- - Combined with loading via `BertModel` (see note above), these changes gave roughly a 3x throughput increase in practice (RTX 4090, batch size ~24).
210
-
211
  ---
212
 
213
  ## Input Format
 
73
 
74
  ## Usage
75
 
 
 
76
  > **Minimal example:** build a batch where only the abstract is truncated (on the right), while the title, tech tokens, and all special tokens are preserved. Then run the model and extract the `[CLS]` embedding for each example.
77
 
78
  ```python
 
84
  # ---------------------------------------------------------------------------
85
  MODEL_NAME = "AndreaTacchella/TechTokenBert"
86
  tokenizer = BertTokenizer.from_pretrained(MODEL_NAME)
87
+ # The model was fine-tuned with an MLM head, kept in the checkpoint for more advanced
88
+ # uses (e.g. tech-code prediction). It is not needed for embeddings, so we load the
89
+ # encoder only via BertModel; the resulting "pooler.dense" newly-initialized warning
90
+ # is expected and harmless -- we only read last_hidden_state, not pooler_output.
91
+ model = BertModel.from_pretrained(MODEL_NAME)
92
  model.eval()
 
 
 
93
 
94
  # ---------------------------------------------------------------------------
95
  # 2. Toy data (3 rows)
 
201
  # a tensor of shape (num_tech_tokens, hidden_size)
202
  ```
203
 
 
 
 
 
 
 
204
  ---
205
 
206
  ## Input Format