Trim README explanation to a brief code comment
Browse files
README.md
CHANGED
|
@@ -73,8 +73,6 @@ For details see the full paper.
|
|
| 73 |
|
| 74 |
## Usage
|
| 75 |
|
| 76 |
-
> **Note on loading the model:** the checkpoint is saved in `BertForMaskedLM` format (used during fine-tuning), but for embedding extraction you should load it with `BertModel` instead, as shown below. `BertModel` loads only the encoder and skips the masked-language-modeling head, which projects every token's hidden state to a logit over the full ~48k vocabulary -- an expensive computation that fine-tuning needs but embedding extraction never uses. This has no effect on the resulting embeddings (the MLM head sits downstream of the hidden states and doesn't feed back into them) but is substantially faster. Only load `BertForMaskedLM` if you actually need masked-token prediction.
|
| 77 |
-
|
| 78 |
> **Minimal example:** build a batch where only the abstract is truncated (on the right), while the title, tech tokens, and all special tokens are preserved. Then run the model and extract the `[CLS]` embedding for each example.
|
| 79 |
|
| 80 |
```python
|
|
@@ -86,11 +84,12 @@ from transformers import BertTokenizer, BertModel
|
|
| 86 |
# ---------------------------------------------------------------------------
|
| 87 |
MODEL_NAME = "AndreaTacchella/TechTokenBert"
|
| 88 |
tokenizer = BertTokenizer.from_pretrained(MODEL_NAME)
|
| 89 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 90 |
model.eval()
|
| 91 |
-
# You will see a warning about "pooler.dense.weight/bias" being newly initialized: this is
|
| 92 |
-
# expected and harmless. The checkpoint has no pooler (BertForMaskedLM doesn't use one), so
|
| 93 |
-
# BertModel creates a random one; we only ever read `last_hidden_state`, never `pooler_output`.
|
| 94 |
|
| 95 |
# ---------------------------------------------------------------------------
|
| 96 |
# 2. Toy data (3 rows)
|
|
@@ -202,12 +201,6 @@ for i, ids in enumerate(input_ids):
|
|
| 202 |
# a tensor of shape (num_tech_tokens, hidden_size)
|
| 203 |
```
|
| 204 |
|
| 205 |
-
### Performance tips for embedding large collections
|
| 206 |
-
|
| 207 |
-
- **Sort by length before batching.** Grouping patents of similar title+abstract length into the same batch minimizes padding, which can nearly double throughput compared to processing patents in arbitrary order. Since embeddings are typically stored keyed by patent ID, results don't need to be restored to the original order afterwards.
|
| 208 |
-
- **Use fp16 autocast on GPU** (`torch.autocast(device_type="cuda", dtype=torch.float16)` around the forward pass) for a further speedup with no measurable loss in embedding quality.
|
| 209 |
-
- Combined with loading via `BertModel` (see note above), these changes gave roughly a 3x throughput increase in practice (RTX 4090, batch size ~24).
|
| 210 |
-
|
| 211 |
---
|
| 212 |
|
| 213 |
## Input Format
|
|
|
|
| 73 |
|
| 74 |
## Usage
|
| 75 |
|
|
|
|
|
|
|
| 76 |
> **Minimal example:** build a batch where only the abstract is truncated (on the right), while the title, tech tokens, and all special tokens are preserved. Then run the model and extract the `[CLS]` embedding for each example.
|
| 77 |
|
| 78 |
```python
|
|
|
|
| 84 |
# ---------------------------------------------------------------------------
|
| 85 |
MODEL_NAME = "AndreaTacchella/TechTokenBert"
|
| 86 |
tokenizer = BertTokenizer.from_pretrained(MODEL_NAME)
|
| 87 |
+
# The model was fine-tuned with an MLM head, kept in the checkpoint for more advanced
|
| 88 |
+
# uses (e.g. tech-code prediction). It is not needed for embeddings, so we load the
|
| 89 |
+
# encoder only via BertModel; the resulting "pooler.dense" newly-initialized warning
|
| 90 |
+
# is expected and harmless -- we only read last_hidden_state, not pooler_output.
|
| 91 |
+
model = BertModel.from_pretrained(MODEL_NAME)
|
| 92 |
model.eval()
|
|
|
|
|
|
|
|
|
|
| 93 |
|
| 94 |
# ---------------------------------------------------------------------------
|
| 95 |
# 2. Toy data (3 rows)
|
|
|
|
| 201 |
# a tensor of shape (num_tech_tokens, hidden_size)
|
| 202 |
```
|
| 203 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 204 |
---
|
| 205 |
|
| 206 |
## Input Format
|