Instructions to use Verah/mistral-japanese-stabalelm-merge with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Verah/mistral-japanese-stabalelm-merge with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Verah/mistral-japanese-stabalelm-merge")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Verah/mistral-japanese-stabalelm-merge")
model = AutoModelForCausalLM.from_pretrained("Verah/mistral-japanese-stabalelm-merge", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Verah/mistral-japanese-stabalelm-merge with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Verah/mistral-japanese-stabalelm-merge"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Verah/mistral-japanese-stabalelm-merge",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/Verah/mistral-japanese-stabalelm-merge

SGLang

How to use Verah/mistral-japanese-stabalelm-merge with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Verah/mistral-japanese-stabalelm-merge" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Verah/mistral-japanese-stabalelm-merge",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Verah/mistral-japanese-stabalelm-merge" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Verah/mistral-japanese-stabalelm-merge",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use Verah/mistral-japanese-stabalelm-merge with Docker Model Runner:
```
docker model run hf.co/Verah/mistral-japanese-stabalelm-merge
```

This is a linear model merge of:

60% https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2

40% https://huggingface.co/stabilityai/japanese-stablelm-instruct-gamma-7b

I recommend following the Mistral chat template and prompting in English.

Evaluation

Tested on correct en-jp translation identification on the first 10k rows of https://huggingface.co/datasets/Verah/tatoeba_dedupe_en-jp_2024-March-01

Desired behaviour is to not accept any translation when we deliberaly test incorrect pairings from the dataset, and to not reject any translation when shown only correctly paired examples.

Model	False Admissions	False Rejections
Mistral Instruct	41	600
(This Model)	13	1839
JP Stable LM Gamma	9679	138
Hermes2DPO	20	598

I made the test harder by concatenating 3 paired sentences together, in the false admissions case 1 out of those 3 was incorrectly paired.

Model	False Admissions	False* Rejections
(This Model)	89	5508
Hermes2DPO	537	1458

This model also wanted to reject many "correct" translations, however 3 unrelated sentences back to back isn't a very correct thing to be doing, either.

Downloads last month: 56

Safetensors

Model size

7B params

Tensor type

F16