Instructions to use ngquocvinh/Humanizer-12B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use ngquocvinh/Humanizer-12B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ngquocvinh/Humanizer-12B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf ngquocvinh/Humanizer-12B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ngquocvinh/Humanizer-12B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf ngquocvinh/Humanizer-12B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf ngquocvinh/Humanizer-12B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf ngquocvinh/Humanizer-12B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf ngquocvinh/Humanizer-12B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf ngquocvinh/Humanizer-12B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/ngquocvinh/Humanizer-12B-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use ngquocvinh/Humanizer-12B-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ngquocvinh/Humanizer-12B-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ngquocvinh/Humanizer-12B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ngquocvinh/Humanizer-12B-GGUF:Q4_K_M

Ollama
How to use ngquocvinh/Humanizer-12B-GGUF with Ollama:
```
ollama run hf.co/ngquocvinh/Humanizer-12B-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use ngquocvinh/Humanizer-12B-GGUF with Docker Model Runner:
```
docker model run hf.co/ngquocvinh/Humanizer-12B-GGUF:Q4_K_M
```

Lemonade

How to use ngquocvinh/Humanizer-12B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull ngquocvinh/Humanizer-12B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Humanizer-12B-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

ngquocvinh commited on 10 days ago

Commit

81fa81a

verified ·

1 Parent(s): e29f954

Add Humanizer 12B Q5_K_M GGUF and measured metadata

Browse files

Files changed (6) hide show

.gitattributes +1 -0
README.md +1 -0
SHA256SUMS.txt +1 -0
humanizer-12b-Q5_K_M.gguf +3 -0
reproducibility/artifact-measurements.tsv +1 -0
reproducibility/manifest.md +8 -0

.gitattributes CHANGED Viewed

@@ -35,3 +35,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 humanizer-12b-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
 humanizer-12b-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text

 *tfevents* filter=lfs diff=lfs merge=lfs -text
 humanizer-12b-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
 humanizer-12b-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
+humanizer-12b-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text

README.md CHANGED Viewed

@@ -41,6 +41,7 @@ Sizes are measured from the local files in decimal GB (`bytes / 1,000,000,000`).
 | [humanizer-12b-bf16.gguf (upstream reference)](https://huggingface.co/jialinyyzz/humanizer/resolve/766072b26fbea83778a0ddc2dd577498098670bd/humanizer-12b-bf16.gguf) | 23.832050 | CPU generation passed in English and Chinese; upstream reference |
 | `humanizer-12b-Q8_0.gguf` | 12.669631 | CPU generation passed in English and Chinese |
 | `humanizer-12b-Q6_K.gguf` | 9.786006 | CPU generation passed in English and Chinese |
 The bilingual CPU smoke test loads each listed file through llama.cpp and confirms a non-empty generation for one English and one Chinese draft. These are runtime checks, not a quality or speed benchmark. No quantization fidelity scores are published yet.

 | [humanizer-12b-bf16.gguf (upstream reference)](https://huggingface.co/jialinyyzz/humanizer/resolve/766072b26fbea83778a0ddc2dd577498098670bd/humanizer-12b-bf16.gguf) | 23.832050 | CPU generation passed in English and Chinese; upstream reference |
 | `humanizer-12b-Q8_0.gguf` | 12.669631 | CPU generation passed in English and Chinese |
 | `humanizer-12b-Q6_K.gguf` | 9.786006 | CPU generation passed in English and Chinese |
+| `humanizer-12b-Q5_K_M.gguf` | 8.547252 | CPU generation passed in English and Chinese |
 The bilingual CPU smoke test loads each listed file through llama.cpp and confirms a non-empty generation for one English and one Chinese draft. These are runtime checks, not a quality or speed benchmark. No quantization fidelity scores are published yet.

SHA256SUMS.txt CHANGED Viewed

@@ -1,4 +1,5 @@
 f5ce56021b797156e85cee85f4363703e49eb82811761c898a1f266d5c77ad36  humanizer-12b-Q8_0.gguf
 bdb3124eda7f13a373560b913fcb19151265d2b9a8b54494d0c2ed7a469ab201  humanizer-12b-Q6_K.gguf
 8eda4ace9d180481d87add6aba5ea70331ee9263ac00e730f037253d4093e50d  prompt_format.json
 cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30  LICENSE

 f5ce56021b797156e85cee85f4363703e49eb82811761c898a1f266d5c77ad36  humanizer-12b-Q8_0.gguf
 bdb3124eda7f13a373560b913fcb19151265d2b9a8b54494d0c2ed7a469ab201  humanizer-12b-Q6_K.gguf
+0521ea308066f16d28c124d6c4b82deffef9965989ea95808b4323e5bb160eda  humanizer-12b-Q5_K_M.gguf
 8eda4ace9d180481d87add6aba5ea70331ee9263ac00e730f037253d4093e50d  prompt_format.json
 cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30  LICENSE

humanizer-12b-Q5_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:0521ea308066f16d28c124d6c4b82deffef9965989ea95808b4323e5bb160eda
+size 8547252448

reproducibility/artifact-measurements.tsv CHANGED Viewed

@@ -2,3 +2,4 @@ file	size_bytes	size_gb	status	sha256
 humanizer-12b-bf16.gguf (upstream reference)	23832049568	23.832050	upstream_bf16_cpu_smoke_pass	47d79b44c3e15ea2540f4edb63556f2b7e456067d7dd252a42ff103a26a51be9
 humanizer-12b-Q8_0.gguf	12669630688	12.669631	cpu_smoke_pass_en_zh	f5ce56021b797156e85cee85f4363703e49eb82811761c898a1f266d5c77ad36
 humanizer-12b-Q6_K.gguf	9786005728	9.786006	cpu_smoke_pass_en_zh	bdb3124eda7f13a373560b913fcb19151265d2b9a8b54494d0c2ed7a469ab201

 humanizer-12b-bf16.gguf (upstream reference)	23832049568	23.832050	upstream_bf16_cpu_smoke_pass	47d79b44c3e15ea2540f4edb63556f2b7e456067d7dd252a42ff103a26a51be9
 humanizer-12b-Q8_0.gguf	12669630688	12.669631	cpu_smoke_pass_en_zh	f5ce56021b797156e85cee85f4363703e49eb82811761c898a1f266d5c77ad36
 humanizer-12b-Q6_K.gguf	9786005728	9.786006	cpu_smoke_pass_en_zh	bdb3124eda7f13a373560b913fcb19151265d2b9a8b54494d0c2ed7a469ab201
+humanizer-12b-Q5_K_M.gguf	8547252448	8.547252	cpu_smoke_pass_en_zh	0521ea308066f16d28c124d6c4b82deffef9965989ea95808b4323e5bb160eda

reproducibility/manifest.md CHANGED Viewed

@@ -44,3 +44,11 @@ built with GNU 13.3.0 for Linux x86_64`.
 built with GNU 13.3.0 for Linux x86_64`.
 - Cgroup: `MemoryMax=32G`, `MemorySwapMax=0`; observed peak RSS-plus-file-cache 14791839744 bytes and peak cgroup swap 0 bytes.

 built with GNU 13.3.0 for Linux x86_64`.
 - Cgroup: `MemoryMax=32G`, `MemorySwapMax=0`; observed peak RSS-plus-file-cache 14791839744 bytes and peak cgroup swap 0 bytes.
+### Q5_K_M
+- Artifact: `humanizer-12b-Q5_K_M.gguf`, 8547252448 bytes, SHA256 `0521ea308066f16d28c124d6c4b82deffef9965989ea95808b4323e5bb160eda`.
+- Quantization command: `llama-quantize --max-buffer-size 512 --imatrix humanizer-12b.imatrix.gguf humanizer-12b-bf16.gguf humanizer-12b-Q5_K_M.gguf Q5_K_M 8`.
+- CPU bilingual `/completion` smoke: `PASS`; runtime `version: 0.6.0-dev (build 1, commit 9c2e0e491)
+built with GNU 13.3.0 for Linux x86_64`.
+- Cgroup: `MemoryMax=32G`, `MemorySwapMax=0`; observed peak RSS-plus-file-cache 13956263936 bytes and peak cgroup swap 0 bytes.