Instructions to use ngquocvinh/Humanizer-12B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use ngquocvinh/Humanizer-12B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ngquocvinh/Humanizer-12B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf ngquocvinh/Humanizer-12B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ngquocvinh/Humanizer-12B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf ngquocvinh/Humanizer-12B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf ngquocvinh/Humanizer-12B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf ngquocvinh/Humanizer-12B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf ngquocvinh/Humanizer-12B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf ngquocvinh/Humanizer-12B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/ngquocvinh/Humanizer-12B-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use ngquocvinh/Humanizer-12B-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ngquocvinh/Humanizer-12B-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ngquocvinh/Humanizer-12B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ngquocvinh/Humanizer-12B-GGUF:Q4_K_M

Ollama
How to use ngquocvinh/Humanizer-12B-GGUF with Ollama:
```
ollama run hf.co/ngquocvinh/Humanizer-12B-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use ngquocvinh/Humanizer-12B-GGUF with Docker Model Runner:
```
docker model run hf.co/ngquocvinh/Humanizer-12B-GGUF:Q4_K_M
```

Lemonade

How to use ngquocvinh/Humanizer-12B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull ngquocvinh/Humanizer-12B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Humanizer-12B-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

ngquocvinh commited on 9 days ago

Commit

5862762

verified ·

1 Parent(s): 81fa81a

Add Humanizer 12B Q4_K_M GGUF and measured metadata

Browse files

Files changed (6) hide show

.gitattributes +1 -0
README.md +1 -0
SHA256SUMS.txt +1 -0
humanizer-12b-Q4_K_M.gguf +3 -0
reproducibility/artifact-measurements.tsv +1 -0
reproducibility/manifest.md +8 -0

.gitattributes CHANGED Viewed

@@ -36,3 +36,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 humanizer-12b-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
 humanizer-12b-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
 humanizer-12b-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text

 humanizer-12b-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
 humanizer-12b-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
 humanizer-12b-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+humanizer-12b-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text

README.md CHANGED Viewed

@@ -42,6 +42,7 @@ Sizes are measured from the local files in decimal GB (`bytes / 1,000,000,000`).
 | `humanizer-12b-Q8_0.gguf` | 12.669631 | CPU generation passed in English and Chinese |
 | `humanizer-12b-Q6_K.gguf` | 9.786006 | CPU generation passed in English and Chinese |
 | `humanizer-12b-Q5_K_M.gguf` | 8.547252 | CPU generation passed in English and Chinese |
 The bilingual CPU smoke test loads each listed file through llama.cpp and confirms a non-empty generation for one English and one Chinese draft. These are runtime checks, not a quality or speed benchmark. No quantization fidelity scores are published yet.

 | `humanizer-12b-Q8_0.gguf` | 12.669631 | CPU generation passed in English and Chinese |
 | `humanizer-12b-Q6_K.gguf` | 9.786006 | CPU generation passed in English and Chinese |
 | `humanizer-12b-Q5_K_M.gguf` | 8.547252 | CPU generation passed in English and Chinese |
+| `humanizer-12b-Q4_K_M.gguf` | 7.381367 | CPU generation passed in English and Chinese |
 The bilingual CPU smoke test loads each listed file through llama.cpp and confirms a non-empty generation for one English and one Chinese draft. These are runtime checks, not a quality or speed benchmark. No quantization fidelity scores are published yet.

SHA256SUMS.txt CHANGED Viewed

@@ -1,5 +1,6 @@
 f5ce56021b797156e85cee85f4363703e49eb82811761c898a1f266d5c77ad36  humanizer-12b-Q8_0.gguf
 bdb3124eda7f13a373560b913fcb19151265d2b9a8b54494d0c2ed7a469ab201  humanizer-12b-Q6_K.gguf
 0521ea308066f16d28c124d6c4b82deffef9965989ea95808b4323e5bb160eda  humanizer-12b-Q5_K_M.gguf
 8eda4ace9d180481d87add6aba5ea70331ee9263ac00e730f037253d4093e50d  prompt_format.json
 cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30  LICENSE

 f5ce56021b797156e85cee85f4363703e49eb82811761c898a1f266d5c77ad36  humanizer-12b-Q8_0.gguf
 bdb3124eda7f13a373560b913fcb19151265d2b9a8b54494d0c2ed7a469ab201  humanizer-12b-Q6_K.gguf
 0521ea308066f16d28c124d6c4b82deffef9965989ea95808b4323e5bb160eda  humanizer-12b-Q5_K_M.gguf
+d6315ff977e287f787352abb74f94240c6d1c6bf255a9f1ec6f33c954f1fd78e  humanizer-12b-Q4_K_M.gguf
 8eda4ace9d180481d87add6aba5ea70331ee9263ac00e730f037253d4093e50d  prompt_format.json
 cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30  LICENSE

humanizer-12b-Q4_K_M.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d6315ff977e287f787352abb74f94240c6d1c6bf255a9f1ec6f33c954f1fd78e
+size 7381367008

reproducibility/artifact-measurements.tsv CHANGED Viewed

@@ -3,3 +3,4 @@ humanizer-12b-bf16.gguf (upstream reference)	23832049568	23.832050	upstream_bf16
 humanizer-12b-Q8_0.gguf	12669630688	12.669631	cpu_smoke_pass_en_zh	f5ce56021b797156e85cee85f4363703e49eb82811761c898a1f266d5c77ad36
 humanizer-12b-Q6_K.gguf	9786005728	9.786006	cpu_smoke_pass_en_zh	bdb3124eda7f13a373560b913fcb19151265d2b9a8b54494d0c2ed7a469ab201
 humanizer-12b-Q5_K_M.gguf	8547252448	8.547252	cpu_smoke_pass_en_zh	0521ea308066f16d28c124d6c4b82deffef9965989ea95808b4323e5bb160eda

 humanizer-12b-Q8_0.gguf	12669630688	12.669631	cpu_smoke_pass_en_zh	f5ce56021b797156e85cee85f4363703e49eb82811761c898a1f266d5c77ad36
 humanizer-12b-Q6_K.gguf	9786005728	9.786006	cpu_smoke_pass_en_zh	bdb3124eda7f13a373560b913fcb19151265d2b9a8b54494d0c2ed7a469ab201
 humanizer-12b-Q5_K_M.gguf	8547252448	8.547252	cpu_smoke_pass_en_zh	0521ea308066f16d28c124d6c4b82deffef9965989ea95808b4323e5bb160eda
+humanizer-12b-Q4_K_M.gguf	7381367008	7.381367	cpu_smoke_pass_en_zh	d6315ff977e287f787352abb74f94240c6d1c6bf255a9f1ec6f33c954f1fd78e

reproducibility/manifest.md CHANGED Viewed

@@ -52,3 +52,11 @@ built with GNU 13.3.0 for Linux x86_64`.
 built with GNU 13.3.0 for Linux x86_64`.
 - Cgroup: `MemoryMax=32G`, `MemorySwapMax=0`; observed peak RSS-plus-file-cache 13956263936 bytes and peak cgroup swap 0 bytes.

 built with GNU 13.3.0 for Linux x86_64`.
 - Cgroup: `MemoryMax=32G`, `MemorySwapMax=0`; observed peak RSS-plus-file-cache 13956263936 bytes and peak cgroup swap 0 bytes.
+### Q4_K_M
+- Artifact: `humanizer-12b-Q4_K_M.gguf`, 7381367008 bytes, SHA256 `d6315ff977e287f787352abb74f94240c6d1c6bf255a9f1ec6f33c954f1fd78e`.
+- Quantization command: `llama-quantize --max-buffer-size 512 --imatrix humanizer-12b.imatrix.gguf humanizer-12b-bf16.gguf humanizer-12b-Q4_K_M.gguf Q4_K_M 8`.
+- CPU bilingual `/completion` smoke: `PASS`; runtime `version: 0.6.0-dev (build 1, commit 9c2e0e491)
+built with GNU 13.3.0 for Linux x86_64`.
+- Cgroup: `MemoryMax=32G`, `MemorySwapMax=0`; observed peak RSS-plus-file-cache 12861640704 bytes and peak cgroup swap 0 bytes.