Instructions to use ngquocvinh/Humanizer-12B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use ngquocvinh/Humanizer-12B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ngquocvinh/Humanizer-12B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf ngquocvinh/Humanizer-12B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ngquocvinh/Humanizer-12B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf ngquocvinh/Humanizer-12B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf ngquocvinh/Humanizer-12B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf ngquocvinh/Humanizer-12B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf ngquocvinh/Humanizer-12B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf ngquocvinh/Humanizer-12B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/ngquocvinh/Humanizer-12B-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use ngquocvinh/Humanizer-12B-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ngquocvinh/Humanizer-12B-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ngquocvinh/Humanizer-12B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ngquocvinh/Humanizer-12B-GGUF:Q4_K_M

Ollama
How to use ngquocvinh/Humanizer-12B-GGUF with Ollama:
```
ollama run hf.co/ngquocvinh/Humanizer-12B-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use ngquocvinh/Humanizer-12B-GGUF with Docker Model Runner:
```
docker model run hf.co/ngquocvinh/Humanizer-12B-GGUF:Q4_K_M
```

Lemonade

How to use ngquocvinh/Humanizer-12B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull ngquocvinh/Humanizer-12B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Humanizer-12B-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

ngquocvinh commited on 9 days ago

Commit

d5cd9fa

verified ·

1 Parent(s): 79b9d62

Add Humanizer 12B Q2_K GGUF and measured metadata

Browse files

Files changed (6) hide show

.gitattributes +1 -0
README.md +1 -0
SHA256SUMS.txt +1 -0
humanizer-12b-Q2_K.gguf +3 -0
reproducibility/artifact-measurements.tsv +1 -0
reproducibility/manifest.md +8 -0

.gitattributes CHANGED Viewed

@@ -38,3 +38,4 @@ humanizer-12b-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
 humanizer-12b-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
 humanizer-12b-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
 humanizer-12b-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text

 humanizer-12b-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
 humanizer-12b-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
 humanizer-12b-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+humanizer-12b-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text

README.md CHANGED Viewed

@@ -44,6 +44,7 @@ Sizes are measured from the local files in decimal GB (`bytes / 1,000,000,000`).
 | `humanizer-12b-Q5_K_M.gguf` | 8.547252 | CPU generation passed in English and Chinese |
 | `humanizer-12b-Q4_K_M.gguf` | 7.381367 | CPU generation passed in English and Chinese |
 | `humanizer-12b-Q3_K_M.gguf` | 6.087072 | CPU generation passed in English and Chinese |
 The bilingual CPU smoke test loads each listed file through llama.cpp and confirms a non-empty generation for one English and one Chinese draft. These are runtime checks, not a quality or speed benchmark. No quantization fidelity scores are published yet.

 | `humanizer-12b-Q5_K_M.gguf` | 8.547252 | CPU generation passed in English and Chinese |
 | `humanizer-12b-Q4_K_M.gguf` | 7.381367 | CPU generation passed in English and Chinese |
 | `humanizer-12b-Q3_K_M.gguf` | 6.087072 | CPU generation passed in English and Chinese |
+| `humanizer-12b-Q2_K.gguf` | 4.830132 | CPU generation passed (EN + ZH); low-bit experimental |
 The bilingual CPU smoke test loads each listed file through llama.cpp and confirms a non-empty generation for one English and one Chinese draft. These are runtime checks, not a quality or speed benchmark. No quantization fidelity scores are published yet.

SHA256SUMS.txt CHANGED Viewed

@@ -3,5 +3,6 @@ bdb3124eda7f13a373560b913fcb19151265d2b9a8b54494d0c2ed7a469ab201  humanizer-12b-
 0521ea308066f16d28c124d6c4b82deffef9965989ea95808b4323e5bb160eda  humanizer-12b-Q5_K_M.gguf
 d6315ff977e287f787352abb74f94240c6d1c6bf255a9f1ec6f33c954f1fd78e  humanizer-12b-Q4_K_M.gguf
 61d0df23e6b0e4069fd6b923e3231eafb5aa8ab79d1d5aa3555c439960e56d30  humanizer-12b-Q3_K_M.gguf
 8eda4ace9d180481d87add6aba5ea70331ee9263ac00e730f037253d4093e50d  prompt_format.json
 cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30  LICENSE

 0521ea308066f16d28c124d6c4b82deffef9965989ea95808b4323e5bb160eda  humanizer-12b-Q5_K_M.gguf
 d6315ff977e287f787352abb74f94240c6d1c6bf255a9f1ec6f33c954f1fd78e  humanizer-12b-Q4_K_M.gguf
 61d0df23e6b0e4069fd6b923e3231eafb5aa8ab79d1d5aa3555c439960e56d30  humanizer-12b-Q3_K_M.gguf
+2aca789be5503d8ad8017017381dcbe1ec53892ae6c58b3eda3bf6ea1af5f58e  humanizer-12b-Q2_K.gguf
 8eda4ace9d180481d87add6aba5ea70331ee9263ac00e730f037253d4093e50d  prompt_format.json
 cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30  LICENSE

humanizer-12b-Q2_K.gguf ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2aca789be5503d8ad8017017381dcbe1ec53892ae6c58b3eda3bf6ea1af5f58e
+size 4830132448

reproducibility/artifact-measurements.tsv CHANGED Viewed

@@ -5,3 +5,4 @@ humanizer-12b-Q6_K.gguf	9786005728	9.786006	cpu_smoke_pass_en_zh	bdb3124eda7f13a
 humanizer-12b-Q5_K_M.gguf	8547252448	8.547252	cpu_smoke_pass_en_zh	0521ea308066f16d28c124d6c4b82deffef9965989ea95808b4323e5bb160eda
 humanizer-12b-Q4_K_M.gguf	7381367008	7.381367	cpu_smoke_pass_en_zh	d6315ff977e287f787352abb74f94240c6d1c6bf255a9f1ec6f33c954f1fd78e
 humanizer-12b-Q3_K_M.gguf	6087071968	6.087072	cpu_smoke_pass_en_zh	61d0df23e6b0e4069fd6b923e3231eafb5aa8ab79d1d5aa3555c439960e56d30

 humanizer-12b-Q5_K_M.gguf	8547252448	8.547252	cpu_smoke_pass_en_zh	0521ea308066f16d28c124d6c4b82deffef9965989ea95808b4323e5bb160eda
 humanizer-12b-Q4_K_M.gguf	7381367008	7.381367	cpu_smoke_pass_en_zh	d6315ff977e287f787352abb74f94240c6d1c6bf255a9f1ec6f33c954f1fd78e
 humanizer-12b-Q3_K_M.gguf	6087071968	6.087072	cpu_smoke_pass_en_zh	61d0df23e6b0e4069fd6b923e3231eafb5aa8ab79d1d5aa3555c439960e56d30
+humanizer-12b-Q2_K.gguf	4830132448	4.830132	cpu_smoke_pass_en_zh	2aca789be5503d8ad8017017381dcbe1ec53892ae6c58b3eda3bf6ea1af5f58e

reproducibility/manifest.md CHANGED Viewed

@@ -68,3 +68,11 @@ built with GNU 13.3.0 for Linux x86_64`.
 built with GNU 13.3.0 for Linux x86_64`.
 - Cgroup: `MemoryMax=32G`, `MemorySwapMax=0`; observed peak RSS-plus-file-cache 11591602176 bytes and peak cgroup swap 0 bytes.

 built with GNU 13.3.0 for Linux x86_64`.
 - Cgroup: `MemoryMax=32G`, `MemorySwapMax=0`; observed peak RSS-plus-file-cache 11591602176 bytes and peak cgroup swap 0 bytes.
+### Q2_K
+- Artifact: `humanizer-12b-Q2_K.gguf`, 4830132448 bytes, SHA256 `2aca789be5503d8ad8017017381dcbe1ec53892ae6c58b3eda3bf6ea1af5f58e`.
+- Quantization command: `llama-quantize --max-buffer-size 512 --imatrix humanizer-12b.imatrix.gguf humanizer-12b-bf16.gguf humanizer-12b-Q2_K.gguf Q2_K 8`.
+- CPU bilingual `/completion` smoke: `PASS`; runtime `version: 0.6.0-dev (build 1, commit 9c2e0e491)
+built with GNU 13.3.0 for Linux x86_64`.
+- Cgroup: `MemoryMax=32G`, `MemorySwapMax=0`; observed peak RSS-plus-file-cache 10395103232 bytes and peak cgroup swap 0 bytes.