Nock-AI/nock-video-4b-GGUF

Nock Video 4B (GGUF, preview)

Ask questions about your own videos, offline: when something happens, what is on screen, or a short summary. A fine-tune of Qwen3.5 4B (Apache-2.0) as one GGUF file for llama.cpp, plus the vision projector.

File	What	Size
nock-video-4b-run2-Q4_K_M.gguf	the model, Q4_K_M (run 2)	2.71 GB
mmproj-F16.gguf	vision projector (unchanged from the base: the vision part was not trained)	0.67 GB
ask_video.py	ask a question about a video file (Python standard library + ffmpeg)
TRAINING_CLIPS.md	the real clips used in training, with authors and licenses

This is a preview: measured on a small test set, and run 2 missed one of the goals we set before training it (sport, by one question). The numbers below are all of them, including where it misses. The first preview (run 1) stays available under the tag run1.

Run it

With llama.cpp (the scores below were measured in llama.cpp; 64 frames peaked at about 5 GB of GPU memory). -hf downloads both the model and the vision projector:

llama-server -hf Nock-AI/nock-video-4b-GGUF --jinja -c 32768 -ngl 99

Then, with ffmpeg on your PATH:

python ask_video.py my_video.mp4 "When does the error message show up?" --moment
python ask_video.py my_video.mp4 "What is written on the whiteboard?"
python ask_video.py my_video.mp4 --summary

ask_video.py is the setup the scores were measured with: up to 64 frames (1 per second for short videos), 768 px wide, each labelled Frame at N s:, then the question and a fixed instruction. Your video and its frames never leave your machine.

Less memory: --max-frames 32 roughly halves the prompt. Run 1 lost accuracy at 32 frames (42 to 35 of 56); run 2 was not measured at 32.

How it was trained

• LoRA (rank 16) on the language part of Qwen3.5 4B; the vision encoder was frozen. 2 epochs, 64 frames per video.
• 160 videos we generated ourselves (app screens, terminal sessions, spreadsheets, lectures and sport-like matches drawn with Remotion). Because we place every event, those answers are exact to the frame. Their summaries are written in varied shapes from what each video shows.
• 40 real clips from Wikimedia Commons (23 sport, 17 family events, 2.2 hours), licensed CC0, public domain or CC BY only, labelled by hand to about 2 seconds, used 3 times each. Authors and licenses: TRAINING_CLIPS.md.
• The test set is kept apart: different apps, people, topics, uploaders and content. 8 more real clips were kept out of training as a separate check.

Results

Test set: 20 videos, 54 minutes (8 drawn by us, 12 licensed real clips: cooking, family events, sport), 56 "when does it happen?" questions, 60 questions about what is on screen, 7 summaries. A moment counts when the answer overlaps the true span by at least 50% (IoU >= 0.5). Short answers and summaries are judged against a rubric written before any model ran. All runs: Q4_K_M in llama.cpp, 64 frames, the same prompts. For the base, questions and summaries come from its first run; the find-the-moment instruction changed later, and that change does not touch the other two prompts.

	Qwen3.5 4B (base)	Run 1 (first preview)	Run 2 (this file)
Find the moment	35 / 56	42 / 56	43 / 56
Screen recordings	10 / 12	12 / 12	11 / 12
Slides and lectures	10 / 12	9 / 12	10 / 12
Cooking	7 / 12	8 / 12	8 / 12
Family events	7 / 12	7 / 12	9 / 12
Sport	1 / 8	6 / 8	5 / 8
Questions about the video	56 / 60	56 / 60	59 / 60
Summaries	6 / 7	6 / 7	6 / 7

Before training run 2 we set the goals for dropping "preview": at least 42 moments, at least 6 sport, more than 7 family, at least 56 questions, at least 6 summaries, and no made-up facts in the long basketball summary. Run 2 met all of them except sport: 5 of 8. The one it lost is the brass band in an 8-minute basketball video (true 374-379 s, run 2 said 376-384 s). So it stays a preview.

On 8 real clips kept out of training (4 sport, 4 family) it found 23 of 28 moments.

Fixed since run 1: it reads "18 tests passed" correctly (run 1 said 10), gives a spreadsheet's final total instead of the old one, and the long basketball summary now sticks to what is on screen (players' points from the captions, the arena from the title card) instead of the made-up "5 coaches are interviewed".

Where it misses:

• It names basil as well as mint for a salsa (basil is only a garnish).
• A bakery summary says "wood-fired oven"; the oven in the video is gas. Check summaries before relying on them.
• The test set is small (20 videos, 8 sport questions), so one answer moves a row.

Not for

Identifying people or faces, surveillance, live streams, or any decision that matters without a person checking the video. It can be wrong about times and details.

License

Apache-2.0, like the base model. The vision projector is the base model's, converted by unsloth/Qwen3.5-4B-GGUF.