Task tags fail when the image placeholder isn't <image>

#7
by stornic56 - opened

I'm running this model in llama.cpp with the chat_template.jinja from the repo. The task tags fail before generation.

expand_tags rebuilds the user message from scratch and only keeps the image when the content contains the literal string <image>:

{%- set has_image = "<image>" in text -%}
{%- set img_idx = text.index("<image>") -%}

llama.cpp uses a unique marker for images (<__media_...__>), so has_image is false. The macro returns only the canonical prompt, the marker is gone, and the runtime stops with:

Number of media markers (0) does not match number of loaded media (1)

The macro also calls str.index(). llama.cpp's Jinja engine has no index or find, so forcing <image> doesn't help.

Could the macro replace just the tag instead of rebuilding the message?

{%- macro expand_tags(text) -%}
  {%- if "<chart2code>" in text -%}
    {{- text.replace("<chart2code>", chart2code_prompt) -}}
  {%- elif "<chart2csv>" in text -%}
    {{- text.replace("<chart2csv>", chart2csv_prompt) -}}
  {%- elif "<chart2summary>" in text -%}
    {{- text.replace("<chart2summary>", chart2summary_prompt) -}}
  {%- elif "<tables_json>" in text -%}
    {{- text.replace("<tables_json>", tables_json_prompt) -}}
  {%- elif "<tables_html>" in text -%}
    {{- text.replace("<tables_html>", tables_html_prompt) -}}
  {%- elif "<tables_otsl>" in text -%}
    {{- text.replace("<tables_otsl>", tables_otsl_prompt) -}}
  {%- else -%}
    {{- text -}}
  {%- endif -%}
{%- endmacro -%}

That keeps whatever image placeholder the runtime uses, keeps text around the tag, and drops the str.index() dependency. For <image>\n<tables_html> the output in transformers is the same.

Repro: transformers 5.23.0 with bf16 returns the table. llama.cpp b12383 with the Q8_0 GGUF and F16 mmproj gives the marker error above.

IBM Granite org

Hi @stornic56 , thanks for raising this. @mrutkows I think this may be the same issue you were discussing with me the other day about the non-<image> placeholders?

IBM Granite org

Thanks @stornic56 for catching this issue
Can you check if the fix here resolves your issue? ๐Ÿ™

Sign up or log in to comment