前回 はGGUFを使ったので、今回はONNX。
llama.cppのセットアップは前回やったので省略。
事前準備
python3 -m venv .venv-onnx
source .venv-onnx/bin/activate
pip install "optimum[onnx]" onnx onnxruntime transformers現在の onnxruntime にはApple Silicon向けのCoreML Execution Providerも含まれているので安心。
確認しておく。
$ python - <<'EOF'
import onnxruntime
print(onnxruntime.get_available_providers())
EOF
['CoreMLExecutionProvider', 'AzureExecutionProvider', 'CPUExecutionProvider']うん、大丈夫そう。Azureなんたらはよくわからんが。
Hagging Faceからモデル取得
今回はHuggingFaceTB/SmolLM2-360M-Instruct。
hf download HuggingFaceTB/SmolLM2-360M-Instruct \
--local-dir ./models/SmolLM2-360M-InstructONNXへexport
optimum-cli export onnx \
--model ./models/SmolLM2-360M-Instruct \
--task text-generation-with-past \
./models/SmolLM2-360M-Instruct-onnx色々みてみる
import onnx
from collections import Counter
model = onnx.load(
"./models/SmolLM2-360M-Instruct-onnx/model.onnx",
load_external_data=False,
)
graph = model.graph
print("graph name:", graph.name)
print("inputs:", len(graph.input))
print("outputs:", len(graph.output))
print("nodes:", len(graph.node))
print("initializers:", len(graph.initializer))
counts = Counter(node.op_type for node in graph.node)
for op, count in counts.most_common():
print(f"{op:30} {count}")$ python graph.py
graph name: main_graph
inputs: 67
outputs: 65
nodes: 7911
initializers: 291
Constant 2891
Unsqueeze 1008
Mul 519
Shape 489
Gather 424
Concat 388
MatMul 290
Cast 271
Reshape 260
Add 228
Transpose 161
Slice 160
Where 130
Div 129
ConstantOfShape 67
Equal 66
Expand 66
Pow 65
ReduceMean 65
Sqrt 65
Neg 64
Softmax 32
IsNaN 32
Sigmoid 32
Range 3
And 2
LessOrEqual 1
Flatten 1
Cos 1
Sin 1最初の30ノードを見てみる。
import onnx
from collections import Counter
model = onnx.load(
"./models/SmolLM2-360M-Instruct-onnx/model.onnx",
load_external_data=False,
)
graph = model.graph
for node in graph.node[:30]:
print(
f"{node.name:60} "
f"{node.op_type:20} "
f"{list(node.input)} -> {list(node.output)}"
)/model/embed_tokens/Gather Gather ['model.embed_tokens.weight', 'input_ids'] -> ['/model/embed_tokens/Gather_output_0']
/model/Shape Shape ['past_key_values.0.key'] -> ['/model/Shape_output_0']
/model/Constant Constant [] -> ['/model/Constant_output_0']
/model/Gather Gather ['/model/Shape_output_0', '/model/Constant_output_0'] -> ['/model/Gather_output_0']
/model/Shape_1 Shape ['/model/embed_tokens/Gather_output_0'] -> ['/model/Shape_1_output_0']
/model/Constant_1 Constant [] -> ['/model/Constant_1_output_0']
/model/Gather_1 Gather ['/model/Shape_1_output_0', '/model/Constant_1_output_0'] -> ['/model/Gather_1_output_0']
/model/Add Add ['/model/Gather_output_0', '/model/Gather_1_output_0'] -> ['/model/Add_output_0']
/model/Cast Cast ['/model/Gather_output_0'] -> ['/model/Cast_output_0']
/model/Cast_1 Cast ['/model/Add_output_0'] -> ['/model/Cast_1_output_0']
/model/Constant_2 Constant [] -> ['/model/Constant_2_output_0']
/model/Range Range ['/model/Cast_output_0', '/model/Cast_1_output_0', '/model/Constant_2_output_0'] -> ['/model/Range_output_0']
/model/Cast_2 Cast ['attention_mask'] -> ['/model/Cast_2_output_0']
/model/Shape_2 Shape ['/model/Range_output_0'] -> ['/model/Shape_2_output_0']
/model/Constant_3 Constant [] -> ['/model/Constant_3_output_0']
/model/Gather_2 Gather ['/model/Shape_2_output_0', '/model/Constant_3_output_0'] -> ['/model/Gather_2_output_0']
/model/Add_1 Add ['/model/Gather_output_0', '/model/Gather_2_output_0'] -> ['/model/Add_1_output_0']
/model/Shape_3 Shape ['/model/embed_tokens/Gather_output_0'] -> ['/model/Shape_3_output_0']
/model/Constant_4 Constant [] -> ['/model/Constant_4_output_0']
/model/Gather_3 Gather ['/model/Shape_3_output_0', '/model/Constant_4_output_0'] -> ['/model/Gather_3_output_0']
/model/Constant_5 Constant [] -> ['/model/Constant_5_output_0']
/model/Unsqueeze Unsqueeze ['/model/Range_output_0', '/model/Constant_5_output_0'] -> ['/model/Unsqueeze_output_0']
/model/Constant_6 Constant [] -> ['/model/Constant_6_output_0']
/model/Unsqueeze_1 Unsqueeze ['/model/Unsqueeze_output_0', '/model/Constant_6_output_0'] -> ['/model/Unsqueeze_1_output_0']
/model/Constant_7 Constant [] -> ['/model/Constant_7_output_0']
/model/Unsqueeze_2 Unsqueeze ['/model/Unsqueeze_1_output_0', '/model/Constant_7_output_0'] -> ['/model/Unsqueeze_2_output_0']
/model/Cast_3 Cast ['/model/Gather_3_output_0'] -> ['/model/Cast_3_output_0']
/model/Constant_8 Constant [] -> ['/model/Constant_8_output_0']
/model/Constant_9 Constant [] -> ['/model/Constant_9_output_0']
/model/Range_1 Range ['/model/Constant_8_output_0', '/model/Cast_3_output_0', '/model/Constant_9_output_0'] -> ['/model/Range_1_output_0']MatMul, Add, Reshape, Transpose, Softmax, Gather みたいな演算が並んでいる。 つまり、forward passそのものがグラフとして保存されている、ってことかな。
入力shapeも見てみる。
import onnxruntime as ort
session = ort.InferenceSession(
"./models/SmolLM2-360M-Instruct-onnx/model.onnx",
providers=["CPUExecutionProvider"],
)
for inp in session.get_inputs():
print(inp.name, inp.shape, inp.type)$ python ./input_shapes.py
input_ids ['batch_size', 'sequence_length'] tensor(int64)
attention_mask ['batch_size', 'past_sequence_length + sequence_length'] tensor(int64)
position_ids ['batch_size', 'sequence_length'] tensor(int64)
past_key_values.0.key ['batch_size', 5, 'past_sequence_length', 64] tensor(float)
past_key_values.0.value ['batch_size', 5, 'past_sequence_length', 64] tensor(float)
past_key_values.1.key ['batch_size', 5, 'past_sequence_length', 64] tensor(float)
past_key_values.1.value ['batch_size', 5, 'past_sequence_length', 64] tensor(float)
...optimum上で実行
import argparse
import time
from transformers import AutoTokenizer
from optimum.onnxruntime import ORTModelForCausalLM
def main():
parser = argparse.ArgumentParser()
parser.add_argument(
"--provider",
choices=[
"CPUExecutionProvider",
"CoreMLExecutionProvider",
],
default="CPUExecutionProvider",
)
parser.add_argument(
"--model-dir",
default="./models/SmolLM2-360M-Instruct-onnx",
)
parser.add_argument(
"--max-new-tokens",
type=int,
default=100,
)
args = parser.parse_args()
tokenizer = AutoTokenizer.from_pretrained(args.model_dir)
model = ORTModelForCausalLM.from_pretrained(
args.model_dir,
provider=args.provider,
)
messages = [
{
"role": "user",
"content": "Explain the difference between TCP and UDP in a few sentences.",
}
]
prompt = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
)
inputs = tokenizer(
prompt,
return_tensors="pt",
)
prompt_tokens = inputs["input_ids"].shape[1]
start = time.perf_counter()
outputs = model.generate(
**inputs,
max_new_tokens=args.max_new_tokens,
do_sample=False,
)
elapsed = time.perf_counter() - start
generated_tokens = outputs[0][prompt_tokens:]
generated_count = len(generated_tokens)
print("=== provider ===")
print(args.provider)
print("\n=== generated ===")
print(
tokenizer.decode(
generated_tokens,
skip_special_tokens=True,
)
)
print("\n=== benchmark ===")
print(f"Prompt tokens: {prompt_tokens}")
print(f"Generated tokens: {generated_count}")
print(f"Elapsed: {elapsed:.3f} sec")
print(f"Generation: {generated_count / elapsed:.2f} tokens/sec")
if __name__ == "__main__":
main()CPU
$ python ./run_optimum.py
=== provider ===
CPUExecutionProvider
=== generated ===
TCP (Transmission Control Protocol) is a connection-oriented protocol that ensures reliable data transfer by establishing a connection between two devices before sending data. It provides a reliable and ordered delivery of data, and it is typically used for applications that require high reliability and performance, such as file transfers and web browsing.
UDP (User Datagram Protocol) is a connectionless protocol that provides fast and reliable data transfer. It does not establish a connection before sending data, and it does not guarantee delivery
=== benchmark ===
Prompt tokens: 42
Generated tokens: 100
Elapsed: 2.846 sec
Generation: 35.13 tokens/secCoreML
$ python run_optimum.py --provider CoreMLExecutionProvider
2026-09-13 23:37:25.168 Python[44794:41607830] 2026-09-13 23:37:25.168588 [W:onnxruntime:, coreml_execution_provider.cc:137 GetCapability] CoreMLExecutionProvider::GetCapability, number of partitions supported by CoreML: 257 number of nodes in the graph: 3812 number of nodes supported by CoreML: 1680
2026-09-13 23:37:29.481 Python[44794:41607830] 2026-09-13 23:37:29.481260 [W:onnxruntime:, session_state.cc:1397 VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.
2026-09-13 23:37:29.481 Python[44794:41607830] 2026-09-13 23:37:29.481298 [W:onnxruntime:, session_state.cc:1399 VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments.
2026-09-13 23:37:30.487 Python[44794:41607830] 2026-09-13 23:37:30.487626 [E:onnxruntime:, sequential_executor.cc:671 ExecuteKernel] Non-zero status code returned while running 17211290811114605102_CoreML_17211290811114605102_0 node. Name:'CoreMLExecutionProvider_17211290811114605102_CoreML_17211290811114605102_0_0' Status Message: coreml_execution_provider.cc:222 operator() Input (past_key_values.0.key) has a dynamic shape ({-1,5,-1,64}) but the runtime shape ({1,5,0,64}) has zero elements. This is not supported by the CoreML EP.
Traceback (most recent call last):
File "/Users/thara/work/20260913/run_optimum.py", line 88, in <module>
main()
File "/Users/thara/work/20260913/run_optimum.py", line 58, in main
outputs = model.generate(
^^^^^^^^^^^^^^^
File "/Users/thara/work/20260913/.venv-onnx/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/thara/work/20260913/.venv-onnx/lib/python3.11/site-packages/transformers/generation/utils.py", line 2566, in generate
result = decoding_method(
^^^^^^^^^^^^^^^^
File "/Users/thara/work/20260913/.venv-onnx/lib/python3.11/site-packages/transformers/generation/utils.py", line 2786, in _sample
outputs = self(**model_inputs, return_dict=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/thara/work/20260913/.venv-onnx/lib/python3.11/site-packages/optimum/onnxruntime/base.py", line 466, in __call__
return self.forward(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/thara/work/20260913/.venv-onnx/lib/python3.11/site-packages/optimum/onnxruntime/modeling_decoder.py", line 412, in forward
onnx_outputs = self.session.run(None, onnx_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/thara/work/20260913/.venv-onnx/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 395, in run
return self._sess.run(output_names, input_feed, run_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running 17211290811114605102_CoreML_17211290811114605102_0 node. Name:'CoreMLExecutionProvider_17211290811114605102_CoreML_17211290811114605102_0_0' Status Message: coreml_execution_provider.cc:222 operator() Input (past_key_values.0.key) has a dynamic shape ({-1,5,-1,64}) but the runtime shape ({1,5,0,64}) has zero elements. This is not supported by the CoreML EP.CPUExecutionProviderは正常動作してるけど、CoreMLExecutionProviderはなんかダメぽ。
Input (past_key_values.0.key) has a dynamic shape ({-1,5,-1,64}) but the runtime shape ({1,5,0,64}) has zero elements. This is not supported by the CoreML EP.
past_key_values.0.key はさっき見たように入力shape。 実行時shapeが{1,5,0,64} になってて、1 * 5 * 0 * 64 = 0 で空tensorになってしまっているのが原因っぽい。 初回のKV cacheが空なのはダメか。
CPUExecutionProviderはOKで、CoreMLExecutionProviderでNG、ということは、 onnxのExecutionProviderは単にハードウェアの差異を吸収するだけでなく、ONNXのグラフをどう実行するかの違いもある、ということがわかる。
CoreML動かしたかったけど、動かせなかった。 ONNX export時にcacheしないようにすればいいんだろうけど、実用的に意味がないのでやる意味なさそう。
まぁこういうこともある。 ONNXを使ってなんかやる時は、EPの実装差異に注意しないといけないかもね。