openedai-vision: ValueError: Pointer argument (at 0) cannot be accessed from Triton (cpu tensor?)

2024-09-22 05:20:56.210 | INFO     | __main__:<module>:146 - Loading VisionQnA[qwen2-vl] with Qwen/Qwen2-VL-72B-Instruct-AWQ
We suggest you to set `torch_dtype=torch.float16` for better efficiency with AWQ.
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11/11 [00:52<00:00,  4.77s/it]
2024-09-22 05:21:58.181 | INFO     | vision_qna:loaded_banner:90 - Loaded Qwen/Qwen2-VL-72B-Instruct-AWQ [ device: cuda:0, dtype: torch.bfloat16, template: None ]
INFO:     Started server process [655646]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:5006 (Press CTRL+C to quit)
INFO:     127.0.0.1:37934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/generation/configuration_utils.py:572: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.001` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`.
  warnings.warn(
ERROR:    Exception in ASGI application
  + Exception Group Traceback (most recent call last):
  |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 188, in __call__
  |     await response(scope, wrapped_receive, send)
  |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 222, in __call__
  |     async for chunk in self.body_iterator:
  |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 179, in body_stream
  |     raise app_exc
  |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 149, in coro
  |     await self.app(scope, receive_or_disconnect, send_no_error)
  |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
  |     await self.app(scope, receive, send)
  |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
  |     raise exc
  |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
  |     await route.handle(scope, receive, send)
  |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
  |     await self.app(scope, receive, send)
  |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
  |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
  |     raise exc
  |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/routing.py", line 74, in app
  |     await response(scope, receive, send)
  |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/sse_starlette/sse.py", line 275, in __call__
  |     async with anyio.create_task_group() as task_group:
  |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 736, in __aexit__
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 406, in run_asgi
    |     result = await app(  # type: ignore[func-returns-value]
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    |     return await self.app(scope, receive, send)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
    |     await super().__call__(scope, receive, send)
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
    |     await self.middleware_stack(scope, receive, send)
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
    |     raise exc
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
    |     await self.app(scope, receive, _send)
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 185, in __call__
    |     with collapse_excgroups():
    |   File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__
    |     self.gen.throw(value)
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/_utils.py", line 83, in collapse_excgroups
    |     raise exc
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/sse_starlette/sse.py", line 278, in wrap
    |     await func()
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/sse_starlette/sse.py", line 258, in stream_response
    |     async for data in self.body_iterator:
    |   File "/home/lissanro/pkgs/openedai-vision/vision.py", line 59, in streamer
    |     async for resp in vision_qna.stream_chat_with_images(request):
    |   File "/home/lissanro/pkgs/openedai-vision/backend/qwen2-vl.py", line 79, in stream_chat_with_images
    |     for new_text in threaded_streaming_generator(generate=self.model.generate, tokenizer=self.processor.tokenizer, generation_kwargs=generation_kwargs):
    |   File "/home/lissanro/pkgs/openedai-vision/vision_qna.py", line 167, in threaded_streaming_generator
    |     raise exq.get_nowait()
    |   File "/home/lissanro/pkgs/openedai-vision/vision_qna.py", line 152, in wrapper
    |     generate(**generation_kwargs)
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    |     return func(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/generation/utils.py", line 2015, in generate
    |     result = self._sample(
    |              ^^^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/generation/utils.py", line 2965, in _sample
    |     outputs = self(**model_inputs, return_dict=True)
    |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    |     return self._call_impl(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    |     return forward_call(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/accelerate/hooks.py", line 170, in new_forward
    |     output = module._old_forward(*args, **kwargs)
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 1607, in forward
    |     outputs = self.model(
    |               ^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    |     return self._call_impl(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    |     return forward_call(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 1144, in forward
    |     layer_outputs = decoder_layer(
    |                     ^^^^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    |     return self._call_impl(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    |     return forward_call(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/accelerate/hooks.py", line 170, in new_forward
    |     output = module._old_forward(*args, **kwargs)
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 900, in forward
    |     hidden_states, self_attn_weights, present_key_value = self.self_attn(
    |                                                           ^^^^^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    |     return self._call_impl(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    |     return forward_call(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/accelerate/hooks.py", line 170, in new_forward
    |     output = module._old_forward(*args, **kwargs)
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 786, in forward
    |     query_states = self.q_proj(hidden_states)
    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    |     return self._call_impl(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    |     return forward_call(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/accelerate/hooks.py", line 170, in new_forward
    |     output = module._old_forward(*args, **kwargs)
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/awq/modules/linear/gemm.py", line 271, in forward
    |     out = WQLinearMMFunction.apply(
    |           ^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/autograd/function.py", line 574, in apply
    |     return super().apply(*args, **kwargs)  # type: ignore[misc]
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/awq/modules/linear/gemm.py", line 63, in forward
    |     out = awq_dequantize_triton(qweight, scales, qzeros)
    |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/awq/modules/triton/gemm.py", line 283, in awq_dequantize_triton
    |     awq_dequantize_kernel[grid](
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/triton/runtime/jit.py", line 345, in <lambda>
    |     return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
    |                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/triton/runtime/jit.py", line 691, in run
    |     kernel.run(grid_0, grid_1, grid_2, stream, kernel.function, kernel.packed_metadata, launch_metadata,
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/triton/backends/nvidia/driver.py", line 365, in __call__
    |     self.launch(*args, **kwargs)
    | ValueError: Pointer argument (at 0) cannot be accessed from Triton (cpu tensor?)
    +------------------------------------

During handling of the above exception, another exception occurred:

  + Exception Group Traceback (most recent call last):
  |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/_utils.py", line 77, in collapse_excgroups
  |     yield
  |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 186, in __call__
  |     async with anyio.create_task_group() as task_group:
  |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 736, in __aexit__
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/sse_starlette/sse.py", line 289, in __call__
    |     await wrap(partial(self.listen_for_disconnect, receive))
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/sse_starlette/sse.py", line 278, in wrap
    |     await func()
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/sse_starlette/sse.py", line 228, in listen_for_disconnect
    |     message = await receive()
    |               ^^^^^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 118, in receive_or_disconnect
    |     async with anyio.create_task_group() as task_group:
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 740, in __aexit__
    |     raise exc_val
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 126, in receive_or_disconnect
    |     message = await wrap(wrapped_receive)
    |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 121, in wrap
    |     result = await func()
    |              ^^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 51, in wrapped_receive
    |     msg = await self.receive()
    |           ^^^^^^^^^^^^^^^^^^^^
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 534, in receive
    |     await self.message_event.wait()
    |   File "/usr/lib/python3.12/asyncio/locks.py", line 212, in wait
    |     await fut
    | asyncio.exceptions.CancelledError: Cancelled by cancel scope 78d8c086c980
    |
    | During handling of the above exception, another exception occurred:
    |
    | Exception Group Traceback (most recent call last):
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 188, in __call__
    |     await response(scope, wrapped_receive, send)
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 222, in __call__
    |     async for chunk in self.body_iterator:
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 179, in body_stream
    |     raise app_exc
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 149, in coro
    |     await self.app(scope, receive_or_disconnect, send_no_error)
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
    |     await self.app(scope, receive, send)
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
    |     raise exc
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
    |     await app(scope, receive, sender)
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
    |     await self.middleware_stack(scope, receive, send)
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
    |     await route.handle(scope, receive, send)
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
    |     await self.app(scope, receive, send)
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
    |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
    |     raise exc
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
    |     await app(scope, receive, sender)
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/routing.py", line 74, in app
    |     await response(scope, receive, send)
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/sse_starlette/sse.py", line 275, in __call__
    |     async with anyio.create_task_group() as task_group:
    |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 736, in __aexit__
    |     raise BaseExceptionGroup(
    | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
    +-+---------------- 1 ----------------
      | Traceback (most recent call last):
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 406, in run_asgi
      |     result = await app(  # type: ignore[func-returns-value]
      |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
      |     return await self.app(scope, receive, send)
      |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
      |     await super().__call__(scope, receive, send)
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
      |     await self.middleware_stack(scope, receive, send)
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
      |     raise exc
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
      |     await self.app(scope, receive, _send)
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 185, in __call__
      |     with collapse_excgroups():
      |   File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__
      |     self.gen.throw(value)
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/_utils.py", line 83, in collapse_excgroups
      |     raise exc
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/sse_starlette/sse.py", line 278, in wrap
      |     await func()
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/sse_starlette/sse.py", line 258, in stream_response
      |     async for data in self.body_iterator:
      |   File "/home/lissanro/pkgs/openedai-vision/vision.py", line 59, in streamer
      |     async for resp in vision_qna.stream_chat_with_images(request):
      |   File "/home/lissanro/pkgs/openedai-vision/backend/qwen2-vl.py", line 79, in stream_chat_with_images
      |     for new_text in threaded_streaming_generator(generate=self.model.generate, tokenizer=self.processor.tokenizer, generation_kwargs=generation_kwargs):
      |   File "/home/lissanro/pkgs/openedai-vision/vision_qna.py", line 167, in threaded_streaming_generator
      |     raise exq.get_nowait()
      |   File "/home/lissanro/pkgs/openedai-vision/vision_qna.py", line 152, in wrapper
      |     generate(**generation_kwargs)
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
      |     return func(*args, **kwargs)
      |            ^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/generation/utils.py", line 2015, in generate
      |     result = self._sample(
      |              ^^^^^^^^^^^^^
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/generation/utils.py", line 2965, in _sample
      |     outputs = self(**model_inputs, return_dict=True)
      |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
      |     return self._call_impl(*args, **kwargs)
      |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
      |     return forward_call(*args, **kwargs)
      |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/accelerate/hooks.py", line 170, in new_forward
      |     output = module._old_forward(*args, **kwargs)
      |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 1607, in forward
      |     outputs = self.model(
      |               ^^^^^^^^^^^
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
      |     return self._call_impl(*args, **kwargs)
      |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
      |     return forward_call(*args, **kwargs)
      |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 1144, in forward
      |     layer_outputs = decoder_layer(
      |                     ^^^^^^^^^^^^^^
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
      |     return self._call_impl(*args, **kwargs)
      |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
      |     return forward_call(*args, **kwargs)
      |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/accelerate/hooks.py", line 170, in new_forward
      |     output = module._old_forward(*args, **kwargs)
      |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 900, in forward
      |     hidden_states, self_attn_weights, present_key_value = self.self_attn(
      |                                                           ^^^^^^^^^^^^^^^
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
      |     return self._call_impl(*args, **kwargs)
      |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
      |     return forward_call(*args, **kwargs)
      |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/accelerate/hooks.py", line 170, in new_forward
      |     output = module._old_forward(*args, **kwargs)
      |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 786, in forward
      |     query_states = self.q_proj(hidden_states)
      |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
      |     return self._call_impl(*args, **kwargs)
      |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
      |     return forward_call(*args, **kwargs)
      |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/accelerate/hooks.py", line 170, in new_forward
      |     output = module._old_forward(*args, **kwargs)
      |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/awq/modules/linear/gemm.py", line 271, in forward
      |     out = WQLinearMMFunction.apply(
      |           ^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/autograd/function.py", line 574, in apply
      |     return super().apply(*args, **kwargs)  # type: ignore[misc]
      |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/awq/modules/linear/gemm.py", line 63, in forward
      |     out = awq_dequantize_triton(qweight, scales, qzeros)
      |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/awq/modules/triton/gemm.py", line 283, in awq_dequantize_triton
      |     awq_dequantize_kernel[grid](
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/triton/runtime/jit.py", line 345, in <lambda>
      |     return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
      |                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/triton/runtime/jit.py", line 691, in run
      |     kernel.run(grid_0, grid_1, grid_2, stream, kernel.function, kernel.packed_metadata, launch_metadata,
      |   File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/triton/backends/nvidia/driver.py", line 365, in __call__
      |     self.launch(*args, **kwargs)
      | ValueError: Pointer argument (at 0) cannot be accessed from Triton (cpu tensor?)
      +------------------------------------

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 406, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 185, in __call__
    with collapse_excgroups():
  File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/_utils.py", line 83, in collapse_excgroups
    raise exc
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/sse_starlette/sse.py", line 278, in wrap
    await func()
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/sse_starlette/sse.py", line 258, in stream_response
    async for data in self.body_iterator:
  File "/home/lissanro/pkgs/openedai-vision/vision.py", line 59, in streamer
    async for resp in vision_qna.stream_chat_with_images(request):
  File "/home/lissanro/pkgs/openedai-vision/backend/qwen2-vl.py", line 79, in stream_chat_with_images
    for new_text in threaded_streaming_generator(generate=self.model.generate, tokenizer=self.processor.tokenizer, generation_kwargs=generation_kwargs):
  File "/home/lissanro/pkgs/openedai-vision/vision_qna.py", line 167, in threaded_streaming_generator
    raise exq.get_nowait()
  File "/home/lissanro/pkgs/openedai-vision/vision_qna.py", line 152, in wrapper
    generate(**generation_kwargs)
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/generation/utils.py", line 2015, in generate
    result = self._sample(
             ^^^^^^^^^^^^^
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/generation/utils.py", line 2965, in _sample
    outputs = self(**model_inputs, return_dict=True)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/accelerate/hooks.py", line 170, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 1607, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 1144, in forward
    layer_outputs = decoder_layer(
                    ^^^^^^^^^^^^^^
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/accelerate/hooks.py", line 170, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 900, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
                                                          ^^^^^^^^^^^^^^^
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/accelerate/hooks.py", line 170, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 786, in forward
    query_states = self.q_proj(hidden_states)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/accelerate/hooks.py", line 170, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/awq/modules/linear/gemm.py", line 271, in forward
    out = WQLinearMMFunction.apply(
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/autograd/function.py", line 574, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/awq/modules/linear/gemm.py", line 63, in forward
    out = awq_dequantize_triton(qweight, scales, qzeros)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/awq/modules/triton/gemm.py", line 283, in awq_dequantize_triton
    awq_dequantize_kernel[grid](
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/triton/runtime/jit.py", line 345, in <lambda>
    return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/triton/runtime/jit.py", line 691, in run
    kernel.run(grid_0, grid_1, grid_2, stream, kernel.function, kernel.packed_metadata, launch_metadata,
  File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/triton/backends/nvidia/driver.py", line 365, in __call__
    self.launch(*args, **kwargs)
ValueError: Pointer argument (at 0) cannot be accessed from Triton (cpu tensor?)