Pastebin
API
tools
faq
paste
Login
Sign up
Please fix the following errors:
New Paste
Syntax Highlighting
2024-09-22 05:20:56.210 | INFO | __main__:<module>:146 - Loading VisionQnA[qwen2-vl] with Qwen/Qwen2-VL-72B-Instruct-AWQ We suggest you to set `torch_dtype=torch.float16` for better efficiency with AWQ. Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11/11 [00:52<00:00, 4.77s/it] 2024-09-22 05:21:58.181 | INFO | vision_qna:loaded_banner:90 - Loaded Qwen/Qwen2-VL-72B-Instruct-AWQ [ device: cuda:0, dtype: torch.bfloat16, template: None ] INFO: Started server process [655646] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:5006 (Press CTRL+C to quit) INFO: 127.0.0.1:37934 - "POST /v1/chat/completions HTTP/1.1" 200 OK /home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/generation/configuration_utils.py:572: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.001` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`. warnings.warn( ERROR: Exception in ASGI application + Exception Group Traceback (most recent call last): | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 188, in __call__ | await response(scope, wrapped_receive, send) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 222, in __call__ | async for chunk in self.body_iterator: | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 179, in body_stream | raise app_exc | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 149, in coro | await self.app(scope, receive_or_disconnect, send_no_error) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__ | await self.app(scope, receive, send) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__ | await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app | raise exc | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app | await app(scope, receive, sender) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__ | await self.middleware_stack(scope, receive, send) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/routing.py", line 735, in app | await route.handle(scope, receive, send) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle | await self.app(scope, receive, send) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/routing.py", line 76, in app | await wrap_app_handling_exceptions(app, request)(scope, receive, send) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app | raise exc | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app | await app(scope, receive, sender) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/routing.py", line 74, in app | await response(scope, receive, send) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/sse_starlette/sse.py", line 275, in __call__ | async with anyio.create_task_group() as task_group: | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 736, in __aexit__ | raise BaseExceptionGroup( | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Traceback (most recent call last): | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 406, in run_asgi | result = await app( # type: ignore[func-returns-value] | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__ | return await self.app(scope, receive, send) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__ | await super().__call__(scope, receive, send) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__ | await self.middleware_stack(scope, receive, send) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__ | raise exc | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__ | await self.app(scope, receive, _send) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 185, in __call__ | with collapse_excgroups(): | File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__ | self.gen.throw(value) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/_utils.py", line 83, in collapse_excgroups | raise exc | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/sse_starlette/sse.py", line 278, in wrap | await func() | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/sse_starlette/sse.py", line 258, in stream_response | async for data in self.body_iterator: | File "/home/lissanro/pkgs/openedai-vision/vision.py", line 59, in streamer | async for resp in vision_qna.stream_chat_with_images(request): | File "/home/lissanro/pkgs/openedai-vision/backend/qwen2-vl.py", line 79, in stream_chat_with_images | for new_text in threaded_streaming_generator(generate=self.model.generate, tokenizer=self.processor.tokenizer, generation_kwargs=generation_kwargs): | File "/home/lissanro/pkgs/openedai-vision/vision_qna.py", line 167, in threaded_streaming_generator | raise exq.get_nowait() | File "/home/lissanro/pkgs/openedai-vision/vision_qna.py", line 152, in wrapper | generate(**generation_kwargs) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context | return func(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/generation/utils.py", line 2015, in generate | result = self._sample( | ^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/generation/utils.py", line 2965, in _sample | outputs = self(**model_inputs, return_dict=True) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl | return self._call_impl(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl | return forward_call(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/accelerate/hooks.py", line 170, in new_forward | output = module._old_forward(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 1607, in forward | outputs = self.model( | ^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl | return self._call_impl(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl | return forward_call(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 1144, in forward | layer_outputs = decoder_layer( | ^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl | return self._call_impl(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl | return forward_call(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/accelerate/hooks.py", line 170, in new_forward | output = module._old_forward(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 900, in forward | hidden_states, self_attn_weights, present_key_value = self.self_attn( | ^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl | return self._call_impl(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl | return forward_call(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/accelerate/hooks.py", line 170, in new_forward | output = module._old_forward(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 786, in forward | query_states = self.q_proj(hidden_states) | ^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl | return self._call_impl(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl | return forward_call(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/accelerate/hooks.py", line 170, in new_forward | output = module._old_forward(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/awq/modules/linear/gemm.py", line 271, in forward | out = WQLinearMMFunction.apply( | ^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/autograd/function.py", line 574, in apply | return super().apply(*args, **kwargs) # type: ignore[misc] | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/awq/modules/linear/gemm.py", line 63, in forward | out = awq_dequantize_triton(qweight, scales, qzeros) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/awq/modules/triton/gemm.py", line 283, in awq_dequantize_triton | awq_dequantize_kernel[grid]( | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/triton/runtime/jit.py", line 345, in <lambda> | return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/triton/runtime/jit.py", line 691, in run | kernel.run(grid_0, grid_1, grid_2, stream, kernel.function, kernel.packed_metadata, launch_metadata, | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/triton/backends/nvidia/driver.py", line 365, in __call__ | self.launch(*args, **kwargs) | ValueError: Pointer argument (at 0) cannot be accessed from Triton (cpu tensor?) +------------------------------------ During handling of the above exception, another exception occurred: + Exception Group Traceback (most recent call last): | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/_utils.py", line 77, in collapse_excgroups | yield | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 186, in __call__ | async with anyio.create_task_group() as task_group: | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 736, in __aexit__ | raise BaseExceptionGroup( | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Traceback (most recent call last): | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/sse_starlette/sse.py", line 289, in __call__ | await wrap(partial(self.listen_for_disconnect, receive)) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/sse_starlette/sse.py", line 278, in wrap | await func() | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/sse_starlette/sse.py", line 228, in listen_for_disconnect | message = await receive() | ^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 118, in receive_or_disconnect | async with anyio.create_task_group() as task_group: | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 740, in __aexit__ | raise exc_val | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 126, in receive_or_disconnect | message = await wrap(wrapped_receive) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 121, in wrap | result = await func() | ^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 51, in wrapped_receive | msg = await self.receive() | ^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 534, in receive | await self.message_event.wait() | File "/usr/lib/python3.12/asyncio/locks.py", line 212, in wait | await fut | asyncio.exceptions.CancelledError: Cancelled by cancel scope 78d8c086c980 | | During handling of the above exception, another exception occurred: | | Exception Group Traceback (most recent call last): | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 188, in __call__ | await response(scope, wrapped_receive, send) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 222, in __call__ | async for chunk in self.body_iterator: | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 179, in body_stream | raise app_exc | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 149, in coro | await self.app(scope, receive_or_disconnect, send_no_error) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__ | await self.app(scope, receive, send) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__ | await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app | raise exc | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app | await app(scope, receive, sender) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__ | await self.middleware_stack(scope, receive, send) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/routing.py", line 735, in app | await route.handle(scope, receive, send) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle | await self.app(scope, receive, send) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/routing.py", line 76, in app | await wrap_app_handling_exceptions(app, request)(scope, receive, send) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app | raise exc | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app | await app(scope, receive, sender) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/routing.py", line 74, in app | await response(scope, receive, send) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/sse_starlette/sse.py", line 275, in __call__ | async with anyio.create_task_group() as task_group: | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 736, in __aexit__ | raise BaseExceptionGroup( | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Traceback (most recent call last): | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 406, in run_asgi | result = await app( # type: ignore[func-returns-value] | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__ | return await self.app(scope, receive, send) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__ | await super().__call__(scope, receive, send) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__ | await self.middleware_stack(scope, receive, send) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__ | raise exc | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__ | await self.app(scope, receive, _send) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 185, in __call__ | with collapse_excgroups(): | File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__ | self.gen.throw(value) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/_utils.py", line 83, in collapse_excgroups | raise exc | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/sse_starlette/sse.py", line 278, in wrap | await func() | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/sse_starlette/sse.py", line 258, in stream_response | async for data in self.body_iterator: | File "/home/lissanro/pkgs/openedai-vision/vision.py", line 59, in streamer | async for resp in vision_qna.stream_chat_with_images(request): | File "/home/lissanro/pkgs/openedai-vision/backend/qwen2-vl.py", line 79, in stream_chat_with_images | for new_text in threaded_streaming_generator(generate=self.model.generate, tokenizer=self.processor.tokenizer, generation_kwargs=generation_kwargs): | File "/home/lissanro/pkgs/openedai-vision/vision_qna.py", line 167, in threaded_streaming_generator | raise exq.get_nowait() | File "/home/lissanro/pkgs/openedai-vision/vision_qna.py", line 152, in wrapper | generate(**generation_kwargs) | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context | return func(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/generation/utils.py", line 2015, in generate | result = self._sample( | ^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/generation/utils.py", line 2965, in _sample | outputs = self(**model_inputs, return_dict=True) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl | return self._call_impl(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl | return forward_call(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/accelerate/hooks.py", line 170, in new_forward | output = module._old_forward(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 1607, in forward | outputs = self.model( | ^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl | return self._call_impl(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl | return forward_call(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 1144, in forward | layer_outputs = decoder_layer( | ^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl | return self._call_impl(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl | return forward_call(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/accelerate/hooks.py", line 170, in new_forward | output = module._old_forward(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 900, in forward | hidden_states, self_attn_weights, present_key_value = self.self_attn( | ^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl | return self._call_impl(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl | return forward_call(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/accelerate/hooks.py", line 170, in new_forward | output = module._old_forward(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 786, in forward | query_states = self.q_proj(hidden_states) | ^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl | return self._call_impl(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl | return forward_call(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/accelerate/hooks.py", line 170, in new_forward | output = module._old_forward(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/awq/modules/linear/gemm.py", line 271, in forward | out = WQLinearMMFunction.apply( | ^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/autograd/function.py", line 574, in apply | return super().apply(*args, **kwargs) # type: ignore[misc] | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/awq/modules/linear/gemm.py", line 63, in forward | out = awq_dequantize_triton(qweight, scales, qzeros) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/awq/modules/triton/gemm.py", line 283, in awq_dequantize_triton | awq_dequantize_kernel[grid]( | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/triton/runtime/jit.py", line 345, in <lambda> | return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/triton/runtime/jit.py", line 691, in run | kernel.run(grid_0, grid_1, grid_2, stream, kernel.function, kernel.packed_metadata, launch_metadata, | File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/triton/backends/nvidia/driver.py", line 365, in __call__ | self.launch(*args, **kwargs) | ValueError: Pointer argument (at 0) cannot be accessed from Triton (cpu tensor?) +------------------------------------ During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 406, in run_asgi result = await app( # type: ignore[func-returns-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__ return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__ await super().__call__(scope, receive, send) File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__ await self.middleware_stack(scope, receive, send) File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__ raise exc File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__ await self.app(scope, receive, _send) File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 185, in __call__ with collapse_excgroups(): File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__ self.gen.throw(value) File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/starlette/_utils.py", line 83, in collapse_excgroups raise exc File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/sse_starlette/sse.py", line 278, in wrap await func() File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/sse_starlette/sse.py", line 258, in stream_response async for data in self.body_iterator: File "/home/lissanro/pkgs/openedai-vision/vision.py", line 59, in streamer async for resp in vision_qna.stream_chat_with_images(request): File "/home/lissanro/pkgs/openedai-vision/backend/qwen2-vl.py", line 79, in stream_chat_with_images for new_text in threaded_streaming_generator(generate=self.model.generate, tokenizer=self.processor.tokenizer, generation_kwargs=generation_kwargs): File "/home/lissanro/pkgs/openedai-vision/vision_qna.py", line 167, in threaded_streaming_generator raise exq.get_nowait() File "/home/lissanro/pkgs/openedai-vision/vision_qna.py", line 152, in wrapper generate(**generation_kwargs) File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/generation/utils.py", line 2015, in generate result = self._sample( ^^^^^^^^^^^^^ File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/generation/utils.py", line 2965, in _sample outputs = self(**model_inputs, return_dict=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/accelerate/hooks.py", line 170, in new_forward output = module._old_forward(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 1607, in forward outputs = self.model( ^^^^^^^^^^^ File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 1144, in forward layer_outputs = decoder_layer( ^^^^^^^^^^^^^^ File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/accelerate/hooks.py", line 170, in new_forward output = module._old_forward(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 900, in forward hidden_states, self_attn_weights, present_key_value = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/accelerate/hooks.py", line 170, in new_forward output = module._old_forward(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 786, in forward query_states = self.q_proj(hidden_states) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/accelerate/hooks.py", line 170, in new_forward output = module._old_forward(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/awq/modules/linear/gemm.py", line 271, in forward out = WQLinearMMFunction.apply( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/torch/autograd/function.py", line 574, in apply return super().apply(*args, **kwargs) # type: ignore[misc] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/awq/modules/linear/gemm.py", line 63, in forward out = awq_dequantize_triton(qweight, scales, qzeros) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/awq/modules/triton/gemm.py", line 283, in awq_dequantize_triton awq_dequantize_kernel[grid]( File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/triton/runtime/jit.py", line 345, in <lambda> return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/triton/runtime/jit.py", line 691, in run kernel.run(grid_0, grid_1, grid_2, stream, kernel.function, kernel.packed_metadata, launch_metadata, File "/home/lissanro/pkgs/openedai-vision/.venv/lib/python3.12/site-packages/triton/backends/nvidia/driver.py", line 365, in __call__ self.launch(*args, **kwargs) ValueError: Pointer argument (at 0) cannot be accessed from Triton (cpu tensor?)
Optional Paste Settings
Category:
None
Cryptocurrency
Cybersecurity
Fixit
Food
Gaming
Haiku
Help
History
Housing
Jokes
Legal
Money
Movies
Music
Pets
Photo
Science
Software
Source Code
Spirit
Sports
Travel
TV
Writing
Tags:
Syntax Highlighting:
None
Bash
C
C#
C++
CSS
HTML
JSON
Java
JavaScript
Lua
Markdown (PRO members only)
Objective C
PHP
Perl
Python
Ruby
Swift
4CS
6502 ACME Cross Assembler
6502 Kick Assembler
6502 TASM/64TASS
ABAP
AIMMS
ALGOL 68
APT Sources
ARM
ASM (NASM)
ASP
ActionScript
ActionScript 3
Ada
Apache Log
AppleScript
Arduino
Asymptote
AutoIt
Autohotkey
Avisynth
Awk
BASCOM AVR
BNF
BOO
Bash
Basic4GL
Batch
BibTeX
Blitz Basic
Blitz3D
BlitzMax
BrainFuck
C
C (WinAPI)
C Intermediate Language
C for Macs
C#
C++
C++ (WinAPI)
C++ (with Qt extensions)
C: Loadrunner
CAD DCL
CAD Lisp
CFDG
CMake
COBOL
CSS
Ceylon
ChaiScript
Chapel
Clojure
Clone C
Clone C++
CoffeeScript
ColdFusion
Cuesheet
D
DCL
DCPU-16
DCS
DIV
DOT
Dart
Delphi
Delphi Prism (Oxygene)
Diff
E
ECMAScript
EPC
Easytrieve
Eiffel
Email
Erlang
Euphoria
F#
FO Language
Falcon
Filemaker
Formula One
Fortran
FreeBasic
FreeSWITCH
GAMBAS
GDB
GDScript
Game Maker
Genero
Genie
GetText
Go
Godot GLSL
Groovy
GwBasic
HQ9 Plus
HTML
HTML 5
Haskell
Haxe
HicEst
IDL
INI file
INTERCAL
IO
ISPF Panel Definition
Icon
Inno Script
J
JCL
JSON
Java
Java 5
JavaScript
Julia
KSP (Kontakt Script)
KiXtart
Kotlin
LDIF
LLVM
LOL Code
LScript
Latex
Liberty BASIC
Linden Scripting
Lisp
Loco Basic
Logtalk
Lotus Formulas
Lotus Script
Lua
M68000 Assembler
MIX Assembler
MK-61/52
MPASM
MXML
MagikSF
Make
MapBasic
Markdown (PRO members only)
MatLab
Mercury
MetaPost
Modula 2
Modula 3
Motorola 68000 HiSoft Dev
MySQL
Nagios
NetRexx
Nginx
Nim
NullSoft Installer
OCaml
OCaml Brief
Oberon 2
Objeck Programming Langua
Objective C
Octave
Open Object Rexx
OpenBSD PACKET FILTER
OpenGL Shading
Openoffice BASIC
Oracle 11
Oracle 8
Oz
PARI/GP
PCRE
PHP
PHP Brief
PL/I
PL/SQL
POV-Ray
ParaSail
Pascal
Pawn
Per
Perl
Perl 6
Phix
Pic 16
Pike
Pixel Bender
PostScript
PostgreSQL
PowerBuilder
PowerShell
ProFTPd
Progress
Prolog
Properties
ProvideX
Puppet
PureBasic
PyCon
Python
Python for S60
QBasic
QML
R
RBScript
REBOL
REG
RPM Spec
Racket
Rails
Rexx
Robots
Roff Manpage
Ruby
Ruby Gnuplot
Rust
SAS
SCL
SPARK
SPARQL
SQF
SQL
SSH Config
Scala
Scheme
Scilab
SdlBasic
Smalltalk
Smarty
StandardML
StoneScript
SuperCollider
Swift
SystemVerilog
T-SQL
TCL
TeXgraph
Tera Term
TypeScript
TypoScript
UPC
Unicon
UnrealScript
Urbi
VB.NET
VBScript
VHDL
VIM
Vala
Vedit
VeriLog
Visual Pro Log
VisualBasic
VisualFoxPro
WHOIS
WhiteSpace
Winbatch
XBasic
XML
XPP
Xojo
Xorg Config
YAML
YARA
Z80 Assembler
ZXBasic
autoconf
jQuery
mIRC
newLISP
q/kdb+
thinBasic
Paste Expiration:
Never
Burn after read
10 Minutes
1 Hour
1 Day
1 Week
2 Weeks
1 Month
6 Months
1 Year
Paste Exposure:
Public
Unlisted
Private
Folder:
(members only)
Password
NEW
Enabled
Disabled
Burn after read
NEW
Paste Name / Title:
Create New Paste
Hello
Guest
Sign Up
or
Login
Sign in with Facebook
Sign in with Twitter
Sign in with Google
You are currently not logged in, this means you can not edit or delete anything you paste.
Sign Up
or
Login
Public Pastes
Altur Song Lyrics
10 hours ago | 13.27 KB
CC:T GPS Beacon
21 hours ago | 0.49 KB
R4 Moon SPI Screen With Wifi
Arduino | 1 day ago | 13.55 KB
Art school
1 day ago | 1.06 KB
Untitled
1 day ago | 6.00 KB
deforestation
2 days ago | 14.70 KB
Untitled
2 days ago | 0.32 KB
Untitled
2 days ago | 0.14 KB
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the
Cookies Policy
.
OK, I Understand
Not a member of Pastebin yet?
Sign Up
, it unlocks many cool features!
