2023-11-02 15:48:16 INFO:Loading TheBloke_openchat_3.5-AWQ...
2023-11-02 15:48:16 WARNING:Auto-assiging --gpu-memory 23 for your GPU to try to prevent out-of-memory errors. You can manually set other values.
Replacing layers...: 100%|█████████████████████████████████████████████████████████████| 32/32 [00:06<00:00,  5.05it/s]
2023-11-02 15:48:35 INFO:Loaded the model in 18.47 seconds.
Output generated in 26.13 seconds (7.62 tokens/s, 199 tokens, context 205, seed 733479273)
Traceback (most recent call last):
  File "E:\text-generation-webui\modules\callbacks.py", line 57, in gentask
    ret = self.mfunc(callback=_callback, *args, **self.kwargs)
  File "E:\text-generation-webui\modules\text_generation.py", line 352, in generate_with_callback
    shared.model.generate(**kwargs)
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\awq\models\base.py", line 36, in generate
    return self.model.generate(*args, **kwargs)
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\transformers\generation\utils.py", line 1652, in generate
    return self.sample(
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\transformers\generation\utils.py", line 2734, in sample
    outputs = self(
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\transformers\models\mistral\modeling_mistral.py", line 1045, in forward
    outputs = self.model(
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\transformers\models\mistral\modeling_mistral.py", line 932, in forward
    layer_outputs = decoder_layer(
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\transformers\models\mistral\modeling_mistral.py", line 621, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\awq\modules\fused\attn.py", line 180, in forward
    scores = scores + attention_mask  # (bs, n_local_heads, slen, cache_len + slen)
RuntimeError: The size of tensor a (809) must match the size of tensor b (405) at non-singleton dimension 3
Output generated in 2.17 seconds (0.46 tokens/s, 1 tokens, context 404, seed 1560756031)
2023-11-02 15:50:54 INFO:Loading TheBloke_openchat_3.5-AWQ...
Replacing layers...: 100%|█████████████████████████████████████████████████████████████| 32/32 [00:06<00:00,  5.02it/s]
2023-11-02 15:51:11 INFO:Loaded the model in 17.56 seconds.
Output generated in 25.10 seconds (7.93 tokens/s, 199 tokens, context 405, seed 1229328071)
Traceback (most recent call last):
  File "E:\text-generation-webui\modules\callbacks.py", line 57, in gentask
    ret = self.mfunc(callback=_callback, *args, **self.kwargs)
  File "E:\text-generation-webui\modules\text_generation.py", line 352, in generate_with_callback
    shared.model.generate(**kwargs)
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\awq\models\base.py", line 36, in generate
    return self.model.generate(*args, **kwargs)
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\transformers\generation\utils.py", line 1652, in generate
    return self.sample(
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\transformers\generation\utils.py", line 2734, in sample
    outputs = self(
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\transformers\models\mistral\modeling_mistral.py", line 1045, in forward
    outputs = self.model(
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\transformers\models\mistral\modeling_mistral.py", line 932, in forward
    layer_outputs = decoder_layer(
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\transformers\models\mistral\modeling_mistral.py", line 621, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\awq\modules\fused\attn.py", line 180, in forward
    scores = scores + attention_mask  # (bs, n_local_heads, slen, cache_len + slen)
RuntimeError: The size of tensor a (1215) must match the size of tensor b (611) at non-singleton dimension 3
Output generated in 1.12 seconds (0.89 tokens/s, 1 tokens, context 610, seed 1077658322)
Traceback (most recent call last):
  File "E:\text-generation-webui\modules\callbacks.py", line 57, in gentask
    ret = self.mfunc(callback=_callback, *args, **self.kwargs)
  File "E:\text-generation-webui\modules\text_generation.py", line 352, in generate_with_callback
    shared.model.generate(**kwargs)
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\awq\models\base.py", line 36, in generate
    return self.model.generate(*args, **kwargs)
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\transformers\generation\utils.py", line 1652, in generate
    return self.sample(
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\transformers\generation\utils.py", line 2734, in sample
    outputs = self(
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\transformers\models\mistral\modeling_mistral.py", line 1045, in forward
    outputs = self.model(
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\transformers\models\mistral\modeling_mistral.py", line 932, in forward
    layer_outputs = decoder_layer(
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\transformers\models\mistral\modeling_mistral.py", line 621, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "E:\text-generation-webui\installer_files\env\lib\site-packages\awq\modules\fused\attn.py", line 180, in forward
    scores = scores + attention_mask  # (bs, n_local_heads, slen, cache_len + slen)
RuntimeError: The size of tensor a (1826) must match the size of tensor b (612) at non-singleton dimension 3
Output generated in 1.14 seconds (0.88 tokens/s, 1 tokens, context 611, seed 595247161)
2023-11-02 15:52:56 INFO:Loading TheBloke_openchat_3.5-AWQ...
Replacing layers...: 100%|█████████████████████████████████████████████████████████████| 32/32 [00:06<00:00,  4.97it/s]
2023-11-02 15:53:13 INFO:Loaded the model in 17.58 seconds.
Output generated in 26.07 seconds (7.63 tokens/s, 199 tokens, context 612, seed 1849376669)