Advertisement
Guest User

Untitled

a guest
Jul 17th, 2019
148
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 4.68 KB | None | 0 0
  1. I am using the local mode example here:
  2. https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker-python-sdk/mxnet_gluon_cifar10/mxnet_cifar10_local_mode.ipynb
  3.  
  4. I managed to get it to work with the example dataset although it was really iffy because I kept getting an error whenever I would run m.fit() or deploy the predictor about a docker-compose command not working. I cannot for the life of me find where in the world this docker compose command is being invoked. It's nowhere in the example code or in the sagemaker-python-sdk. I can only get it to work by running this docker-compose command manually, having it fail to run, then running the code block again. I have no clue why this works but it does and sometimes it seems like it gets fixed spontaneously. This isn't the actual issue I am trying to fix but I thought it would be worth mentioning.
  5.  
  6. Once I get the m.fit() training session to run and it succeeds then deploy the predictor and confirm it's running by running "docker ps", I run the code which sends a request to the predictor. I then get one of two errors.
  7.  
  8. When I leave the example code alone the predictor server returns a dictionary with the 503 error code and a message that says "Prediction Failed". I can see where the error comes from in the sagemaker-python-sdk source code but it's extremely vague and is probably the result of something external.
  9.  
  10. If I change around the "shape" parameter on line 48 and 50 in cifar10.py
  11. {code} train_data = get_train_data(num_cpus, data_dir, batch_size, (3, 32, 32),
  12. ...
  13. test_data = get_test_data(num_cpus, data_dir, batch_size, (3, 32, 32))
  14. {code}
  15. so that it matches the dimensions of the images in my dataset, it gives this error.
  16.  
  17. {code}---------------------------------------------------------------------------
  18. JSONDecodeError Traceback (most recent call last)
  19. <ipython-input-12-9c7c3c7ff3ff> in <module>
  20. 1 for i, img in enumerate(image_data):
  21. 2 try:
  22. ----> 3 response = predictor.predict(img)
  23. 4 print("THIS IS THE SERVER RESPONSE: ", response)
  24. 5 print('image {}: class: {}'.format(i, int(response)))
  25. /usr/local/lib/python3.6/dist-packages/sagemaker/predictor.py in predict(self, data, initial_args)
  26. 86 request_args = self._create_request_args(data, initial_args)
  27. 87 response = self.sagemaker_session.sagemaker_runtime_client.invoke_endpoint(**request_args)
  28. ---> 88 return self._handle_response(response)
  29. 89
  30. 90 def _handle_response(self, response):
  31. /usr/local/lib/python3.6/dist-packages/sagemaker/predictor.py in _handle_response(self, response)
  32. 92 if self.deserializer is not None:
  33. 93 # It's the deserializer's responsibility to close the stream
  34. ---> 94 return self.deserializer(response_body, response["ContentType"])
  35. 95 data = response_body.read()
  36. 96 response_body.close()
  37. /usr/local/lib/python3.6/dist-packages/sagemaker/predictor.py in __call__(self, stream, content_type)
  38. 360 """
  39. 361 try:
  40. --> 362 return json.load(codecs.getreader("utf-8")(stream))
  41. 363 finally:
  42. 364 stream.close()
  43. /usr/lib/python3.6/json/__init__.py in load(fp, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
  44. 297 cls=cls, object_hook=object_hook,
  45. 298 parse_float=parse_float, parse_int=parse_int,
  46. --> 299 parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  47. 300
  48. 301
  49. /usr/lib/python3.6/json/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
  50. 352 parse_int is None and parse_float is None and
  51. 353 parse_constant is None and object_pairs_hook is None and not kw):
  52. --> 354 return _default_decoder.decode(s)
  53. 355 if cls is None:
  54. 356 cls = JSONDecoder
  55. /usr/lib/python3.6/json/decoder.py in decode(self, s, _w)
  56. 337
  57. 338 """
  58. --> 339 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  59. 340 end = _w(s, end).end()
  60. 341 if end != len(s):
  61. /usr/lib/python3.6/json/decoder.py in raw_decode(self, s, idx)
  62. 355 obj, end = self.scan_once(s, idx)
  63. 356 except StopIteration as err:
  64. --> 357 raise JSONDecodeError("Expecting value", s, err.value) from None
  65. 358 return obj, end
  66. JSONDecodeError: Expecting value: line 1 column 1 (char 0) {code}
  67.  
  68. How am I supposed to debug these types of issues anyway??? Where do I look? I am very new to machine learning in general.
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement