Advertisement
Not a member of Pastebin yet?
Sign Up,
it unlocks many cool features!
- I am using the local mode example here:
- https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker-python-sdk/mxnet_gluon_cifar10/mxnet_cifar10_local_mode.ipynb
- I managed to get it to work with the example dataset although it was really iffy because I kept getting an error whenever I would run m.fit() or deploy the predictor about a docker-compose command not working. I cannot for the life of me find where in the world this docker compose command is being invoked. It's nowhere in the example code or in the sagemaker-python-sdk. I can only get it to work by running this docker-compose command manually, having it fail to run, then running the code block again. I have no clue why this works but it does and sometimes it seems like it gets fixed spontaneously. This isn't the actual issue I am trying to fix but I thought it would be worth mentioning.
- Once I get the m.fit() training session to run and it succeeds then deploy the predictor and confirm it's running by running "docker ps", I run the code which sends a request to the predictor. I then get one of two errors.
- When I leave the example code alone the predictor server returns a dictionary with the 503 error code and a message that says "Prediction Failed". I can see where the error comes from in the sagemaker-python-sdk source code but it's extremely vague and is probably the result of something external.
- If I change around the "shape" parameter on line 48 and 50 in cifar10.py
- {code} train_data = get_train_data(num_cpus, data_dir, batch_size, (3, 32, 32),
- ...
- test_data = get_test_data(num_cpus, data_dir, batch_size, (3, 32, 32))
- {code}
- so that it matches the dimensions of the images in my dataset, it gives this error.
- {code}---------------------------------------------------------------------------
- JSONDecodeError Traceback (most recent call last)
- <ipython-input-12-9c7c3c7ff3ff> in <module>
- 1 for i, img in enumerate(image_data):
- 2 try:
- ----> 3 response = predictor.predict(img)
- 4 print("THIS IS THE SERVER RESPONSE: ", response)
- 5 print('image {}: class: {}'.format(i, int(response)))
- /usr/local/lib/python3.6/dist-packages/sagemaker/predictor.py in predict(self, data, initial_args)
- 86 request_args = self._create_request_args(data, initial_args)
- 87 response = self.sagemaker_session.sagemaker_runtime_client.invoke_endpoint(**request_args)
- ---> 88 return self._handle_response(response)
- 89
- 90 def _handle_response(self, response):
- /usr/local/lib/python3.6/dist-packages/sagemaker/predictor.py in _handle_response(self, response)
- 92 if self.deserializer is not None:
- 93 # It's the deserializer's responsibility to close the stream
- ---> 94 return self.deserializer(response_body, response["ContentType"])
- 95 data = response_body.read()
- 96 response_body.close()
- /usr/local/lib/python3.6/dist-packages/sagemaker/predictor.py in __call__(self, stream, content_type)
- 360 """
- 361 try:
- --> 362 return json.load(codecs.getreader("utf-8")(stream))
- 363 finally:
- 364 stream.close()
- /usr/lib/python3.6/json/__init__.py in load(fp, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
- 297 cls=cls, object_hook=object_hook,
- 298 parse_float=parse_float, parse_int=parse_int,
- --> 299 parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
- 300
- 301
- /usr/lib/python3.6/json/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
- 352 parse_int is None and parse_float is None and
- 353 parse_constant is None and object_pairs_hook is None and not kw):
- --> 354 return _default_decoder.decode(s)
- 355 if cls is None:
- 356 cls = JSONDecoder
- /usr/lib/python3.6/json/decoder.py in decode(self, s, _w)
- 337
- 338 """
- --> 339 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
- 340 end = _w(s, end).end()
- 341 if end != len(s):
- /usr/lib/python3.6/json/decoder.py in raw_decode(self, s, idx)
- 355 obj, end = self.scan_once(s, idx)
- 356 except StopIteration as err:
- --> 357 raise JSONDecodeError("Expecting value", s, err.value) from None
- 358 return obj, end
- JSONDecodeError: Expecting value: line 1 column 1 (char 0) {code}
- How am I supposed to debug these types of issues anyway??? Where do I look? I am very new to machine learning in general.
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement