SHARE
TWEET

deep_get_size_of.py

mvaganov May 10th, 2019 70 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. def deep_get_size_of(o, ids=None, progress_callback=None):
  2.     """Find the memory footprint of a Python object
  3.  
  4.    This is a recursive function that drills down a Python object graph
  5.    like a dictionary holding nested dictionaries with lists of lists
  6.    and tuples and sets.
  7.  
  8.    The sys.getsizeof function does a shallow size of only. It counts each
  9.    object inside a container as pointer only regardless of how big it
  10.    really is.
  11.    https://code.tutsplus.com/tutorials/understand-how-much-memory-your-python-objects-use--cms-25609
  12.  
  13.    :param o: the object
  14.    :param ids:
  15.    :param progress_callback
  16.    :return:
  17.    """
  18.     try:
  19.         import collections.abc as collections_abc  # only works on python 3.3+
  20.     except ImportError:
  21.         import collections as collections_abc
  22.     import sys
  23.     if ids is None:
  24.         ids = set()
  25.     d = deep_get_size_of
  26.     if id(o) in ids:
  27.         return 0
  28.     r = sys.getsizeof(o)
  29.     ids.add(id(o))
  30.     if isinstance(o, str):
  31.         return r
  32.     if isinstance(o, collections_abc.Mapping):
  33.         total = 0
  34.         count = len(o.iteritems())
  35.         iterator = 0
  36.         for k, v in o.iteritems():
  37.             total += d(k, ids) + d(v, ids)
  38.             if progress_callback is not None:
  39.                 iterator += 1.0
  40.                 progress_callback(iterator/count)
  41.         return r + total
  42.         # return r + sum(d(k, ids) + d(v, ids) for k, v in o.iteritems())
  43.         # this single line does what the above lines do, but without calling progress_callback
  44.     if isinstance(o, collections_abc.Container):
  45.         total = 0
  46.         count = len(o)
  47.         iterator = 0
  48.         for x in o:
  49.             total += d(x, ids)
  50.             if progress_callback is not None:
  51.                 iterator += 1.0
  52.                 progress_callback(iterator/count)
  53.         return r + total
  54.         # return r + sum(d(x, ids) for x in o)
  55.     return r
RAW Paste Data
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy. OK, I Understand
 
Top