SHARE
TWEET

Untitled

a guest Jan 11th, 2011 377 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. def queryset_iterator(queryset, chunksize=1000):
  2.     """
  3.    Iterate over a Django Queryset ordered by the primary key.
  4.  
  5.    This method loads a maximum of chunksize (default: 1000) rows in it's
  6.    memory at the same time while django normally would load all rows in it's
  7.    memory.
  8.  
  9.    Note that this iterator does not support ordered query sets.
  10.  
  11.    from http://www.mellowmorning.com/2010/03/03/django-query-set-iterator-for-really-large-querysets/
  12.    with some local optimizations.
  13.    """
  14.  
  15.     last_pk = queryset.order_by('-pk')[0].pk
  16.     queryset = queryset.order_by('pk')
  17.     pk = queryset[0].pk
  18.     while pk < last_pk:
  19.         for row in queryset.filter(pk__gte=pk, pk__lte=last_pk)[:chunksize]:
  20.             yield row
  21.         pk += chunksize
RAW Paste Data
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy. OK, I Understand
 
Top