a guest Jan 11th, 2011 377 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
- def queryset_iterator(queryset, chunksize=1000):
- Iterate over a Django Queryset ordered by the primary key.
- This method loads a maximum of chunksize (default: 1000) rows in it's
- memory at the same time while django normally would load all rows in it's
- Note that this iterator does not support ordered query sets.
- from http://www.mellowmorning.com/2010/03/03/django-query-set-iterator-for-really-large-querysets/
- with some local optimizations.
- last_pk = queryset.order_by('-pk').pk
- queryset = queryset.order_by('pk')
- pk = queryset.pk
- while pk < last_pk:
- for row in queryset.filter(pk__gte=pk, pk__lte=last_pk)[:chunksize]:
- yield row
- pk += chunksize
RAW Paste Data