Guest User

Untitled

a guest
Jan 24th, 2019
88
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 0.54 KB | None | 0 0
  1. def iterator2dataframes(iterator, chunk_size: int):
  2. """Turn an iterator into multiple small pandas.DataFrame
  3. This is a balance between memory and efficiency"""
  4. records = []
  5. frames = []
  6. for i, record in enumerate(iterator):
  7. records.append(record)
  8. if i % chunk_size == chunk_size - 1:
  9. frames.append(pd.DataFrame(records))
  10. records = []
  11. if records:
  12. frames.append(pd.DataFrame(records))
  13. return pd.concat(frames)
  14. df = iterator2dataframes(cursor, 25000)
Add Comment
Please, Sign In to add comment