Guest User

Untitled

a guest
Oct 17th, 2017
75
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 1.90 KB | None | 0 0
  1. def reindex(self, entities):
  2. # Note - for some period of time, this doubles the memory usage of your ES cluster
  3. # If this is a problem, the ES reindex function can take source/destination cluster so you can
  4. # throw away the entire old cluster after everything is migrated.
  5. # This assumes you use an alias (shown as `self.index` in this example) for your API calls
  6.  
  7. # 1. Create a new index with a unique name
  8. # Set all of the index settings on the new index
  9. # Set all of the doctype mappings on it
  10. # 2. Use the Elasticsearch reindex helper to copy all documents over to new index
  11. # 3. Swap the alias so API calls will use the new index. Drop the old index.
  12. # 4. Reload all documents onto the new index to get any newly mapped fields that may need to be indexed
  13.  
  14. old_index = list(self.es.indices.get_alias(self.index).keys())[0]
  15.  
  16. new_index = self._generate_actual_index() # Unique name of the underlying index
  17. self.create_index(new_index) # create the index with any index and analyzer settings
  18.  
  19. # initialize the DocType mappings on the new index
  20. for entity in entities:
  21. # All of the entities in ES implement `doc_type()`
  22. doc_type = entity.doc_type()
  23. doc_type.init(new_index, using=self.es)
  24.  
  25. helpers.reindex(self.es, old_index, new_index)
  26. self.es.indices.put_alias(new_index, self.index)
  27. self.es.indices.update_aliases({
  28. "actions": [
  29. {"remove": {"index": old_index, "alias": self.index}},
  30. {"add": {"index": new_index, "alias": self.index}}
  31. ]
  32. })
  33. self.es.indices.delete(old_index)
  34.  
  35. for entity in entities:
  36. # Some function to pull batches out of the database (limit/offset)
  37. # Then use the `streaming_bulk` ES helper to push them
  38. pass
Add Comment
Please, Sign In to add comment