Guest User

Untitled

a guest
Apr 5th, 2018
144
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 2.48 KB | None | 0 0
  1. import psycopg2
  2. import string
  3. import random
  4. from multiprocessing import Pool
  5.  
  6. '''
  7. Machine:
  8. 1 core
  9. 1 GB RAM
  10.  
  11. Improvements:
  12.  
  13. #1 Removed the autocommit in populate_database
  14.  
  15. While the option is needed to create a database, it is not to work within a given database.
  16. I removed the option (conn.set_isolation_level(psycopg2.extensions.ISOLATION_LEVEL_AUTOCOMMIT)
  17. and explicitely added a commit to the connection further down (conn.commit()).
  18. This limits the amount of transaction and makes the queries significantly faster.
  19. 29 seconds -> 9 seconds
  20.  
  21. #2 Added a Pool
  22.  
  23. The operations done on the different databases are independent and can be executed simultaneously.
  24. I've added a Pool to run create_and_populate_database for each db simultaneously.
  25. Python Global Interpreter Lock allows better results even on a single threaded machines as it uses subprocesses and not threads.
  26. Here is the code:
  27. p = Pool(DATABASES_COUNT)
  28. p.map(create_and_populate_database, db_names)
  29. 9 seconds -> 4 seconds
  30.  
  31. '''
  32.  
  33.  
  34. DATABASES_COUNT = 5
  35. DB_NAME_SIZE = 20
  36.  
  37. TABLES_COUNT = 4
  38. TABLE_NAME_SIZE = 10
  39.  
  40. ROWS_COUNT = 500
  41. STRING_NAME_SIZE = 8
  42.  
  43. WORKERS = 0
  44. PG_HOST = "127.0.0.1"
  45. PG_USER = "postgres"
  46.  
  47. def generate_random_string(chars):
  48. return ''.join(random.choice(string.ascii_lowercase + string.digits) for _ in range(chars))
  49.  
  50. def populate_database(database_name):
  51. conn = psycopg2.connect("host='%s' dbname='%s' password='none' user='%s'" % (PG_HOST, database_name, PG_USER))
  52. cursor = conn.cursor()
  53. for i in range(0, TABLES_COUNT):
  54. table_name = 't' + generate_random_string(TABLE_NAME_SIZE)
  55. print("CREATING TABLE %s" % table_name)
  56. cursor.execute("CREATE TABLE %s (id serial, name varchar(%s) not null)" % (table_name, STRING_NAME_SIZE))
  57. for j in range(0, ROWS_COUNT):
  58. cursor.execute("INSERT INTO %s (name) VALUES ('%s')" % (table_name, generate_random_string(STRING_NAME_SIZE)))
  59. conn.commit()
  60. conn.close()
  61.  
  62. def create_and_populate_database(database_name):
  63. print("Creating database %s" % database_name)
  64. conn = psycopg2.connect("host='%s' dbname='postgres' password='none' user='%s'" % (PG_HOST, PG_USER))
  65. conn.set_isolation_level(psycopg2.extensions.ISOLATION_LEVEL_AUTOCOMMIT)
  66. cursor = conn.cursor()
  67. cursor.execute("CREATE DATABASE %s" % (database_name))
  68. conn.close()
  69. populate_database(database_name)
  70.  
  71. db_names = []
  72. for i in range(0, DATABASES_COUNT):
  73. db_names.append('d' + generate_random_string(DB_NAME_SIZE))
  74.  
  75. if __name__ == '__main__':
  76. p = Pool(DATABASES_COUNT)
  77. p.map(create_and_populate_database, db_names)
Add Comment
Please, Sign In to add comment