1. $ time python fastq-unique.py < /usr/local/src/methylcode/data/dme/en.wt.1.fastq > unique.fastq
  2. 36754782 records in file
  3. checking 2525155 potential duplicates in a python set
  4. 4699 false-positive duplicates in the bloom filter
  5.  
  6. real 11m31.077s
  7. user 11m1.937s
  8. sys 0m23.093s