Ch4 Ex1-Programming Collective Intelligence

from string import punctuation

def separatewords(text):
    punctuation1 = punctuation.replace('+','').replace('$','').replace('#','')
    res = []
    result = text.replace('\n',' ').replace('\t',' ').replace('\r',' ').replace('\x0b',' ').replace('\x0c',' ').lower()
    result1 = result.split(' ')
    # Passes reiterates over the cleaned text to get, for example, a ? within quotes
    for passes in range(2):
        for a in range(len(result1)):
            for punk in punctuation1:
                if result1[a] == punk:
                    result1[a] = ""
                if result1[a][-1:] == punk:
                    result1[a] = result1[a].replace(punk, ' ')[:-1]
                if result1[a][:0] in punk:
                    result1[a] = result1[a].replace(punk, '')[0:]
    for b in result1:
        if b != '': res.append(b)
    return res


"""   COLLECTIVE INTELLIGENCE - CHAPTER 4 EXERCISE 1

1. Word separation. The separatewords() method currently considers any non-alphanumeric character to be a separator, meaning it will not properly index entries like "C++," "$20," "Ph.D.," or "617-555-1212." What is a better way to separate words? Does using whitespace as a separator work? Write a better word separation function.

Note: There is no real way of catching all exceptions
      example  chillin'  or  'chilian'
      example  C++  or 1+ 1 = 3

      There are no perfect solutions with humans especially where spoken languages are concerned.
      The method described above was designed to index the entries in the exercise,
      but there's unicode to take into account, and french, spanish, japanese........
"""