Store twitter data in a way such that for each tweet we have:

- Original tweet (+)
- Clean preprocessed tweet
- List of Urls from tweet
- List of Hashtags from tweet
- tweet type (+)
- author (+)
- timestamp (+)
- number of likes
- number of retweets
- number of comments
- language
- mentioned users (@)

With such an approach we will not loose any information, only restructure it in a useful way.

With (+) I marked what we have now.

Clean preprocessed tweet will be used for converting tweet into vector.
List of Urls will be useful in future if we want to follow this URL and download info from webpage.
List of Hashtags will be used for vector creation, we haven't decided yet how exactly it will happen.


Regarding deleting the hashtags, I think we need to distinguish 2 cases:
1) Hashtag is a word inside sentence, removing this hashtag will break the meaning of the sentence. In this case we just remove # sign and keep the hashtag word.
2) There is a list of hashtags outside the sentence, removing them will not cause sentence break. In this case we just delete the hashtags from the tweet and put them separately to the list of hashtags.

PREPROCESSING (how to get Clean preprocessed tweet)

- delete nonEnglish tweets (or detect language and store it as feature of tweet)
- move all URLs to the the list and save as tweet feature, remove URLs from tweet text.
- move all hashtag words to the list of hashtags, save this list as new feature. From original text detect all hashtag who has another hashtag as a neighbor, delete these hashtags. Remove all '#' signs.
- delete @user names mentioned in tweet, and move it to another feature.

Here I described  the basic preprocessing steps we will need at any case. Further steps will be specified after first results.