Developing A Secure Email/IM Replacement

Rethinking e-mail and IM for the "new" age of the Internet we've found ourselves in. Would appreciate feedback. First cut of a design:

NuMail - secure e-mail / IM replacement for secure communications ('comms')

General ideas:
- server never sees plain-text comms
- clients handle encryption/decryption
- minimize attack vectors to client-only (keyloggers etc)
- easy to use: users don't see private/public keys etc, they just need to remember one passphrase

Account Creation:
- client sends server:
-- username
-- scrypt(passphrase) ('hash')
-- private key encrypted using hash
-- plaintext public key
- server generates salt, stores salt & scrypt(hash,salt), encrypted private key and plaintext public key
- since hash was used to encrypt private key but server stores scrypt(hash,salt), server has just enough information to authenticate client but not enough to decrypt comms

Have chosen scrypt as there are no known ASIC implementations.

Login:
- client sends server username and hash (via SSL - which CA's can be trusted?)
- server checks that scrypt(hash,salt) matches stored data
- upon success, client keeps in memory hash for duration of session.

User A sends comms to User B (same server):
- A's client asks server for B's public key (via SSL so no middle man can sub their own pub key)
- A's client encrypts comms using B's public key => payload B
- A's client encrypts comms using A's public key => payload A
- A's client sends payloads B,A to server
- Server stores payload B against B's account, payload A against A's account

User A sends comms to User B (different server):
- A's server asks B's server for B's public key, provides it to A's client
- A's client encrypts comms using B's public key => payload B
- A's client encrypts comms using A's public key => payload A
- A's client sends payloads B,A to A's server
- A's server stores payload A against account A, sends payload B to B's server which stores it against B's account.

How Does B Read Payload B?
- B asks server for B's encrypted private key
- client uses in memory hash which was originally used to encrypt private key to now decrypt it, then uses decrypted private key to decrypt payload B
- A can read payload A in the same way. Payloads A & B are identical in content
  when decrypted but require separate keys (A & B's respectively) to be decrypted by their owner

Only way for server to spy on user comms would be to wait for user to login and record submitted hash, then use it to decrypt encrypted private key. This could be eliminated by storing encrypted private key client-side, but then the system becomes harder to use across multiple devices.

I understand that javascript in it's current form isn't completely up to scratch for web-based client-side crypto (http://www.matasano.com/articles/javascript-cryptography/). But I don't think all of his points hold for this particular use case, particularly the point around using SSL making javascript crypto obsolete; not obsolete at all in the case of encrypting a message payload to ensure the server never receives and thus can never see/store/recover plaintext.

Code would be open source. Anyone could host a NuMail server and charge for storage or other features.

Thoughts?