nacl proposal v2

Intro

Native Client offers the nacl_io library, which allows developers to mount filesystems like html5fs and access them using regular POSIX calls. At the moment, filesystem support is restricted to the local machine and to public http content using httpfs. The aim of this project is to extend nacl_io by adding new filesystems to it - specifically, cloud storage. Cloud is the future of computing, and letting developers mount a cloud storage as a regular filesystem would allow for more powerful Native Client applications in future.

Notes:

- Since the Pepper API provides classes for interacting with URL resources, my current focus in the project is to interface with HTTP based APIs. If time permits, JavaScript based APIs may be implemented.


Project goals

- Expose at least one cloud storage( which uses HTTP based API) as a filesystem, starting with Dropbox.

- Implement local caching in the filesystem, so that files accessed from the cloud storage are available afterwards.

- Tests and demos for both of the above.


Implementation

1 Most cloud storage APIs support OAuth verification. The assumption in the project is that the developer completes the verification, and passes the access token to the filesystem in the mount string parameter as follows:

mount("","/mount/target","cloudfs","","provider=dropbox,access_token=somedata")

2 In the initializer function for cloudfs, the parameter string is parsed. Different provider classes will be available, and cloudfs instantiates an object depending on the provider. The access_token data is stored in the class itself. Any time cloudfs needs to interface with the server, it uses the provider object. The provider classes will interface with the cloud server, according to the API.

3 cloudfs will keep an array of structures fileMap, where fileMap contains {fileName,access_time,modified_time}. Every file in the cache will have an entry in fileMap. The time fields can be populated using the metadata from the cloud server and the local time from the machine. Caching functions can use the time fields to various ways. This array can be written to the cache storage periodically or before unmounting, so that this data persists.

4 cloudfs shall implement two mounts inside /mount/target - /mount/target/cloud (which provides direct access to the cloud), and /mount/target/cached_cloud( which will be the cache system implemented in html5fs). All calls will first be routed to this cache system. If these calls fail due to absence in the cache, cloudfs will use the provider object to call the server. The returned file from the server is then written to the cache. This behavior will change depending on the function being implemented, but the general idea should remain the same.

5 Any operation on a file will modify fileMap appropriately. Also, before any operation is performed on a file already in the cache, the server is queried to check if a more updated version is present. The behavior for such version clashes will have to be specified.

6 The caching operation described here is rudimentary at best, and will not support differential caching or journaling. Depending on the results of the community bonding period, these ideas will be refined.


Timeline and deliverables

Community bonding period
- Get familiar with nacl_io library and how developers use it in production code.
- Decide between using httpfs or Pepper objects(URLLoader etc) in the base filesystem. httpfs already implements a lot of http access methods, so leveraging them would be advantageous. Also, httpfs has a caching system which I'd like to investigate, if it can be extended further.
- Refine the implementation of the base filesystem(the interface with the cloud storage) currently planned.
- Research the different APIs for their specific methods, and plan ways to create interfaces for them.
- Research about how to efficiently implement caching in a filesystem, and ways to introduce differential caching and delta patching if possible. Also study about the use of IOCTLs and how to use them for more efficient operations in the project.
- Create templates for the planned implementations.
- Get familiar with the jsfs system, and plan an implementation to interface JavaScript APIs with the proposed filesystem.

Week 1,2,3
- Create the cloudfs class and the base implementation for dropbox.
- Testing the implementation

Week 4,5
- Creating a demo for the created implementation
- Buffer week for further testing and refining, if any

Mid term evaluation deliverables
- cloudfs class
- Implemention of the dropbox interface
- Demo for the same

Week 6,7,8
- Implementation of caching functions.
- Testing of the implementation.

Week 9,10
- Working demo for cloudfs, with dropbox as the provider.

Week 11,12 ( Tentative)
- Implement more interfaces for other cloud storages.
- Testing of the implementations.

Final deliverables
- cloudfs class, with local caching implemented.
- Interface implementation for dropbox.
- (Tentative) Interface implementations for other cloud storages


about me

My name is Mainak Gachhui, and I am currently studying in Manipal Institute of Technnology. I am passionate about technology, and very interested in contributing to open source software. I do not have any major open source contribution yet, but I hope to change that with this project and continue further.

contact details

Email: mgachhui@gmail.com