web2py - model less modules web2pyapps

Model Less Apps (using data models and modules in web2py)
Recipe by Bruno Rocha (rochacbruno) on 2012-03-05 in Application, Model, Module
Views (7062)Favorite (7)Like (6)Dislike (0)Subscribe (3)

    Create your datamodels in modules

Share

Recently many people have asked me why to avoid the use of models in large apps. The main fact is that the models are run each (and all) request, so that all objects, connections, and variables defined there will be global for the whole application.

In medium apps that have actions using objects globally there would be no problem, but in apps where size and execution time of actions varies with the complexity, always have all models run may be the bottleneck.

A simple example in a single app:
This page needs a lot of data models and functions: http://www.reddit.com/
It does not need: http://www.reddit.com/help/privacypolicy

So the best thing is to choose in each action, which data models and functions you want to have available, and thus avoid unnecessary loads.

Another case is when we have an ajax call, and this action should only return a JSON, or sometimes only validate a value. in this case do not want to have as many objects Auth, db, crud and all the tables.

So, I created a simple base structure for apps without models.

A model less approach to web2py applications


This app uses the following components.

modules/appname.py

This is the main module will serve as a proxy for other components including db, Auth, Crud, Mail etc.. And this module will also be responsible for loading configuration files that can come from, json, ini, xml or sqlite databases.

    from gluon.tools import Auth, Crud, Mail
    from gluon.dal import DAL
    from datamodel.user import User
    from gluon.storage import Storage
    from gluon import current


    class MyApp(object):
        def __init__(self):
            self.session, self.request, self.response, self.T = \
                current.session, current.request, current.response, current.T
            self.config = Storage(db=Storage(),
                                  auth=Storage(),
                                  crud=Storage(),
                                  mail=Storage())
            # Global app configs
            # here you can choose to load and parse your configs
            # from JSON, XML, INI or db
            # Also you can put configs in a cache
            #### -- LOADING CONFIGS -- ####
            self.config.db.uri = "sqlite://myapp.sqlite"
            self.config.db.migrate = True
            self.config.db.migrate_enabled = True
            self.config.db.check_reserved = ['all']
            self.config.auth.server = "default"
            self.config.auth.formstyle = "divs"
            self.config.mail.server = "logging"
            self.config.mail.sender = "me@mydomain.com"
            self.config.mail.login = "me:1234"
            self.config.crud.formstyle = "divs"
            #### -- CONFIGS LOADED -- ####

        def db(self, datamodels=None):
            # here we need to avoid redefinition of db
            # and allow the inclusion of new entities
            if not hasattr(self, "_db"):
                self._db = DataBase(self.config, datamodels)
            if datamodels:
                self._db.define_datamodels(datamodels)
            return self._db

        @property
        def auth(self):
            # avoid redefinition of Auth
            # here you can also include logic to del
            # with facebook based in session, request, response
            if not hasattr(self, "_auth"):
                self._auth = Account(self.db())
            return self._auth

        @property
        def crud(self):
            # avoid redefinition of Crud
            if not hasattr(self, "_crud"):
                self._crud = FormCreator(self.db())
            return self._crud

        @property
        def mail(self):
            # avoid redefinition of Mail
            if not hasattr(self, "_mail"):
                self._mail = Mailer(self.config)
            return self._mail


    class DataBase(DAL):
        """
        Subclass of DAL
        auto configured based in config Storage object
        auto instantiate datamodels
        """
        def __init__(self, config, datamodels=None):
            self.config = config
            DAL.__init__(self,
                         **config.db)

            if datamodels:
                self.define_datamodels(datamodels)

        def define_datamodels(self, datamodels):
            # Datamodels will define tables
            # datamodel ClassName becomes db attribute
            # so you can do
            # db.MyEntity.insert(**values)
            # db.MyEntity(value="some")
            for datamodel in datamodels:
                obj = datamodel(self)
                self.__setattr__(datamodel.__name__, obj.entity)
                if obj.__class__.__name__ == "Account":
                    self.__setattr__("auth", obj)


    class Account(Auth):
        """Auto configured Auth"""
        def __init__(self, db):
            self.db = db
            self.hmac_key = Auth.get_or_create_key()
            Auth.__init__(self, self.db, hmac_key=self.hmac_key)
            user = User(self)
            self.entity = user.entity

            # READ AUTH CONFIGURATION FROM CONFIG
            self.settings.formstyle = self.db.config.auth.formstyle
            if self.db.config.auth.server == "default":
                self.settings.mailer = Mailer(self.db.config)
            else:
                self.settings.mailer.server = self.db.config.auth.server
                self.settings.mailer.sender = self.db.config.auth.sender
                self.settings.mailer.login = self.db.config.auth.login


    class Mailer(Mail):
        def __init__(self, config):
            Mail.__init__(self)
            self.settings.server = config.mail.server
            self.settings.sender = config.mail.sender
            self.settings.login = config.mail.login


    class FormCreator(Crud):
        def __init__(self, db):
            Crud.__init__(db)
            self.settings.auth = None
            self.settings.formstyle = self.db.config.crud.formstyle



modules/basemodel.py

This module is an abstraction of the DAL, specifically abstracts the method define_tables. It may seem strange or unnecessary. But I concluded that placing it in a more object-oriented means that the writing is more organized and better reuse of code.



    from gluon.dal import DAL
    from gluon.tools import Auth


    class BaseModel(object):
        """Base Model Class
        all define_ methods will be called, then
        all set_ methods (hooks) will be called."""

        hooks = ['set_table',
                 'set_validators',
                 'set_visibility',
                 'set_representation',
                 'set_widgets',
                 'set_labels',
                 'set_comments',
                 'set_computations',
                 'set_updates',
                 'set_fixtures']

        def __init__(self, db=None, migrate=None, format=None):
            self.db = db
            assert isinstance(self.db, DAL)
            self.config = db.config
            if migrate != None:
                self.migrate = migrate
            elif not hasattr(self, 'migrate'):
                self.migrate = self.config.db.migrate
            if format != None or not hasattr(self, 'format'):
                self.format = format
            self.set_properties()
            self.check_properties()
            self.define_table()
            self.define_validators()
            self.define_visibility()
            self.define_representation()
            self.define_widgets()
            self.define_labels()
            self.define_comments()
            self.define_computations()
            self.define_updates()
            self.pre_load()

        def check_properties(self):
            pass

        def define_table(self):
            fakeauth = Auth(DAL(None))
            self.fields.extend([fakeauth.signature])
            self.entity = self.db.define_table(self.tablename,
                                               *self.fields,
                                               **dict(migrate=self.migrate,
                                               format=self.format))

        def define_validators(self):
            validators = self.validators if hasattr(self, 'validators') else {}
            for field, value in validators.items():
                self.entity[field].requires = value

        def define_visibility(self):
            try:
                self.entity.is_active.writable = self.entity.is_active.readable = False
            except:
                pass
            visibility = self.visibility if hasattr(self, 'visibility') else {}
            for field, value in visibility.items():
                self.entity[field].writable, self.entity[field].readable = value

        def define_representation(self):
            representation = self.representation if hasattr(self, 'representation') else {}
            for field, value in representation.items():
                self.entity[field].represent = value

        def define_widgets(self):
            widgets = self.widgets if hasattr(self, 'widgets') else {}
            for field, value in widgets.items():
                self.entity[field].widget = value

        def define_labels(self):
            labels = self.labels if hasattr(self, 'labels') else {}
            for field, value in labels.items():
                self.entity[field].label = value

        def define_comments(self):
            comments = self.comments if hasattr(self, 'comments') else {}
            for field, value in comments.items():
                self.entity[field].comment = value

        def define_computations(self):
            computations = self.computations if hasattr(self, 'computations') else {}
            for field, value in computations.items():
                self.entity[field].compute = value

        def define_updates(self):
            updates = self.updates if hasattr(self, 'updates') else {}
            for field, value in updates.items():
                self.entity[field].update = value

        def pre_load(self):
            for method in self.hooks:
                if hasattr(self, method):
                    self.__getattribute__(method)()


    class BaseAuth(BaseModel):
        def __init__(self, auth, migrate=None):
            self.auth = auth
            assert isinstance(self.auth, Auth)
            self.db = auth.db
            from gluon import current
            self.request = current.request
            self.config = self.db.config
            self.migrate = migrate or self.config.db.migrate
            self.set_properties()
            self.define_extra_fields()
            self.auth.define_tables(migrate=self.migrate)
            self.entity = self.auth.settings.table_user
            self.define_validators()
            self.hide_all()
            self.define_visibility()
            self.define_register_visibility()
            self.define_profile_visibility()
            self.define_representation()
            self.define_widgets()
            self.define_labels()
            self.define_comments()
            self.define_computations()
            self.define_updates()
            self.pre_load()

        def define_extra_fields(self):
            self.auth.settings.extra_fields['auth_user'] = self.fields

        def hide_all(self):
            alwaysvisible = ['first_name', 'last_name', 'password', 'email']
            for field in self.entity.fields:
                if not field in alwaysvisible:
                    self.entity[field].writable = self.entity[field].readable = False

        def define_register_visibility(self):
            if 'register' in self.request.args:
                register_visibility = self.register_visibility if hasattr(self, 'register_visibility') else {}
                for field, value in register_visibility.items():
                    self.entity[field].writable, self.entity[field].readable = value

        def define_profile_visibility(self):
            if 'profile' in self.request.args:
                profile_visibility = self.profile_visibility if hasattr(self, 'profile_visibility') else {}
                for field, value in profile_visibility.items():
                    self.entity[field].writable, self.entity[field].readable = value



modules/datamodel/<some entity>.py

Here is where you will create data models, define the fields, validation, fixtures etc ... the API here is different from the normal web2py mode, bmodules/handlers/base.pyut you can still use the same objects and methods.


    from gluon.dal import Field
    from basemodel import BaseModel
    from gluon.validators import IS_NOT_EMPTY, IS_SLUG
    from gluon import current
    from plugin_ckeditor import CKEditor


    class Post(BaseModel):
        tablename = "blog_post"

        def set_properties(self):
            ckeditor = CKEditor(self.db)
            T = current.T
            self.fields = [
                Field("author", "reference auth_user"),
                Field("title", "string", notnull=True),
                Field("description", "text"),
                Field("body_text", "text", notnull=True),
                Field("slug", "text", notnull=True),
            ]

            self.widgets = {
                "body_text": ckeditor.widget
            }

            self.visibility = {
                "author": (False, False)
            }

            self.representation = {
                "body_text": lambda row, value: XML(value)
            }

            self.validators = {
                "title": IS_NOT_EMPTY(),
                "body_text": IS_NOT_EMPTY()
            }

            self.computations = {
              "slug": lambda r: IS_SLUG()(r.title)[0],
            }

            self.labels = {
                "title": T("Your post title"),
                "description": T("Describe your post (markmin allowed)"),
                "body_text": T("The content")
            }




modules/handlers/base.py

This is a rendering engine using the web2py template, just a base class that initializes our handlers with everything you need, here you can inject common objects in to render context, also you can implement cache or extend in any way you want, here you can choose another template language if needed, it is very easy to use chetah, jinja or mako here instead of web2py template (if you really want or need). This structure allows you to easily have an app with multiple themes, and the views can be in any directory or even in the database (I am using it for email templates stored in database).


    rom gluon import URL
    from gluon.tools import prettydate


    class Base(object):
        def __init__(
            self,
            hooks=[],
            meta=None,
            context=None
            ):
            from gluon.storage import Storage
            self.meta = meta or Storage()
            self.context = context or Storage()
            # you can user alers for response flash
            self.context.alerts = []

            self.context.prettydate = prettydate

            # hooks call
            self.start()
            self.build()
            self.pre_render()
            self.load_menus()

            # aditional hooks
            if not isinstance(hooks, list):
                hooks = [hooks]

            for hook in hooks:
                self.__getattribute__(hook)()

        def start(self):
            pass

        def build(self):
            pass

        def load_menus(self):
            self.response.menu = [
               (self.T('Home'), False, URL('default', 'index'), []),
               (self.T('New post'), False, URL('post', 'new'), []),
            ]

        def pre_render(self):
            from gluon import current
            self.response = current.response
            self.request = current.request
            self.session = current.session
            self.T = current.T

        def render(self, view=None):
            viewfile = "%s.%s" % (view, self.request.extension)
            return self.response.render(viewfile, self.context)


modules/handlers/<some entity>.py

Here is where the logic of action should occur, consult the database, calculate, assemble objects such as forms, tables, etc. .. and here also will check the permissions and authentication, you have to create one handler for each entity of your app, example: contacts, person, article, product...


    from handlers.base import Base
    from myapp import MyApp
    from datamodel.post import Post as PostModel
    from gluon import SQLFORM, URL, redirect


    class Post(Base):
        def start(self):
            self.app = MyApp()
            self.auth = self.app.auth  # you need to access this to define users
            self.db = self.app.db([PostModel])

            # this is needed to inject auth in template render
            # only needed to use auth.navbar()
            self.context.auth = self.auth

        def list_all(self):
            self.context.posts = self.db(self.db.Post).select(orderby=~self.db.Post.created_on)

        def create_new(self):
            # permission is checked here
            if self.auth.has_membership("author", self.auth.user_id):
                self.db.Post.author.default = self.auth.user_id
                self.context.form = SQLFORM(self.db.Post, formstyle='divs').process(onsuccess=lambda form: redirect(URL('show', args=form.vars.id)))
            else:
                self.context.form = "You can't post, only logged in users, members of 'author' group can post"

        def edit_post(self, post_id):
            post = self.db.Post[post_id]
            # permission is checked here
            if not post or post.author != self.auth.user_id:
                redirect(URL("post", "index"))
            self.context.form = SQLFORM(self.db.Post, post.id, formstyle='divs').process(onsuccess=lambda form: redirect(URL('show', args=form.vars.id)))

        def show(self, post_id):
            self.context.post = self.db.Post[post_id]
            if not self.context.post:
                redirect(URL("post", "index"))




controllers/<some entity>.py

Here is the entry point to call the handler, will only create an instance of a handler and pass arguments to it, then return will be handler.render() ready and able to cache.



    from handlers.post import Post


    def index():
        post = Post('list_all')
        return post.render("mytheme/listposts")


    def new():
        post = Post('create_new')
        return post.render("mytheme/newpost")


    def edit():
        post = Post()
        post.edit_post(request.args(0))
        return post.render("mytheme/editpost")


    def show():
        post = Post()
        post.show(request.args(0))
        return post.render("mytheme/showpost")



So the views (template files) will be at views/yourtheme/somefilename



example of the view for show() action

    {{extend 'layout.html'}}
    <div class="row">
        <div class="three columns alpha">
            <img src="{{=URL('default', 'download', args=post.author.thumbnail)}}" width=100>


            <strong>{{="%(first_name)s %(last_name)s (%(nickname)s)" % post.author}}</strong>


            <small>{{=prettydate(post.created_on)}}</small>
        </div>
        <div class="eleven columns" style="border-left:1px solid #444;padding:15px;">
        <h2><a href="{{=URL('show', args=[post.id, post.slug])}}">{{=post.title}}</a></h2>
        {{=XML(post.body_text)}}
        </div>
        <div class="one columns omega">
          {{if auth.user_id == post.author:}}
              <a href="{{=URL('edit', args=[post.id, post.slug])}}" class="button">Edit</a>
          {{pass}}
        </div>
    </div>
    <hr/>


I am testing and I found it is very performatic, but, you can help testing it more.

The code is here: https://github.com/rochacbruno/web2py_model_less_app
Download the app here: https://github.com/rochacbruno/web2py_model_less_app/downloads

This sample is a blog system with just one entity 'blog_post' and also the auth and users, but you can use as a template to create more entities.

Can you help testing the gain of performance of this approach?