If you benefit from web2py hope you feel encouraged to pay it forward by contributing back to society in whatever form you choose!

Recently many people have asked me why to avoid the use of models in large apps. The main fact is that the models are run each (and all) request, so that all objects, connections, and variables defined there will be global for the whole application.

In medium apps that have actions using objects globally there would be no problem, but in apps where size and execution time of actions varies with the complexity, always have all models run may be the bottleneck.

A simple example in a single app:
This page needs a lot of data models and functions: http://www.reddit.com/
It does not need: http://www.reddit.com/help/privacypolicy

So the best thing is to choose in each action, which data models and functions you want to have available, and thus avoid unnecessary loads.

Another case is when we have an ajax call, and this action should only return a JSON, or sometimes only validate a value. in this case do not want to have as many objects Auth, db, crud and all the tables.

So, I created a simple base structure for apps without models.

 

A model less approach to web2py applications


This app uses the following components.

modules/appname.py

This is the main module will serve as a proxy for other components including db, Auth, Crud, Mail etc.. And this module will also be responsible for loading configuration files that can come from, json, ini, xml or sqlite databases.

from gluon.tools import Auth, Crud, Mail
from gluon.dal import DAL
from datamodel.user import User
from gluon.storage import Storage
from gluon import current


class MyApp(object):
    def __init__(self):
        self.session, self.request, self.response, self.T = \
            current.session, current.request, current.response, current.T
        self.config = Storage(db=Storage(),
                              auth=Storage(),
                              crud=Storage(),
                              mail=Storage())
        # Global app configs
        # here you can choose to load and parse your configs
        # from JSON, XML, INI or db
        # Also you can put configs in a cache
        #### -- LOADING CONFIGS -- ####
        self.config.db.uri = "sqlite://myapp.sqlite"
        self.config.db.migrate = True
        self.config.db.migrate_enabled = True
        self.config.db.check_reserved = ['all']
        self.config.auth.server = "default"
        self.config.auth.formstyle = "divs"
        self.config.mail.server = "logging"
        self.config.mail.sender = "me@mydomain.com"
        self.config.mail.login = "me:1234"
        self.config.crud.formstyle = "divs"
        #### -- CONFIGS LOADED -- ####

    def db(self, datamodels=None):
        # here we need to avoid redefinition of db
        # and allow the inclusion of new entities
        if not hasattr(self, "_db"):
            self._db = DataBase(self.config, datamodels)
        if datamodels:
            self._db.define_datamodels(datamodels)
        return self._db

    @property
    def auth(self):
        # avoid redefinition of Auth
        # here you can also include logic to del
        # with facebook based in session, request, response
        if not hasattr(self, "_auth"):
            self._auth = Account(self.db())
        return self._auth

    @property
    def crud(self):
        # avoid redefinition of Crud
        if not hasattr(self, "_crud"):
            self._crud = FormCreator(self.db())
        return self._crud

    @property
    def mail(self):
        # avoid redefinition of Mail
        if not hasattr(self, "_mail"):
            self._mail = Mailer(self.config)
        return self._mail


class DataBase(DAL):
    """
    Subclass of DAL
    auto configured based in config Storage object
    auto instantiate datamodels
    """
    def __init__(self, config, datamodels=None):
        self.config = config
        DAL.__init__(self,
                     **config.db)

        if datamodels:
            self.define_datamodels(datamodels)

    def define_datamodels(self, datamodels):
        # Datamodels will define tables
        # datamodel ClassName becomes db attribute
        # so you can do
        # db.MyEntity.insert(**values)
        # db.MyEntity(value="some")
        for datamodel in datamodels:
            obj = datamodel(self)
            self.__setattr__(datamodel.__name__, obj.entity)
            if obj.__class__.__name__ == "Account":
                self.__setattr__("auth", obj)


class Account(Auth):
    """Auto configured Auth"""
    def __init__(self, db):
        self.db = db
        self.hmac_key = Auth.get_or_create_key()
        Auth.__init__(self, self.db, hmac_key=self.hmac_key)
        user = User(self)
        self.entity = user.entity

        # READ AUTH CONFIGURATION FROM CONFIG
        self.settings.formstyle = self.db.config.auth.formstyle
        if self.db.config.auth.server == "default":
            self.settings.mailer = Mailer(self.db.config)
        else:
            self.settings.mailer.server = self.db.config.auth.server
            self.settings.mailer.sender = self.db.config.auth.sender
            self.settings.mailer.login = self.db.config.auth.login


class Mailer(Mail):
    def __init__(self, config):
        Mail.__init__(self)
        self.settings.server = config.mail.server
        self.settings.sender = config.mail.sender
        self.settings.login = config.mail.login


class FormCreator(Crud):
    def __init__(self, db):
        Crud.__init__(db)
        self.settings.auth = None
        self.settings.formstyle = self.db.config.crud.formstyle



modules/basemodel.py

This module is an abstraction of the DAL, specifically
abstracts the method define_tables. It may seem strange or unnecessary. But I concluded that placing it in a more object-oriented means that the writing is more organized and better reuse of code.

 

from gluon.dal import DAL
from gluon.tools import Auth


class BaseModel(object):
    """Base Model Class
    all define_ methods will be called, then
    all set_ methods (hooks) will be called."""

    hooks = ['set_table',
             'set_validators',
             'set_visibility',
             'set_representation',
             'set_widgets',
             'set_labels',
             'set_comments',
             'set_computations',
             'set_updates',
             'set_fixtures']

    def __init__(self, db=None, migrate=None, format=None):
        self.db = db
        assert isinstance(self.db, DAL)
        self.config = db.config
        if migrate != None:
            self.migrate = migrate
        elif not hasattr(self, 'migrate'):
            self.migrate = self.config.db.migrate
        if format != None or not hasattr(self, 'format'):
            self.format = format
        self.set_properties()
        self.check_properties()
        self.define_table()
        self.define_validators()
        self.define_visibility()
        self.define_representation()
        self.define_widgets()
        self.define_labels()
        self.define_comments()
        self.define_computations()
        self.define_updates()
        self.pre_load()

    def check_properties(self):
        pass

    def define_table(self):
        fakeauth = Auth(DAL(None))
        self.fields.extend([fakeauth.signature])
        self.entity = self.db.define_table(self.tablename,
                                           *self.fields,
                                           **dict(migrate=self.migrate,
                                           format=self.format))

    def define_validators(self):
        validators = self.validators if hasattr(self, 'validators') else {}
        for field, value in validators.items():
            self.entity[field].requires = value

    def define_visibility(self):
        try:
            self.entity.is_active.writable = self.entity.is_active.readable = False
        except:
            pass
        visibility = self.visibility if hasattr(self, 'visibility') else {}
        for field, value in visibility.items():
            self.entity[field].writable, self.entity[field].readable = value

    def define_representation(self):
        representation = self.representation if hasattr(self, 'representation') else {}
        for field, value in representation.items():
            self.entity[field].represent = value

    def define_widgets(self):
        widgets = self.widgets if hasattr(self, 'widgets') else {}
        for field, value in widgets.items():
            self.entity[field].widget = value

    def define_labels(self):
        labels = self.labels if hasattr(self, 'labels') else {}
        for field, value in labels.items():
            self.entity[field].label = value

    def define_comments(self):
        comments = self.comments if hasattr(self, 'comments') else {}
        for field, value in comments.items():
            self.entity[field].comment = value

    def define_computations(self):
        computations = self.computations if hasattr(self, 'computations') else {}
        for field, value in computations.items():
            self.entity[field].compute = value

    def define_updates(self):
        updates = self.updates if hasattr(self, 'updates') else {}
        for field, value in updates.items():
            self.entity[field].update = value

    def pre_load(self):
        for method in self.hooks:
            if hasattr(self, method):
                self.__getattribute__(method)()


class BaseAuth(BaseModel):
    def __init__(self, auth, migrate=None):
        self.auth = auth
        assert isinstance(self.auth, Auth)
        self.db = auth.db
        from gluon import current
        self.request = current.request
        self.config = self.db.config
        self.migrate = migrate or self.config.db.migrate
        self.set_properties()
        self.define_extra_fields()
        self.auth.define_tables(migrate=self.migrate)
        self.entity = self.auth.settings.table_user
        self.define_validators()
        self.hide_all()
        self.define_visibility()
        self.define_register_visibility()
        self.define_profile_visibility()
        self.define_representation()
        self.define_widgets()
        self.define_labels()
        self.define_comments()
        self.define_computations()
        self.define_updates()
        self.pre_load()

    def define_extra_fields(self):
        self.auth.settings.extra_fields['auth_user'] = self.fields

    def hide_all(self):
        alwaysvisible = ['first_name', 'last_name', 'password', 'email']
        for field in self.entity.fields:
            if not field in alwaysvisible:
                self.entity[field].writable = self.entity[field].readable = False

    def define_register_visibility(self):
        if 'register' in self.request.args:
            register_visibility = self.register_visibility if hasattr(self, 'register_visibility') else {}
            for field, value in register_visibility.items():
                self.entity[field].writable, self.entity[field].readable = value

    def define_profile_visibility(self):
        if 'profile' in self.request.args:
            profile_visibility = self.profile_visibility if hasattr(self, 'profile_visibility') else {}
            for field, value in profile_visibility.items():
                self.entity[field].writable, self.entity[field].readable = value



modules/datamodel/<some entity>.py

Here is where you will create data models, define the fields, validation, fixtures etc ... the API here is different from the normal web2py mode, b
modules/handlers/base.pyut you can still use the same objects and methods.
 

from gluon.dal import Field
from basemodel import BaseModel
from gluon.validators import IS_NOT_EMPTY, IS_SLUG
from gluon import current
from plugin_ckeditor import CKEditor


class Post(BaseModel):
    tablename = "blog_post"

    def set_properties(self):
        ckeditor = CKEditor(self.db)
        T = current.T
        self.fields = [
            Field("author", "reference auth_user"),
            Field("title", "string", notnull=True),
            Field("description", "text"),
            Field("body_text", "text", notnull=True),
            Field("slug", "text", notnull=True),
        ]

        self.widgets = {
            "body_text": ckeditor.widget
        }

        self.visibility = {
            "author": (False, False)
        }

        self.representation = {
            "body_text": lambda row, value: XML(value)
        }

        self.validators = {
            "title": IS_NOT_EMPTY(),
            "body_text": IS_NOT_EMPTY()
        }

        self.computations = {
          "slug": lambda r: IS_SLUG()(r.title)[0],
        }

        self.labels = {
            "title": T("Your post title"),
            "description": T("Describe your post (markmin allowed)"),
            "body_text": T("The content")
        }

 


modules/handlers/base.py

This is a rendering engine using the web2py template, just a base class that initializes our handlers with everything you need, here you can inject common objects in to render context, also you can implement cache or extend in any way you want, here you can choose another template language if needed, it is very easy to use chetah, jinja or mako here instead of web2py template (if you really want or need). This structure allows you to easily have an app with multiple themes, and the views can be in any directory or even in the database (I am using it for email templates stored in database).

 

rom gluon import URL
from gluon.tools import prettydate


class Base(object):
    def __init__(
        self,
        hooks=[],
        meta=None,
        context=None
        ):
        from gluon.storage import Storage
        self.meta = meta or Storage()
        self.context = context or Storage()
        # you can user alers for response flash
        self.context.alerts = []

        self.context.prettydate = prettydate

        # hooks call
        self.start()
        self.build()
        self.pre_render()
        self.load_menus()

        # aditional hooks
        if not isinstance(hooks, list):
            hooks = [hooks]

        for hook in hooks:
            self.__getattribute__(hook)()

    def start(self):
        pass

    def build(self):
        pass

    def load_menus(self):
        self.response.menu = [
           (self.T('Home'), False, URL('default', 'index'), []),
           (self.T('New post'), False, URL('post', 'new'), []),
        ]

    def pre_render(self):
        from gluon import current
        self.response = current.response
        self.request = current.request
        self.session = current.session
        self.T = current.T

    def render(self, view=None):
        viewfile = "%s.%s" % (view, self.request.extension)
        return self.response.render(viewfile, self.context)


modules/handlers/<some entity>.py

Here is where the logic of action should occur, consult the database, calculate, assemble objects such as forms, tables, etc. .. and here also will check the permissions and authentication, you have to create one handler for each entity of your app, example: contacts, person, article, product...

 

from handlers.base import Base
from myapp import MyApp
from datamodel.post import Post as PostModel
from gluon import SQLFORM, URL, redirect


class Post(Base):
    def start(self):
        self.app = MyApp()
        self.auth = self.app.auth  # you need to access this to define users
        self.db = self.app.db([PostModel])

        # this is needed to inject auth in template render
        # only needed to use auth.navbar()
        self.context.auth = self.auth

    def list_all(self):
        self.context.posts = self.db(self.db.Post).select(orderby=~self.db.Post.created_on)

    def create_new(self):
        # permission is checked here
        if self.auth.has_membership("author", self.auth.user_id):
            self.db.Post.author.default = self.auth.user_id
            self.context.form = SQLFORM(self.db.Post, formstyle='divs').process(onsuccess=lambda form: redirect(URL('show', args=form.vars.id)))
        else:
            self.context.form = "You can't post, only logged in users, members of 'author' group can post"

    def edit_post(self, post_id):
        post = self.db.Post[post_id]
        # permission is checked here
        if not post or post.author != self.auth.user_id:
            redirect(URL("post", "index"))
        self.context.form = SQLFORM(self.db.Post, post.id, formstyle='divs').process(onsuccess=lambda form: redirect(URL('show', args=form.vars.id)))

    def show(self, post_id):
        self.context.post = self.db.Post[post_id]
        if not self.context.post:
            redirect(URL("post", "index"))

 


controllers/<some entity>.py

Here is the entry point to call the handler, will only create an instance of a handler and pass arguments to it, then return will be handler.render() ready and able to cache.

 

from handlers.post import Post


def index():
    post = Post('list_all')
    return post.render("mytheme/listposts")


def new():
    post = Post('create_new')
    return post.render("mytheme/newpost")


def edit():
    post = Post()
    post.edit_post(request.args(0))
    return post.render("mytheme/editpost")


def show():
    post = Post()
    post.show(request.args(0))
    return post.render("mytheme/showpost")

 

So the views (template files) will be at views/yourtheme/somefilename

 

example of the view for show() action

{{extend 'layout.html'}}
<div class="row">
    <div class="three columns alpha">
        <img src="{{=URL('default', 'download', args=post.author.thumbnail)}}" width=100>
        

        <strong>{{="%(first_name)s %(last_name)s (%(nickname)s)" % post.author}}</strong>
        

        <small>{{=prettydate(post.created_on)}}</small>
    </div> 
    <div class="eleven columns" style="border-left:1px solid #444;padding:15px;">
    <h2><a href="{{=URL('show', args=[post.id, post.slug])}}">{{=post.title}}</a></h2>
    {{=XML(post.body_text)}}
    </div>
    <div class="one columns omega">
      {{if auth.user_id == post.author:}}
          <a href="{{=URL('edit', args=[post.id, post.slug])}}" class="button">Edit</a>
      {{pass}}
    </div>
</div>
<hr/>


I am testing and I found it is very performatic, but, you can help testing it more.

The code is here: https://github.com/rochacbruno/web2py_model_less_app
Download the app here: https://github.com/rochacbruno/web2py_model_less_app/downloads

This sample is a blog system with just one entity 'blog_post' and also the auth and users, but you can use as a template to create more entities.

Can you help testing the gain of performance of this approach?

Related slices

Comments (5)

  • Login to post



  • 0
    kevin-krac-10916 5 years ago

    How do you unit test the controller/action?

    def index():
        post = Post('list_all')
        return post.render("mytheme/listposts")

     

    My first thought would be to mock the post handler Post('list_all') but not sure how this is done since the object is instantiated inside the action and is not injected via a method or constructor. So when I unit test this it fails when trying to access the session object downstream, when the handler instantiates the MyApp class.

    I come from the .NET world where it is common to mock the dependencies of the class under test, and that's it.

    But I'm not sure how to do it in this case? Could you help out?

     

    Thanks!


  • 0
    kevin-krac-10916 5 years ago

    Great post!

    Whenever I want to unit test the post controller it fails in the init() method of the MyApp class because the session is not instantiated:
    AttributeError: 'thread._local' object has no attribute 'session'

    So what is the correct way of testing the controller/action? How should I mock the post handler object?

    I come from the .NET world in which with IoC/DI, you can mock the dependency of the controller (the handler, in this case) and pass it to the controller/action as a parameter.
    But in this case in which the handler is instantiated inside the controller/action, how can that be done? Or should I mock the session, request and response objects themselves?

    Thanks!


  • 0
    samuel-bonilla-11088 6 years ago

    the database does not work, example:

    I change self.config.db.uri = "sqlite :/ / nuevabase.sqlite" but it still connects to "sqlite :/ / myapp.sqlite".

    the solution:

    class DataBase(DAL, MyApp):
        """
        Subclass of DAL
        auto configured based in config Storage object
        auto instantiate datamodels
        """
        def __init__(self, config, datamodels=None):
            self._LAZY_TABLES = dict()
            self._tables = dict()
            #self.config = config
            db = "sqlite://nuevabase.sqlite"
            DAL.__init__(self,
                         db)


  • 0
    josedesoto 6 years ago

    Thanks Bruno for the post!!!

    I am developing one application using this way, but I am not sure how to organize the order when the app creates the tables. They have reference between them. In SQLite does not matter the order how to create the tables, but in Mysql yes, having the error: Can't create table 'xxx' (errno: 150). Any recomendation how to do it?

    replies (1)
    • josedesoto 6 years ago

      I have resolved this issue creating a controller fuction (install) where it creates an instance of a handler in the order the app has to create the tables. Not sure if is the best solution...


  • 0
    samuel-bonilla-11088 6 years ago

    thanks bruno is grate..........


Hosting graciously provided by:
Python Anywhere