Django GSOC 2012

--------
GSOC 2012 Customizable serialization
-------

Django has a framework for serialization but it is simple tool. The main problem is impossibility to define own serialization structure and no support for related models. Below I present a proposal to improve current framework.
In my opinion it is not possible to create a framework completely independent of formats that will be user to serialize objects. For instance XML has richer syntax than json (e.g. fields can be tags or attributes) so we must provide functions to handle it which wont be useful in JSON serialization.

-------
Features to implement:
-------

Based on presented issues to consider, GSOC proposal from last years and django-developers group threads I prepare a list of features that good solution should have.

1. Defining the structure of serialized object
    1.1. Object fields can be at any position in output tree.
    1.2. Renaming fields
    1.3. Serializing non-database attributes/properties
    1.4. Serializing any subset of object fields.
2. Defining own fields
    2.1. Related model fields
        2.1.1. Serializing foreign keys, m2m and reverse relations
        2.1.2. Choose depth of serialization
        2.1.3. Handling natural keys
        2.1.4. Handling objects serialized before (in other location of output tree)
        2.1.5. Object of same type can be differently handled depends on location
    2.2. Other fields - custom serialization (e.g. only date in datetime fields)
3. One definition can support multiple serialization formats (XML, JSON, YAML)
4. Backward compatible
5. Solution should be simple. Easy to write own serialization scheme.

Below I have tags like (F2.1.2) - means support for feature 2.1.2.

------
Concept:
------

Output structure will be defined declarative using classes. For sure there is needed class for model definition. In my solution I define also model fields with classes. It's the simplest way to provide free output structure.

Suppose we want to serialize this model:

class Comment(Model):
    user = ForeignKey(Profile)
    photo = ForeignKey(Photo)
    topic = CharField()
    content = CharField()
    created_at = DateTimeField()
    ip_address = IPAddressField()

class User(Model):
    fname = CharField()
    lname = CharField()

class Photo(Model):
    sender = ForeignKey(User)
    image = ImageField()


Below we have definition of serializer classes CommentSerializer.

If we want to serialize  comment queryset:
serializers.serialize('json/xml/yaml', queryset, serializer=CommentSerializer, **options)
If 'serializer' isn't provided we have defaults serializer for each format (F3)

class CommentSerializer(BaseModelSerializer):

    @attribute
    def content(self, obj):
        return obj.content.truncate(20)

    def photo(self, obj):
        return CommentSerializer.serialize(obj, PhotoSerializer) #(F2.1.5)

    def topic(self, obj):
        return ModelSerializer.serialize(obj, TopicFieldSerializer)

    def x(self, obj): #(F1.3)
        return 5

    def y(self, obj):
        return 10

    def unicode__datetime(self, obj): #(F2.2)
        return smart_unicode(obj.date())

    class Meta:
        aliases = {'topic' : 'subject'}
        #fields = (,)
        exclude = ('ip_address',)
        fk_level = 1
        fk_natural_keys = False|True
        fk_reserialize = False|True
        default_field_serializer = FieldSerializer
        default_related_serializer = NestedRelationSerializer # subclass of BaseModelSerializer or BaseFieldSerializer
        model_name = "my_obj"
        structure = "__fields pk__field special[topic__field x__field] special2{created_at y__field}"

Class has only methods and definition of Meta class. Default each field is serialized by Meta.default_field_serializer or Meta.default_related_serializer. Class methods redefining this behavior. unicode__xxx method redefining serialization for type xxx

Methods can return:
1. Base type like str, int, float etc.
2. Object subclassing BaseFieldSerializer or BaseModelSerializer
3. List of 1,2

Meta Class
    a) aliases - redefine field name: topic : "..." => subject : "...". Can do 'topic' : '' - return of topic method is one level higher. There is metatag __fields - rename all fields.  #(F1.2)
    b) fields - fields to serialize #(F1.4)
    c) exclude - fields to not serialize #(F1.4)
    e) fk_level - depth of related models serialization. If 0 then use flat serialization else nested. Each nested model has level-=1 #(F2.1.2)
    f) fk_natural_keys - use natural_keys instead of pk #(F2.1.3)
    g) fk_reselialize - serialize previous serialized model if nested? If True then yes else fallback to flat #(F2.1.4)
    h) default_field_serializer - default field serializer
    h) default_related_serializer - default realted serializer #(F2.1.1)
    i) model_name - if it isn't empty returns <model_name_value>serialized object</model_name_value>  else return serialized object. Useful with nested serialization.
    j) structure - I don't like it :/ Declaration of structure. It's some shortcut for declaring fields (in methods way). #(F1.1)
        __fields - all fields and declared methods (parametrized by Meta.fields and Meta.exclude)
        xxx__field where xxx is model field or defined method. xxx_field won't be in  __fields
        xxx[x,y,z] - For xml: <xxx><x_name>x_value</x_name></xxx><xxx>... For json: xxx : [x,y,z]
        xxx{x,y,z} - For xml: <xxx><x_name>x_value</x_name><y_name>... For json xxx : { x_name : x_value ... }


class TopicFieldSerializer(BaseFieldSerializer):

    def lower_topic(self, name, obj):
        return obj.topic.lower()

    def __name__(self, name, obj):
        return "value"

    def __value__(self, name, obj):
        return getattr(obj, name)

    class Meta:
        structure="__fields additional[]"

Field serializer has two special methods __name__ and __value__. __value__ is the primary value returned by field.
E.g.
    In some model class (topic="Django")
        def topic(self, obj):
            return Model.serialize(obj, FieldSerializer)
    xml: <topic>Django</topic>
    json "topic" : "Django"

But what if we want to add come custom attribute (like lower_topic above).
    xml: <topic><lower_topic>django</lower_topic>Django</topic> - far i know it's correct but it's what we want?
    json topic : {lower_topic : django, ??? : Django}
We have __name__ to provide some name for field:
    def __name__(self, obj):
        return "value"
    xml: <topic><lower_topic>django</lower_topic><value>Django</value></topic>
    json topic : {lower_topic : django, value : Django}


structure in Meta of FieldSerializer is like in ModelSerializer

class PhotoSerializer(BaseModelSerializer):
    @attribute
    def image(self, obj):
        return obj.image.url

    class Meta:
        model_name = ""
        structure = "__fields"


What we got:

XML
<my_obj content="Some truncated conte">
    <user>
        <fname>Piotr</fname>
        <lname>Grabowski</fname>
    </user>
    <photo image="/images/1.jpg">
        <sender>1</sender>
    </photo>
    <pk>10</pk>
    <special>
        <subject>
            <lower_topic>extra topic</lover_topic>
            <value>EXTRA topic</value>
            <additional></additional>
        </subject>
    </special>
    <special>
        <x>5</x>
    </special>
    <special2>
        <created_at>2011-03-20</created_at>
        <y>10</y>
    </special2>
</my_obj>

JSON
{
    "user" : {
        "fname" : "Piotr",
        "lname" : "Grabowski"
    },
    "photo" : {
        "image" : "/images/1.jpg",
        "sender" : 1
    },
    "pk" : 10,
    "special" : [
        { subject : { "lower_topic" : ..., ... },
        {"x" : 5}
    ],
    "special2" {
        "created_at" : "...",
        "y" : 10
    }
}

-----
Prove of concept
-----

class JSONSerializer(BaseModelSerializer):

    def pk(self, obj):
        return smart_unicode(obj._get_pk_val(), strings_only=True)

    def model(self, obj):
        return smart_unicode(obj._meta)

    class Meta:
        fk_level=0
        fk_natural_keys = False
        default_field_serializer = FieldSerializer
        structure = "pk__field model__field fields{__fields}"


class XMLSerializer(BaseModelSerializer):
    @attribute
    def pk(self, obj):
        return smart_unicode(obj._get_pk_val(), strings_only=True)

    @attribute
    def model(self, obj):
        return smart_unicode(obj._meta)

    class Meta:
        aliases = {'__fields' : 'field'}
        fk_level=0
        fk_natural_keys = False
        default_field_serializer = XMLFieldSerializer
        default_relation_serializer = XMLFlatRelationSerializer
        model_name="object"
        structure = "pk__field model__field __fields"


XMLFieldSerializer(BaseFieldSerializer):
    @attribute
    def name(self, name, obj):
        ...

    @attribute
    def type(self, name, obj):
        ...
XMLFlatRelationSerializer(BaseFieldSerializer):
    @attribute
    def to
        ...

    @attribute
    def name
        ...

    @attribute
    def rel
        ...
-----
Supported features:
-----

1. Defining the structure of serialized object
    1.1. Object fields can be at any position in output tree. YES
    1.2. Renaming fields. YES
    1.3. Serializing non-database attributes/properties. YES
    1.4. Serializing any subset of object fields. YES
2. Defining own fields
    2.1. Related model fields
        2.1.1. Serializing foreign keys, m2m and reverse relations PARTIAL - FK OUT OF BOX, M2M AND REVERSE RELATION CAN BE MANUAL CODED
        2.1.2. Choose depth of serialization YES
        2.1.3. Handling natural keys YES
        2.1.4. Handling objects serialized before (in other location of output tree) YES
        2.1.5. Object of same type can be differently handled depends on location YES
    2.2. Other fields - custom serialization (e.g. only date in datetime fields) YES
3. One definition can support multiple serialization formats (XML, JSON, YAML) DEPENDS OF DEFINITION - YES IN MOST CASES
4. Solution should be simple. Easy to write own serialization scheme. ???

-----
What I should consider now:
-----
    a) 2.1.1 - Handling of m2m fields - handling intermediate models
    b) What object should be passed to FieldSerialization
    c) Serialization of heterogeneous lists - it's possible now but needs to manual coding class.
    d) 4. It is really simple?
    e) 3. Can one definition support any format?
    f) Think about YAML - something betwen JSON and XML. Can solution be incompatible with it?


-----
Shedule
-----
TODO

-----
About
-----
My name is Piotr Grabowski. I'm last year student at the Institute of Computer Science University of Wrocław (Poland). I've been working with Django for 2 years. Python is my preffered programing language but I have been using also Ruby(&Rails) and JavaScript.