Exploration of Metaprogramming in Python

Many people are unfamiliar with the concept of "metaprogramming", and there isn't a highly precise definition for it. This article centers on metaprogramming within Python. However, in reality, what is discussed here may not fully adhere to the strict definition of "metaprogramming". It's just that I couldn't find a more apt term to represent the theme of this article, so I borrowed this one.

The subtitle is "Control Everything You Want to Control". Essentially, this article focuses on one thing: leveraging the features provided by Python to make the code as elegant and concise as possible. Specifically, through programming techniques, we modify the characteristics of an abstraction at a higher level of abstraction.

First and foremost, it's a well - known cliché that everything in Python is an object. Additionally, Python offers numerous "metaprogramming" mechanisms, such as special methods and metaclasses. Operations like dynamically adding attributes and methods to an object are not regarded as "metaprogramming" in Python at all. But in some static languages, achieving this requires certain skills. Let's discuss some aspects that can easily perplex Python programmers.

Let's start by classifying objects into different levels. Commonly, we know that an object has its type, and Python has long implemented types as objects. Thus, we have instance objects and class objects, which are two levels. Readers with a basic understanding will be aware of the existence of metaclasses. In brief, a metaclass is the "class" of a "class", meaning it is at a higher level than a class. This adds another level. Are there more?

ImportTime vs RunTime

If we view it from a different perspective and don't need to apply the same criteria as the previous three levels, we can distinguish between two concepts: ImportTime and RunTime. The boundaries between them are not distinct. As the names suggest, they refer to two moments: the time of import and the time of execution.

What occurs when a module is imported? Statements in the global scope (non - definitional statements) are executed. What about function definitions? A function object is created, but the code within it is not executed. For class definitions, a class object is created, the code in the class definition scope is executed, and the code in the class methods is not executed naturally.

What about during execution? The code in functions and methods will be executed. Of course, you need to call them first.

Metaclasses

So, we can say that metaclasses and classes belong to ImportTime. After a module is imported, they are created. Instance objects belong to RunTime. Simply importing a module won't create instance objects. However, we can't be too dogmatic because if you instantiate a class within the module scope, instance objects will also be created. It's just that we usually write the instantiation inside functions, hence this classification.

If you want to control the characteristics of the created instance objects, what should you do? It's quite simple. Override the __init__ method in the class definition. Then, what if we want to control some properties of the class? Is there such a need? Definitely!

Regarding the classic singleton pattern, everyone knows there are multiple ways to implement it. The requirement is that a class can only have one instance.

The simplest implementation is as follows:

class _Spam:
    def __init__(self):
        print("Spam!!！")

_spam_singleton = None

def Spam():
    global _spam_singleton
    if _spam_singleton is not None:
        return _spam_singleton
    else:
        _spam_singleton = _Spam()
        return _spam_singleton

This factory - like pattern isn't very elegant. Let's review the requirement once more. We desire a class to have only one instance. The methods we define in a class are the behaviors of instance objects. So, if we want to change the behavior of a class, we need something at a higher level. This is where metaclasses come into play. As mentioned earlier, a metaclass is the class of a class. That is to say, the __init__ method of a metaclass is the initialization method of a class. We know there is also the __call__ method, which enables an instance to be called like a function. Then, this method of a metaclass is the one called when a class is instantiated.

The code can be written like this:

class Singleton(type):
    def __init__(self, *args, **kwargs):
        self._instance = None
        super().__init__(*args, **kwargs)

    def __call__(self, *args, **kwargs):
        if self._instance is None:
            self._instance = super().__call__(*args, **kwargs)
            return self._instance
        else:
            return self._instance


class Spam(metaclass = Singleton):
    def __init__(self):
        print("Spam!!!")

There are two main differences compared to a general class definition. One is that the base class of Singleton is type, and the other is that there is a metaclass = Singleton in the definition of Spam. What is type? It is a subclass of object, and object is its instance. That is to say, type is the class of all classes, the most fundamental metaclass. It stipulates some operations that all classes need when they are created. So, our custom metaclass needs to subclass type. At the same time, type is also an object, so it is a subclass of object. It's a bit hard to grasp, but just get a general idea.

Decorators

Let's talk about decorators. Most people consider decorators one of the most challenging concepts to understand in Python. In fact, it's just syntactic sugar. Once you understand that functions are also objects, you can easily write your own decorators.

from functools import wraps


def print_result(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        result = func(*args, **kwargs)
        print(result)
        return result

    return wrapper


@print_result
def add(x, y):
    return x + y


# Equivalent to:
# add = print_result(add)

add(1, 3)

Here, we also use a decorator @wraps, which is used to make the returned inner function wrapper have the same function signature as the original function. Basically, we should add it when writing decorators.

As I wrote in the comments, the form of @decorator is equivalent to func = decorator(func). Understanding this point allows us to write more types of decorators. For example, class decorators, and writing a decorator as a class.

def attr_upper(cls):
    for attrname, value in cls.__dict__.items():
        if isinstance(value, str):
            if not value.startswith('__'):
                setattr(cls, attrname, bytes.decode(str.encode(value).upper()))
    return cls


@attr_upper
class Person:
    sex ='man'


print(Person.sex)  # MAN

Pay attention to the differences between the implementation of ordinary decorators and class decorators.

Data Abstraction - Descriptors

If we want some classes to possess certain common characteristics or be able to exercise control over them within the class definition, we can customize a metaclass and make it the metaclass of these classes. If we want some functions to have certain common functions and avoid code duplication, we can define a decorator. Then, what if we want the attributes of instances to have some common characteristics? Some may say we can use property, and indeed we can. But this logic must be written in each class definition. If we want some attributes of the instances of these classes to have the same characteristics, we can customize a descriptor class.

Regarding descriptors, this article https://docs.python.org/3/howto/descriptor.html explains it very well. At the same time, it also elaborates on how descriptors are hidden behind functions to achieve the unification and differences between functions and methods. Here are some examples.

class TypedField:
    def __init__(self, _type):
        self._type = _type

    def __get__(self, instance, cls):
        if instance is None:
            return self
        else:
            return getattr(instance, self.name)

    def __set_name__(self, cls, name):
        self.name = name

    def __set__(self, instance, value):
        if not isinstance(value, self._type):
            raise TypeError('Expected' + str(self._type))
        instance.__dict__[self.name] = value


class Person:
    age = TypedField(int)
    name = TypedField(str)

    def __init__(self, age, name):
        self.age = age
        self.name = name


jack = Person(15, 'Jack')
jack.age = '15'  # Will raise an error

There are several roles here. TypedField is a descriptor class, and the attributes of Person are instances of the descriptor class. It seems that the descriptor exists as an attribute of Person, that is, a class attribute rather than an instance attribute. But in fact, once an instance of Person accesses an attribute with the same name, the descriptor will take effect. It should be noted that in Python 3.5 and earlier versions, there is no __set_name__ special method. This means that if you want to know what name the descriptor is given in the class definition, you need to explicitly pass it to the descriptor when instantiating it, that is, you need one more parameter. However, in Python 3.6, this problem is solved. You just need to override the __set_name__ method in the descriptor class definition. Also, note the writing of __get__. Basically, the judgment of instance is necessary, otherwise it will raise an error. The reason isn't difficult to understand, so I won't go into details.

Controlling Subclass Creation - An Alternative to Metaclasses

In Python 3.6, we can customize the creation of subclasses by implementing the __init_subclass__ special method. In this way, we can avoid using the somewhat cumbersome metaclasses in some cases.

class PluginBase:
    subclasses = []

    def __init_subclass__(cls, **kwargs):
        super().__init_subclass__(**kwargs)
        cls.subclasses.append(cls)


class Plugin1(PluginBase):
    pass


class Plugin2(PluginBase):
    pass

Summary

Metaprogramming techniques such as metaclasses are somewhat obscure and difficult to comprehend for most people, and most of the time, we don't need to use them. However, the implementation of most frameworks utilizes these techniques so that the code written by users can be concise and easy to understand. If you want to gain a deeper understanding of these techniques, you can refer to some books such as Fluent Python and Python Cookbook (some content of this article is referenced from them), or read some chapters in the official documentation, such as the descriptor How - To mentioned above, and the Data Model section, etc. Or directly examine the Python source code, including the source code written in Python and the CPython source code.

Remember, only use these techniques after fully understanding them, and don't attempt to use them everywhere.