Meta-Programming
The most underrated feature of Python
Introduction
It’s no news to anyone that I like python. And since Python was the 6th most loved language in the 2021 StackOverflow developer survey, I doubt that I’m the only one. When you ask Python programmers about why they love the language, they talk about the ease of use, the intuitive nature of the language, duck typing, the amazing ecosystem of open source packages, and so on.
And while those are great points, they aren’t really specific to python. Ruby for example fits most of those bills and even statically typed languages like C# have come a long way in embracing the inherent simplicity of something like python(the most recent example being Top-Level statements in C# 10)
This brings me to what I think sets python apart from every other language - the ease of meta-programming. That might be a bit confusing so let’s set the stage first.
Objects
Every Object-Oriented language has a concept of an object - a way to encapsulate data and methods. Once you have an object, most of the time it’s trivial to set an attribute or change some property value.
In dynamic languages like python, the attributes aren’t even restricted by the class definition(as is the case in languages like Java and C++). For example, the following code is perfectly valid
class Foo:
pass
foo = Foo()
foo.greet = 'hello there'
print(foo.greet) # hello there
You can even add functions
foo.greet = lambda: 'hello there'
print(foo.greet()) # hello there
So what does this mean? It means that once you have an object, you can change its attributes as you wish. There are (for the most part) no restrictions.
How is this possible? This is because the object has a dictionary to back it. Think about it - a dictionary and class aren’t that different. Yes, a key in a dictionary can be any hashable data structure while the attributes of a class must be strings, but other than that, there are no major differences.
So how can we see this dictionary? By accessing the __dict__ attribute of the object.
class Person:
def __init__(self, name):
self.name = name
def greet(self):
print(f"Hello {self.name}")
person = Person('Shashwat')
person.greet() # Hello Shashwat
print(person.__dict__) # {'name': 'Shashwat'}
So that seems expected. But where is the function greet stored? Or for that matter, where’s __init__? Didn’t we just establish that all data and functions are stored in __dict__?
Well, yes. Functions and data are both stored in __dict__. But to understand why we can’t find the functions above, we need to understand how member functions(or methods) work in python.
Take a look at the following code. Say instead of doing person.greet(), we did greet(person). Does it make any difference?
def greet(self):
print(f"Hello {self.name}")
person = Person('Shashwat')
greet(person) # Hello Shashwat
In some languages, it might. But in python, (again, for the most part) it doesn’t. And in fact, this is why the function doesn’t belong in the object’s dictionary in the first place - the function remains the same for each instance of the class. It belongs to the class - not the object. The object is simply passed to this function as an argument - like any other argument - there’s nothing special about self
This is why you can’t see the function in person.__dict__
So now the question arises, how does a class store these functions? And since we know you can have static variables as well, does a class also have a __dict__?
Classes
Of course, it does! Let’s try the same example as before -
class Person:
def __init__(self, name):
self.name = name
def greet(self):
print(f"Hello {self.name}")
print(Person.__dict__)
# {
# ...,
# '__init__': <function Person.__init__ ...>,
# 'greet': <function Person.greet at ...>,
# ...
# }
The actual output contains a lot more stuff so I’ve only included the relevant bits.
So now we’ve established that classes and objects both have a __dict__ which is used to store functions and data. And that you can mutate any object dynamically.
But since a class is also a __dict__, can you mutate it as well? Let’s try!
class Person:
def __init__(self, name):
self.name = name
def greet(self):
print(f"Hello {self.name}")
person = Person('Shashwat')
Person.greet = lambda self: 'Hello Mike'
person.greet() # Hello Mike
This is weird - it looks like you can mutate both classes and objects! If you think this is magic, you would be even more amazed when I tell you that the reason classes can be mutated is because classes are objects too.
You might have heard of the “Everything is an object” approach of python. Classes, dictionaries, functions, integers, strings - what have you - everything is an object.
This is why calling functions on integers, which is something unheard of in other languages, is commonplace in python.
print(bin(10)) # 0b1010
print((10).bit_count()) # 2
However, before you go on telling your friends this, keep this in mind - Python’s definition of an object is probably different from what you learned in a C++ or Java course. In those languages, objects are instances of a class on which you can set attributes and call member functions. This is a very narrow interpretation of what constitutes an object.
Python takes a more liberal approach - everything that you can store in a variable is an object. Most of the time, you will also be able to set attributes and call methods on these “objects”. But there are exceptions - say for example the print function.
The following for example works
my_print = print
my_print('Hello World') # Hello World
But trying to set an attribute on print fails. So does trying to access an attribute
# AttributeError:
# 'builtin_function_or_method' object has no attribute 'x'
print.foo = 10
Of course, its very unlikely that you’ll ever need to set a property on a builtin - which is the only place this restriction applies. That is apart from when you restrict a class yourself - one such way would be to use __slots__.
class Foo:
__slots__ = ('y')
pass
f = Foo()
f.y = 10
print(f.y) # 10
# AttributeError: 'Foo' object has no attribute 'x'
f.x = 10
Meta-Programming
So now that we know how objects, classes, and functions work in Python, we can finally appreciate meta programming.
To put it simply, meta programming can be defined as code generating or altering other code. Say for example you want to generate getters and setters for all the attributes of a class - programmatically. Doing so might sound difficult and indeed it is in most programming languages(although a lot of languages offer Records or Data classes - which do the same). But is relatively straightforward in Python.
But why is that? At the expense of sounding like a broken record, remember that everything in python is an object. This means that mutating a class, object, function or even a module is as simple as setting an attribute on your object.
Want to change the name of a member function? Just reassign the attribute on the class.
class Person:
def hi(self):
print('hello world')
p = Person()
Person.new_func = Person.hi
del Person.hi
p.new_func()
Most languages aren’t as flexible as this - which is what makes dealing with Reflection such a pain in statically typed languages.
However, we must start small. Say you want to measure the time it takes to run a function. You could do something like
# start timer
func()
# end timer
But this clutters the code and if you’re timing lots of functions, it gets cumbersome. What you need to do is to alter the behavior of this function without actually altering its code yourself.
Enter decorators. These aren’t the focus of this article so I won’t explain them in detail. But in a nutshell, a decorator is a function that takes a function as an argument and returns a new, desired function.
For the timing example, you might make the following decorator
def time_it(func):
def my_func(*args, **kwargs):
start = time.perf_counter()
val = func(*args, **kwargs)
end = time.perf_counter()
print(f"Took {end - start} ns")
return val
return my_func
and then, simply “decorate” the function you wish to time
from math import factorial
@time_it
def fact(x):
return factorial(x)
fact(10_000) # Took 0.002677972999663325 ns
In less than 10 lines of code, we have been able to time a function without duplicating our code. And of course, this isn’t limited to functions. Since classes are objects too, you can easily decorate them as well. In this case, instead of returning a function, you return a class from your decorator.
But that’s not where any of this stops. You see, since classes are objects too, they must be an instance of a “class” too? In python, you can use type(x) to find the type(which in Python 3 is the same as class) of x
For example,
print(type('x')) # <class 'str'>
print(type(10)) # <class 'int'>
class Foo:
...
foo = Foo()
print(type(foo)) # <class '__main__.Foo'>
But what is the type of int, str, and Foo? Let’s try it out!
print(type(int)) # <class 'type'>
print(type(str)) # <class 'type'>
print(type(Foo)) # <class 'type'>
So. All 3 of them are instances of the type(or class) type
And for good measure, let’s see what’s the type of type
print(type(type)) # <class 'type'>
So the type of type is type itself. How does that make any sense? You see, type isn’t a class. It’s a metaclass. And while I don’t have time to discuss metaclasses in detail today, they are a fascinating topic and make you think outside of the normal OOP bubble.
If you must know, however, think of metaclasses as things that create “classes”. We’ve already established that you can create and modify classes on the fly(just like any other object in python) - Metaclasses are used to generate those classes.
So in the end we have two main ways of altering python code at runtime - decorators and metaclasses. Both of them greatly simplify the code you have to write by generating a lot of repetitive and boring code so that you can focus on the business logic of your application. And that is what makes meta programming so attractive - the less code you write, the lesser chances of you messing something up.
Bigger Picture
A lot of the stuff I’ve mentioned today sounds cool but also complicated. Fortunately, you don’t have to worry about most of it. Just because python has rich support for meta programming doesn’t mean that you must make use of it every day. But it does mean that libraries that make use of these language features provide a far better developer experience than if they did things the traditional way.
Take data classes for example.
from dataclasses import dataclass
@dataclass(order=True, frozen=True)
class Person:
age: int
name: str
This class looks simple but if you were to implement everything by hand, it would likely take you 100 lines of code. Think about it - the object is frozen, that is, it’s immutable. Since python objects are mutable by default, you’d have to implement a property for each of the attributes to make it immutable. Then the fact that the objects have relative ordering - add another 6 functions for <, <=, >, >=, == and !=. Not to mention that you’d have to have an __init__, __repr__, __str__ and a __hash__ function
And this is with just 2 attributes. Think what happens when you have 10 attributes. And what happens if you need to ignore some attributes when comparing two objects? At what point does the code become impossible to reason about?
With data classes, it doesn’t matter how complicated the logic is. Your class will be generated on the fly by python code.
Other languages do have support for similar things - there are records in Java and C# but they are quite limited in their configuration. Python exposes you to all the tools you need to create your solutions. Most of the time, the ones provided in the standard library will be enough. But when they aren’t, you can just code something yourself.
Conclusion
And that’s what I think is the best thing about Python. As Guido van Rossum said, “We are all consenting adults here”. Python as a language doesn’t try to hide its internal workings.
Not only does it have the tools required to aid such rich language features, but it’s also kind enough to let us play with them - and create even better, fancier tools around them.