Python 3.11 has introduced many new exciting features and improvements that are described here. The CPython implementation is significantly faster than in the 3.10 version. It introduces new syntax features, built-ins, and interpreter enhancements that are themed around making Python more robust and reliable. Let's take a dive and look at some of the new things!
Table of Contents
Probably the most prominent feature that stands out is the addition of a new concept of an exception group. Exception groups were proposed in PEP 654. Basically, they allow for raising multiple exceptions at once.
This can be helpful in the case of running complex code where numerous exceptions can occur. Python 3.11 provides two new builtins: BaseExceptionGroup(BaseException) and ExceptionGroup(BaseExceptionGroup, Exception). Let's examine the basic syntax for the ExceptionGroup:
It's also possible to "filter" exceptions by means of the subgroup method. subgroup returns exception group with the same metadata, however, it contains only exceptions that satisfy provided condition:
Similarly, it is possible to obtain a complement of the filtered set by use of the split method:
Obviously, Python 3.11 allows also for handling these new exception groups. They can be handled just like the "old" Exception type - in the try ... except statement.
If we imagine handling a complex group of exceptions where we're only concerned with one kind of error, we may use, for example, the subgroup method. Here's an idea of how it might be implemented, again, based on PEP 654.
LogicError was handled.
That way, we raised regular exception handler, the logging_handler, that might've existed in our code before 3.11, to handle the exception group.
You may also like: AWS X-Ray - A Distributed Tracing System for Debugging Applications
The except* clause
This new clause was introduced in Python 3.11 to make working with exception groups easier and is again based on the PEP 654. It is a generalization of the standard except clause.
In short, it extracts all exceptions being subtypes of a given type as an exception group and leaves the remainder for further propagation. Here's an example of code using this new syntax:
Regular Exception caught by the except* (so called naked exception) is wrapped as an ExceptionGroup for consistency:
In case of the exception raised not being caught by the except*, then, when caught by the regular except clause, it appears as an Exception again:
Might be interesting: How to manage AWS ECS environment variables with Chamber?
In 3.11 significant amount of typing features was introduced.
As proposed by PEP-646, TypeVarTuple was introduced to the typing module. Similarly to how the TypeVar can be parametrized with a single type, the TypeVarTuple can be parametrized with a variadic number of types thus enabling variadic generics.
Use cases for that were originally identified in the numerical computing libraries like NumPy or TensorFlow, where array types could be parametrized with array shape. More about potential use cases can be read here. We may imagine multidimensional array generic to be defined as follows:
Required and NotRequired in TypedDict
Proposed in PEP-655. Enables to declare required and optional elements on a dict in a simple and readable way:
The Self type was implemented as proposed in the PEP-673. It offers great help in designing any kind of fluent interface being much more readable than the TypeVar approach used prior to 3.11:
As proposed in PEP-675. The main motivation for this feature was, again, to make Python more secure. For a secure API, it's best to accept only literal strings or string constructed from literal values.
As described in the original PEP, code that executes raw SQL queries can greatly benefit from this:
Data class transforms
Proposed in PEP-681. While many libraries with dataclass-like behavior exist, it wasn't really possible to be annotated using standard typing. By dataclass-like we understand decorators that perform synthesis of dunder methods, define frozen behavior and the field specifiers.
Therefore, a new annotation, dataclass_transform, was introduced to be used with classes, metaclasses, and functions that are decorators and which represent said behavior.
This decorator being used indicates to a static type checker that this function, class, or metaclass alters the target class to act in a dataclass-like manner.
dataclass_transform decorator has a runtime effect of setting __dataclass_transform__ dict on a decorated item. Below is a simplistic example of a dataclass_transform usage with a function decorator.
Decorated as_model function itself creates the init method that accepts all class annotations as kwargs and sets as respective attributes at init. We may also observe the __dataclass_transform__ being set on the as_model decorator:
Here's what would be the class equivalent of the above:
And, for sake of completeness, the metaclass variant:
New tomllib module
As of now, it's used for packaging and configuration of multiple popular tools. The inclusion of a dedicated TOML parsing module certainly is a step towards making the python ecosystem more consistent as it will encourage more projects to support configuration via TOML.
The new module allows for reading a file-like binary object via the toml.load function. Results are written to a dict that consists of basic python types:
Similarly to other parsing libraries,like json for instance,tomllib comes equipped with a loadsfunction to read data from string:
Unfortunately, this is where the similarities end - tomllib doesn't have a way to export data. There is no dump or dumps functionality provided. The reasons are described here. In short - reading is sufficient to satisfy most use cases in the standard python ecosystem.
Additionally, both load and loads provide a hook for parsing floats in the TOML files. This enables configuring how the float values are represented in the result dict, thus results of parsing floats can be types other than the basic python types.
Other module changes
WSGI static typing
Another new module, called wsgiref.types was added for static checking of WSGI-related types. More here.
Changes in existing modules
Python 3.11 introduced a lot of exciting changes to existing modules. A full list is available here.
In version 3.11, Python received a significant performance boost. Few builtins like lists and dicts were optimized. List comprehensions are now between 20 and 30 % faster. list.append() method was improved as well by ~15% on average.
For the dictionaries, when all of the keys are Unicode objects, hash values aren't stored which in turn reduced their size by a factor of ~20%. Other built-in optimizations were made to integers. For integers smaller than 2 ** 30, both sum() and // are significantly faster on x86-64 architectures.
On top of that optimizations, the CPython interpreter itself was optimized greatly for both startup and runtime. Bytecode objects are now statically allocated and stored in the __pycache__.
This results in a 10-15% speedup of the startup which is of great importance for short-lived python programs as for them the startup contributes to most of the total time.
Runtime was boosted by optimizing python frames. Frame objects are now smaller and contain less low-level data. The process of creation and memory allocation was improved.
In fact, frames aren't created at all for most of the user code. They're evaluated lazily whenever they're requested by the debugger. The other performance improvement was inlining python functions.
Prior to 3.11, whenever a python function was called, an underlying C function was called to interpret it. In the new version, whenever the interpreted detects a function call from another python function, it sets a new frame for that function and executes it without calling the C function, which effectively inlines the function call. This results in a significantly bigger recursion limit as well as faster-running recursive functions.
Finally, the 3.11 version introduces the adaptive python interpreter specialization proposed in PEP-659. The general idea is that while python is a dynamic languages, most types rarely change at runtime - a concept known as type stability.
The interpreter is able to detect such code fragments and replace bytecode with more efficient variants in a process called quickening. This optimization is only applied for hot code - that is code that's run multiple types and results in up to 25% performance gain in some scenarios.
Want to learn more from professional backend developers? Check out the other blog post from our Feel The Tech series.