Suppose you want to write a new Django/DRF API service tomorrow or have inherited a large but messy Django codebase.
Introduction
If your application is useful, it will do more than enable crud operations on relational database tables via HTTP. Let's assume that your application will talk to several cloud services and third-party APIs. It may make use of a message broker, or noSQL databases in addition to an RDBMS via Django's ORM. Let's also assume that it will contain a fair amount of complex business logic that enables the software to add value in whatever the business domain happens to be.
Where does all this business logic go?
Views?
Serializers?
Models?
Custom Managers and QuerySets?
In his popular post a few years ago Against service layers in Django, James Bennett argues in favour of putting most of it in Model methods and some in custom Managers and QuerySets, rather than allotting a dedicated architectural layer to business logic. For Bennett, Django's data models do double-duty as domain models works quite well with Django's active-record ORM.
This seems like a fine idea until you look at a real codebase and encounter something like methods on a Model class for processing a deeply nested YAML document. I'm not suggesting that Bennett or anyone for that matter would approve of such an abuse of fat models. However, I think it's a possible consequence of not giving busy engineers, who may be in a hurry to ship something, an easy answer to "where do I put this?" that doesn't involve Django classes.
What I'm arguing in this post is that while full-blown enterprise architecture patterns don't work well with Django, larger applications can still benefit from a service layer. By service layer, I don't mean pass-through methods to ORM manager methods as shown in Bennett's example. Services don't need to map to data models at all. They can encapsulate any concept within the business domain. When every concept has its own service, the Framework classes don't end up polluted with logic that doesn't fit anywhereelse.
Before delving further into this argument, it is helpful to identify exactly what Bennett was warning us against.
Two kinds of service layer
There are at least two kinds of service layers that are relevant to Django. There is no doubt more, but even splitting the concept in two should help address some confusion and misunderstandings.
What I call the Enterprise Service Layer, is what you might arrive at if you applied the book Architecture Patterns with Python (Available here for free) to a Django application. You would make use of enterprise architecture patterns, likely hide Django's ORM behind abstractions and have your business logic only interact with these abstractions.
The second kind of service layer is simpler and more pragmatic. It is exemplified by the software consultancy Hacksoft's Django style guide. We'll call this implementation Simple Service Layer.
In this post, I'm arguing that Bennett's objections to the service layer are aimed more at the former than the latter.
Simple Service Layer
The simple service layer represents a lightweight attempt to solve the problem of separating business logic from infrastructural code like framework classes. This kind of service layer typically restricts all ORM queries for a particular ORM model to a single Python module. However, a service module can just as well encapsulate any concept from the business domain, even one that requires querying multiple ORM models or other databases such as Redis.
According to Hacksoft's style guide, these service modules contain type-annotated functions, each of which has a single well-defined responsibility.
Why functions?
Motivation
Service modules provide a simple way to ensure that all creation, modification, and deletion of rows in a given table happen in one place. This allows developers to ensure that any additional operations that we want to happen in conjunction with create, update and delete happen consistently and are abstracted away for users of the module.
This is valuable because create, update and delete are basic verbs in the business logic, that should live alongside more specific ones e.g. create_from_x, transition_from_state_x_to_state_y, get_objects_shared_with_user, offboard_user. We want all of our business logic to be captured in the services module, rather than being distributed throughout the codebase (possibly with inconsistencies).
Although service modules work well for encapsulating a single database table, they should not be limited to a one-to-one relationship with an ORM model. They can be used for more complex use cases. They should capture whatever complexity is entailed by delivering a particular piece of business logic or service to the rest of the codebase.
Generally, having a service layer encourages developers to design more modular systems, rather than attempting to fit all code within Django/DRF’s models, serializer and views. Once freed from acting like Django has made all architectural and design decisions for us, we can be more creative and design our software more elegantly.
Advantages of This Approach
Simple and less effort to maintain
Easy to enforce using static analysis
DRF’s generic views work
QuerySet and Model instances, they work with DRF's generic views, serializer, and filter backends.Promotes separation of concerns
Trade-offs of This Approach
All business logic depends directly on Django
Doesn’t encourage fast unit tests, but slow integration tests
Gives calling code database access through Model and QuerySet APIs
Queryset and Model APIs.Enterprise Service Layer
This is a more complex approach to the service layer. Its core concept is the Dependency Inversion Principle. Normally high-level components that carry out business logic depend on (read import) low-level application logic components such as ORMs and I/O libraries. These typical dependencies can be inverted by defining abstractions that embody what the business logic requires of external systems. The infrastructural (non-business-logic) code consists of concrete implementations of these abstractions. In this way, the business logic isn't aware of what database backend or file system it is using; the infrastructural code only needs to know enough to satisfy its interface with the business logic. Business logic that depends on abstractions can continue to be used unchanged if the concrete implementations of these abstractions change.
Implementing Dependency Inversion: Ports, Adapters, and Services
There are many components described in Architecture Patterns with Python, most of which won’t be covered here. This section will only deal with service modules, ports, and adaptors.
Ports are the abstract interfaces depended on by the services.
Adaptors are the concrete implementations of abstract interfaces. Many different adaptors can slot into one type of port. They are like electronic adaptors that plug into ports on a computer, for example allowing an HDMI port on a computer to connect to an old monitor via VGA or a newer one via DVI.
Services are modules that handle business logic. They all contain a main function some of whose parameters are ports (the abstract interfaces); the arguments passed in using those parameters are adaptors (the concrete implementations). Services carry out high-level business logic and are ignorant of the implementation details of application logic.
Advantages of This Approach
Fast unit tests without unittest.mock
Swappable dependencies
Ease of microservice-ing:
Business logic doesn’t depend on Django
Promotes even better separation of concerns
Prevents calling code getting database access through Model and QuerySet APIs
Trade-offs of This Approach
Complex and more effort to maintain
Reliance on fake implementations when testing
FakeManager, for example, behaves exactly like a real manager talking to the database via the ORM.)Would require building abstraction around advanced features of Django’s ORM
select_related and prefetch_related? What about select_for_update in concurrency situations? What about annotations, F objects, and other advanced ORM features that let us shift some of the computation to Postgres?Generic DRF views would not work
APIViews. This is because generic views only work with QuerySets, not plain iterables like lists. This is arguably a good thing but it would take considerable engineering time. The advantage of simpler superclasses for views would be that new developers unfamiliar with DRF would understand what was actually going on in the views without first getting to grips with Byzantine inheritance hierarchies. It’s worth noting, over-reliance on inheritance isn’t even considered good design by proponents of OOP (See Composition over inheritance); the generic views are arguably poorly designed – so why depend on them?You'd maintain data classes for each model
Weighing up the Pros and Cons
Bennett argues that the benefits of the Enterprise service layer are not justified by the cost of implementing it. He observes that Django codebases seldom swap out the data access layer; for better or worse, that's tightly coupled to just every other part of Django. What then, is the point of swappable dependencies?
The same goes for pure unit tests. In a follow-up post called More on service layers in Django, he makes the following observations:
here are a lot of things that can go wrong when using mocks, fake-factories and other tools to simulate a dependency, and it’s easy to wind up with misplaced confidence because the beautifully-isolated tests were operating with incorrect or incomplete simulations of key dependencies, or even just testing against the wrong things. So generally, the more complex or crucial to the application a dependency is, the less likely I am to try to isolate/mock it away and the more likely I am to use the real thing. A hybrid approach of trying to have some “pure” tests that use simulated dependencies and other less-pure tests that use the real thing is possible, of course, but raises questions about why so much effort is put into isolating the “logic” from the “dependencies” if test runs are going to have to use the real dependencies at some point anyway.
I can't deny that there's something unsatisfying about the impurity of typical Django tests. However, the way that the test database is abstracted away is a feature, not a bug. As someone who appreciates speed and elegance, I grudgingly accept that integration tests are more valuable than unit tests. Yes, they're slow, but there is Pytest tooling for Django that runs them in parallel with multiple test databases. The simple service layer works well with these Django integration tests but also gives engineers the freedom to write some business logic as unit-testable pure functions.
A common theme here is that Django's batteries-included design philosophy isn't compatible with enterprise patterns. Bennett recommends a data-mapper ORM like SQLAlchmy over Django's ORM and the book Architecture Patterns in Python is aimed at developers using a microframework like Flask or FastAPI. This is well illustrated by the unit-of-work pattern that wraps each action in a database transaction. Django had a decorator/context manager that does the same. It also wraps all requests in a transaction if ATOMIC_REQUESTS = True is in the project's settings.
Conclusion
This post is largely inspired by watching a team try to tame a legacy Django codebase where the majority of previous engineers threw business logic into models, views and serialisers with reckless abandon. Attempts to refactor a small part of the codebase using enterprise patterns generally confused those not directly involved. We had far more success following Hacksoft's simple service layer; no one had to read half a book to understand it. We got the benefits of separation of concerns without restricting our ability to interact with the database. Nor did we have to build further abstractions around the towering abstraction that is Django's ORM.
Some may say that developers should simply know how to write modular code and avoid putting inappropriate logic in framework classes. Sadly this wasn't the case for me as a novice. So many design decisions were made for me by Django that my design thinking atrophied somewhat. The simple service layer empowers developers to creatively model their business domain and avoid being stifled by the few abstractions that Django provides. The Django framework classes can be viewed as nouns that serve well-defined purposes such as data modelling, HTTP request handling and "serde". Services provide a place to put the verbs of your software, where it does the things that make it valuable to users.