Wayland Architecture: Separating the Compositor and Window Manager

Why Separating the Wayland Compositor and Window Manager Matters

The Linux ecosystem is steadily moving toward Wayland as its primary display protocol, but for developers, the line between the “compositor” and the “window manager” remains a hot topic. Historically, X11-based systems had a clear division: the X server rendered graphics, while a separate window manager handled placement, focus, and policies. Wayland, by design, merges these roles—but does it have to?

Projects like Sway, KWin, and GNOME Shell have blurred this boundary. But with the growing demand for modularity, security, and user customization, there’s a push to re-explore this separation. The implications for code maintainability, user experience, and security are significant—and the technical trade-offs aren’t just academic.

Key Takeaways:

• Separating the compositor and window manager in Wayland is technically feasible, but not the default.
• This separation can improve modularity, security, and testability, but introduces real-world complexity.
• Several modern compositors are experimenting with this split, but integration and IPC challenges persist.
• Developers should carefully evaluate performance, flexibility, and maintainability trade-offs before adopting a split design.

To illustrate, consider a scenario where a user wants to swap out their window management behavior (e.g., switch from tiling to stacking) without replacing the entire compositor. In a monolithic design, this would require patching or forking the compositor itself. With a separated architecture, the user could simply run a different window manager process, increasing flexibility while potentially reducing maintenance headaches for developers.

Wayland Architecture Essentials

Wayland replaces the legacy X11 protocol with a simpler, more modern model. In Wayland, the compositor is responsible for directly managing both graphics rendering and input event distribution. The window manager role—placement, focus, tiling, and decorations—has traditionally been subsumed into the compositor process.

Technical Terms:

• Compositor: A process that combines the contents of multiple application windows (surfaces) into a single display output, handling how they are rendered and where.
• Window Manager (WM): A process or component that controls the placement and appearance of windows, dictating policies like focus, stacking, tiling, and decorations.
• Buffer Management: The allocation and management of memory regions (buffers) used to store rendered graphics before they are displayed on screen.
• Input Event Dispatch: The process of routing input events (keyboard, mouse, touch) to the correct client window.

But what if you want a clean separation? Let’s look at the architectural components:

• Compositor: Handles rendering, buffer management, and input event dispatch. Examples: wlroots, weston. For example, wlroots is a modular library that provides building blocks for writing compositors, making it easier to experiment with custom architectures.
• Window Manager: Implements window policies: how windows are arranged, focused, and decorated. In X11, this was a standalone process (like openbox or i3), allowing users to choose their preferred WM independently of the X server.

With Wayland, the default is monolithic: the compositor (e.g., Sway, KWin) embeds window policy logic. But some projects, especially those using wlroots, experiment with extracting window management into a separate process for flexibility and security.

Transitioning from the conceptual overview, let’s look at how these ideas manifest in actual projects.

Practical Separation: Real-World Examples

To understand how separation works in practice, let’s examine real projects and approaches. The following table compares several well-known Wayland compositors and their strategies for handling window management:

For example, Sway (a tiling window manager and compositor for Wayland) implements both roles in a single process. This results in tight integration and low latency, but limits the ability to swap out window management logic without modifying the compositor itself. On the other hand, River takes a different approach by delegating window management decisions to an external process via Inter-Process Communication (IPC). This allows for greater modularity, as the window manager can be restarted or replaced independently of the compositor.

For a deeper dive, see the discussions and proposals on LWN.net: Modular window management for Wayland compositors.

Now that we’ve seen how separation plays out in real projects, let’s explore what this looks like in code.

Code Examples: Implementing Separation

Let’s get concrete: how can you actually implement a separation between compositor and window manager logic in a modern Wayland environment? Below are real-world, runnable examples using wlroots (a popular Wayland compositor library) and a simple custom window manager process communicating via IPC.

Example 1: Minimal wlroots-based Compositor (C, wlroots v0.16.0)

This example shows a barebones Wayland compositor setup using wlroots. In this architecture, the compositor initializes the display and backend, and sets up an event listener for new surfaces (windows). Instead of handling window placement itself, it forwards these events to an external window manager via IPC.

#include <wlr/backend.h>
#include <wlr/types/wlr_compositor.h>
#include <wlr/types/wlr_output.h>
#include <wlr/types/wlr_seat.h>
#include <wlr/types/wlr_xdg_shell.h>

int main(int argc, char *argv[]) {
    struct wl_display *display = wl_display_create();
    struct wlr_backend *backend = wlr_backend_autocreate(display, NULL);
    struct wlr_compositor *compositor = wlr_compositor_create(display, backend);
    struct wlr_xdg_shell *xdg_shell = wlr_xdg_shell_create(display);

    // Forward new surface events to WM process (IPC demo)
    wl_signal_add(&xdg_shell->events.new_surface, &new_surface_listener);

    wlr_backend_start(backend);
    wl_display_run(display);
    wl_display_destroy(display);
    return 0;
}
/*
 * Output: A minimal compositor that delegates window management to an external process.
 * (Needs additional IPC glue code for real use.)
 */

In this setup, wl_signal_add attaches a listener to the “new_surface” event, which will be responsible for notifying the window manager process. The compositor itself remains agnostic to window placement policies.

Example 2: Window Manager Process (Python, using UNIX domain sockets)

This example illustrates a simple window manager process written in Python. It listens for incoming window events from the compositor over a UNIX domain socket (a common form of IPC on Unix-like systems). When a new window event arrives, it decides where to place the window and sends placement instructions back to the compositor.

import socket
import json
import os

SOCKET_PATH = '/tmp/wm_ipc.sock'

def handle_window_event(window_event):
    # Example: Always place new windows at (100, 100)
    print(f"Received window event: {window_event}")
    window_id = window_event['id']
    return {'window_id': window_id, 'x': 100, 'y': 100}

if os.path.exists(SOCKET_PATH):
    os.remove(SOCKET_PATH)

with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as server:
    server.bind(SOCKET_PATH)
    server.listen()
    print("Window Manager listening for events")
    while True:
        conn, _ = server.accept()
        with conn:
            data = conn.recv(4096)
            if not data:
                break
            event = json.loads(data.decode())
            response = handle_window_event(event)
            conn.send(json.dumps(response).encode())

# Output: Receives window events and returns placement decisions over IPC

Here, SOCKET_PATH defines where the compositor and window manager will communicate. The handle_window_event function, though simplistic, demonstrates how window management decisions can be made externally.

Example 3: Client Event Dispatch (C, sending events to the WM process)

This example demonstrates how the compositor might send window events to the window manager process. It opens a UNIX domain socket, sends a JSON-encoded event, and waits for the window manager’s response (such as coordinates for window placement).

#include <sys/socket.h>
#include <sys/un.h>
#include <unistd.h>
#include <stdio.h>
#include <string.h>

void notify_wm_of_new_surface(int window_id) {
    int sock = socket(AF_UNIX, SOCK_STREAM, 0);
    struct sockaddr_un addr = {0};
    addr.sun_family = AF_UNIX;
    strcpy(addr.sun_path, "/tmp/wm_ipc.sock");
    connect(sock, (struct sockaddr*)&addr, sizeof(addr));
    char buf[128];
    snprintf(buf, sizeof(buf), "{\"id\": %d}", window_id);
    write(sock, buf, strlen(buf));
    read(sock, buf, sizeof(buf));
    printf("WM response: %s\n", buf);
    close(sock);
}

/*
 * Output: Sends window events to the WM process and prints the response.
 * This would be called from your compositor's new_surface event handler.
 */

Such IPC-based communication allows for complete decoupling of window management logic from the compositor, enabling runtime replacement or upgrades of the window manager process.

Trade-Offs and Comparison

Why is this separation rare? The answer comes down to complexity and performance. Here’s a brief comparison of the two approaches:

For example, in a monolithic Sway-like setup, window management decisions (such as how to tile or stack windows) are implemented directly in the compositor’s codebase, ensuring fast and predictable behavior. In a separated architecture (like River), the compositor must send window events to the external WM process, which could introduce latency if the WM process is overloaded or delayed.

For high-throughput desktop environments, the monolithic approach still dominates. But for kiosks, customizable workstations, or environments where attack surface must be minimized, separation is increasingly attractive.

Next, let’s consider the specific edge cases and pitfalls that arise when deploying such separation in real-world systems.

Edge Cases and Pitfalls in Production

If you pursue separation, here are issues you’ll likely encounter—many of which are under-discussed in tutorials:

• Atomicity: Window management decisions must be atomic. Atomicity refers to operations being indivisible—either they happen completely or not at all. Race conditions between the compositor and WM process can cause flicker or incorrect stacking, especially if both try to update window state simultaneously.
• Crash Recovery: If the WM process crashes, windows may be left unmanaged. You must design robust reconnection and state sync logic, such as periodically polling the WM or having the compositor cache the last known good state.
• Timing: Fast-moving input (drag, resize) can expose IPC latency. Users notice even 10ms lag. For example, if a user drags a window, the compositor must send rapid updates to the WM process, which then responds with new positions; any delay can result in choppy movement.
• Protocol Versioning: As your WM logic evolves, you’ll need to version your IPC protocol to avoid mismatches. If the compositor and WM process disagree on message formats, window placement may break or become unpredictable.
• Security: Exposed IPC sockets are a potential attack vector. Always use UNIX domain sockets, validate all messages, and restrict socket permissions. For instance, making the socket accessible only to the intended user reduces the risk of malicious injection of window events.

Real-world compositors like river and Wayfire are experimenting with plugin- or IPC-based window management, but production deployments should test for these edge cases early and often.

With these potential pitfalls in mind, it’s important to keep up with ongoing developments in the field.

What to Watch Next

The separation of compositing and window management in Wayland is gaining mindshare but remains niche. However, as recent discussions on LWN.net and developer mailing lists show, there’s growing interest in modular, scriptable, and secure desktop architectures.

Key trends to watch:

• Evolution of the wlroots library to better support external WM processes.
• Adoption of scripting or plugin APIs (Lua, JavaScript) in compositors for runtime window policy changes.
• Security research into privilege separation between rendering and window management.
• Development of shared IPC protocol standards to allow third-party window managers to interoperate.

For developers building the next generation of Linux desktops—or shipping secure kiosks—understanding and experimenting with compositor/WM separation is both a technical challenge and an opportunity. For example, developers interested in minimizing the trusted computing base for critical deployments (such as ATMs or digital signage) may find process separation especially valuable for reducing attack surface.

References:

• Modular window management for Wayland compositors (LWN.net)