Every year someone says that this is the year of the Linux desktop.

It is never the year of the Linux desktop.

There are many reasons for this. Drivers. Games. Adobe. Microsoft Office. Battery life. The thing where you close the lid of a laptop and open it again later to find that it passed into the good night. These explanations are all correct in the small and unsatisfying in the large. They explain why a person did not switch to Linux last Thursday. They do not explain why the desktop, as an institution, will continue to belong to Apple and Microsoft.

And now there is a new and more depressing explanation.

The future computer user is not a person.

Or at least not only a person. The robots are coming for the desktop. The interesting part is that the ramps were already there.

They were called accessibility APIs.

If you use a Mac and open the Accessibility Inspector tool that’s built into the system (you really should try it), you can see a second version of the computer, hiding inside the first one. The first version is the one you look at: windows, shadows, rounded rectangles, a little bouncing icon in the Dock from Slack announcing that you are falling behind.

The second version is a tree. A literal hierarchy of objects. Window. Group. Button. Text field. Scroll area. Static text. Each object has properties. Some have values. Some have actions. Some will tell you where they are. Some will tell you what they contain. Some will let you press them without moving the mouse at all.

This is not how computers were initially designed to be used, if by “used” you mean “used by sighted people moving a pointer around.” It is how computers had to be exposed to people who could not rely on pixels. VoiceOver needed it. Switch control needed it. Dictation systems needed it. The operating system had to learn to describe itself.

And now the agents need it too.

You can see this most clearly in OpenAI’s Codex Computer Use feature, which on macOS doesn’t just take a screenshot. It also pulls “available text” out of the frontmost window including text the app makes available outside the visible scroll area, which is to say, content that is technically not on the screen at all. It also allows the agent to interact with your entire Mac without interrupting your usage as it has its own independent mouse that can work in the background.

OpenAI bought the company that built this in October 2025: a twelve-person shop called Software Applications Incorporated, whose product, Sky, had never been publicly released. Sam Altman had personally invested in the seed round. The founders had previously sold Workflow to Apple, where it became Shortcuts. What OpenAI got for an undisclosed but evidently real amount of money was the team’s bet about the right way for an AI model to drive a Mac. The bet appears to have been correct. The binary that runs this inside Codex today is still named SkyComputerUseClient.

This is the part where you might expect me to say that the reason macOS is suddenly so good for agents is the accessibility API. But that’s not really the full story. Windows has accessibility APIs. Linux has accessibility APIs. APIs are easy to have. You write them down in a header file, give a conference talk about them, and then spend the next twenty years explaining why nobody used them correctly.

The reason macOS is so far ahead is because of defaults.

Apple did not, when most of this was being soldered into place in the late 1990s, anticipate that a stochastic parrot with an $800+ billion valuation would one day need to change a setting in Finder. Apple just decided that if you build a normal Mac app out of normal Mac controls with things like NSButton, NSTextField, WKWebView, the boring stock pieces then your app should be accessible by default. The developer didn’t have to do anything. They wrote a regular app and got a high-fidelity accessibility tree for free, because Apple put the cost of compliance into the SDK instead of the application. The blind user got the tree. The accidental beneficiary, all these years later, is Codex.

This is one of those situations where a moral concern turns out, in retrospect, to have also been infrastructure.

For most of software history, accessibility was treated by most engineering teams as either a compliance chore, an act of kindness, or a thing you would get to at the end if there was time, which there never was, because the only features that were ever truly protected were the ones that affected someone’s bonus.

This was always wrong! But it is now wrong in a way that rich people can understand. A bad accessibility tree no longer excludes only disabled users. It also excludes agents. Accessibility is, by accident, becoming agent compatibility.

Agents are now new customers. History is not sentimental about motives. The accessibility tree was built for assistive technology, and now the robots in the machine wants to use it to book a flight.

And in this area, the Mac is truly far ahead.

Windows, in its defense, has a very serious accessibility tree. Microsoft UI Automation (UIA) is, in some ways, the most Microsoft thing imaginable.

It is a complete object model of the desktop with three filtered views: raw, control, and content. Because of course Microsoft looked at the question of “what is on screen” and decided one ontology would not suffice. It has a real pattern system: InvokePattern for buttons, TextPattern for documents, ValuePattern for inputs, and an enumeration of verbs that controls admit to supporting.

Microsoft’s own documentation cheerfully observes that this same API can be used by assistive technologies and by automated test scripts, which has turned out to be the most prescient sentence Microsoft has written about Windows in many years.

UI Automation is, by any reasonable engineering standard, excellent.

The problem with Windows is not the API. The problem is archaeology. Every Windows machine is a museum of electricity. There’s not one type of app. There is Win32. There is WPF. There is WinForms. There is UWP. There is WinUI. There is Electron. There is some custom line-of-business application written by a contractor in 2009 who has since moved to a farm and cannot be reached. There is a settings panel that is secretly a web page. There is a desktop app that is secretly Chromium wearing a fake mustache. The list goes on.

UIA can be very good. But the app has to meet it halfway. And on Windows the app frequently does not meet it halfway. It’s nearly unusable. A UIA tree scanned across a real Windows desktop is full of regions that respond, with admirable consistency, the way an empty house responds to a knock.

The recurring theme here is that an agent does not just need an API. It needs a civilization of apps that conform to the API well enough that the agent can trust what they say. A button that admits to being a button. A text field that admits to containing text. A table that does not expose itself as fourteen hundred unnamed rectangles and a prayer.

Which brings us to the mess of Linux.

To be fair, and one should be fair about this, because Linux folks can smell imprecision through concrete, Linux does have an accessibility stack! It is called AT-SPI , the Assistive Technology Service Provider Interface, and it is real. It runs over D-Bus. It exposes Accessible, Action, Component, Document, Text, Value, and so on. GTK apps support it. Qt apps support it. Firefox supports it. LibreOffice supports it. Orca, the GNOME screen reader, has been in production on it since 2006.

But agents do not just need an accessibility tree. They need to enumerate windows. They need to capture the screen. They need to synthesize input. They need a coherent permission model. They need to do all of this without the user feeling like they are watching a haunted mouse perform community theater.

On a Mac, this is one Accessibility toggle and one Screen Recording toggle, both clearly named, both stored in the same general place. On Linux under Wayland, the screen capture is a portal, the input synthesis is a different portal or libei, the window enumeration is a per-compositor protocol, and the cross-compositor accessibility evolution, called Newton, is a prototype being developed by a man named Matt Campbell on a grant from the Sovereign Tech Fund. A GNOME Foundation report from April 2025 describes the protocol as “not yet rigorously defined” and notes it “has not yet seen any cross-desktop discussions.” KDE has not committed to it.

Every step of the loop is available, after installing the correct backend and selecting the correct session type and, depending on the day, sacrificing a small goat to the compositor. Apple can force attention. Microsoft can institutionalize it. Linux has to convene it.

The problem is that Linux can make almost anything exist, but it cannot make almost everyone agree to care about it at the same time. This is the part that has kept the year-of-the-Linux-desktop joke alive long after Linux became, on most days, a perfectly usable desktop. StatCounter has it at 2.99% of global desktop usage in April 2026, up from 2.76% in 2022. Continents move faster. But you can put Ubuntu on a ThinkPad and do most of what a normal person needs to do, and people do. The desktop got pretty good.

The mission, by its original definition, is mostly accomplished.

Nobody is throwing a party because the target is about to move.

The target is moving because the standard for “usable desktop” is no longer whether you would enjoy using it. The standard is whether a thing that is not you can use it on your behalf.

The high-fidelity accessibility tree, the reliable input synthesis, the standardized window enumeration, the portable screen capture, the coherent permission model. Apple has been building exactly this for thirty years, paid for by Cupertino, almost entirely for the benefit of users who number in the low millions. The work is now, accidentally and irreversibly, also the substrate for agents, which are about to number in the billions.

Microsoft has been engineering most of it but letting half the platform skip the homework. The Linux community has been building parts of it, in scattered repositories, by ones and twos, often on grants, often by one guy in Nebraska.

This is not the kind of gap a community closes by writing better software. It is the kind of gap that takes a decade of full-time employees auditing every label in every default app, a market mechanism that punishes you when you don’t, and a centralized review process to enforce it from above.

None of that exists for Linux. None of it is coming.

There was, for a long time, a wall.

If you walked into a book store, past the magazines and the cookbooks, you’d arrive at the computer section, and along one wall there was a stretch of books with cartoon animals on their covers. A rhino for JavaScript. A camel for Perl. A python (obviously) for Python. And whatever this was:

They were thick, they cost about $50, and they had titles like “Learning React” and “HTTP: The Definitive Guide”. If you wanted to learn how to do a thing on a computer, you bought one of these, took it home, and opened it up next to your computer and typed what it said until the thing worked.

That wall is smaller now. If it’s even still there. In some stores the wall is gone and relegated to a small rack that has six books on it, three of which are about ChatGPT.

Through the first nine months of 2023, sales in the “computer book” category at Circana BookScan (the industry’s standard tracker, which costs roughly the price of a small used car to subscribe to) were down 16.9% year over year. Publishers Weekly, which had been dutifully reporting these figures in its quarterly narrative summaries, kept doing so right up through that 16.9% figure, and then in 2024 and 2025 simply stopped mentioning the category by name.

To be clear, books in general are doing fine. Total U.S. print sales reached 762.4 million units in 2025, up 0.3% over 2024, which was itself up 0.5% over 2023. The category that is in trouble is the part of it that teaches you how to make software. The American Association of Publishers’ “professional books” segment, which is the rough corporate proxy for “books your employer might buy you,” fell 22.3% in August 2025.

The book industry is fine but the technical end is bleeding out.

Quickly and quietly. There was no Napster moment for the programming book. Nobody filed a lawsuit. The publishers did not, as far as I can tell, even hold a press conference. We simply found one day that they stopped reporting the category itself. The category doesn’t die, it just stops being talked about.

You already know why, more or less. ChatGPT has over 900 million monthly active users. GitHub Copilot has 4.7 million paying subscribers as of January 2026, up roughly 75% in a year. You can’t imagine writing software without Claude Code anymore.

Stack Overflow is receiving about 3,800 questions a month, which is what it was getting in 2008, before it had finished being launched. The chatbots have eaten the demand for the kinds of answers that programming books used to provide.

The programming book was, when you look at it squarely, always a slightly absurd object. Printed text on bound paper, describing software that lived on screens, which the reader had to retype, by hand, into a screen of their own. I loved doing this and they remain some of my very fondest childhood memories. But the medium was wrong for the content. People put up with it because there was no better way to get a careful sustained explanation of a technical thing into one person’s head from another person’s.

What the book was good at, despite being the wrong format, was forcing both the writer and the reader to be slow. You cannot fake your way through 400 pages. It took a certain discipline to get through.

The chatbot does not have this discipline. The chatbot has read every book and forgotten the point of every one of them. It will explain idempotency in the precise number of words you require, and you will close the tab, and you will not remember what it told you, because you did not type it.

That last sentence is the whole thing. Knowledge, for working programmers, was always the residue of typing. Of doing. The typing was the practice! What is going away is the typing.

Which, on balance, may be fine. I don’t know. People used to lose weekends to installing Linux from a stack of floppies and struggling with WinModems, and nobody pretends that was character-building (though I now consider them fond memories too). Tools get easier. Skills shift. The kid who is right now learning to code by chatting with an agent is not a worse programmer than I was at 12, hunched over Learning Perl, retyping examples that would not run because I missed a semicolon.

That kid is a different programmer. They are, in some ways I don’t fully understand, working at a higher level of abstraction than I ever did at that age, and the things they will build with that abstraction will surprise me.

But somewhere in a used bookstore in San Francisco or Seattle or wherever used bookstores still exist, there is a 1997 edition of Learning Perl. It smells faintly of basement. Someone wrote their name in the front of it in pencil. There is a furiously underlined sentence in chapter 7 about regular expressions that was made in anger. On page 112 there are coffee stains where the caffeine blots are somehow still a valid Perl program.

The book costs three dollars. Nobody is going to buy it.

One of the current trends in modern software is for developers to slap an API call to OpenAI or Anthropic for features within their app. Reasonable people can quibble with whether those features are actually bringing value to users, but what I want to discuss is the fundamental concept of taking on a dependency to a cloud hosted AI model for applications.

This laziness is creating a generation of software that is fragile, invades your privacy, and fundamentally broken. We are building applications that stop working the moment the server crashes or a credit card expires.

We need to return to a habit of building software where our local devices do the work. The silicon in our pocket is mind bogglingly faster than what was available a decade ago. It has a dedicated Neural Engine sitting there, mostly idle, while we wait for a JSON response from a server farm in Virginia. That’s ridiculous.

Even if your intentions are pure, the moment you stream user content to a third party AI provider, you’ve changed the nature of your product. You now have data retention questions and all the baggage that comes with that (consent, audit, breach, government request, training, etc.)

On top of that you also substantially complicated your stack because your feature now depends on network conditions, external vendor uptime, rate limits, account billing, and your own backend health.

Congratulations! You took a UX feature and turned it into a distributed system that costs you money.

If the feature can be done locally, opting into this mess is self inflicted damage.

“AI everywhere” is not the goal. Useful software is the goal.

Concrete Example: Brutalist Report’s On-Device Summaries

Years ago I launched a fun side project named The Brutalist Report which is a news aggregator service inspired by the 1990s style web.

Recently, I decided to build a native iOS client for it with the design goal of ensuring it would remain a high-density news reading experience. Headlines in a stark list, a reader mode that strips the cancer that has overtaken the web, and (optionally) an “intelligence” view that generates a summary of the article.

Here’s the key point though: the summary is generated on-device using Apple’s local model APIs. No server detours. No prompt or user logs. No vendor account. No “we store your content for 30 days” footnotes needed.

It has become so normal for folks that any AI use is happening server-side. We have a lot of work to do to turn this around as an industry.

It’s not lost on me that sometimes the use-cases you have will demand the intelligence that only a cloud hosted model can provide, but that’s not the case with every use-case you’re trying to solve. We need to be thoughtful here.

Available Tooling

I can only speak on the tooling available within the Apple ecosystem since that’s what I focused initial development efforts on. In the last year, Apple has invested heavily here to allow developers to make use of a built-in local AI model easily.

The core flow looks roughly like this:

import FoundationModels

let model = SystemLanguageModel.default
guard model.availability == .available else { return }

let session = LanguageModelSession {
  """
  Provide a brutalist, information-dense summary in Markdown format.
  - Use **bold** for key concepts.
  - Use bullet points for facts.
  - No fluff. Just facts.
  """
}

let response = try await session.respond(options: .init(maximumResponseTokens: 1_000)) {
  articleText
}

let markdown = response.content

And for longer content, we can chunk the plain text (around 10k characters per chunk), produce concise “facts only” notes per chunk, then runs a second pass to combine them into a final summary.

This is the kind of work local models are perfect for. The input data is already on the device (because the user is reading it). The output is lightweight. It’s fast and private. It’s okay if it’s not a superhuman PhD level intelligence because it’s summarizing the page you just loaded, not inventing world knowledge.

Local AI shines when the model’s job is transforming user-owned data, not acting as a search engine for the universe.

There are plenty of AI features that people want but don’t trust. Summarizing emails, extract action items from notes, categorize this document, etc.

The usual cloud approach turns every one of those into a trust exercise. “Please send your data to our servers. We promise to be cool about it.”

Local AI changes that. Your device already has the data. We’ll do the work right here.

You don’t build trust with your users by writing a 2,000 word privacy policy. You build trust by not needing one to begin with.

The tooling available on the platform goes even further.

One of the best moves Apple has made recently is pushing “AI output” away from unstructured blobs of text and toward typed data.

Instead of “ask the model for JSON and pray”, the newer and better pattern is to define a Swift struct that represents the thing you want. Give the model guidance for each field in natural language. Ask the model to generate an instance of that type.

That’s it.

Conceptually, it looks like this:

import FoundationModels

@Generable
struct ArticleIntel {
  @Guide(description: "One sentence. No hype.") var tldr: String
  @Guide(description: "3–7 bullets. Facts only.") var bullets: [String]
  @Guide(description: "Comma-separated keywords.") var keywords: [String]
}

let session = LanguageModelSession()
let response = try await session.respond(
  to: "Extract structured notes from the article.",
  generating: ArticleIntel.self
) {
  articleText
}

let intel = response.content

Now your UI doesn’t have to scrape bullet points out of Markdown or hope the model remembered your JSON schema. You get a real type with real fields, and you can render it consistently. It produces structured output your app can actually use. And it’s all running locally!

This isn’t just nicer ergonomics. It’s an engineering improvement.

And if you’re building a local first app, this is the difference between “AI as novelty” and “AI as a trustworthy subsystem”.

“But Local Models Aren’t As Smart”

Correct.

But also so what?

Most app features don’t need a model that can write Shakespeare, explain quantum mechanics, and pass the bar exam. They need a model that can do one of these reliably: summarize, classify, extract, rewrite, or normalize.

And for those tasks, local models can be truly excellent.

If you try to use a local model as a replacement for the entire internet, you will be disappointed. If you use it as a “data transformer” sitting inside your app, you’ll wonder why you ever sent this stuff to a server.

Use cloud models only when they’re genuinely necessary. Keep the user’s data where it belongs. And when you do use AI, don’t just glue it as a chat box. Use it as a real subsystem with typed outputs and predictable behavior.

Stop shipping distributed systems when you meant to ship a feature.

In June of 2023 Red Hat made a controversial decision to change how they distribute the source code behind Red Hat Enterprise Linux (RHEL). There have been a lot of keyboards tapped angrily across social media that left many uncertain about the ramifications of the decision. There were many questions about the future viability of downstream rebuilds of RHEL affecting distributions like Rocky Linux, AlmaLinux, Oracle Linux, and others. Each have since made announcements to try and calm their communities.

Still. Many in the open source community have interpreted Red Hat’s decision for what it really was: A dick move.

There has been a steady uptick of people stating that they will migrate (or already have) to Debian – seeking refuge from what they see as greedy corporate influence. I understand the sentiment fully. However, there’s a problem here that I want to talk about: security.

The ugly truth is that security is hard. It’s tedious. Unpleasant. And requires a lot of work to get right.

Debian does not do enough here to protect users.

Long ago, Red Hat embraced the usage of SELinux . And they took it beyond just enabling the feature in their kernel. They put in the arduous work of crafting default SELinux policies for their distribution.

These policies ship enabled by default in their distribution. The policies help protect a variety of daemons that run by default on RHEL, as well as many of the most popular daemons folks tend to use on the server.

Apache, nginx, MariaDB, PostgreSQL, OpenSSH, etc. are all covered by SELinux policies that ship on RHEL distributions.

The protection even extends to containers. Containers are increasingly the preferred method for developers to deploy their software – myself included. A common misconception is that if you run something in a container, it’s inherently secure. This is absolutely not true. Containers by themselves do not solve a security problem. They solve a software distribution problem. They give a false impression of security to those that run them.

On Red Hat based distributions, you can use a drop-in Docker alternative named podman which allows you to run containers without needing a daemon (unlike Docker) and it provides other benefits like being able to run fully root-less. But Red Hat takes it a step further here and applies strong default SELinux policies that separate the container from the host OS and even from other containers!

There have been numerous examples of being able to escape from a container and touch the host OS or other containers. This is where tools like SELinux step in. The application of SELinux policies on a container allows for a hardened sarcophagus to place your application in which mitigates the risk of unknown future exploits. And it’s nearly effortless to use on RHEL.

Red Hat was aware that unless they put in the work on these default policies, their users would simply not embrace the technology and millions of servers would remain vulnerable. Because let’s be real here. SELinux is hard. The policy language and tooling is cumbersome, obtuse, and is about as appealing as filling out tax forms. It frankly sucks to use – if you are manually creating your own policies that is.

But due to the work Red Hat has put in, the usage of SELinux on RHEL is mostly transparent and provides real security benefits to their users.

Debian’s Approach

Debian, a stalwart of the open-source community, is revered for its stability and extensive software library. I am a fan and donate to the project every year (you should too!) even though I don’t run it in production environments.

However, its default security framework leaves much to be desired. Debian’s decision to enable AppArmor by default starting with version 10 signifies a positive step towards improved security, yet it falls short due to the half-baked implementation across the system.

Debian’s reliance on AppArmor and its default configurations reveals a systemic issue with its approach to security. While AppArmor is capable of providing robust security when properly configured, Debian’s out-of-the-box settings fail to leverage its full potential:

Limited Default Profiles: Debian ships with a minimal set of AppArmor profiles, leaving many critical services unprotected.

Reactive vs. Proactive Stance: Debian’s security model often relies on users to implement stricter policies, rather than providing a secure-by-default configuration.

Inconsistent Application: Unlike SELinux in Red Hat systems, which applies to the entire system consistently, AppArmor in Debian is applied piecemeal, leading to potential security gaps.

Lack of Resources: Debian as a community-driven project lacks the resources to develop and maintain comprehensive security policies comparable to those provided by Red Hat.

It’s very common for folks to run container workloads on Debian via Docker – which does automatically generate and load a default AppArmor profile for containers named docker-default. Unfortunately, it’s not a very strong profile for security and is overly permissive.

This profile, while providing some protection, leaves significant attack surfaces exposed. For instance:

  network,
  capability,
  file,
  umount,
  # Host (privileged) processes may send signals to container processes.
  signal (receive) peer=unconfined,
  # runc may send signals to container processes (for "docker stop").
  signal (receive) peer=runc,
  # crun may send signals to container processes (for "docker stop" when used with crun OCI runtime).
  signal (receive) peer=crun,
  # dockerd may send signals to container processes (for "docker kill").
  signal (receive) peer={{.DaemonProfile}},
  # Container processes may send signals amongst themselves.
  signal (send,receive) peer={{.Name}},
  deny @{PROC}/* w,   # deny write for all files directly in /proc (not in a subdir)
  # deny write to files not in /proc/<number>/** or /proc/sys/**
  deny @{PROC}/{[^1-9],[^1-9][^0-9],[^1-9s][^0-9y][^0-9s],[^1-9][^0-9][^0-9][^0-9/]*}/** w,
  deny @{PROC}/sys/[^k]** w,  # deny /proc/sys except /proc/sys/k* (effectively /proc/sys/kernel)
  deny @{PROC}/sys/kernel/{?,??,[^s][^h][^m]**} w,  # deny everything except shm* in /proc/sys/kernel/
  deny @{PROC}/sysrq-trigger rwklx,
  deny @{PROC}/kcore rwklx,
  deny mount,
  deny /sys/[^f]*/** wklx,
  deny /sys/f[^s]*/** wklx,
  deny /sys/fs/[^c]*/** wklx,
  deny /sys/fs/c[^g]*/** wklx,
  deny /sys/fs/cg[^r]*/** wklx,
  deny /sys/firmware/** rwklx,
  deny /sys/devices/virtual/powercap/** rwklx,
  deny /sys/kernel/security/** rwklx,

The network rule allows all network-related syscalls without restriction.

The capability rule, without specific denials, permits most capabilities by default.

The file rule grants broad file access permissions, relying on specific deny rules for protection.

AppArmor vs. SELinux

The fundamental difference between AppArmor and SELinux lies in their approach to Mandatory Access Control (MAC). AppArmor operates on a path-based model, while SELinux employs a significantly more complex type enforcement system. This distinction becomes particularly evident in container environments.

SELinux applies a type to every object in the system - files, processes, ports, you name it. When you launch a container on a SELinux-enabled RHEL system, it’s immediately assigned the container_t type – a strict access control mechanism. The container_t type effectively cordons off the container, preventing it from interacting with any object not explicitly labeled for container use.

But SELinux doesn’t stop at type enforcement. It takes container isolation a step further with Multi-Category Security (MCS) labels. These labels function as an additional layer of segregation, ensuring that even containers of the same type (container_t) remain isolated from each other. Each container gets its own unique MCS label, creating what amounts to a private sandbox within the broader container_t environment.

AppArmor, in contrast, doesn’t concern itself with types or categories. It focuses on limiting the capabilities of specific programs based on pre-defined profiles. These profiles specify which files a program can access and which operations it can perform. While this approach is more straightforward to implement and understand, it lacks the granularity and system-wide consistency of SELinux’s type enforcement. Almost no mainstream Linux distribution distributes comprehensive AppArmor profiles for all common network-facing daemons by default.

The practical implications of these differences are significant. In a SELinux environment, a compromised container faces substantial hurdles in accessing or affecting the host system or other containers, thanks to the dual barriers of type enforcement and MCS labels.

This isn’t to say one is universally superior to the other. SELinux offers more robust isolation but at the cost of increased complexity and potential performance overhead. AppArmor provides a simpler, more approachable security model that can still be quite effective when configured properly. The root of my point though is that Red Hat has put in the work here to make the use of SELinux and containers seamless and easy for its users. You aren’t left to fend for yourself.

In the end, the choice between Debian and Red Hat isn’t just about corporate influence versus community-driven development. It’s also a choice between a system that assumes the best and one that prepares for the worst. Unfortunately in today’s highly connected world, pessimism is a necessity.

The Enterprise Linux model works something like this:

It’s decided to snapshot a collection of upstream open source projects at a specific version, including the Linux kernel, to form as the basis of a new cohesive version of an Enterprise Linux distribution. This collection of software will remain locked at its specific version throughout the lifespan of that Enterprise Linux distribution release – which is often 10 years or more.

The engineers, maintainers, and testers behind these distributions then put in the enormous amount of thankless work of bug fixes, tests, QA, documentation, etc. It’s a massive undertaking of tedious painstaking work. At the end of the process, you end up with a relatively stable Linux distribution that works well and they release it to the world.

The next phase of hard work begins for that release. Keeping the large collection of software secured and as bug-free as possible as their upstream counterparts continue to churn and release new versions. The maintainers of your Enterprise Linux distribution must carefully keep track of the ocean of changes.

The process often looks like this:

In practice, what this works out to is they freeze packages at a specific version for a long time and only ever backport fixes for the most egregious of bugs or security issues that receive a CVE.

Sounds good, right? Nope. Not for security. There’s a lot wrong with this model.

As research shows : “Despite the National Vulnerability Database (NVD) publishes identified vulnerabilities, a vast majority of vulnerabilities and their corresponding security patches remain beyond public exposure”.

Let’s focus on the example of one of the largest ongoing open source projects: The Linux kernel. According to the 2020 Linux Kernel History Report, the Linux kernel project receives about 10 changes per hour, or approximately 225 changes per day. The team behind the kernel does not treat security bugs as something special. It’s simply another bug. All bugs are equal in the eyes of the project. They do not designate a particular bugfix as security related in the commit logs. They do not publish advisories. In fact, the vast majority of CVEs found for the Linux kernel are only retroactively identified as security researchers and interested parties pore over the changes and realize that a few of those bugfixes are indeed closing security gaps. Only then do they file for a CVE for the issue.

Quite often, most identified CVEs were fixed months ago in the stable kernel release. That’s not even thinking about the huge pile of unidentified CVEs lurking in the sea of bugfixes. Multiply this problem times the total number of packages that make up an Enterprise Linux distribution and you end up with a huge vulnerability window.

The fundamental issue is that the Enterprise Linux model optimizes for stability at the expense of security. By locking packages to specific versions and only backporting select fixes, these distributions lag significantly behind upstream versions when it comes to security bugfixes, albeit unintentionally.

Defenders will counter that this is a necessary tradeoff – you can’t have both stability and being fully up-to-date on security fixes. I’m not convinced that’s true as there are other Linux distributions that have shown you can have both. Rolling release distributions like OpenSUSE Tumbleweed follow upstream much more closely while still maintaining stability through thorough automated testing . Additionally, distributions like Fedora Linux, while not technically a rolling release, do tend to hew close to upstream versions at a more regular cadence.

Even so, these type of distributions are not a panacea and operational challenges may present themselves with staying close to upstream given your application or team needs.

There does exist a middle ground, but you’re probably not going to like hearing it.

The Case for Oracle Linux

I need you to set aside your automatic (and justified) repulsion to what you just read. Yes, I know that Oracle has earned every bit of their reputation amongst engineers. I’m totally with you on that – I am not affiliated with Oracle in any way.

But stop screaming for a moment and hear me out.

The Oracle Linux project was started in October 2006 as a downstream rebuild of RHEL. They simply took Red Hat’s sources, recompiled it and put their branding on it, and sold their own support contracts for it in an attempt to undercut Red Hat’s business.

Throughout the years, they’ve continued this work and publish all their work on yum.oracle.com which is freely available. As their very own site and explanation states in the FAQ :

Q. Wait, doesn’t Oracle Linux cost money?

A. Oracle Linux support costs money. If you just want the software, it’s 100% free. And it’s all in our yum repo at yum.oracle.com. Major releases, errata, the whole shebang. Free source code, free binaries, free updates, freely redistributable, free for production use. Yes, we know that this is Oracle, but it’s actually free. Seriously.

Where it got interesting was in September 2010 when Oracle announced one optional departure for their RHEL clone: the pompously named Unbreakable Enterprise Kernel.

Oracle was now offering everyone two kernels: The Red Hat Compatible Kernel (RHCK) and their own Unbreakable Enterprise Kernel (UEK).

Over time, UEK has evolved into a unique Enterprise Linux kernel offering that more closely tracks the upstream Linux kernel. Rather than cherry picking individual fixes to backport to their kernel the way most Enterprise Linux distributions do, the model they’ve found works best “has been to lag behind the tip of the LTS tree by less than four weeks”.

They use that small lag time to perform their typical “enterprise” validation and testing process before shipping an updated UEK release.

By sticking much closer to the upstream kernel releases at regular intervals, they are able to keep up with the enormous amount of bugfixes that go into the kernel – whether they were security related or not.

It’s an interesting middle ground option for one of the more critical components of an Enterprise Linux distribution. You can rely on the operational stability of the overall distribution tooling, but still have a relatively updated (and validated) kernel.

Oracle has also publicly continued to commit to keeping “…the binaries and source code for that distribution publicly and freely available”. At this point their distribution has been around for 17 years, and they’ve never tried to restrict access to their source code in that time.

It feels wrong to speak positively about an Oracle product, but Oracle Linux is actually a good option worthy of our consideration.

Just don’t tell Larry I said that.