Recovering from a Bad Ignition Config on Fedora CoreOS

Fedora CoreOS is deliberately unusual: there’s no traditional console login to fall back on, no default password, and essentially no configuration happens interactively. Everything is set up once, at first boot, by an Ignition config – SSH keys, users, systemd units, filesystem layout, the lot. We covered how that process works in provisioning CoreOS with an Ignition config.

That design is a big part of why CoreOS is reliable and reproducible – but it also means that if your Ignition config has a mistake in it, you can end up with a server that’s running, reachable on the network, and completely inaccessible to you: a malformed SSH key, a typo’d username, or a broken systemd unit that never lets a service start. This post covers how to avoid that, how to diagnose it if it happens anyway, and what your realistic options are once it has.

The core problem: Ignition only runs once

Ignition does its work during the very first boot and then it’s done – it isn’t a configuration management system that reapplies your config on every boot the way something like cloud-init’s later stages or a config management agent might. If the config was wrong, the mistake gets baked in at first boot, and there generally isn’t a mechanism to “reapply” a corrected version to an already-provisioned instance. This is exactly why validating your config before you ever boot a machine with it is so important – it’s the point in the process where a mistake is cheap to fix.

Validate before you boot, not after

If you’re writing your config in Butane (the human-friendly YAML format that compiles down to Ignition’s JSON), run it through butane with strict mode before you use it anywhere:

butane --strict --output config.ign config.bu

--strict will refuse to compile the config if it contains anything Butane considers invalid or risky (like validation warnings), rather than silently ignoring them. It’s a good habit to make this non-negotiable in any provisioning pipeline.

It’s important to understand what this does and doesn’t catch, though. butane --strict validates syntax and schema – that your YAML is well-formed, that fields are the right type, that you haven’t used an option that doesn’t exist. It cannot tell you that the SSH public key you pasted in doesn’t match the private key you actually have, that you meant to type ubuntu and typed ubunt, or that a systemd unit you’ve defined references a file path that will never actually exist on the running system. Those are logical mistakes, not syntax mistakes, and no validator can catch them for you – only careful review, and ideally a test run, can.

Two practices help here:

• Boot a throwaway test instance with the exact config you’re about to use in production, confirm you can actually SSH in and that your units start correctly, then destroy it. This costs a few minutes and catches almost everything.
• Double-check the SSH key you’re embedding is the public key, copied without truncation, and that it corresponds to a private key you actually hold – a single dropped character is enough to lock you out permanently.

If you’re already stuck: console access

If you’ve already booted an instance with a bad config and can’t reach it over SSH, your first move is whatever console or serial access your VPS provider offers – this is typically a web-based console (sometimes called VNC console, serial console, or “recovery console” depending on the provider) that gives you a view of the machine’s boot output and, in some cases, an interactive terminal independent of the network.

What you can usefully do from there depends heavily on what actually went wrong:

• If the machine boots fully but SSH access is broken (wrong key, missing user), console access may still get you a login prompt if a password or alternate access path exists – but on a stock CoreOS Ignition setup, there typically isn’t a fallback password login configured, since SSH keys are the intended access method. In that case, console access confirms the diagnosis (system’s up, network’s fine, it’s specifically the credential that’s wrong) without actually getting you in.
• If a systemd unit from your config is failing, console output during boot may show you exactly why – a missing file, a bad path, a syntax error in the unit itself – which is useful for correcting the config for your next attempt even if it doesn’t get you into this instance.
• Check whether your provider offers a way to attach the disk to a rescue environment or a second instance. This is the one path that can sometimes let you directly edit files on the broken instance’s filesystem – mounting its disk elsewhere, correcting an authorized_keys file or a unit file, then reattaching it. Not every provider supports this for every plan, so check what’s actually on offer before assuming it’s an option.

How this differs from a failed OS update

It’s worth being clear about a distinction that trips people up: CoreOS does have a genuine, well-designed automatic recovery mechanism, but it’s for a different problem than a bad Ignition config.

CoreOS auto-updates itself using rpm-ostree, which manages your OS as a set of versioned, atomic deployments rather than patching files in place. When an update is applied, rpm-ostree creates a new deployment and keeps the previous one intact – it doesn’t delete it. That means a straightforward manual rollback is always available if a newer OS deployment turns out to be broken:

rpm-ostree rollback

On top of that, CoreOS’s bootloader supports boot counting, and health-check tooling like greenboot can act on repeated boot failures automatically – after a set number of failed boots on the new deployment, the bootloader falls back to the previous, known-good one without anyone needing to intervene. Note that greenboot is not installed by default on Fedora CoreOS itself (it ships by default on Fedora IoT) – you need to add it yourself with rpm-ostree install greenboot greenboot-default-health-checks if you want this protection.

The key difference is what’s being rolled back. That mechanism protects you against a bad OS update – the previous deployment is right there, known-good, and switching back to it is a well-defined operation. A bad Ignition config isn’t an OS deployment at all; it’s the one-time setup that ran before any of this versioning existed. There’s no “previous Ignition run” to roll back to, because Ignition only ran once, at the very start, and whatever it did (or failed to do) is simply the state the machine is in.

The practical fix: redeploy, don’t repair

If your initial Ignition config was wrong and you can’t reach the instance through any of the options above, the pragmatic answer is usually to stop trying to repair the running instance and provision a new one with a corrected config instead. This isn’t a failure of troubleshooting – it’s the intended model. CoreOS instances are meant to be treated as disposable and reproducible: fix the config file, validate it properly this time, and redeploy.

This is much less painful if you’re already managing your Ignition/Butane configs as version-controlled files rather than something typed once into a provisioning form and forgotten – if that’s not already how you work, it’s worth adopting after an incident like this, since it turns “redeploy with a fix” into a fast, low-drama operation instead of a scramble to remember what the original config even said.

Summary

Ignition’s one-shot-at-first-boot design is what makes CoreOS predictable, but it also means a broken config generally can’t be patched on a live instance the way you’d fix a traditional server. Validate with butane --strict, test-boot configs before using them for real, keep them version controlled, and treat console access as a diagnostic tool rather than a guaranteed way back in. When the initial config was genuinely wrong, redeploying with a corrected version is normally faster and more reliable than trying to reach into a stuck instance – and it’s exactly the workflow CoreOS is designed around.