Skip to the demos
Home
TRACK

The work, as postmortems

A résumé bullet says what was built. A postmortem says what was wrong, how anyone found out and what it cost — which is the part that tells you whether someone understands their own work.

  1. sev1 · customer visibleDell · 2024

    Bringing a customer online took days

    impact
    New infrastructure could not be used until someone had walked it through provisioning by hand. Customers waited days to touch hardware they had already paid for, and every wait was somebody's afternoon.
    detection
    It was never hidden. Onboarding time was a number the business quoted to customers, and it was the number nobody wanted to quote.
    root cause
    Provisioning was a sequence of manual steps across systems that did not speak to each other. Each step was small; the handoffs between them were where the days went.
    mitigation
    Zero-touch provisioning: React workflows over Go APIs, with NATS JetStream carrying the events between stages so each step began when the previous one finished rather than when a person noticed it had.
    resolution
    Days to under two hours.
    what I take from it
    The engineering was in the seams, not the steps. Almost none of the elapsed time was work being done — it was work waiting to be noticed. Worth remembering the next time a process looks slow: measure the gaps before optimising the tasks.
  2. sev2 · degradedDell · 2024

    The asset table fell over past a few thousand rows

    impact
    Operators managing large estates watched the interface stutter and stall on the one screen they used most. The data was correct; it simply could not be looked at.
    detection
    Reported as 'the page is slow', which is where most performance work starts. Profiling put it on the DOM, not the network.
    root cause
    Every row was a real element. Ten thousand assets meant ten thousand rows and everything the browser has to do for each of them, on every interaction.
    mitigation
    A virtualized grid built as a custom element with its own shadow root, keeping a recycled pool of row elements so the document holds a windowful regardless of the table's length, with RTK Query owning paging and caching behind it.
    resolution
    Smooth interaction past 10,000 assets, with the DOM flat.
    what I take from it
    Shadow DOM was the decision I would defend hardest. Sealing the grid's internals off meant its performance characteristics stopped depending on whatever the surrounding application did to its stylesheet.
    Run this one
  3. sev1 · customer visibleDell · 2025

    Two operators could see two different truths

    impact
    State changed underneath people. Two operators looking at the same estate could act on different pictures of it, and the only fix a user had was to refresh and hope.
    detection
    Surfaced as intermittent 'wrong data' reports — the kind that cannot be reproduced on demand because the bug is a race, not a value.
    root cause
    Views were fetched, not subscribed. Once rendered, a page had no way of learning that the world had moved on.
    mitigation
    Real-time state synchronisation: WebSocket delta updates applied into Redux Toolkit, so a change reaches every connected client as a small patch rather than a re-fetch, across multi-tenant environments.
    resolution
    4,000+ concurrent live-sync connections.
    what I take from it
    Deltas are the easy half. The half that matters is what happens when a client misses one — a version on every mutation, so a client that has drifted knows it and can ask to be repaired rather than quietly rendering the past.
    Break this one
  4. sev2 · degradedDell · 2025

    The database was answering the same question all day

    impact
    APIs were slower than the work they did justified, and the database carried load that had nothing to do with new information.
    detection
    Visible in query volume: the same expensive reads, repeatedly, for data that changed far less often than it was asked for.
    root cause
    Caching existed in places, added where someone had noticed a problem. Different TTLs, different key conventions, and no shared idea of when an entry stopped being true.
    mitigation
    A unified caching architecture on Redis: dynamic keys, configurable TTLs, and invalidation driven by dependency and by event rather than by timeout alone, with warming for the entries that were always going to be asked for.
    resolution
    Redundant database workload removed; API responsiveness improved.
    what I take from it
    Time-based expiry is a guess about how long something stays true. The work was in replacing that guess with a statement of what each entry depended on — after which invalidation stops being a judgement call.
    Break this one
  5. sev3 · internalDell · 2025

    Nobody trusted the test suite

    impact
    Pre-merge feedback was slow enough to context-switch away from and flaky enough to re-run on faith. A suite in that state stops being a signal and starts being a toll.
    detection
    The tell was behavioural: people re-running failed jobs without reading them.
    root cause
    A framework that made the expensive choice the default, with no way to select the subset of tests a change actually implicated.
    mitigation
    Rebuilt the end-to-end framework and migrated it from Cypress to Playwright: stubbed pre-merge runs against live post-merge validation, environment-specific configuration, tag-based selection so a change runs what it affects, and Jenkins orchestration.
    resolution
    Feedback fast and selective enough to be read rather than re-run.
    what I take from it
    The migration was the visible part and the least important one. What changed the culture was tag-based selection: when a suite only runs what your change touches, a red result is about you, and people start reading them.
  6. sev3 · internalDell · 2025

    Every capability shipped on everyone else's schedule

    impact
    Independent teams' work was welded into a single deployable. One team's readiness set the release date for all of them.
    detection
    Release coordination overhead — the meetings that exist only because a build cannot be split.
    root cause
    A single application shell that compiled every capability into itself.
    mitigation
    React micro-frontends over Webpack Module Federation, so independently developed capabilities compose into a shared shell at runtime instead of at build time.
    resolution
    Capabilities that ship on their own schedule.
    what I take from it
    Runtime composition moves a class of failure from build time to production, which is a real cost and worth naming. It buys independence, and it obliges the shell to degrade gracefully when a remote does not load.