When Perfect Isn’t Possible: How An Engineer Solved a Complex Engineering Challenge
Phebe Tememu knows software engineering isn’t always about building the perfect solution. More often, it requires smart trade-offs, adapting to constraints, and delivering reliable systems that work in the real world. In this conversation, she reflects on solving a complex multi-tenant architecture challenge, the lessons she learned about balancing engineering ideals with business realities, and the habits every aspiring engineer should develop.
You’ve been an engineer for a few years now. Walk me through a recent challenge, from the first moment you noticed something was wrong to the moment it was resolved.
I joined a project where the architectural decision was already set before I came on board. As I got into the work, I realised the base architecture could have been done more efficiently, but we’d already gone too far to start over. Redoing it from scratch wasn’t a real option, so the real question became: how do I get what we need while working around the architecture that already existed?
The first shift had to be mental. I used to have a rigid mindset — not very open to change. But the job is really about accepting that things won’t always go the way you planned, and being able to say, “this isn’t working, let’s try something else,” without getting emotionally attached to your own implementation.
Once I got past that, research did the rest. Google is your friend. I don’t think there are many truly unique problems in this field. Testing helped too: instead of committing to a full implementation, I’d abstract a small piece, test it, and only build it out further once I knew it held up.
What I landed on wasn’t the most elegant solution; it was a workaround. I chose a solution that worked immediately and deployed it. Early in my career, I was taught strict rules about how code should look and how implementations should be structured. Those rules are useful guides, but sometimes constraints mean you take the solution that works now, ship it, and iterate later.
Can you describe the actual system involved well enough that another engineer would recognise it?
The architecture was multi-tenant with a fully isolated, multi-database setup: every tenant had their own database, and a super-admin needed oversight across all of them.
The problem is obvious once you say it out loud: the super-admin can’t efficiently query ten different databases with a thousand transactions each. You’d be opening that many connections and burning memory just to pull a unified view.
Instead of querying each tenant database individually, transaction events were streamed into a central master database. This allowed the super admin to view all transactions from a single source while preserving tenant isolation. The main engineering challenge became ensuring every transaction was reliably written to both databases, with retry mechanisms and dead-letter queues handling failures.
Was there documentation or a specific resource you leaned on for this?
Actually, no formal documentation; it was a Reddit thread. I’m a big fan of Reddit for this kind of thing, because more often than not, someone has already run into your exact problem and either solved it or linked to something that gets you most of the way there. It’s less about learning something brand new and more about being reminded, “oh, I can actually do it this way.”
What did solving this actually cost you: time, performance, complexity, or something else?
I care a lot about elegant code, so the honest answer is: it felt hacky, and that bothered me more than it probably should have. The implementation itself didn’t take long once I knew the shape of the solution, but there was repetitive code, the kind of duplication you’re generally taught to avoid.
There’s also a version of this where that duplicated data becomes useful later; for analytics, for training models, whatever comes next. But at the time, its only job was letting the super-admin see everything in one place without querying every tenant database by hand. I had to be okay with building something narrowly useful now rather than holding out for something more broadly reusable later.
Were you working alone, or with a team?
I started essentially alone; the one teammate on the project before me had resigned. By the time others joined, I’d already finished the core implementation, so they built on top of it rather than shaping the direction themselves.
So there wasn’t much room for a different take, since the groundwork was already done?
I wouldn’t put it that way, because I gleaned ideas from others. People outside the project looked at it and confirmed the implementation did what it needed to do. That was enough validation for me to move forward.
What did the two engineers who joined later actually contribute?
Both were full-stack, but one leaned front-end. I owned the backend; the other built the customer-facing side. There were also a few endpoints I handed off, since I trusted them to take that piece while I stayed focused on the core implementation.
Is there anything about this project you’re still not fully happy with?
It’s less about being happy with it and more about whether our assumptions held up. Software rarely ships in its final form; you go to market with a set of assumptions, some turn out right, some don’t, and you adjust. I’m naturally engineering-focused rather than business-focused; I can spend hours making something look a particular way, and then take it to market and realise nobody cares how much craft went into it — they just want something that makes their life easier. That’s a lesson I keep relearning.
Was there a point where things got worse before they got better?
Constantly. This project connected to an external IoT device — an EV charger — and I hadn’t worked with hardware talking to a backend system before. Testing that connection meant using the actual charger in our office. It stopped working. The CEO was, understandably, not thrilled, and I wasn’t allowed to touch it for weeks afterwards. “Move fast and break things” doesn’t really apply when the thing you might break is an expensive external system.
Looking back, is there something you said or believed in the moment that you now think was wrong?
Payment integrations are always nerve-wracking, because so much can go wrong on the other end of an external gateway. I used to assume that if a payment gateway went down, the fix was simple: fail over to a different one. It’s not that simple.
Handling payments taught me that success isn’t just about the happy path. Payment gateways can fail in unpredictable ways, so you need safeguards like webhooks, reconciliation, circuit breakers, and idempotency keys to prevent duplicate transactions and maintain consistency.
I also had to get more comfortable thinking about the customer’s experience, not just the backend logic. As a backend engineer, my instinct is to reason about the system. But the customer just wants to know: did my payment go through or not?
What’s one piece of advice you’d give a newer engineer, based on all this?
Don’t get stuck on a single implementation. You can spend hours on something and eventually realise it isn’t what it needs to be, and you have to be okay with that. It’s frustrating, especially if you’re inheriting someone else’s architecture and assuming they should have done it differently. I’ve learned to assume there was usually a reason. Be ready to pivot, and research constantly — the internet almost always has an answer close to your situation.
Did this experience change how the team works now — code review, testing, monitoring?
When I joined, we were shipping straight to main with no review process. One of the things I put in place was a peer-review workflow: pull requests, a second set of eyes, then approval before merge. Two people catch more than one.
It also reshaped how I think about architecture generally. You might not know why an earlier engineer chose data isolation over a shared database; maybe it’s a compliance requirement, maybe it’s an industry standard, like the extra isolation healthcare data often needs. A shared multi-tenant database would have let us query everything directly with no streaming layer at all. But there’s usually a trade-off behind a decision you didn’t make yourself. Trade-offs are your bread and butter. Why are we doing this? What are we losing? What are we gaining?
Phebe studied Quantity Surveying at Obafemi Awolowo University, has been a software engineer for three years and has been at Revocube for almost a year.