Introduction
As a Solution Architect I lost a $1,000,000 client while working for software service company because of a wrong decision.
We were creating an order management system for clothes replenishment. From excel and mails, we had to transition to a proper system with roles, statuses etc. The problem was that I underestimated the load of the system and failed data modelling, and picked the wrong database. Half a year after the inception of the project the client fired us, and the service company lost more than $1,000,000 of potential profit.
So, architecture decisions matter! Today we discuss the biggest challenges in making architecture decisions.
Challenge 1: Lack of expertise
Let's say you start with a new project. One of the many decisions yet to make is the database. In JetBrains’ 2022 Developer Ecosystem survey, for example, 69% of respondents used 2 or more databases concurrently, meaning though about 31% worked with a single database. A single database experience makes it impossible to make a well-weighted informed decision.
Of course, going PostgreSQL or Mysql or any other battle-tested database is not a bad choice for the first couple of years. However this strategy is closing your eyes and hoping for the best rather that making a rational decision. It may render false if the situation requires prioritising flexibility over schema strictness. Sure, you can just emulate a document-storage with JSON fields in Postgres - if you know about it and aware how to work with it. It is frequently not the case.
Solution
Never stop learning. Learn the different storage options and related tradeoffs. Read appropriate books. Study system design cases. Study papers. Grow!
Challenge 2: Wrong Focus
You just landed a new job and joined a new team. You clone the main repo and immediately jump into conclusions: lack of integration tests, suboptimal code quality, custom validation of endpoints input instead of formal schemes. Who the hell wrote this code?!
You start convincing the team and the Engineering Manager the team absolutely needs to address the technical debt right away. Every team member nodes in agreement, but do not express any excitement. I saw that many times in my career and the reason is always the same: a new member just lacks the story and context to make right conclusions.
Indeed, part of the problem might be engineers never cared or knew about convenient library and tools. However the real reason is that technical debt is always a trade-off. Overinvestment in technical excellence before understanding final picture can bite you direly. Sometimes it's an overkill for a fully-blow solution as they come with their own complexity too.
Solution
Ask for project priorities and project history. How it was setup? What was the original goal? How the project roadmap looks like? What is the most impactful thing you should work on right now?
Challenge 3: Lack or misunderstanding of the requirements
How many times you saw people going with micro services from the day 1? Or starting with Kubernetes cluster instead of single VM or at least a managed container service? The internet is full of such stories like "I went with Kubernetes and ruined my startup as a CTO".
This is not the story of CV-driven development. It is the story of misunderstood requirements. Engineers will tell you they need scalability or they have to go paranoid security. It's not a bad faith: it is their previous experience talking ignoring actual business requirements at the same time.
How many customers you expect? How the load will change over time? Is the security the primary concern instead of velocity let's say?
In some cases you will know the answers, in some you won't. It's ok - as long as you make reasonable assumptions.
Solution
Talk to product people. Ask them how the business should look like in 3-6 months. How the load would look like? How paranoid you should be about data? Maybe you need to pass an audit, how to prepare for that? Make yourself a business person, not a coder.
Challenge 4: Insufficient analysis
Any decision has consequences and bear risks. Even going with the most traditional choices like PostgreSQL bear the risk of write amplification problems . It also have the consequences like suboptimal horizontal sharding.
Another example of missing consequences analysis is the cost calculations: alright, you decided to use Fivetran as your data ingestion mechanism because of hundreds of ready integration. But how much you are going to pay for it? Does it worth it in your particular business case?
I observe made decisions in the ADRs lacking this analysis.
Solution
Put a research effort into such analysis. Adopt the discipline and dedication to making well informed decisions.
Challenge 5: Emotional attachment
Olaf Zimmerman highlights that engineers attach themselves to their technical choices. I see this in practice too: people will literally put only the pros to their preferred solution and only the cons to all the other options.
While this behaviour is expected from the meatbags, the things never work this way: any solution have pros and cons, and making decisions is always about the tradeoffs.
Solution
Be honest to yourself and your colleagues. If you don't see any cons, then you lack information: make deeper research and/or ask for the feedback. Find out what you're missing. Be aware of your emotions.
Those are the biggest challenges of making architecture decisions. Do you have any in your mind? Let me know!
Business Oriented System Design Course Cohort #6 is officially open!
Looking for a way to advance your career? Felt you overgrew the mere feature development, but lack skills to design complete systems? This course got you covered. 10 hours of content packed lectures, engaging practice and the final work you will be proud to showcase as well as Credly(by Pearson)-based digital certificate proving your experience. More than 70 engineers already passed the course with amazing feedback and advanced their careers. New cohort starts on 23rd of July. Details, Feedbacks and Enrollment into the course is here.
