Architecture Weekly #173

Architecture Weekly Issue #173. Articles, books, and playlists on architecture and related topics. Split by sections, highlighted with complexity: 🤟 means hardcore, 👷‍♂️ is technically applicable right away,  🍼 - is an introduction to the topic or an overview. Now in telegram and Substack as well.

Highlights

Architecture Advise Process 🍼

Typically the architecture decisions are made by someone with "Architect" in the title. But you either get an ivory tower architect distant from the reality or an ultimate bottleneck in the process. Andrew Harmel-Law recommends to use Architecture Advisory Process decentralising making the architecture decisions. How exactly? Find in his own article on Martin Fowler's blog. Btw, the interview with Andrew Harmel-Law early access is already on my Patreon.

Scaling the Practice of Architecture, Conversationally
Scaling architecture through a structured series of conversations

#adr #architecturedecisions

Data quality pipeline with AWS Glue and S3 Tables 👷‍♂️

Data quality is the corner stone of multiple solutions and organizations, and getsupplied.ai is no exception. While looking for a proper solution, I found this guide recommending using AWS Glue jobs to split the raw data into quality data and poor data through the set of validators.

From raw to refined: building a data quality pipeline with AWS Glue and Amazon S3 Tables | Amazon Web Services
Organizations often struggle to extract maximum value from their data lakes when running generative AI and analytics workloads due to data quality challenges. Although data lakes excel at storing massive amounts of raw, diverse data, they need robust governance and management practices to prevent common quality issues. Without proper data validation, cleansing processes, and ongoing […]

#dataengineering #aws

Wrong ways to use the database or decision pendulum 👷‍♂️

Observing downsides of a particular technology - like a database - can force you to switch to a complete opposite, say picking up MongoDB after issues with scaling PostgreSQL. The problems start when the new tech is completely incapable of providing you even a fraction of necessary guarantees. But no more spoilers - grab an interesting story of one technical rewrite.

Wrong ways to use the databases, when the pendulum swung too far | LUU.IO

#db #casestudy

Still waiting on this staff engineer bump?

 Stop just shipping tickets - Start designing systems your CFO can brag about.  Business oriented system design course cohort number six goes live on 23rd of July. 10 weeks live sessions and async personalized feedback included!  70+ alumni from   booking.com, Bolt and Flo Health report a 15% average compensation increase  within six months.  11 seats remaining. New cohort starts on 23rd of July. Apply now here. 

Follow-Up

An Illustrated Guide to OAuth and OpenID Connect 👷‍♂️

OAuth might seem a bit complicated at first - I personally struggled for a while. With such concepts a clear explanation is invaluable. Find this illustrated guide to OAuth and forget about cross-service authentication struggle for good.

An Illustrated Guide to OAuth and OpenID Connect
An illustrated guide to explain OAuth and OpenID Connect!

#security

With AI, everything is complicated and it's great new 🍼

Ever heard of a Cynefin classification? It helps prioritize the domains for example in DDD; Vlad Khononov now uses it to show that AI creates a tectonic shift of domains from complicated to clear, and complex to complicated. Folow Vlad's small note.

With AI, everything is so complicated... and this is great news!
Lately I’ve been spending more and more time researching AI models and their effects on software engineering and architecture, so it’s time to share my findings. First, let me explain what I mean by “complicated.

#ddd #ai

Practical Design Patterns for Modern AI Systems 👷‍♂️

From one hand side everyone is building AI-powered systems. From the other hand side the patterns always emerged - in in OOP, in ML Ops and indeed they emerge with AI-driven systems. Find the set of patterns for improved performance, ethical output and pleasant user experience.

Beyond the Gang of Four: Practical Design Patterns for Modern AI Systems
In this article, author Rahul Suresh discusses emerging AI patterns in the areas of prompting, responsible AI, user experience, AI-Ops, and optimization, with code examples for each design pattern.

#ai

Awesome CTO List 🍼

As I am officially a CTO now, I want to make sure I don't have any obvious weaknesses for the position. I got into this list which covers a long list of topics: position expectations, hiring, architecture, technologies, finances and many more. I obviously could not get through all of this, but the list looks promising and valuable.

GitHub - kuchin/awesome-cto: A curated and opinionated list of resources for Chief Technology Officers, with the emphasis on startups
A curated and opinionated list of resources for Chief Technology Officers, with the emphasis on startups - kuchin/awesome-cto

#reading #cto

Minimizing S3 API Costs with Distributed mmap 👷‍♂️

If you bring Kafka installation to a cloud, you're going to pay triple digits a day with less than half a gigabit a sec, which is huge. WarpStream is the solution exactly to this problem: batching across all small logical streams before persisting, keeping objects inS3 big and cheap while metadata lives elsewhere and compacting cold data makes it truly cost efficient.

Minimizing S3 API Costs with Distributed mmap
We first introduced WarpStream in our blog post: “Kafka is Dead, Long Live Kafka”, but to summarize: WarpStream is a Kafka protocol compatible data streaming system built directly on top of object storage.

#streaming

Software in the era of AI 👷‍♂️

And here's the kicker: Andrej Karpathy shares some thoughts on the future of software engineering. Rethinking our approach to coding, building for agents and many more in this inspiring video.

#ai

Big thanks to Nikita, Constantin, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman, Egor, Roman, Evgeniy, Nadia, Daria, Dzmitry, Mikhail, Nikita, Dmytro, Denis and Mikhail for supporting the newsletter on Patreon!