Architecture Weekly #79
Architecture Weekly Issue #79. Articles, books, and playlists on architecture and related topics. Split by sections, highlighted with complexity: 🤟 means hardcore, 👷♂️ is technically applicable right away, 🍼 - is an introduction to the topic or an overview. Now in telegram as well.
WARNING 🇺🇦
It's already been a year since Russia's crazy, brutal and unjustified war against Ukraine. We condemn this war and want it to stop ASAP. We continue this newsletter so you can advance your skill and help the millions of Ukrainian people in any way possible. If you want to help directly, visit this fund.
Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman, Egor, Roman, Evgeniy, Nadia, Daria and Dzmitry for supporting the newsletter. They receive early access to the articles, influence the content and participate in the closed group where we discuss the architecture problems. They also see my daily updates on all the things I am working on. Join them at Patreon or Boosty!
Highlights
HTTP/3 from A to Z: Core Concepts 🤟
We as an industry just standardized HTTP/2 in 2015, and yet we're doing HTTP/3. Turns out the performance promises of version 2 didn't really make in the implementation, so instead we decided to completely change the attitude and reconsider the foundation HTTP resides on: the transport protocol. Meet QUIC: a transport protocol built on top of UDP. It is reliable and secure but quick! You will find out exactly how in the awesome article in Smashing Magazine.
#web #protocols
Performance isolation in a multi-tenant database environment 🤟
The blog post from Cloudflare discusses the challenges of maintaining performance in a multi-tenant Postgres database environment. They faced issues with throughput and latency due to tenants overusing shared resources. Their initial solution involved manually limiting server-side connections, but it was time-consuming and didn't prevent resource-intensive queries. To improve this, they implemented connection throttling at their database proxy server, PgBouncer, allowing them to control and kill existing connections. They're also considering future solutions like congestion avoidance, tenant resource quotas, and gateway query queuing to further enhance performance isolation.
#performance #db
Cost Efficiency in Startups 🍼
When you begin a startup, you rarely care about infrastructure cost, signing up for SaaS services, using clouds extensively etc. This is justified, as you need to find the product market fit. But once you gain some traction, it can become a pain, as you need to be cost-efficient to improve unit economics. Find an article in Martin Fowler's blog on the how the technological cost can be a bottleneck in the growing startup.
#startups #costefficiency
Follow-Up
AWS Lambda Enhances Support for Larger Container Images 🤟
AWS Lambda has unveiled support for deploying container images up to 10GiB, enabling customers to utilize larger code bases and dependencies. To achieve this while maintaining performance goals, AWS developed a specialized storage and caching system. This system integrates caching, deduplication, convergent encryption, erasure coding, and block-level demand loading. Since its inception, it has successfully processed hundreds of trillions of Lambda invocations for over a million customers, showcasing robust resilience. This advancement marks a significant step in providing more flexibility and efficiency in serverless computing.
#aws #serverless #paper
MLOps: Building a Feature Store 👷♂️
In the realm of machine learning, data scientists were caught in a cycle of repetitive tasks, slowing their progress. The hero of this tale, the Feature Store, emerged to streamline these processes. It was a magical tool with three key components: Transform, Store, and Serve. The Transform layer processed data into features, the Store layer safeguarded these features and their metadata, and the Serve layer made the stored features accessible. But the Feature Store was more than a simple tool; it was a complex system offering a variety of functionalities. It validated and joined features, created datasets, output data in suitable formats, performed temporal operations, visualized and monitored features, recommended features, and governed them to prevent chaos. Find the illustrations inside!
#ml #devops
Pfizer and their Serverless Architecture on AWS 👷♂️
Pfizer has enhanced its serverless architecture on AWS for processing digital biomarker data, making it more adaptable and configurable. The new framework uses AWS Step Functions and other serverless services for a file processing pipeline, along with a custom Python package for data ingestion and processing. Digital biomarkers, which are quantitative measures of physiological and behavioral data, require a scalable solution for processing. Pfizer's improved architecture supports multiple data sources and different algorithms. They also developed a specialized Python package, SciKit-Digital-Health (SKDH), for computing digital biomarkers. The updated solution uses AWS services like S3 for storage, DynamoDB for metadata, AWS Step Functions for process orchestration, AWS Batch with Fargate for data processing, and AWS SQS for messaging. The architecture includes two processing workflows for scanning new configuration files and processing study data and metadata files.
#casestudy #aws #serverless
What is Change Data Capture? 👷♂️
The blog post from Fivetran introduces Change Data Capture (CDC), a real-time data transfer method that improves upon the traditional ETL (Extract, Transform, Load) process. CDC extracts, transforms, and loads data almost simultaneously, offering benefits like real-time operations, reduced system resource impact, faster database migrations, and synchronization across multiple data systems.
CDC can be implemented in various ways, including log-based, trigger-based, timestamp-based, and difference-based methods, each suitable for different database and data processing needs. Fivetran provides a data integration solution built around CDC, enabling efficient data movement from various sources into cloud destinations.
#data
Reverse Engineer iOS App 🤟
iOS App Developers frequently think that iOS as an Operating System is secure enough and skip on applying the proper security controls on their apps. In order to fight that perception I am sharing an article showing how one can reverse engineer an iOS app, extract the secrets, understand the logic and repack the application. Toolset included!
#ios #security
What Being a Staff Developer Means at Shopify 🍼
And the last one for today is the article by Shopify about the role of Staff Developer. Technical excellence, leadership, business orientation and other attributes of the role inside!