Architecture Weekly Issue #38. Articles, books, and playlists on architecture and related topics. Split by sections, highlighted with complexity: 🤟 means hardcore, 👷♂️ is technically applicable right away, 🍼 - is an introduction to the topic or an overview. Now in telegram as well.
It's already been 221 days since Russia's crazy, brutal, and unjustified war against Ukraine. We condemn this war and want it to stop ASAP. We continue this newsletter so you can advance your skill and help the millions of Ukrainian people in any way possible.
Video version of this issue is available on YouTube. Subscribe!
Reducing Logging Cost by Two Orders of Magnitude using CLP 🤟
The more your business grows the more you need to understand what happens within your system. With Uber and tons of analytical data and therefore log data generated each day, it became an issue. They need to retain a sufficient amount of logs to understand what happens within their Spark jobs and not pay for it too much. So they used a Compressed Log Processor, which was able to reduce the number of writes to SSD and store the logs in a searchable manner but occupying two orders of magnitude less space. More details inside!
#logging #costoptimization #bigdata
Complete System Design Series 🍼
I stumbled upon a good series on System Design on Medium. Naina Chaturvedi makes the introduction to what System Design is, and proceeds to scaling, load balancing, database sharding and many more. She also has a series of posts about designing real-life services like Dropbox, Instagram, Web Crawler and many more.
Distributed Architecture Concepts 🍼
Gergely Orosz is famous for his "The pragmatic engineer" newsletter, but it started small and gain its first pick with an article on distributed architecture concepts. He explains what are SLAs, scaling, consistency, data durability and more, he learned during his work at Uber.
End-to-end field-level encryption for Apache Kafka Connect 👷♂️
In some markets and business domains, you will face high requirements for security and privacy. If you use Apache Kafka for data processing, you may find yourself in a situation where you can't allow PII data into the streaming platform. One of the solutions would be to use e2e field-level encryption. Please find an article by Hans-Peter Grahsh from Red Hat on how it can be realized.
How Block Unified on One Graph 👷♂️
GraphQL is a technology which provides API clients to control what fields they want to query. Block(previously Square) adopted GraphQL for different parts of their systems, including data about merchants, devices, payments and more. They decided to unify those APIs under a single Federated Graph. Find a story about how they achieved that.
No-code permissions with Kong and Permit.io 🍼
Kong API is one of the popular API Gateways out there. And as with such systems, you always want to check if an API user can call a particular API. From that perspective, a solution called Permit.io can help, as they provide a platform for easy permissions management. Nowadays, you can combine those two to minimize coding effort for policy management for your API. A short guide on how to configure the two together inside.
#security #api #nocode
Are you spending too much on Kubernetes? 👷♂️
Kubernetes is pretty widespread in modern distributed systems. Part of the architect's job is to improve the cost efficiency of the solutions, e.g. not spending too much on the infrastructure. Denilson Nastacio from IBM wrote about six considerations that you might find useful and thought provocative about where the cost comes from within a Kubernetes cluster.
#k8s #cost #bestpractices
Discord supercharge for network disks for low latency 🤟
Discord runs 4 billion messages each day. They want to be reliable and fast. Discord also uses GCP for its hosting, and local SSDs do not fully fulfil their requirements. Please find out how they created their own write-through cache to achieve their goal.
StackOverflow architecture 👷♂️
This is an old, but very well-aged post on how the biggest programming(and not only programming) Q&A site works. Despite expectations of heavy caching, microservices and other stuff, they manage to do the job with a pretty basic setup. Read the whole story down below!
Logging and monitoring solution for Azure exercise 👷♂️
Logging and monitoring solution is a must-have part of every production software system. Near 30% of the real-world systems running in the cloud do it on Azure. In turn, Azure has a good exercise on how to build a proper monitoring solution for your Azure resources. Totally recommend passing if you happen to work in the cloud by Microsoft.
#azure #cloud #training
Like the newsletter? Consider helping to run it at Patreon or Boosty. Patrons and Boosty subscribers of certain levels also get access to a private Architecture Community. Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel and Robert for already supporting the newsletter.