Architecture Weekly Issue #71. Articles, books, and playlists on architecture and related topics. Split by sections, highlighted with complexity: 🤟 means hardcore, 👷‍♂️ is technically applicable right away,  🍼 - is an introduction to the topic or an overview. Now in telegram as well.

WARNING 🇺🇦

It's already been a year since Russia's crazy, brutal and unjustified war against Ukraine. We condemn this war and want it to stop ASAP. We continue this newsletter so you can advance your skill and help the millions of Ukrainian people in any way possible. If you want to help directly, visit this fund.

Big thanks to Nikita, Anatoly, Oleksandr, Dima, Pavel B, Pavel, Robert, Roman, Iyri, Andrey, Lidia, Vladimir, August, Roman, Egor, Roman, Evgeniy and Nadia for supporting the newsletter. They receive early access to the articles, influence the content and participate in the closed group where we discuss the architecture problems. They also see my daily updates on all the things I am working on. Join them at Patreon or Boosty!  

Highlights

The State of Data Engineering 2023 🍼

Data Engineering is a huge domain. That's why state reports are so valuable! Please find one here: Ingestion tools, Data Lakes, Metadata management, Analytics management and many more!  

The State of Data Engineering 2023 - Data Version Control at Scale
What are the leading tools and trends that shaped the state of data engineering in 2023? Find out the results in this year’s report.
Git for Data - lakeFSEinat Orr, PhD.

#dataengineering

Intro to Data Engineering 🍼

Speaking about Data Engineering: if the words above do not tell you much, please follow the conversation I had with Pasha Finkelshteyn, a Developer Advocate at JetBrains. He explains what data engineering is all about, what are the biggest problems there and much more!

#dataengineering #video

Building Efficient Experimentation Environments for ML Projects 👷‍♂️

Looks like the 71st Highlights are all about Data Engineering :) SwirlAI newsletter gives us a post on organizing the Experimentation Environment for Data Scientists to improve the ML models. The article looks into the properties that an efficient Experimentation Environments should have. And, as a MLOps engineer, you should strive to provide these to your users and as a Data Scientist, you should know what you should be demanding for.

SAI Notes #05: Building efficient Experimentation Environments for ML Projects.
Let’s look into what it takes for an Experimentation Environment to be efficient in Machine Learning Projects.
SwirlAI NewsletterAurimas Griciūnas

#ml #dataengineering

Follow-Up

Contract Testing Case Study 👷‍♂️

We covered Contract Testing several times. This time I would like to share the article by Ebay Engineering. The folks wanted to ensure the API provided by the Notification team was functioning well for all of it's consumers. They considered the BDD, but found some caveats and pivoted to the Contract Testing. Find out what they managed to achieve!  

API Evolution Is a Challenge. Could Contract Testing Be the Solution?
Contract testing has grown in popularity in recent years with the widespread adoption of microservice architectures. In this article, we will share our experiences with contract testing at eBay.
HomeXiaoye Wang

#api #contracttesting

Six Ways to shoot yourself in the foot with health checks 👷‍♂️

Health checks seem to be pretty simple: check if the application responds in time and connects to a database and a Kafka topic... But it's just first impression. You can easily break your application with small health check tweaks. Find out 6 ways to do that!  

Six ways to shoot yourself in the foot with healthchecks
One reason I like working at startups isyou get to wear many hats.Of course,by "wear many hats" I really mean"suffer occasional periods of extreme stress when things failand there are no grownups you can go to for help".I like to think of it as Extreme Learning.

#devops #observability

Microsoft Azure Well-Architected Framework 🍼

We covered the Amazon Well-Architectured framework previously. But obviously, they are not the only cloud provider there. It's time for the Azure Well-Architected Framework, which focuses on Reliability, Security, Cost Optimization, Operational Excellence and Performance Efficiency.

Microsoft Azure Well-Architected Framework - Azure Well-Architected Framework
Learn about the five pillars of the Azure Well-Architected Framework and how they can produce a high quality, stable, and efficient cloud architecture.
Microsoft Learnmartinekuan

#cloud #architecture #bestpractice

The Full Circle on Developer Productivity 🍼

Steve Yegge used to work at Amazon, Google, Grab and now working at Sourcegraph. He shares his long journey and discusses why the developer tooling is so important and why it's so hard to make it right. I couldn't stop reading!

The Full Circle on Developer Productivity with Steve Yegge
Where does Steve’s passion for developer tools come from? He talked through his career and learnings, and threw in a few stories that hadn’t been previously shared anywhere.
The Pragmatic EngineerGergely Orosz

#philosophy

Securing the API access 👷‍♂️

When you expose the API you need to protect it from the malicious access. The first idea is to use some type of a token, but it comes with the problems of invalidation, limiting the scope and many more. OAuth tokens combined with JWT yields more efficient solution without comprising the security and even solving the mentioned problems. Follow the Zapier's blog for further info!  

API Keys vs OAuth Tokens vs JSON Web Tokens
For an API to be a powerful extension of a product, it almost certainly needs authentication. By building API calls that can read, write, and delete user data, you can magnify an app’s influence on its users’ lives. So, if authentication is a given, the method is the real choice....
ZapierZapier

#api #security

Mock System Design Interview: Video Portal 👷‍♂️

Another bit of content from my YouTube Channel: mock system design interview. Here we are architecting a simple video portal, which is capable of uploading the videos, converting it to different qualities and stream it to mobile and web clients. Follow the video for the designing process!

#video #mockinterview #systemdesign