I spent 3 days at GOTO Berlin conference and it was amazing. Picked up many interesting subjects, exchanged ideas with others and had so much fun. I am surely going back to Düsseldorf smarter, wiser and humbler. Here are a few good thoughts I picked up along the way – part 1/2
Building ‘good enough architecture’
The number of tools a software engineer has at her disposal is overwhelming. This can easily cause her to lose track of what she is trying to achieve. Here’s a story:
Sometimes back, we had to build a reporting pipeline. The goal was for our product owner and the leadership team to get metrics on how many records pass-through one of our data pipelines. We took our inspiration from a Netflix blog post which involved scheduled Jupyter notebooks which would interact with a database. The notebook was orchestrated by an airflow server and the reports synced to an S3 bucket. It was amazing to work on. It was very customizable and one of our selling points was that the final report was interactive.
When we presented our progress in a sprint review session, someone from the leadership team challenged us why we had to build a whole pipeline and why not use an already available reporting platform like Looker. We already had several reports on Looker. I made a long speech about how customizable the pipeline is and how interactive the report is.
Now when I look back, our Product Owner just needed the number and they did not care if they had an interactive report built using bleeding-edge tech. We spent several days building this pipeline, but we could have made the same impact on our stakeholders in a few hours using a pre-existing platform.
Moral of the story: Everyone wants to be Netflix, but you are not into the video streaming business with millions of concurrent users.
The systems you build WILL be dynamic
While designing systems or pipelines, you can spend all your effort and brainstorm to come up with all the possible use cases. But, the requirements will be dynamic. New use cases will come in and throw a wrench into your roadmap. This is the norm.
The best course of action is to build the simplest thing that will work. BUT, make sure you build evolvable structures. You should design your systems in a way that any new requirement which comes in does not push you back to the drawing board. The systems you build will continue to evolve, and you should be on top of it. The right amount of governance can guide evolution in the right way.
As David Patterson and John Hennessy wrote in Computer Organization and Design, one of the great principles of computer architecture (also software engineering) is to “make the common case fast”
Making the common case fast will tend to enhance performance better than optimizing the rare case. Ironically, the common case is often simpler than the rare case. This common sense advice implies that you know what the common case is, which is only possible with careful experimentation and measurement.
David Patterson and John Hennessy
Focus on the common case and build the simplest solution that works. Iterate later.
Mars Explorer and the lessons it can teach us
I learnt about the story of NASA’s Mars orbiter. The orbiter was lost and the reason was a mistake in how the two parts of the system communicated. The software that calculated the thrust and the software that calculated the trajectory, used entirely different units. There are two major lessons that we can learn from this.
When you have several microservices which talk to each other, you should make sure that the contracts for each component are defined. This also gives the freedom to change your implementation, while you make sure the contracts are not broken.
The importance of integration tests is not well understood! If there are two services which work together, build an integration test. Think of the integration test as the little angel on your shoulder who tells you when you are about to do something wrong. If all the systems involved in the Mars orbiter project were integration tested, things would have turned out different.
Hello Ruby and our kids growing up with computers
Our kids are growing up in a world where they will interact with computers much more than us. Linda Liukas gave a keynote where she made a point that teaching kids how computers work should be part of primary school education. Last month, my 4-year-old daughter asked me how Amazon Alexa works. I’m sure more questions are coming.
Linda talked about a few illustrated books about computers and the internet designed for primary school kids. It was amazing to see how she can connect with children to teach them about basic concepts of the Internet, CPU, Input and Output.
CI, CD and Resume Driven Development
A colleague of mine challenged someone when they said they used CI-CD at their previous role. He said CI and CD are two separate concepts and when we asked more, we realised what they really did was Continuous Integration (CI) and not Continuous Deployment (CD). Both are different concepts and have separate use cases.
Many of us talk about how dynamic we are, releasing multiple versions of our software to production every day. I do not mean to say this is a bad thing, not at all. I mean to say that you may not need it. If you are in a business which needs you to constantly make changes and innovate, and you have all the resources to do so, then it makes sense. But I feel we should ask ourselves if we create value for our users by doing so.
You may be in a situation where all you need is CI, where you want to ensure the quality of code being pushed to the master branch. The deployment may not be done so rapidly and can have some manual intervention. Just don’t fall into the trap of using CI & CD just because it looks fancy on your CV.
This was highlighted by Ken from ThoughtWorks and Continuous Integration is not a new concept as we all think.