umesh k singla

SRE skills, goals, culture: Notes from Google's SRE Book (#1)

I will be trying to put my notes from the famous Google SRE book whatever I felt important from a section or a paragraph. I didn't care for similar kind of notes or summaries which may be available on the internet. This is just for my motivational purposes to help me continue reading and post some of my learnings here.

The plan is to have a post each week for 10 weeks in a row. Let's see how it works out.

The following is the first set of my notes from Sep 30:

50-60% are SEs and 40-50% are close to SEs with 85-99% of the skillset required, and who in addition had a set of technical skills that are useful to SRE but is rare for most SEs …

- common additional skills for SRE:

UNIX system internals
networking (later 1 to 3)

- - -

the team tasked with managing a service needs to code or it will drown. Google places a 50% cap on the aggregator “ops” work for all SREs — tickets, on-call, manual tasks, etc.

- - -

... have to measure how SRE time is spent. With that measurement in hand, we ensure that the teams consistently spending less than 50% of their time on development change their practices. this could mean shifting some of the ops work to the dev team. or at times, adding a staff.

- while this may seem ridiculous at the time, but the principle which has been set at the core, has to be followed in order to not fall apart later. SREs should have the bandwidth to engage in creative, autonomous engineering (which is the ideal goal) constantly. Not able to manage this is a weakness in the foundation of a company from the very beginning which aims to scale to millions.

- - -

Google operates under a blame-free postmortem culture, with the goal of exposing faults and applying engineering to fix these faults, rather than avoiding or minimizing them.

- - -

SRE’s goal is no longer ”zero outages”; rather, SREs and product developers aim to spend the error budget getting maximum feature velocity. … An outage is no longer a “bad” thing — it is an expected part of the process of innovation, and an occurrence that both development and SRE teams manage rather than fear.

- 100% reliability is wrong reliability target. No user can tell the difference between 100% and 99.999% available system. so, putting a huge effort in that 0.001% has no benefit. What level of availability will the users be happy with, given how they use the product - should be taken into account while setting a reliability target. `1 - that target` is the error budget which we can spend on anything we want, example, taking risks while launching new features quickly.

- - -

The hero jack-of-all-trades on-call engineer does work, but the practice on-call engineer armed with a playbook works much better.

- when humans are necessary in emergency response, thinking through and recording the best practices ahead of time in a “playbook” produces roughly a three times improvement in mean-time-to-repair as compared to acting without preparation (“winging it”). This also ensures the newcomers get the chance to learn about the system and the company has less dependency on any single person (“hero”).

- - -

By removing humans from the loop (of change management), these practices avoid the normal problems of fatigue, familiarity/contempt, and inattention to highly repetitive tasks. As a result, both release velocity and safety increase.

- changes in a live system are the prominent source of outages. Having progressive rollouts, quickly detecting problems and rolling back changes in case of problems, minimize the number of users exposed.

- - -

… that a surprising number of services and teams don’t take the steps necessary to ensure that the required capacity (for future demand) is in place by the team it is needed.

- both organic and inorganic demand should be taken into account while demand forecasting and then, provision accordingly. Regular load testing to correlate infrastructure-capacity to service-capacity should be done.

- - -

Resource use is a function of load, capacity, and software efficiency. SREs predict demand, provision capacity, and can modify the software.

- - -

Reference book:

Learning brush strokes I

I was today looking at one of my glasses paintings I made in junior school years. I had drawn two butterflies with free hand using a black liner. While this is not something big I noticed, both the butterflies have different sizes and shapes but they have the same curves. The depth, the arcs at a very minute level (half a centimeter) reflect the same hand which drew them, anyone looking at the two pictures can infer it.

I wonder if there is a way to learn this. Not just learning what type of object it is or what species of butterflies it is but the curves, brush strokes and the drawing style which is very identifiable to a person and truly personalized. Learning the way different painters move their hands rendering every drawing ever created unique.

Hope to see more on it.

GSoC#9: It's already time?

This is it.

I triggered a discussion in MacPorts which led to a meeting involving the whole community. I'm so noob rn I get goosebumps at such a statement:

"If we start at 15 UTC or later, we could hopefully cover all time zones from the west USA to India, including even China in the evening."

With this, the GSoC period is officially off to an end and what has been (and will continue to be) a wonderful journey.

Now, I await the final verdict.

EDIT: Passed. :D

GSoC#8: Project and pre-final evaluation period

It's nearly over and I don't like it.

Effectively, my project had three courses of actions to implement - snapshot, migrate and restore. Out of them, migrate and restore both depended on the snapshot action largely and also, I divided each of these actions into procedures or simpler steps further as you probably know from my previous emails throughout the term. I have the following points to make. I welcome all kinds of comments.

1. snapshot action is entirely finished for now.

All the Tcl and C functions for creating a snapshot, fetching a snapshot given its id, fetching a list of recent snapshots, fetching properties of a certain snapshot have been finished.
This required me to create a new entity named reg_snapshot and wrote all the helper functions as well in util.c as were there for entry, file, portgroup etc.

2. restore action is also finished for now.

It takes an optional argument as 'port restore --snapshot-id <id>' if you want to pass in a certain snapshot or without the argument, it lists the snapshots on the console and asks the user to select an id and proceeds.
It first deactivates all the currently active ports and then reproduces the install commands for the selected snapshot. NOTE that it doesn't uninstall the original ports, as desired for the minimal build.
But before deactivating, it first sorts the portlist in a way that dependencies come later.
And also, before installing the selected snapshot, it first sorts the ports from the snapshot in a way that dependencies come first.
I have used "registry::run_target" directly for doing all this and tried testing by modifying the registry.db locally with many of the scenarios I could think of.

3. migrate action had essentially four steps - a) creating a snapshot, b) uninstall the current ports, c) upgrade port command and d) restore the last snapshot.

Out of these 4, creating a snapshot and restoring are simply imported from the respective modules.
I have finished the procedure for uninstalling the ports and verified.
The one task that is left to commit is to cover, if even possible, all the scenarios where we need to look for upgrading port command, that is, basically check for the change in OS or arch in macports.conf.
While I have added the most simple checks for it like here [0], there was a wonderful discussion thread on it by Mojca [1] and Clemens [2] yesterday covering many good cases like a change of compiler, for one or change in libraries by Apple, for two. This is what I am working on as informed by my last update.

4. I have added most of the documentation for all the modules, functions and any important steps, listed any minor TODOs in the code itself, added ui_msg statements, cleaned the code etc.

5. Apart from this, I have been trying to get involved more in the community on the dev mailing list and as Ken pointed out that writing a portfile is not a task too difficult, I have been looking at portfile tutorial as well.

6. Future Plans related to the project:

First, finish the part of migrate action left.
Work on the test cases for all the three modules.
On mportinit, we should suggest the user run `port migrate` instead of (or in addition to) the link to migration guide.

Attachments: Link 1 Link 2 Link 3

GSoC#7: Global variables are bad.

I couldn't be happier to see this, after eight hours of struggle:

Why did I get stuck? Because I'm used to global variables in C from lower division classes at my university. I always tried to escape the pain of passing by reference or address or value and simply use a global pointer. But when it comes to large code bases and across modules, like the macports one, it's not possible anymore. How often someone passes a pointer by address?

It was a good lesson. The next challenge is to pass this struct to Tcl.

GSoC#6: Project and pre-second evaluation period

Unlike the previous one, this was a more productive period. There were some important discoveries I made about how the port command works and how the macports-base is written to serve the Tcl and registry APIs for non-core modules.

I'll finally update you on the three phases of my project in a more formal way.

A snapshot is a list of all commands that created the current installed state.
Restoring a snapshot deactivates the active ports and reproduce the install commands for the selected snapshot.
Migrate creates a new snapshot, uninstalls installed ports and reproduces the install commands for the last snapshot.

I described the snapshot in great detail in the last post. There have been changes proposed to it when I was confused about how to get the whole snapshot action behave as a single transaction and Rainer pointed out the need to have the snapshot logic begin at a higher level, probably using Tcl wrappers.

Second Phase: In all, the action `migrate` aims to create a new snapshot, uninstalls all ports, upgrades the port command for the new architecture and finally restores the last snapshot which involves installing all ports from the snapshot we created before uninstalling. Till now, I have finished the following procedures.

uninstall_installed to uninstall all the currently installed ports.
recover_ports_state to install and activate (according to the snapshot) the ports to bring out the state as close to before migrating.
sort_portlist_by_dependents and port_dependencies to get the dependents and dependencies in the topologically sorted order in order to avoid installation of broken ports.

There is a great deal of project left for the next phase. At a higher level, I still haven't written for fetching the snapshot from the database, needs rigorous testing and error handling. From the previous phase, the above-proposed changes are still left.

The next phase will include finishing the migrate action, connecting it to the snapshot action but majorly, the implementation of restore action. I'll very likely need to take the help of my mentor in the coming phase.

In this period, I have also tried to improve on updates for the community with all the issues, leftovers and other points separately laid out. This helped to bring out the best in the discussion.

Find the work log here.

GSoC#5: A few words on Tcl (and an advice!)

After spending a couple of hours reddit-ing and other articles, I found some really interesting insights about Tcl which otherwise you might not never know about a programming language. So the following are some of the facts about Tcl which I found interesting and just stating them here:

To quote Philip Greenspun, as a software developer, you're unlikely to get rich. So you might as well try to get through your life in such a way that you make a difference to the world (like every startup shouts "make the world a better place"). Tcl illustrates one way:

make something that is simple enough for almost everyone to understand,
give away your source code
explain how to "weave" your source code in with other systems

Looks something we can do? Yesss. But do we? Noo.

Tcl rocks! Well, everything IS a string, like in a fundamental way. Just look at the tape from a Turing Machine which is just an infinite mutable string. This is really, really true. Like, when you do something like a for loop, you're basically running a "for" command and passing it four strings.

 for {set x 0} {$x<10} {incr x} {  
   puts "hey"  
 }

is equivalent to:

 for "set x 0" {$x<10} "incr x" "puts \"hey\""

:o :o

Numbers and strings being interchangeable actually works. Everything being a string, is damn powerful but takes a while to get into the mindset that you are not just coding the solution up but programming the programming to the solution.

Oh, and the "uplevel". "uplevel" is a terrifying and baffling feature. It gives you access to the local variables of any function that called you.. Say, whaaat?

Tcl : scripting languages :: C : compiled languages.

As jaymaj21 comments on reddit, "simple small footprint, least hairy implementation." One of the most efficiently parsed scripting languages, Tcl is simple and way easy to implement. You can just write up your own Tcl interpreter very quick. You can write a Tcl program that pokes around the run-time environment, for example, using "info exists x".

A language that is not powerful by itself can become powerful if a procedure can invoke the interpreter, that is, if a program can write a program and then ask to have it evaluated. In Tcl, you do this by calling eval!

Tcl is more of "Tcl, C Library" because you can access about everything from C in a way that's pleasant to use.

God was I lucky to get a Tcl project for my GSoC. Finger lickin' good!