GSoC Mentor Summit 2019

I was lucky to get the chance to attend the Google Summer of Code Mentor Summit two years in a row. It was hosted in Munich this year, first time outside Silicon Valley, from October 17 to October 20. I was a student participant with MacPorts organization in 2017, and later took the role of org admin in 2018 and co-mentor in 2019.

If you are not familiar with the Google Summer of Code, it is a global program focused on introducing students to open-source software development. Student participants are paired with a mentor from the participating organizations, gaining exposure to real-world software development and techniques. Students have the opportunity to spend the break between their school semesters earning a stipend while working in areas related to their interests. Google invites two mentors each year in autumn for an unconference.

MacPorts is a package manager for macOS which tries to simplify the packaging and installation of open-source software on Mac systems. It is the only package management system that supports macOS as far back as 10.5 for a majority of the packages. Luckily or unluckily, Homebrew wasn't there.

Together with Mojca Miklavec, I mentored Arjun Salyan who did a great job at implementing the long-standing feature in MacPorts to be able to view ports' information on a web interface along with open issues, build history, and user installation statistics. We would have a two-hour hangout call on Sunday afternoons the last summer to discuss the progress. We left a huge list of improvements and features for another summer maybe. GitHub Issues and the dev mailing list made it convenient for the community to review code and give feedback alongside the development.

My blurred shot of Allianz Arena

Google arranged a scavenger hunt across the city of Munich and provided us with all-day public transit passes. Munich is a very well-connected city and is home to centuries-old buildings even in the middle of the city. You can basically see glass elevators in the middle of the roads.

The two remaining days were reserved for unconference sessions on various topics. I attended a number of different sessions but also used the “hallway track” to get in touch with others. I had a far better experience this time than my first time at the summit in 2018. It definitely felt good to discuss python packaging issues I am having at work with Francesco Bonazzi who works for SymPy and also at Max Planck Institute in Germany on symbolic computing. I also got feedback (or an enhancement request I'd say) on my migration project from 2017 from a senior developer at RTEMS.

There were multiple notable sessions. One of them discussed the possibility of having a GSoC-like program for documentation and technical writers, and the challenges behind it. How people who are not accustomed to coding practices and version control can leverage Github.

A set of sessions I attended was around the growth of the smaller open-source communities, how they can finance themselves and employ people, and how more permissible and less vague software licenses may help. This might need convincing large companies to use open-source. A big problem we face at MacPorts is the availability of free hardware for testing pull requests and building binaries because of the expensive nature of Macs. It's been solved to a large extent by the recent release of GitHub Actions and Azure pipelines, providing 6 hours of build time which is a significant increase from 50 mins at Travis CI but is still a hurdle in bringing new contributors on board.

People associated with universities and research labs had a very interesting series of sessions on open science. It does help to know how the work done in physics, biology or neuroscience needs to be reproducible and accessible to practically realize its impact. There is a huge lack of practice/awareness among science academia to document, code, and release their collected datasets and experiment details in a manner that it can be reused by scientific communities outside their research groups. Scanning the brains of 20 people from North America for hints of drug use is largely ineffective unless it can be reproduced across demographics.

My favorite session was a loose discussion on the importance of #offtopic or #random channels hosted by coala's Maximilian Scholz. By sharing content unrelated to work and just "fooling around", it helps the developers to know more about the personality of the community, and the senior members and hence, more potential to connect, or in some cases, lifelong friendships.

I noticed a number of projects working on building apps for helping younger kids learn to code through mobile games. (I think?) I made friends with people from NumFOCUS (which supports scipy, numpy, pandas among many other scientific computing tools) and Zulip (an open-source alternative to Slack).

I found it fascinating how the Google OSPO team managed to get 330 people from 42 different countries to agree on the discussion topics. They would put up a board with time slots and rooms, and people would suggest topics and indicate interests using mere post-it notes and dot stickers. No exhausting long discussions and voting procedures just for deciding the topics.

I thoroughly enjoyed the evenings and discussions with other open-source projects. Google had authentic Bavarian cuisine arranged for us. This certainly included the beer, of which we managed to finish 3 whole barrels on the first evening itself. I cannot forget the chocolate room, full of the weirdest chocolates which people brought from all corners of the world.

25% on-call rule, paging, actionable alerts: Notes from Google's SRE Book (#8)

SREs are on-call for the services they support. The SRE teams are quite different from purely operational teams in that they place heavy emphasis on the use of engineering to approach problems. These problems, which typically fall in the operational domain, exist at a scale that would be intractable without software engineering solutions.

- - -

The quantity of on-call can be calculated by the percent of time spent by engineers on on-call duties. The quality of on-call can be calculated by the number of incidents that occur during an on-call shift.

- - -

Using the 25% on-call rule, we can derive the minimum number of SREs required to sustain a 24/7 on-call rotation. Assuming that there are always two people on-call (primary and secondary, with different duties), the minimum number of engineers needed for on-call duty from a single-site team is eight: assuming week-long shifts, each engineer is on-call (primary or secondary) for one week every month. For dual-site teams, a reasonable minimum size of each team is six, both to honor the 25% rule and to ensure a substantial and critical mass of engineers for the team.

- - -

We’ve found that on average, dealing with the tasks involved in an on-call incident—root-cause analysis, remediation, and follow-up activities like writing a postmortem and fixing bugs—takes 6 hours. It follows that the maximum number of incidents per day is 2 per 12-hour on-call shift. In order to stay within this upper bound, the distribution of paging events should be very flat over time,

- - -

Google offers time-off-in-lieu or straight cash compensation, capped at some proportion of overall salary. The compensation cap represents, in practice, a limit on the amount of on-call work that will be taken on by any individual.

- - -

To make sure that the engineers are in the appropriate frame of mind to leverage the latter mindset, it’s important to reduce the stress related to being on-call.

- - -

And the most important point relevant to the companies trying to monitor their services at any scale here:
... when the same alert pages for the fourth time in the week, and the previous three pages were initiated by an external infrastructure system, it is extremely tempting to exercise confirmation bias by automatically associating this fourth occurrence of the problem with the previous cause.

- - -

Paging alerts should be aligned with the symptoms that threaten a service’s SLOs. All paging alerts should also be actionable. Low-priority alerts that bother the on-call engineer every hour (or more frequently) disrupt productivity, and the fatigue such alerts induce can also cause serious alerts to be treated with less attention than necessary.

Sometimes the changes that cause operational overload are not under the control of the SRE teams. For example, the application developers might introduce changes that cause the system to be more noisy, less reliable, or both. In this case, it is appropriate to work together with the application developers to set common goals to improve the system.

In extreme cases, SRE teams may have the option to "give back the pager"—SRE can ask the developer team to be exclusively on-call for the system until it meets the standards of the SRE team in question.

- - -

... while knowledge gaps are discovered only when an incident occurs. ... To counteract this eventuality, SRE teams should be sized to allow every engineer to be on-call at least once or twice a quarter, thus ensuring that each team member is sufficiently exposed to production.
- - -

Metrics, monitoring, dashboards: Notes from Google's SRE Book (#7)

It's very easy to make dashboards in Datadog or New Relic or CloudWatch or whatever the monitoring system you use, but unless you know about what metric needs to be monitored for your SLOs, there will be infinite ways of missing ways. There can be hundreds of reasons for your application to not run and monitoring the container status or disk space or CPU Utilization is just not going to catch those.

Below I try to quote some ideas from the book which helped Google evolve to maintaining over tens of 9s now and appreciate the art of monitoring. I am fascinated by the part how Google leveraged all those problem-solving and design skills to make something this useful and efficient, telling us that even if we think we understand the ideas of simplicity or readability or scaling, we probably don't.


Google’s monitoring systems don’t just measure simple metrics, such as the average response time of an unladen European web server; we also need to understand the distribution of those response times across all web servers in that region. This knowledge enables us to identify the factors contributing to the latency tail.

- - -

This new model made the collection of time-series a first-class role of the monitoring system, and replaced those check scripts with a rich language for manipulating time-series into charts and alerts. ... the history of the collected data can be used for that alert computation as well.

- - -

Instead of executing custom scripts to detect system failures, Borgmon relies on a common data exposition format; this enables mass data collection with low overheads and avoids the costs of subprocess execution and network connection setup.

- - -

To facilitate mass collection, the metrics format had to be standardized.

- - -

Typically, a team runs a single Borgmon per cluster, and a pair at the global level.

- - -

each of the major languages used at Google has an implementation of the exported variable interface that automagically registers with the HTTP server built into every Google binary by default.

- - -

... using service discovery reduces the cost of maintaining it and allows the monitoring to scale.

- - -

... Borgmon rules, consists of simple algebraic expressions that compute time-series from other time-series. These rules can be quite powerful because they can query the history of a single time-series (i.e., the time axis), query different subsets of labels from many time-series at once (i.e., the space axis), and apply many mathematical operations.

- - -

... it’s better to use counters, because they don’t lose meaning when events occur between sampling intervals. Should any activity or changes occur between sampling intervals, a gauge collection is likely to miss that activity.

- - -

... example also uses a Google convention that helps readability. Each computed variable name contains a colon-separated triplet indicating the aggregation level, the variable name, and the operation that created that name.

- - -

... teams send their page-worthy alerts to their on-call rotation and their important but subcritical alerts to their ticket queues. All other alerts should be retained as informational data for status dashboards.

- - -

This decoupling allows the size of the system being monitored to scale independently of the size of alerting rules. These rules cost less to maintain because they’re abstracted over a common time-series format. New applications come ready with metric exports in all components and libraries to which they link, ...

- - -

Last quote could be taken as an action item from this post.

Wonderful day it is!

Releases, builds, versions: Notes from Google's SRE Book (#6)

About the messy releases.

Making sure that our tools behave correctly by default and are adequately documented makes it easy for teams to stay focused on features and users, rather than spending time reinventing the wheel (poorly) when it comes to releasing software.

- - -

… run their own release processes. Although we have thousands of engineers and products, we can achieve a high release velocity.

- We have embraced the philosophy that frequent releases result in fewer changes between versions. This approach makes testing and troubleshooting easier.

- - -

Our builds are hermetic, meaning that they are insensitive to the libraries and other software installed on the build machine. Instead, builds depend on known versions of build tools, such as compilers, and dependencies, such as libraries. The build process is self-contained and must not rely on services that are external to the build environment.

- - -

Instead, we branch from the mainline at a specific revision and never merge changes from the branch back into the mainline.

- - -

We also recommend creating releases at the revision number (version) of the last continuous test build that successfully completed all tests. These measures decrease the chance that subsequent changes made to the mainline will cause failures during the build performed at release time.

- - -

Most companies deal with the same set of release engineering problems regardless of their size or the tools they use: How should you handle versioning of your packages? Should you use a continuous build and deploy model, or perform periodic builds? How often should you release? What configuration management policies should you use? What release metrics are of interest?

- - -

The release engineer needs to understand the intention of how the code should be built and deployed. The developers shouldn’t build and “throw the results over the fence” to be handled by the release engineers.

- - -

In fact, SRE’s experience has found that reliable processes tend to actually increase developer agility: rapid, reliable production rollouts make changes in production easier to see. As a result, once a bug surfaces, it takes less time to find and fix that bug. Building reliability into development allows developers to focus their attention on what we really do care about—the functionality and performance of their software and systems.

- - -

… when you consider a web service that’s expected to be available 24/7, to some extent, every new line of code written is a liability.

- - -

If we release 100 unrelated changes to a system at the same time and performance gets worse, understanding which changes impacted performance, and how they did so, will take considerable effort or additional instrumentation.

- - -

A point for sure to be perceived negatively by most of the overly ambitious startups:

Every time we say "no" to a feature, we are not restricting innovation; we are keeping the environment uncluttered of distractions so that focus remains squarely on innovation, and real engineering can proceed.

- - -

Toil, complaints, understanding role: Notes from Google's SRE Book (#5)

This chapter was a particularly interesting read which deals with day-to-day non-optimal ways of doing work and it goes unnoticed. It is tiring and gives a false sense of accomplishment.

Our SRE organization has an advertised goal of keeping operational work (i.e., toil) below 50% of each SRE’s time.

- - -

Furthermore, when we hire new SREs, we promise them that SRE is not a typical Ops organization, quoting the 50% rule just mentioned. We need to keep that promise by not allowing the SRE organization or any subteam within it to devolve into an Ops team.

- There’s a floor on the amount of toil any SRE has to handle if they are on-call. A typical SRE has one week of primary on-call and one week of secondary on-call in each cycle (for discussion of primary versus secondary on-call shifts. It follows that in a 6-person rotation, at least 2 of every 6 weeks are dedicated to on-call shifts and interrupt handling, which means the lower bound on potential toil is 2/6 = 33% of an SRE’s time. In an 8-person rotation, the lower bound is 2/8 = 25%.

- - -

It’s fine in small doses, and if you’re happy with those small doses, toil is not a problem. Toil becomes toxic when experienced in large quantities. If you’re burdened with too much toil, you should be very concerned and complain loudly.

- Your career progress will slow down or grind to a halt if you spend too little time on projects.

- - -

We work hard to ensure that everyone who works in or with the SRE organization understands that we are an engineering organization. Individuals or teams within SRE that engage in too much toil undermine the clarity of that communication and confuse people about our role.

- - -

If you’re too willing to take on toil, your Dev counterparts will have incentives to load you down with even more toil, sometimes shifting operational tasks that should rightfully be performed by Devs to SRE. Other teams may also start expecting SREs to take on such work.

- - -

System behavior, percentiles, SLOs: Notes from Google's SRE Book (#4)

Introduced a good alternative to Averages used in monitoring with practical examples. Usually, everyone knows that taking averages are just not enough but here, I feel, is a solution:

because not measuring behavior at the client can miss a range of problems that affect users but don’t affect server-side metrics.

- for example, measuring response latency of the API does not cover the poor user latency which could be due to a slow page load or problems with the page’s JavaScript.

- - -

We generally prefer to work with percentiles rather than the mean (arithmetic average) of a set of values. … … User studies have shown that people typically prefer a slightly slower system to one with high variance in response time, so some SRE teams focus only on high percentile values

- Averaging can hide a large number of slow requests and still be a good average. Percentiles help you to know how much percent of the total requests were served under a value and how much wasn’t. The behavior of an nth-percentile value will tell you the variance in response times. Good/consistent behavior, that is less variation, determines reliability than being really fast at some times and being slow at other. 

- Other examples could be, like, instantaneous load than measuring average load and seeing how well a system is serving the real-time load. 

Also, having standard templates for various SLIs, like measurement window, how frequently to measure, what all clusters to include, how data is acquired (example, at server side or client side), time to last byte vs the first byte as latency, etc help save effort and confusion.

- - -

Start by thinking about (or finding out!) what your users care about, not what you can measure.

- - -

Choose just enough SLOs to provide good coverage of your system’s attributes. Defend the SLOs you pick: if you can’t ever win a conversation about priorities by quoting a particular SLO, it’s probably not worth having that SLO.

- - -

A good SLO is a helpful, legitimate forcing function for a development team.

- - -

Compare the SLIs to the SLOs, and decide whether or not action is needed. … Without the SLO, you wouldn’t know whether (or when) to take action.

- - -

Defining objectives, performance indicators, agreements: Notes from Google's SRE Book (#3)

SLIs, SLOs and SLAs, concepts which the Indian tech companies lack.

We use intuition, experience, and an understanding of what users want to define service level indicators (SLIs), objectives (SLOs), and agreements (SLAs).

- SLI is a quantitative measure of a level of service.

- SLO is a bound on SLIs (upper or lower or target).

- SLA has the consequences if SLOs are met or not met.

Seems like SLA is not just something you can shut people on but more objective.

- - -

Without an explicit SLO, users often develop their own beliefs about desired performance, which may be unrelated to the beliefs held by the people designing and operating the service. This dynamic can lead to both over-reliance on the service, when users incorrectly believe that a service will be more available than it actually is, and under-reliance, when prospective users believe a system is flakier and less reliable than it actually is.

- - -

On the Chubby example:

In any given quarter, if a true failure has not dropped availability below the target, a controlled outage will be synthesized by intentionally taking down the system. In this way, we are able to flush out unreasonable dependencies on Chubby shortly after they are added. Doing so forces service owners to reckon with the reality of distributed systems sooner rather than later.

- - -

SRE doesn’t typically get involved in constructing SLAs, because SLAs are closely tied to business and product decisions. SRE does, however, get involved in helping to avoid triggering the consequences of missed SLOs.

- - -

Whether or not a particular service has an SLA, it’s valuable to define SLIs and SLOs and use them to manage the service.

- - -

Downtime, outages, service risk: Notes from Google's SRE Book (#2)

I failed to deliver upon my own commitment to reading some pages of the book each day but it hasn't demotivated me from continuing because the book is too good.

To make this problem tractable and consistent across many types of systems we run, we focus on unplanned downtime.

- For service risk, it is not clear how to reduce all the potential factors of degraded performance into a metric. Degraded or unreliable performance can impact user satisfaction, revenue, loss of trust, and most of these factors are hard to measure. 

- A metric which measures an unexpected service downtime can be a property of the system we want to optimize. Unplanned downtime is not just when the system went down but it could accommodate every other request which failed or delay in serving the request or anything which affects the users.

- - -

… instead of using metrics around uptime, we define availability in terms of the request success rate.

- measuring service risk in terms of objective metrics:

1. time-based availability over a period of time (last 30 days or last 3 months): uptime/uptime+downtime 
2. another measure could be 'request success rate': successful/total-requests over a 1-day window

the second metric can be made to use for batch processes, pipelines, storage etc with minimal modifications. Example, for a pipeline which reads records from a CSV file, transforms and puts in a database, the ratio of the number of records successfully processed to unsuccessfully processed can be a similar availability measure.

- - -

In 2006, … … We set a lower availability target for YouTube than for our enterprise products because rapid feature development was correspondingly more important.

- - -

Which is worse for the service: a constant low rate of failures, or an occasional full-site outage? Both types of failure may result in the same absolute number of errors, but may have vastly different impacts on the business.

- - -

The requirements for building and running infrastructure components differ from the requirements for consumer products in a number of ways.

- Example, consumer services using BigTable in the path of a user request require low latency, empty request queues almost all the time whereas infrastructure services using BigTable for offline analysis require the queue of tasks to never be empty and want more throughput. 

- - -

Note that we can run multiple classes of services using identical hardware and software. We can provide vastly different service guarantees by adjusting a variety of service characteristics, … 

- such as the quantities of resources, the degree of redundancy, the geographical provisioning constraints, and, critically, the infrastructure software configuration.

- - -

A point which I feel is very much applicable to startups:

Usually, pre-existing teams have worked out some kind of informal balance between them as to where the risk/effort boundary lies. Unfortunately, one can rarely prove that this balance is optimal, rather than just a function of the negotiating skills of the engineers involved. Nor should such decisions be driven by politics, fear, or hope.

- Instead it’s better to come up with an objective metric agreed upon by both the engineering team and the product team which can be used to guide the negotiations in a reproducible way and without resent. The more data-based the decision can be, the better in terms of defending the decision.

- - -

AWS instance reservation simplified

A while ago I was working on AWS cost reduction at my company and reserving instances is one essential step in such a project. This required me to go through so many AWS docs on different types of reservation options provided to get the whole picture. I tried to summarize the important points here, some of them which could be missed if one does not go through all the documents carefully:

Reserving, Modification and Exchanges are different.

• how much you want to reserve and what you do with them is up to you, discounts will get applied accordingly.
• While modifying the reserved, instance footprint should match (normalization factor x number of instances of that family (example, m4.))
• You can't modify it to a lesser footprint. If you need a greater reserved footprint, then simply purchase more reserved.
• You cannot modify instances for which only one option is there, like i3.metal.
• The instances you want to modify cannot be in the marketplace at the same time, of course.
• While Exchanging offers exchanging the instance with any type, or size, you cannot control the number of instances. AWS will just give you any number of instances to maintain the resulting cost, equal to or higher than what you are paying for the original instances.
• Another IMPORTANT point after saying above is that you can exchange any number of any type of instances but for only 1 type of instance.
• The above also means you can split the reservation and exchange them for different types of instances.

The normalization factor table is given at https://aws.amazon.com/blogs/aws/new-instance-size-flexibility-for-ec2-reserved-instances/.

The math works out as long as we keep the total normalization units the same or higher and in most cases, keep the instance family the same as well.

For example, in case of modification, let’s say you already own an RI for a c4.8xlarge. This RI now applies to any usage of a Linux/UNIX C4 instance with shared tenancy in the region. This could be:

One c4.8xlarge instance, or
Two c4.4xlarge instances, or
Four c4.2xlarge instances, or
Sixteen c4.large instances, or
One c4.4xlarge and eight c4.large, and more.

For example, in the case of an exchange, if you have a t2.large CRI, to change it to a smaller t2.medium instance and an m3.medium instance:

Step 1. Modify the one t2.large CRI by splitting it into two t2.medium CRIs z
Step 2. Exchange one of the new t2.medium CRIs for an m3.medium CRI, then.

It's easier to think about reserving in terms of normalization units now.

DOUBT:
I wonder if someone would do something like this:

• You have a reservation with two t2.micro instances (giving you a footprint of 1) and a reservation with one t2.small instance (giving you a footprint of 1). You merge both reservations to a single reservation with one t2.medium instance—the combined instance size footprint of the two original reservations equals the footprint of the modified reservation.
• In both cases, you will get charged the same. Not sure.

References:
Apply RI
Modify RI
Exchange RI

SRE skills, goals, culture: Notes from Google's SRE Book (#1)

I will be trying to put my notes from the famous Google SRE book whatever I felt important from a section or a paragraph. I didn't care for similar kind of notes or summaries which may be available on the internet. This is just for my motivational purposes to help me continue reading and post some of my learnings here.

The plan is to have a post each week for 10 weeks in a row. Let's see how it works out.

The following is the first set of my notes from Sep 30:

50-60% are SEs and 40-50% are close to SEs with 85-99% of the skillset required, and who in addition had a set of technical skills that are useful to SRE but is rare for most SEs …

- common additional skills for SRE:

• UNIX system internals
• networking (later 1 to 3)

- - -

the team tasked with managing a service needs to code or it will drown. Google places a 50% cap on the aggregator “ops” work for all SREs — tickets, on-call, manual tasks, etc.

- - -

... have to measure how SRE time is spent. With that measurement in hand, we ensure that the teams consistently spending less than 50% of their time on development change their practices. this could mean shifting some of the ops work to the dev team. or at times, adding a staff.

- while this may seem ridiculous at the time, but the principle which has been set at the core, has to be followed in order to not fall apart later. SREs should have the bandwidth to engage in creative, autonomous engineering (which is the ideal goal) constantly. Not able to manage this is a weakness in the foundation of a company from the very beginning which aims to scale to millions.

- - -

Google operates under a blame-free postmortem culture, with the goal of exposing faults and applying engineering to fix these faults, rather than avoiding or minimizing them.

- - -

SRE’s goal is no longer ”zero outages”; rather, SREs and product developers aim to spend the error budget getting maximum feature velocity. … An outage is no longer a “bad” thing — it is an expected part of the process of innovation, and an occurrence that both development and SRE teams manage rather than fear.

- 100% reliability is wrong reliability target. No user can tell the difference between 100% and 99.999% available system. so, putting a huge effort in that 0.001% has no benefit. What level of availability will the users be happy with, given how they use the product - should be taken into account while setting a reliability target. `1 - that target` is the error budget which we can spend on anything we want, example, taking risks while launching new features quickly.

- - -

The hero jack-of-all-trades on-call engineer does work, but the practice on-call engineer armed with a playbook works much better.

- when humans are necessary in emergency response, thinking through and recording the best practices ahead of time in a “playbook” produces roughly a three times improvement in mean-time-to-repair as compared to acting without preparation (“winging it”). This also ensures the newcomers get the chance to learn about the system and the company has less dependency on any single person (“hero”).

- - -

By removing humans from the loop (of change management), these practices avoid the normal problems of fatigue, familiarity/contempt, and inattention to highly repetitive tasks. As a result, both release velocity and safety increase.

- changes in a live system are the prominent source of outages. Having progressive rollouts, quickly detecting problems and rolling back changes in case of problems, minimize the number of users exposed.

- - -

… that a surprising number of services and teams don’t take the steps necessary to ensure that the required capacity (for future demand) is in place by the team it is needed.

- both organic and inorganic demand should be taken into account while demand forecasting and then, provision accordingly. Regular load testing to correlate infrastructure-capacity to service-capacity should be done.

- - -

Resource use is a function of load, capacity, and software efficiency. SREs predict demand, provision capacity, and can modify the software.

- - -

Reference book:

Site Reliability Engineering, edited by Betsy Beyer, Chris Jones, Jennifer Petoff, and Niall Richard Murphy (O’Reilly). Copyright 2016 Google, Inc., 978-1-491-92912-4.

Learning brush strokes I

I was today looking at one of my glasses paintings I made in junior school years. I had drawn two butterflies with free hand using a black liner. While this is not something big I noticed, both the butterflies have different sizes and shapes but they have the same curves. The depth, the arcs at a very minute level (half a centimeter) reflect the same hand which drew them, anyone looking at the two pictures can infer it.

I wonder if there is a way to learn this. Not just learning what type of object it is or what species of butterflies it is but the curves, brush strokes and the drawing style which is very identifiable to a person and truly personalized. Learning the way different painters move their hands rendering every drawing ever created unique.

 

Hope to see more on it.

GSoC#9: It's already time?

This is it.

I triggered a discussion in MacPorts which led to a meeting involving the whole community. I'm so noob rn I get goosebumps at such a statement:

"If we start at 15 UTC or later, we could hopefully cover all time zones from the west USA to India, including even China in the evening."

With this, the GSoC period is officially off to an end and what has been (and will continue to be) a wonderful journey.

Now, I await the final verdict.


EDIT: Passed. :D

GSoC#8: Project and pre-final evaluation period

It's nearly over and I don't like it.

Effectively, my project had three courses of actions to implement - snapshot, migrate and restore. Out of them, migrate and restore both depended on the snapshot action largely and also, I divided each of these actions into procedures or simpler steps further as you probably know from my previous emails throughout the term. I have the following points to make. I welcome all kinds of comments.

1. snapshot action is entirely finished for now.

• All the Tcl and C functions for creating a snapshot, fetching a snapshot given its id, fetching a list of recent snapshots, fetching properties of a certain snapshot have been finished.
• This required me to create a new entity named reg_snapshot and wrote all the helper functions as well in util.c as were there for entry, file, portgroup etc.

2. restore action is also finished for now.

• It takes an optional argument as 'port restore --snapshot-id <id>' if you want to pass in a certain snapshot or without the argument, it lists the snapshots on the console and asks the user to select an id and proceeds.
• It first deactivates all the currently active ports and then reproduces the install commands for the selected snapshot. NOTE that it doesn't uninstall the original ports, as desired for the minimal build.
• But before deactivating, it first sorts the portlist in a way that dependencies come later. 
• And also, before installing the selected snapshot, it first sorts the ports from the snapshot in a way that dependencies come first. 
• I have used "registry::run_target" directly for doing all this and tried testing by modifying the registry.db locally with many of the scenarios I could think of.

3. migrate action had essentially four steps - a) creating a snapshot, b) uninstall the current ports, c) upgrade port command and d) restore the last snapshot.

• Out of these 4, creating a snapshot and restoring are simply imported from the respective modules.
• I have finished the procedure for uninstalling the ports and verified.
• The one task that is left to commit is to cover, if even possible, all the scenarios where we need to look for upgrading port command, that is, basically check for the change in OS or arch in macports.conf.
• While I have added the most simple checks for it like here [0], there was a wonderful discussion thread on it by Mojca [1] and Clemens [2] yesterday covering many good cases like a change of compiler, for one or change in libraries by Apple, for two. This is what I am working on as informed by my last update.

4. I have added most of the documentation for all the modules, functions and any important steps, listed any minor TODOs in the code itself, added ui_msg statements, cleaned the code etc.

5. Apart from this, I have been trying to get involved more in the community on the dev mailing list and as Ken pointed out that writing a portfile is not a task too difficult, I have been looking at portfile tutorial as well.

6. Future Plans related to the project:

• First, finish the part of migrate action left.
• Work on the test cases for all the three modules.
• On mportinit, we should suggest the user run `port migrate` instead of (or in addition to) the link to migration guide.

Attachments: Link 1 Link 2 Link 3

GSoC#7: Global variables are bad.

I couldn't be happier to see this, after eight hours of struggle:

Why did I get stuck? Because I'm used to global variables in C from lower division classes at my university. I always tried to escape the pain of passing by reference or address or value and simply use a global pointer. But when it comes to large code bases and across modules, like the macports one, it's not possible anymore. How often someone passes a pointer by address?

It was a good lesson. The next challenge is to pass this struct to Tcl.

GSoC#6: Project and pre-second evaluation period

Unlike the previous one, this was a more productive period. There were some important discoveries I made about how the port command works and how the macports-base is written to serve the Tcl and registry APIs for non-core modules.

I'll finally update you on the three phases of my project in a more formal way.

1. A snapshot is a list of all commands that created the current installed state.
2. Restoring a snapshot deactivates the active ports and reproduce the install commands for the selected snapshot.
3. Migrate creates a new snapshot, uninstalls installed ports and reproduces the install commands for the last snapshot.

I described the snapshot in great detail in the last post. There have been changes proposed to it when I was confused about how to get the whole snapshot action behave as a single transaction and Rainer pointed out the need to have the snapshot logic begin at a higher level, probably using Tcl wrappers.

Second Phase: In all, the action `migrate` aims to create a new snapshot, uninstalls all ports, upgrades the port command for the new architecture and finally restores the last snapshot which involves installing all ports from the snapshot we created before uninstalling. Till now, I have finished the following procedures.

• uninstall_installed to uninstall all the currently installed ports.
• recover_ports_state to install and activate (according to the snapshot) the ports to bring out the state as close to before migrating.
• sort_portlist_by_dependents and port_dependencies to get the dependents and dependencies in the topologically sorted order in order to avoid installation of broken ports.

There is a great deal of project left for the next phase. At a higher level, I still haven't written for fetching the snapshot from the database, needs rigorous testing and error handling. From the previous phase, the above-proposed changes are still left.

The next phase will include finishing the migrate action, connecting it to the snapshot action but majorly, the implementation of restore action. I'll very likely need to take the help of my mentor in the coming phase.

In this period, I have also tried to improve on updates for the community with all the issues, leftovers and other points separately laid out. This helped to bring out the best in the discussion.

Find the work log here.

GSoC#5: A few words on Tcl (and an advice!)

After spending a couple of hours reddit-ing and other articles, I found some really interesting insights about Tcl which otherwise you might not never know about a programming language. So the following are some of the facts about Tcl which I found interesting and just stating them here:

To quote Philip Greenspun, as a software developer, you're unlikely to get rich. So you might as well try to get through your life in such a way that you make a difference to the world (like every startup shouts "make the world a better place"). Tcl illustrates one way:

• make something that is simple enough for almost everyone to understand,
• give away your source code
• explain how to "weave" your source code in with other systems

Looks something we can do? Yesss. But do we? Noo.

Tcl rocks! Well, everything IS a string, like in a fundamental way. Just look at the tape from a Turing Machine which is just an infinite mutable string. This is really, really true. Like, when you do something like a for loop, you're basically running a "for" command and passing it four strings.

 for {set x 0} {$x<10} {incr x} {  
   puts "hey"  
 }  

is equivalent to:

 for "set x 0" {$x<10} "incr x" "puts \"hey\""  

:o :o

Numbers and strings being interchangeable actually works. Everything being a string, is damn powerful but takes a while to get into the mindset that you are not just coding the solution up but programming the programming to the solution. 

Oh, and the "uplevel". "uplevel" is a terrifying and baffling feature. It gives you access to the local variables of any function that called you.. Say, whaaat?

Tcl : scripting languages :: C : compiled languages.

 As jaymaj21 comments on reddit, "simple small footprint, least hairy implementation." One of the most efficiently parsed scripting languages, Tcl is simple and way easy to implement. You can just write up your own Tcl interpreter very quick. You can write a Tcl program that pokes around the run-time environment, for example, using "info exists x".

A language that is not powerful by itself can become powerful if a procedure can invoke the interpreter, that is, if a program can write a program and then ask to have it evaluated. In Tcl, you do this by calling eval!

Tcl is more of "Tcl, C Library" because you can access about everything from C in a way that's pleasant to use. 

God was I lucky to get a Tcl project for my GSoC. Finger lickin' good!

GSoC#4: Project and pre-first evaluation period

My project is about adding support to automate the migration process, in particular, after an OS upgrade or hardware change. So, from a set of manual migration instructions, the project aims to narrow it down to `port migrate`.

The project is divided into three phases: getting the information (in our database), removing the (existing) information, restoring the (updated) information. I know, it doesn’t seem to be a really efficient logic but we plan to improve it further as we go.

First Phase: The action `snapshot` aims to replicate the user install commands. The plan was to implement this using database and not files, so it justifies the need to add support for the tables to the entire stack up from cregistry.

• The snapshot action basically takes a complete copy of all the installed ports and the metadata. Till now,
• 3 new sqlite tables have been added to the registry’s create and update function and regsitrydb version renewed so that existing installations get them too.
• snapshot_create in registry2.0/entry.c enables access to the sqlite database from Tcl frontend using registry APIs. It calls cregistry/entry.c reg_snapshot_create internally.
• reg_snapshot_create creates a snapshot in the snapshots table, calls snapshot_store_ports.
• snapshot_store_ports stores info on all currently installed ports in the snapshot_ports table (and requested flag)
• snapshot_store_port_variants stores port variants with the sign in the corresponding table. It depends on getting parsed variants which I have not written yet (reason below).
• Time spent on figuring out things initially counts as well, right? Only if this does not become an endless excuse.

Talk about Future: Go according to the Timeline mentioned in the proposal. Quickly finish the left outs in this 4-day planned buffer and start with the migrate action. Things seem to be picked up fast compared to the initial weeks. I’m now more aware of where lies what, hopefully.

I was stuck at.. 

• for a long time, Tcl or C? In what language do I have to write the above three functions, Tcl or C? Believe me, this is not something you ask your mentor or community or perhaps, anyone. Then I started looking at the implementation of existing commands, one by one, took 3 to 4 days, and then one fine morning, I saw the mapping array from command to function. It was there, right at the bottom of entry.c. Yes, if you give this argument to registry::entry, this method will be called. Had a sound sleep that afternoon.
• for a short time, probably for a day, that there exists something like reg_entry_propget to help you see through dark, to get what you want, just give the key. Should have spent more time going through C files than Tcl ones during community bonding period :/.

I am stuck at ..

• Only one variant of a port can be active at a time. Makes sense. Running `port install apache2 -preforkmpm +workermpm` and then `port install apache2 +openldap` saves apache2 as two different ports in the registry. So, reg_entry_propget only returns one variant of a port when passed as reg_entry pointer. Now, how do I link the two variants installed (note installed and not active) to the same port in my database? If not, all we are left is with one variant mapped to “each” port, questioning the need for snapshot_port_variants table.
• Till ^ gets resolved, I won’t be able to write get_variants_parsed function.
• IMPORTANT: where to add the OS check? I tried starting a discussion on this but never arrived at a decent conclusion.

Still not clear ..

• where to include the --list for listing snapshots? if with `snapshot` action, then it creates the need for --create which I do not prefer personally. But if with `restore`, then `port restore --list` doesn’t appeal as well. Only if a command existed which lists the choices first and then proceeds so that I could simply copy their design strategy and save myself from confusion every time I decide to do things nicely.
• limiting the number of snapshots that a user can take?
• and more to come :).

Screenshot if someone enjoys looking at it, I just like the shutter sound:

I like this one:

Also, I forget, the unreviewed commits are here.

EDIT:

A doubt, so fundamental to my project, that got cleared while writing this post:

• We’re solving for these two end use cases (correct me if I'm wrong):
• running simply `migrate` is (snapshot the current installed state + restore).
• `snapshot` followed by `restore`: restoring back some existing snapshot.
• Interactive restore could be a solution for --list?