Daniel Lobato

Why

3 years after I started to work at Red Hat, I realized I was getting very specialized in everything related to systems management. There was another area in this software field that I just didn’t understand, Machine Learning / AI. During my undergraduate years, I took courses on Knowledge Engineering and AI but they did not take me anywhere near competent in a professional setting. For many years, I have been mesmerized at the things DeepMind, OpenAI,  Tesla (autopilot?!) and others have done.  Even just “simple” Counter Strike bots seemed magic. Over time, the outlook of “this is just magic, I don’t understand” made me itchy and I had to get rid of it. It was time to dive in and learn about ML, image recognition, and the math and research behind these.

I applied, got in, and here’s the story about my first year at OMSCS.

How

It takes a lot of time. In a nutshell, I had to say goodbye to many weekends, performing above and beyond at work and having time in the evenings. Forget about lying down on the couch for hours. From my estimations, you should be ready to spend 2-3 years committed to this. Side projects while working full time in grad school? Grad school is your “side-project”. If you enjoy programming (I do), you won’t even miss side projects after all the coding you will do. Most courses have a strict policy of not sharing code for previous assignments, but some allow it.

I started with Advanced Operating Systems, since I already have some experience at work with different OSs and virtualization, I thought it would be a great idea to learn more deeply about these topics as my first grad-level course. It was a fantastic choice. The pace was fine, and it helped me get familiar with reading research papers and learn better about OS fundamentals. It brought me to a point where I was able to read “Linux Kernel Development” without major problems and I got to dust off my C skills. As a kind of extracurricular, I recommend doing the Eudyptula challenge if you want to go from “I know basic C” to “I can contribute to the Linux Kernel”.

On my second term, I made a choice of taking two courses at the same time. Data and Visual Analytics and Computational Photography. It was a terrible idea. Even if you have good time management skills, I found it too exhausting to do two courses plus a full time job, which means that slowly you will feel more tired mentally until the end of the term. A tired mind affects all areas of your life, at home, and at my job I was just not as “present” as I wanted to be. On the other side, I learned about feature analysis, transformations, brushed up on statistics and R. Computational Photography taught me a lot about how to work with pixels and 3D representations, and I learned how to use OpenCV to a level where I could be comfortable using it in a professional setting. Here’s what we did (source code available upon request, it’s forbidden to publicly share it):

On my last term so far I took Reinforcement Learning. Summer terms are shorter, and RL suggests a prerequisite on Machine Learning which I have not taken yet. Going back to one course a term was a big relief. At the end of the previous term I slowly turned down certain lead responsibilities at Red Hat as I was clearly not fit for it – not with the self-imposed burden of grad school. Not only RL has been my favorite class so far, but I’ve also been able to go out some Sundays and just.. live life at a pace that I could enjoy. Hell, years after watching A Beautiful Mind I finally understood what a Nash equilibrium is – and other kinds of equilibria too. The assignments were very academic, in most of them we had to read a paper and replicate the results. We also got a chance to play with OpenAI gym. It was so satisfying to train this Lunar Lander and watch it land. If you’re curious, it’s deep Q-learning + neural networks I used to train it (we were free to experiment).

Academic papers always looked daunting to me. I had very little exposure to papers when I finished my undergrad degree. I remember reading about 4 papers, maybe. After one year of reading a few papers every month, I published some summaries for the ones I read and found interesting. Check out my repo paper-notes. This advantage of grad school is not to be taken lightly. Being able to stay on top of your field by following people on Google Scholar/Arxiv is very satisfying to me and much saner than following Hacker News/Twitter/Reddit/<insert fad of the day>.

Some of the stuff learned in the courses will be applicable at your job right away. I’m not talking about the discipline, the grit, how ever you want to call it. I’m talking about specific technologies like Protocol Buffers, Keras, ggplot2, and more.  The community of students is just great. If you are near other students in your class, you can always meet with them in person, but even if that isn’t possible, the Slack chat and forums will help you bond with your peers.

Graduate school -even through distance learning- is being one of the most rewarding academic experiences I’ve ever had.  I hope you are ready to apply now!

If your goal is to learn, reading books is one of the best habits you could add to your life. It’s no wonder all successful people (by a traditional measure, wealth – Gates, Buffett…), and all interesting people I know are voracious readers. I mostly concur with that opinion, learning by reading builds up like compound interest.

There’s another habit for which I don’t have any ad-hominem argument to convince you to pick up. Make a conscious effort to re-read certain books.

Why this works for me? Lessons from books are just not obvious at every point of your life. There are certain experiences you have to share with the author in order to relate to them better. Maybe it’s also that the books I keep on my ‘re-read list’ are also very dense and it seems impossible to remember all they try to teach. From a more scientific point of view, spaced repetition has proven to be a great technique to retain stuff in your memory.

The way I do it is certainly no-frills. I just keep certain books on my tablet (Kindle Fire HD 7′ with Android, battery lasts like an ebook!) and my phone, and I read them using Bookari premium (previously Mantano Reader). You can configure it to read books using some text-to-speech engine so it works perfect for me.
There isn’t much more to it, than that. From time to time I get bored in a flight/train trip/beach/whatever… so I start reading and these books are always with me.

Here are some of the books in my list:

Peopleware: Productive Projects and Teams (2nd edition) – Tom DeMarco, Timothy Lister
Practical Object-Oriented Design in Ruby – Sandi Metz
Team Geek – Brian W. Fitzpatrick, Ben Collins-Sussman
A guide to the good life: The ancient art of stoic joy – William B. Irvine
Apprenticeship Patterns – Dave Hoover, Adewale Oshineye
The algorithm design manual – Steven Skiena
Release it! – Michael T. Nygard
Programming Pearls (2nd ed) – Jon Bentley

Which ones have you found worth reading over and over?

PS: It’s been a long time since the last blog post. Mostly because I thought of my blog as a place for ‘hardcore’ technical content. I don’t feel that should be the case any more, and I will not stop myself from writing about less ‘quantitative’ topics. What I aim with this is to share more, and learn more through this medium.

As Major Hayden mentioned more than a year ago, deploying CoreOS is a bit of a different beast than deploying other operating systems. In this case, we are going to do it by PXE booting the image, then applying a cloud-config script which will will set the SSH keys, core user password, CoreOS version, and register in etcd.

We are going to pass parameters that will set these options. That way we can define a host group, with certain parameters, such as the authorized keys, etcd discovery url, and virtual disk. This will simplify booting hosts in our deployment so that creating a new CoreOS node in the cluster will be reduced to three clicks, New Host -> Hostgroup: CoreOS cluster > Submit.

As in my previous tutorial for unattended Atomic deployments, I will assume you have Foreman installed, and a PXE Smart Proxy in the network (or networks) you want to launch your cluster. If not, please go to theforeman.org and get a default installation. In my opinion, Libvirt is the easiest way to get this PXE “enabled” network.

The PXE templates are already in community-templates , make sure to add the snippet too. You can create them manually by going to Hosts > Provisioning Templates > New template. However, it is much easier to install the foreman_templates plugin, then run:

foreman-rake templates:sync

And you’ll get all of the templates in the community-templates repository.

Create a new operating system with the following options, in this case it will use CoreOS 647.0.0 from the stable channel.

Time to create the host group. Go to Configure > Host groups > New host group, and create a group with the following parameters. The network must be the one you can PXE boot on, and you can add a parameter ssh_authorized_keys, value should be your public ssh key, usually located in .ssh/id_rsa.pub. I did not add it here as I have a global parameter ssh_authorized_keys with that value. Get a discovery code by going to discovery.etcd.io/new , and put the value you got in etcd_discovery_url.

  That should be enough for the group. Now create a new host in that cluster, and as soon as it boots, it will connect to etcd. Remember the URL we used for discovery? Go to that URL and you should see all hosts that have registered in the cluster.

Enjoy it, and please point out any mistakes on the comments section, or let me know on Twitter.

Project Atomic is a new initiative to have a family of well-known, enterprise-tested operating systems ready for massive container deployments.

Atomic operating systems focus on:

• Minimal size
• Immutable
• Easy to update and rollback
• Container cluster and runtime provided (currently via Docker and Kubernetes)

It comes with a set of tools. ostree, fleet, kubectl to manage your OS updates, network configurations and cluster health. As I manage my virtual machines using Foreman, and I used to spend some time developing the Docker plugin, the project piqued my interest, as it helps me work with containers more efficiently.

For this deployment, I will assume you have a Foreman host with a Smart Proxy with at least TFTP. I will be using domains in the examples, but you could use IPs instead. I think it could be possible to skip the TFTP part for PXE too, but I have not got that far yet. You need a subnet in which you can PXE boot hosts, an example of such a subnet on Libvirt can be found on Dominic Cleal’s blog.

Step one, download Fedora 22 Atomic iso or RHEL 7 (installer) Atomic iso in your Foreman host. For the Fedora case, it’ll be possible to fetch the content straight from the repo through ostreeupdate , however for the moment we will need the image to get vmlinux, initrd, and a few other files.

# wget https://dl.fedoraproject.org/pub/alt/stage/current/Cloud_Atomic/x86_64/iso/Fedora-Cloud_Atomic-x86_64-22.iso -O fedora-atomic.iso

Mount these images in a public directory so that they can be reached from the virtual machine. To keep any existent vmlinux available for non-atomic hosts, we will copy and rename it to vmlinuz_atomic from the mounted iso. By default /var/www/html/pub/atomic would work:

# mkdir /var/www/html/pub/atomic
# mount -o loop fedora-atomic.iso /var/www/html/pub/atomic/
# cp /var/www/html/pub/atomic/isolinux/vmlinuz /var/lib/tftpboot/vmlinuz_atomic

We will need now an installation medium in Foreman pointing to this location. Go to Host > Installation media and create a mirror:

Create the operating system. We will go back to this operating system to associate it with the appropriate partition table and templates afterwards. For the moment, just make sure you choose the right major version 7 for RHEL, 22 in the case of Fedora, as these are the only Atomic ones. Go to Hosts > Operating systems and click on New operating system.

Create a new partition table in Foreman to provide an initial /boot and / in the Atomic virtual machine.  Go to Hosts > Partition tables, and click on New partition table.

zerombr
clearpart --all --initlabel
part /boot --size=300 --fstype="ext4"
part pv.01 --grow
volgroup atomicos pv.01
logvol / --size=3000 --fstype="xfs" --name=root --vgname=atomicos

Head to /config_templates (or Hosts > Provisioning templates) and create a new PXE template. Choose a name, then go click on the type tab and select PXELinux. Associate the template with the operating system you created previously, and use this as the content:

DEFAULT pxeboot
TIMEOUT 20
PROMPT 0
LABEL pxeboot
kernel vmlinuz_atomic
append initrd=<%= @host.medium.path %>isolinux/initrd.img repo=<%= @host.medium.path %> ks=<%= foreman_url('provision')%> ks.device=bootif network ks.sendmac
IPAPPEND 2

Stay on provisioning templates, and create the kickstart. To do so,  click on New Template, choose a type ‘provision’, associate it with the operating system you created previously, and add this code in the editor:

lang <%= @host.params['lang'] || 'en_US.UTF-8' %>
keyboard <%= @host.params['keyboard'] || 'us' %>
timezone --utc <%= @host.params['time-zone'] || 'UTC' %>

# Partition table should create /boot and a volume atomicos
<% if @dynamic -%>
%include /tmp/diskpart.cfg
<% else -%>
<%= @host.diskLayout %>
<% end -%>


bootloader --timeout=3
<% if @host.operatingsystem.name =~ /.*Fedora.*/ -%>
ostreesetup --nogpg --osname=fedora-atomic --remote=fedora-atomic --url=<%= @host.medium.path %>/content/repo/ --ref=fedora-atomic/f<%= @host.os.major %>/<%= @host.architecture %>/docker-host
<% else -%>
ostreesetup --nogpg --osname=rhel-atomic-host --remote=rhel-atomic-host --url=file:///install/ostree --ref=rhel-atomic-host/<%= @host.os.major %>/<%= @host.architecture %>/standard
<% end -%>
services --disabled cloud-init,cloud-config,cloud-final,cloud-init-local
rootpw --iscrypted <%= root_pass %>

reboot

%post
(
# Report success back to Foreman
curl -s -o /dev/null --insecure <%= foreman_url %>
) 2>&1 | tee /mnt/sysimage/root/install.post.log

exit 0

%end

This template will pull the content from the –ref you specify, and the URL would be $FOREMANSERVER/pub/atomic/content/repo/. If you visit this URL, you should be able to find the docker-host file at the end of the hierarchy, specifically heads/fedora-atomic/f22/x86_64/docker-host. If you cannot find docker-host there because you’re using this tutorial for Fedora 23 and it has changed, I would recommend you to peruse your /pub/atomic folder and find the correct URL. Currently this is the structure for RHEL 7 and Fedora 22.

It’s time to associate these templates with the operating system, and deploy the host. Go back to Hosts > Operating systems, and click on your Atomic operating system to associate the templates:

We’re all set. Time to deploy the host. I’m doing this in Libvirt as it’s where my PXE network is configured.  Go to Hosts > New host and choose a name for your Atomic host. The Puppet options are irrelevant, as Puppet is not able to modify the Atomic ostree for the moment. Select the right domain and subnet, to ensure you’re booting in a PXE-enabled network. Something I found is that Anaconda tends to get stuck when I have tried to provision Atomic systems with less than 1GB of RAM, so I would recommend to assign that amount of RAM to your Atomic host at least.

The operating systems tab should look similar to this. Remember this is network-based provisioning.

If everything went well, your system will PXE boot and start Anaconda right away. If you have VNC access to the machine, it will look similar to this:

After Anaconda finishes the installation, you should be able to SSH into that machine using the root password you provided in Foreman.

We’re done! I suggest you use this host as a Kubernetes master, minion, or as a Docker host. For now, I will investigate how to pass the proper parameters through Foreman to provide Atomic/Kubernetes cluster provisioning.

After breakfast, hopefully 7:15am, I make a list of things I want to carry out on that day. I write this list by hand on a notepad to always keep it in front of me. This includes some tasks that might not have to do with work.  I estimate how long each of these will take, so I know at the start of the day what can I realistically accomplish. Anything that I haven’t done and it’s important, I move it to the next day, or I add a reminder in my calendar. Anything that I haven’t done and is not overly important is left behind.

Task estimations allows me to say – “There’s a meeting in 15 minutes. Let’s pick a 10 minutes task and do it.”  This helps me to avoid the paradox of choice. Without estimations, I would have hesitated for 5 minutes, picked one task, and most likely run late to the meeting so I could finish it. They also allow me to know beforehand how busy will my day be, and how much spare time will I get.

Aside from daily Japanese study, which I do every single day, I mark three tasks with a black line, meaning these are the 3 priorities for the day. Almost invariably, code reviews are one of them.  I start the day off by doing around 30-40 minutes of email, but I don’t count it as part of the 3 priorities. I use offlineimap and mutt for that, and unless necessary, I don’t check email for the rest of the day. After lunch I might refresh to see if anyone needs something urgently, but that’s it.

I work in chunks of about 50 minutes, taking breaks of 5 minutes which I spend cleaning, reviewing kanji, playing with the cat, or just not sitting.  Working for more than 1 hour will usually result on a longer break. I use workrave to keep track of this, and it works like a charm. Even if I’m not working, I still work in this fashion as I know I have a tendency to get RSI or tendinitis.

The way I choose what tasks to put on the list -work related- is defined by a pyramid I made. Top means most important, bottom means least important.

• level 1 – Security bugs, CVEs
• level 2 – Release blockers (upstream or downstream)
• level 3 – Learning opportunities. Feature team work
• level 4 – Features that will bring new users. Important bug fixes
• level 5 – Features that might bring new users. Minor bug fixes

If it isn’t clear from the priorities ‘pyramid’ I’m mostly focused on getting more people to our projects, and delivering timely, stable releases. That doesn’t mean I let minor bugs ‘starve’ on the queue, but it’s unlikely they would be solved by me if there is a release soon and other issues are blocking it. As any human, there are times where I simply don’t have the energy to properly work or review on level 1,2,3 items, and I resort to simpler issues as a way to clear my mind and get something done.

Since I work on an open source project, reviewing contributions is also part of my work at Red Hat. It is not a minor part of my job, in fact I probably spend 1/3rd of my working hours doing that. I apply the same priorities as I listed above. However, I have not yet found a way to deal with IRC support and Redmine emails. For the former, I normally do it when I’m working on something that does not require my full attention, or if I see someone asking questions about topics I know few people are familiar with. I would love to find a way to process Redmine emails more efficiently, I used to only check closed issues, and change the release tag if needed, but it was very time consuming and I would miss support tickets and not learn about feature requests and bugs.

Any suggestions, criticisms or comments would be highly appreciated.

[1] Some people asked me why would I not use some TO DO app to keep it in sync with my phone. The reason is simple, I want to always keep that list in mind when I’m in work mode and I don’t want to look at it when I’m not. I do use Evernote to keep track of yearly goals I set through a weekly system.

This is roughly a repost of an internal newsletter I run at Red Hat.

Thanks to Edmond Lau, whose book The Effective Engineer helped me to design this system.

Nearly every time the word ‘metric’ comes up in software development, is to drop another diss about it. This post is not meant to rant about the latest fad on how to measure code quality (wtfs/minute), monitor your developers (ROI), or count your sushi.

Instead, I would like to share how complicated it is to measure software developers quality, and how I struggle to get better at some areas.

The wall

I struggle because either me or this profession lack a standard ladder of merits to climb. It sort of looks like a maze in some ways. Surely other developers trying to improve their skills can relate to the feeling of studying up some exotic topic until you hit the wall and think: “How is this really making me improve as a software developer?”.

At that point, you either walk back your steps and rethink what do you want to learn next, or you break the wall and connect other paths to it. The decision is up to you, but there are a number of factors that can help you lean towards smashing the wall. I can’t really help with this, but here are some of the questions I pose to myself before continuing:

• Are there plenty of positive reviews of the topic?
• What do my friends say about it?
• Is this something fundamental to Computer Science or some applied technology?
• Will this broaden my knowledge in unexpected ways?

Again, these questions are just personal and more often than not their answer is blurry when you have just hit the wall. However, I do think it’s a good exercise to outline what are the benefits of devoting a big chunk of your free time to learning something.

These questions help me decide whether the wall is worth breaking or not. Moreover, how do I know how thick is it, and when will I start making out paths cross?

Skill acquisition measurement

There are not so many models that try to measure how far ahead are we with a certain skill. Dreyfus model of skill acquisition, four stages of competence, are two possible ways of merely figuring it out. I see at least two cons with the way these models can be applied to software development.

First off, software development is a vast field. It covers stuff from splay trees, to linear bounded automata or device drivers programming. The point is, it’s quite difficult to measure ‘raw software developer quality’. It’s not so difficult to measure skills.

Nonetheless an expert on some skills is needed to measure these skills. I would venture to say most people exploring a field, do not have access to experts who can let them know how much have they learned. Students get hopeless, and walk back without proper knowledge of the topic, and mostly having wasted a lot of time. I acknowledge and I’m grateful universities and moocs help fixing this issue, but still I’d love to find a way self-learners can mimic the experience themselves at home.

According to Dreyfus model of skill acquisition, experts in a topic make 1-5 % of the population, they write the books, push the research boundaries, and have developed great intuition. I believe this intuition solving problems is what makes the rest of the world believe they are geniuses.

In the end, no one is neither an expert nor a novice at all things software. Systematizing human knowledge into an array with defined boxes is hard. Hell, even knowing what ‘quantity’ to put in each of the boxes is really hard. Is it even worth it worrying about this?

To put things in context, I am spending a lot of free time digging these walls:

• SICP
• Contributions to MRI
• Conway’s game of life
• Clojure

Before you head on to more interesting places, two questions:

• How do you know whether a topic is worth exploring or not?
• How do you know when should you stop digging and move on?

Some useful links:

Pragmatic thinking and learning

Programmer competency matrix

5 stages of programmer incompetence

The four stages of programming competence