I was honoured to be asked to pay tribute to Phil Rowlands at SRECon EMEA this morning. Below is a transcript of the remarks, and some photos, which did the inadequate-by-design task of representing someone’s life in ten minutes. But it needed to be done, however inadequate it was destined to be.

Introduction #

A large majority of you are new, so most of you won’t know - or have encountered - Philip Rowlands, commonly known as Phil.

Here he is, looking rather devilish due to the cast of light. But I think he’d prefer daemonic.

Why? #

The reason I’m on stage now is to introduce him to you, you who mostly didn’t know him, because he was the co-chair of SRECon EMEA, and largely responsible for the programme and the event you are about to experience, before he was suddenly taken from us in August of this year.

I want to say before I go much further that I appreciate being able to borrow from the words of others - particularly Alexandra, Trevor, Julian, Steve, and other ex-colleagues, as well as my own memories, to paint this picture. [Which is partial, to be clear! I only really knew him for a small slice of his career, before he went back to England.]

Jane Street #

And I also want to thank Jane Street since they have made a very generous donation to USENIX in his memory, and this marks one of those increasingly frequent occasions where hedge funds are doing the morally decent thing. (Do you remember when they were the enemy?)

Career #

Phil’s career is a shopping list of a set of organisations famously hard to get into, famously successful, and famously demanding. Many organisations talk a good game about only hiring the best, but we can safely say that someone with this many years at Google, Daiwa, Two Sigma, and Jane Street does not do that without being one of the best.

And Phil was technically one of the best.

He was also one of the best, technically.

Rimshot #

Early Google #

I myself first met Phil in the combination pressure cooker and chocolate factory of early stage Google, just across the river here. He was one of the first people working for Google Engineering in Dublin in early ’05, and one of the first people to take the extremely non-trivial burden of on-call for all of Google’s user-facing infrastructure.

That team had a 3-minute SLA, which is “bring your laptop into the toilet with you” territory (as I unfortunately know).

Testing on the Toilet #

But as well as helping to demonstrate that Google could in fact actually extrude production authority across the globe – which really is no easy matter – Phil, in the words of Claire Stapleton, thought infrastructure was an art.

Elegance #

He made systems that were necessary, efficient, well-behaved, pragmatic, and if you closed your eyelids and squinted a little and rotated what you were looking a little so it caught the light just so, they could even occasionally pass as elegant.

NTP #

For example! He designed the first production deployment of NTP for Google, and when the famous leap second incident of late 2005 happened - which messed Google storage up real good - he led the response to keep 2008 from messing us up as much. His design document was canonical for NTP for about a decade, which is an astonishingly long time in computers.

But it was probably DNS where he had the most effect, suggesting a way to get the same load-balancing magic for unqualified domains as happens for qualified ones – which is partially why you can go to google.com rather than www.google.com and get low latency – Phil did that. He also helped out with some of the public standards work required to get client subnet DNS loadbalancing working properly in RFC 7871.

RFC 7871 #

…where he is immortalised today. Not a bad list of names to be beside!

He had a strong orientation towards community, which is why he presented here, at SRECon EMEA, on both those topics.

Postcards from the Edge (Networking) #

He covered about 2000 years of time-keeping history in 45 minutes, which is a pretty good compression ratio.

Paul Mousematapetris #

I greatly enjoyed his DNS talk, and not just because I won one of the mouse-mat prizes for knowing the surname of Paul Mockapetris.

Chair #

Well, regardless, his instinct and talent for communication was I think partially why he was co-chair.

And I think that’s a moment to stop and reflect on character, rather than achievement. I mean - obviously they are related.

Character #

I found Phil a gentle and sympathetic soul, immensely talented and unassumingly clever, though everyone who worked with him knew his impish sense of humour could not be suppressed. The sheer delight he took in choosing the most literal - and trolliest - interpretation was something to behold. A true pedant’s pedant.

RFC 5321 #

For example! The RFCs for electronic mail – as if anyone still used that - require that all domains maintain a reachable email address. In the words of Phil’s ex-boss and friend Trevor, “a standard written for a different kind of Internet. Phil would dutifully wade into the morass of that mailbox, cherry-pick the ‘best’ messages from Internet cranks, and read them aloud to the rest of the team–weighing with tremendous gravity the various evidence and arguments being presented. It would be hard to describe those emails as being actionable, but Phil was insistent that it was a vital function, required by standard, and thus his sacred duty, despite what his management chain might think.” 

Traffic light system #

But Phil also had a kind of genius for the opportunity to form community around physical objects.

Phil decided one day that we needed actual traffic lights for traffic SRE, and found a Greek company with an online wizard to design and procure two identical street-legal three-color traffic lights which were, of course, wired into pages and alerts.

I mean of course you would do that, that’s just obvious.

Over time, this grew and spread to other SRE offices and completely unrelated teams on other continents.

In 2020, a full decade after Phil had moved on from Google, there was even a COVID home edition with a self-assembly kit for people to build their own “SRE traffic light” for their home office.

Ribbon development #

As Lerna said: “I got the chance to get to know Phil more when we co-chaired lighting talks.  He was kind, clever and diligent, always giving his all. At some point we noticed that the lightning talks chairs didn’t get ribbons for their badge and somehow (I don’t know how) he managed to get us these light blue ribbons for co-chairing that year. We also joked about borrowing the bell from the building staff to keep the lightning talks speakers from going over time and next thing, Phil had managed to get the bell.”

The hatchet job #

Or that time that he brought a hatchet on a flight from the US for an office Halloween decoration, and then turned it into a potentially lethal pinyata club, with a giant Toblerone suspended from the ceiling.

Several health and safety violations there I should think.

SRE shanty town #

Or that time he insisted it couldn’t be a proper disaster recovery test without emergency shelter, and brought a tent into the office. And I think there was later schwag of tents associated with that project.

Generlensity #

He was generous to his friends as well – as a photography hobbyist, he bought Julian of Jane Street a special lens for his phone that allowed him to take different kinds of shots you couldn’t get without it – and as you’ll appreciate, something like that you put on your phone doesn’t come cheap.

Mornington #

But I think ultimately community came before accessories for Phil, and one of his major occupations was choral singing, in some sense the purest expression of collective effort towards beauty without harm. This is him with Mornington Singers - also my wife’s choir - and where he met his wife Alexandra.

Choral singers are also nerds. This is known. There were 2 or 3 SREs in Mornington at one point! And while we’re talking about nerdery… 

MOTD #

If anyone remembers what MOTD files are on UNIX systems, then you might know what this is already. It’s basically a text file that gets shown to you every time to log into a system.

Phil was very annoyed about this. Question to the audience - can you see what upset Phil?

Yes, that’s right - the wing-tips are misaligned!

Phil got a patch submitted to fix them, so your dragon is not greeting you every time with broken wings.

But after he passed away, Jane Street patched it again in turn, with his initials - lots and lots of people will see it every day.

Goodbye Phil, you will be missed. And specifically, you are missed today.