on the nature of short conversation and Facebook

A friend of mine died very suddenly this week. He was 37, they suspect a heart attack, which seems very odd. On some level, I hope it was due to a heart defect.

http://oandnfh.funeralplan2.com/obituaries.aspx?turl=http://hosting-24758.tributes.com/obituary/show/William-Anderson-Moore-101204852

I posted about it on my Facebook because I treat it a lot like I treat anything I have access to... it's a way to share myself, my thoughts, my feelings. While I am private about some aspects of my life, most of it I have been very open about all along.

I mused that it was troubling me. Mostly it troubled me because most of the other people in my life who have died... it wasn't a surprise. Theresa Koerbacher down the street? She got cancer as a kid and didn't win. David White? Car accident. My grandmother? She was old as dirt and it was probably for the best. My father? He had COPD and his life quality had severely diminished. Johnny Law? Well, he was a really large guy, we were shocked to find out he just fell down some stairs.

But this? The guy was only a few months older than me. And the thing was, he had such an interesting Facebook. He had taken up photography and his pictures were really good. I mean, I know we all have the amateur professionals in our friend groups, but his subjects and work always kept me interested. And despite all this, in about 2.5 years of being friends on Facebook, it looks like we interacted about twice. And the last time we saw each other was probably when we worked together back in the mid-late 90s.

All this said, I came to look forward to his FB postings because they provided a window into someone else's life. Someone's life that had maybe found a bit of clarity and contentment, even if it maybe wasn't exactly what he wanted. (Then again, how was I to know either way?)

I posted about this and the responses came. And I felt weird because the goal was not to make it about me. I was not looking for sympathy for my feelings. I was looking to connect with people, to interact, to come away with something new. Someone had left my life and it leaves a hole; I wanted to express that and to hear something from somebody else, even if it was just someone they had lost and felt the same way about.

I felt weird about this (the same way I did when I posted about Johnny Law) because I was lamenting more the loss of the person's creative work than the person themselves, being as I wasn't as close to the person as many other people were. It wasn't like a close friend died, it was someone that I actually knew and admired their work, no matter the medium.

So I was frustrated by the "I'm sorry for your loss" responses.

Because one day you're posting pictures of owls (and self-marveling at such) and then a few days later, you're gone.

That's it.

No more pictures of owls. Or dogs. Or sunrises. Or anything else.

Minnebar 9

Here's how I spent my time on Saturday at the local 900-person "unconference":

Paul Cantrell, "Scheduling Minnebar with Simulated Annealing" Great talk as is usual from Paul. He intro'd by saying that simulated annealing sounds very fancy but really the algorithm is very dumb. For Minnebar, the key metric is whether people can attend all of the sessions they have expressed interest in through the session web tool. So the function being optimized is the mean, over all attendees, of the percent of their sessions they cannot attend due to scheduling. This may mean it's possible to game the system by registering for fewer sessions. Somebody who registers for 6 will have 0.17/N weight for having two sessions scheduled opposite each other, while somebody who expresses interest in just 2 will have 0.5/N weight on their preference. Certainly you should not register for any you don't much care about.

I asked about whether it was possible to do better on assignment to rooms. Currently Paul just assigns time slots first, then greedily puts the session with the largest number of interested people in the largest room. But I think it may be possible to do better than this--- some recurring sessions are known to be very popular and yet do not get placed in the largest venues.

There were quite a few questions of the form "couldn't you find a better/faster solution with X" to which is response was (correctly) "no, we only run this once a year so we don't care very much."

"Pack Your Own Chute - the Personal Decision to Join a Startup", a panel discussion led by Neal Tovsen, with Paul DeBettignies, Liz Tupper, Matt Hardy, and Todd H Gardner. This one was fairly depressing but a good conversation. They talked about the need to set boundaries and have clear communication with your co-founder when forming a startup. How much money are you willing to lose on this project? Where's your bail-out point? What is the worst-case scenario? (They also talked about some of the fun and freedom too.)

What struck me is that everyone on the panel was talking about creating a startup rather than joining one. Nobody talked about taking a smaller plunge by going to work for somebody else's startup first. (I know I found my time at Kealia very valuable when we started Tintri--- and I wasn't even doing the heavy lifting on the corporate/financial/HR side!) But just like you might save up money to make sure you could survive the 18 months it took to bootstrap your startup, it may make total sense to work for 2-3 years in another startup to build connections and experience for your own.

Another issue that was not confronted directly was moving. Relocating is a big cost and one that I totally understand not wanting to pay--- I didn't. But it should be asked: if the only way to make this startup succeed is to move to San Francisco, would you do so?

I learned about Track:js from one of its founders on the panel (and the other seated next to me in the next session!) A good-looking tool for debugging Javascript problems that happen in production systems, and one Tintri might try out.

Reed Robinson, "Lessons from a Failed Startup". The failed startup was Heroic, a local attempt at a recommendation service for home services. (His dad repairs garage openers.) He had a good list which I'll try to reproduce here, but I wish his slides were available:

  1. Validate your assumptions. (Like, "people care about my product", "the team can build the product", and "people will pay for my product.")
  2. Sell it, then build it. Heroic spent lots of time on UX but decided not to ask people to pay right away. For his next venture he wants to sell the idea and find people willing to fund the development. (I am not a huge fan of this model but VC looks like that if you squint at it correctly.)
  3. Acknowledge holes in your team ASAP. He talked about how Heroic lacked marketing, finance, and sales expertise. It's possible to go overboard as well here--- but you also don't want to dig a hole.
  4. See things to fruition. Reed gave an example of promotions that were tried and quickly discarded after only a week or so. Some strategic moves need time.
  5. Do things that other people don't want to do. His father gets ten emails a week from people wanting to partner with him. Very crowded space, maybe focus on something less "sexy".
  6. Intimately know your cash situation. Ties in with holes in the team.
  7. Have confidence, but don't take yourself too seriously. Celebrate successes.


Bridget Kromhout, Monitoring at a SaaS Startup. Talked about her work in ops at 8thBridge, and a wide variety of tools. One thing the Tintri support lead emphasizes is that alerts need to be actionable--- don't wake somebody up in the night for something they can't fix. Bridget reinforced this message strongly. Nagios is old but still a good tool for ensuring alerts are what you want. Some of the newer tools like Sensu are so complicated that ops people fear they need monitoring on their monitoring tool.

She shared a couple stories in which 8thBridged goofed some by not using some of the information they had available. MongoDB's MMS was telling them about a global write lock problem but they weren't monitoring or alerting on it. Etsy has a very good monitoring team and tries to monitor even those things that don't seem to be moving, "just in case they make a run for it."

Tools: Graphite and StatsD for collecting and showing stats (they feed info into nagios from them for actual alerting), Whisper for storing time-seris data, Carbon for buffering and storing stats. Slides with more references here: http://www.slideshare.net/bridgetkromhout/monitoring-at-a-saas-startup

Maybe Tintri should look at providing stats directly into some of these tools. She talked about Logstache (?), Kibana, and ElasticSearch as some next-generation tools it might be worth looking into.)

Jeff Lin of BustOut Solutions, "Chasing Ninja Rockstars: Searching for Top Talent and Why We're Doing it All Wrong". I should have given one this a miss. The key problem, as Jeff admitted, is that he doesn't do much hiring. His 13-person team is 5x smaller than the number of hires Tintri made last quarter. So when we asked questions like how to source more diverse candidates, or how to get team buy-in to change hiring practices, he didn't have any suggestions.

He had some good points about team culture mattering, and creativity requiring diversity, but it was not very practical advice on how to get there.

Jeff told an anecdote that pissed me off. At a previous company (not BustOut) his boss took the resumes and filtered out all those with Master's or PhD degrees and said he basically didn't want to hire anybody who had spent too much time in the "ivory tower" because they were out of touch with technology trends.

I'm not sure what Jeff thought the point of this story was (I think it was in support of skills over credentialism) but it directly contradicted much of the rest of what he said about looking for good problem-solvers and people who were enthusiastic learners. Nobody goes for their PhD to get a better salary. (The same may not necessarily be true of MSCS.) I suspect a not-so-subtle side effect is that Jeff's old boss didn't want to pay more for developers.

James Renkin, "Building A Global, Privacy-Conscious CDN On $20 A Day". This talk was pure geeky joy. James located 11 sites throughout the globe that were willing to let his virtual server peer with their BGP (the Internet's routing protocol.) He was thus able to create a content distribution network on his own using BGP anycast to direct users to the "closest" (sort of) server. This is what the big guys do (some root DNS servers use it, Akamai might but usually just uses DNS redirection.) But he did it for an outlay of about $7000/year. About $500 of that is the IP address range (which you have to jump through multiple hoops to get--- he's got an ISP side business which he was able to leverage.) The remainder is running the servers and additional charges for injecting his route into BGP. I learned a lot of this stuff during my PhD work, so this talk made me really happy. I have no intention of duplicating his work, though.

The conference had a couple innovations that didn't work so well. Best Buy let us use their employee parking garage this year instead of visitor parking, but frankly it's a maze (not very well marked, poor traffic flow.) They only had one exit gate opening to leave and more than one person got into the wrong lane.

The organizers brought in food trucks to serve lunch but the lines were long and it was not all that warm a day to be standing outside. I think three trucks was insufficient; it didn't help that it was not clear there were three separate lines, and that one of the trucks opened late.

There was also a keynote from an agile coach that I didn't get much out of. I was amused that he emphasized subtracting things to get more productivity but he's usually called in as an "add" to the team.

"Love and Math" vs "Flash Boys"

"Love and Math" is by Edward Frenkel. It tries to show the beauty of modern mathematics by taking the reader on an autobiographical tour of Frenkel's career, explaining the concepts and connections he's worked with over the years. Mathematical beauty, he says, ought to be accessible to non-mathematicians in the same way that beautiful paintings are accessible to non-painters.

"Flash Boys" is by Michael Lewis. It tries to show the workings of modern stock markets by taking the reader on a biographical tour of Brad Katsuyama's career and the founding of IEX. Wall Street, he says, is screwing the little guy (and big institutional investors) by delivering their trades into the hands of high-frequency traders.

Frenkel doesn't succeed in making the math simple. I had some undergraduate courses in "modern algebra" (not really modern at all), including Galois theory, and have maintained an amateur interest since then. I found myself flailing about halfway through the book, and never really developed a good sense what the Langlands Program actually was. The analogy he uses is a Rosetta stone tying together number theory, curves over finite fields (which he does a good job explaining), and Reimann surfaces (ditto.) Quantum theory comes in at the end too! But few of the conjectures that tie the different languages together are stated explicitly. Much of the mathematical meat is buried in the endnotes. This makes it harder for a somewhat-mathematically-sophisticated reader like me to follow the flow, but I can't see how somebody with less background than I comprehends the ideas and examples at all. There are a lot of concepts floating around that make me echo Enrico Fermi: "If I could remember the names of all these particles, I'd be a botanist."

Here's an example: I understand finite fields, and curves over finite fields. I kind of understand a manifold, and his glossary provides the definition "A smooth geoemetric shape such as a circle, a sphere, or the surface of a donut." He does a good job developing these ideas. Then later we get "manifold over a finite field" and I'm lost--- finite fields aren't smooth! Google or Wikipedia are no help, I just get dumped into reading about Calabi-Yau manifolds, whatever the heck they are.

That said, I applaud Frenkel for the attempt. I really appreciate having a book about what modern mathematicians are really doing. The sections on being a Jewish mathematician in the USSR are alone worth the price of admission. Antisemitism kept him out of Moscow University (and would have placed further roadblocks on graduate study) but the mathematicians of Moscow created informal networks that trained an entire generation of brilliant students.

Frenkel is also unabashedly Platonist which causes some hair-tearing. Perhaps pure mathematicians should be introduced to the concept of "confirmation bias."

Lewis's book succeeds where Frenkel does not. It not only tells a good story, but it also lays out very clearly how high-frequency trading works and affects institutional investors. This picture may not be accurate or complete--- but it's at least part of the story.

In brief, today's stock market is not a singular location. There are numerous exchanges each with their own rules, practices, and costs. When a broker gets a buy order he typically has to fill it from multiple exchanges (in fact a regulation requires him to go find the best prices!) But a high-frequency trader can monitor his activity on a subset of exchanges (or even one) and beat the broker to other exchanges, buy the available stock and re-sell it immediately at a slightly higher price. This is how high-frequency traders account for 50% of the shares traded--- they're not actually providing any "liquidity", they're just stepping in front of party-to-party sales that would otherwise happen anyway.

There are, of course, numerous criticisms of Lewis's account. Some claim this dynamic was already well known and published in books at the time his protagonist was puzzling it out. Or that the same was true before electronic exchanges. (This is not a positive.) Or that sophisticated investors were already deploying countermeasures before IEX, and IEX is a scam of some sort under the thumb of known insider traders. Or that Lewis is getting only the "buy-sider" picture of the world and HFT provides incalculable benefits in ways that only people who really know the stock exchanges can understand. That the whole book is nothing but an advertisement for IEX, who are less savvy and ethical than actually portrayed.

Some of these criticisms I think miss the mark. It's not important who was first to understand HFT or develop countermeasures; it's important to explain it in a clear and engaging fashion. Frankel likes to think that his mathematics is clear, elegant, and beautiful to an outsider--- but ends up, frankly, in a morass of jargon. Financial insiders like to cloud their basic operation in a cloud of strange terms and vague feel-good explanations--- but where their money comes from can usually be explained very simply.

The silliest criticism I hear is that individual investors ("the little guy") shouldn't care because their trades are so small that HFT can't exploit them. This is an out-and-out lie in two forms. The first is that some individual trades are certainly big enough to suffer. One of the telling anecdotes in the book is a trader with Bloomberg access performing a late-night trade on his personal account, and setting off a flurry of activity which he can observe. The second is that individual investors often put their money into mutual funds which are--- guess what--- big institutional investors who are affected.

Michael Lewis told Salon that IEX has been flooded with resumes and would-be whistleblowers since the publication of his book. Sadly, I think Frenkel's book is not likely to have a similar impact.