wrong tool

You are finite. Zathras is finite. This is wrong tool.

Facebook hitting a billion people – is this the day that open messaging died?

by Leave a Comment

In the 1980’s visionary technologists created e-mail as an open non-proprietary messaging system. This allowed anyone to communicate with anyone on open networks.

With Facebook hitting a billion, and WhatsApp hitting 900 million people, we now have a new proprietary network that has the reach that email does.

In the open messaging world, messages were owned by the sender, and the recipient and were portable. Your social network – the set of people you interact with – and your chats were owned by the people who created them.

Facebook and Whatsapp have now up-ended that open communication channel. They own your social network and they own your messages and they have the reach to displace open communications. A private entity owns your friends and the relationship to your friends.

And hence snapchat and wickr. If there is no portability and durability that is independent of the service provider, then you may as well treat the messages as ephemeral.

Who would have thought, my most private and important data would be owned by someone else.

Oh brave new world!

The completely misunderstood IOPS

by Leave a Comment

I was recently in a meeting about performance where the discussion turned to how many IOPS was the database doing.

And what was interesting was how much of our thinking about performance is formed in a world where IOPS are a scarce resource because the underlying media was soooo slow.

In the modern, post spinning rust world, IOPS are practically free. The bottleneck is not the underlying media, SSD’s and later things like 3D Xpoint memory (what a horrible, horrible name for such an important technology) have essentially free IOPS. The bottleneck is no longer the media (disk drive) but instead the electronics that sit in front of the media.

The electronics include things like networks, memory busses, and CPU’s. We are now bandwidth and CPU constrained, no longer media constrained. What that means is – of course – interesting.

One practical consideration is that looking to optimize IOPS is no longer a worthy effort. Instead, we should be looking at CPU and Memory cost per IOP. And we should be willing to trade off some CPU and Memory for more IOPS to improve overall system behavior.

For folks, like myself, who grew up working really hard to try and avoid doing disk operations, embracing IO is going to be hard…

And like a buddy of mine once said, these material scientists keep investing these new exotic technologies that keep us system software engineers busy.

It’s a good time to work in systems.

Metrics over usability 

by Leave a Comment

  
This is the kind of shit that drove Zynga customers nuts.

In an attempt to drive metrics to other features … We add friction to the top activity… I didn’t know about collages and not do I care to know about them and I certainly don’t want to be reminded of them all of the time.

I used to be able to just enter a status, now I have to pick one.

This is just another example of egregious Facebook metric driven feature – like the hyper aggressive attempts to get me to turn on notifications.

The end of storage tiers

by Leave a Comment

I wrote about this in 2008 on my now defunct corporate blog at NetApp. It’s fun to be working at a company that can actually create the IOPS tier.

Flash has once again thrown into stark relief the absurd classification of storage into tiers

Talk to a storage vendor and Tier 1 is their most expensive stuff. Talk to a storage architect and Tier 1 is their most expensive stuff. If you’re lucky there is some overlap.

Then we have Flash. Is it Tier 0? Does Flash make Disk Tier 5? What is the role of Flash and Disk? Is Disk the new Tape? So do we need to have Tier -1 for storage that is faster than Flash?

Then there is the whole disk storage is secondary storage. Secondary to what?

I never really did get all of those classifications of storage into tiers. I tend to think of storage in terms of how it is used.

So instead let me propose a new model for storage tiers based on the ratio of application CPU and memory, the amount of IOPS required and the capacity needs of the application or the ratio CPU:memory:IOPS:Capacity.

Based on that ratio there are three storage tiers

  1. Captive IOPS, where IOPS are all dedicated to a single application. In this deployment the ratio is 1:1:1:1. Add more CPU and Memory and you add more IOPS and Capacity. Because of the nature of the application and how many IOPS it consumes, there is nothing left over for another application.
  2. Shared IOPS, where IOPS are shared across a collection of applications. In this deployment the ratio is M:N:1:1. As you add more CPU and memory, the number of IOPS increase but not at the same rate. So you can share the IOPS across a number of applications rather than dedicating them to a single application.
  3. Capacity Efficient where the number of IOPS is dwarfed by the capacity requirements. In this deployment the ration M:N:1:Q. where as M and N increase, Q increases but IOPS do not. A good example is a backup server. As more data gets backed you need more capacity, but you don’t actually need more IOPS. Another good example is a home directory where capacity needs increase, but actual IOPS do not.

Next, I’ll explore the implications of these three tiers.

The declining value of stock options in an IPO free world

by 1 Comment

One of the tricky bits about massive valuations, and long drawn out periods before an IPO is cash compensation for employees.

This article does a really good job of pointing out another emerging crisis.

The basic challenge is that as the time to IPO expands because of secondary private equity markets, employees begin to devalue the value of their options because they are not liquid putting upward pressure on their salaries.

Let me explain.

Suppose you earn 100k a year. A pretty awesome chunk of change mind you … Now at Big Corp you can expect to earn 400k over four years.

A startup makes an offer, and it has less cash and more equity. The cash component is 70k. And so at the startup you can expect to earn 280k over four years.

The gap between Big Corp and Startup is 120k over four years. Over eight years the gap is 240k.

Basically every four years at the Startup is equivalent to working one less year at Big Company.

Thank goodness you say for options. The options will cover the difference between the Startup and the Big Company, hopefully by a lot.

And here’s where things get dicey, the longer it takes to get the option money from Startup, the bigger the financial exposure from a startup failure.

In a normal environment to compensate for the increase risk exposure, you would need more equity. The longer it takes to see if an investment pans out, the higher the rate of return should be.

Let’s put it this way:

Big Corp Comp = 400k

Startup Comp = 280k  and 2 million dollar option outcome with 10% certainty.

The expected value of Big Corp is 400k

The expected value of Startup is 480k (280  + 2000*.10)

This looks pretty good! Except I am probably overstating the probability of the 2 million dollar option outcome… And if we start adding notions of present value of money, the startup outcome may not look so good…

If the time horizon gets extended out to 8 years,and option outcome is the same.

Big Corp Comp = 800k expected outcome

Startup Comp = 760k expected out come

Now you’re starting to lose out by staying at a startup.  And again if we consider that Big Corp gives you 30k a year more than Startup Comp, and that money can be invested then it’s even more favorable for Big Corp.

Therefore employees at Startups either have to larger salaries or bigger equity packages or have to believe that the option outcome is going to be much higher.

Given that the time horizon to liquidity events is increasing, my expectation is that salaries are going to increase.  If salaries do not then we may be in a bubble because we are all collectively underestimating the risk of the option outcome.

 

Android’s reach is the only thing keeping it relevant

by 2 Comments

Every so often some analyst publishes an article talking about changes in market share of iOS vs Android as a way to get people to read their articles. Like marketwatch just did. 

And those articles are just so wrong and simplistic in their analysis that I want to scream.

Let me help you read them a little bit better…

There are three ways you make money in the mobile market:

  •  Selling devices
  •  In app purchases / subscription services
  •  Selling advertising

There are two costs in the mobile market:

  • user acquisition (basically getting people to download the app)
  • development costs

And then there is the reality of income inequality  – there are a small number of people with a lot of money to spend on shit. 

Given all of this the most important question isn’t what is apple’s market share but:

of the people who have money how many does apple reach?

In this game, apple is dominating the competition and the competition is getting screwed in three ways:

(a) Apple’s lead in manufacturing is astonishing. 

(i) They are simply building shit that other people can’t 

When they build the Mac Pro with the sold aluminum case they bought all of the lathes that could make that case and two years worth of production of the lathes. That meant their competitors could only build a case like Apple’s 3 years after apple had already shipped the mac pro. 

(ii) Their supply chain is magical. Tim Cook, a supply chain dude, became CEO of Apple to reflect the importance of supply chain management. 

And this lead results in – better products – than their competitors can make

(b) Because of the share of people who have dollars, app developers are making better apps for iOS than android

If you have an android device, buy an ipHone and use the same apps, and you’ll see the difference immediately. The iOS are simply better built. And Apple has gone to great lengths to make it necessary to build two copies of the App. And given 5$ of investment, companies will go with 4$ on Apple and 1$ on Android. 

(c) Because of carrier subsidies

People are paying significantly less than list price for a phone and because carriers want to get people to use their networks, they will happily subsidize phones … This allows Apple to sell a high value product at a low price. 

Put differently:

Consumers always buy the best value, and the best value is the best built phone running the best apps at the best price and Apple has an incredible lead in both areas.  

Then why do people focus on reach?

Because I lied about the three ways.There is a fourth:

Use the phone to sell a service (Uber/Google Search/Facebook/Amazon) or enterprise customers like Microsoft Outlook/Prezi etc. 

For those vendors, having an app on the widest set of platforms is crucial to business success because it affects reach.

But because they have to be an all platforms, they have to port to every platform.

The real observation hear is that as long as Android has 80+% market share, for reasons of reach app developers will port to Android. But because the $$$ they expect to make, the ports will be crappier than the iOS versions. Maybe for some guys – like Facebook or Uber – they will have apps that are equivalently great – who knows. 

There is this theory  – Android has reach, and eventually it will make enough money. There are two problems with this theory:

Reaching the top 10% of income earners is more valuable than the remaining 90% because of income distribution. I would rather have 100% of the top 10% than 100% of the bottom 90% of global earners. 

Reaching 90% of the people is more expensive than reaching 10% of the people – in raw dollars. And so the value of each guy with less money is less than one guy with more money. 

If Android had no reach, we would be talking about anti-trust not about Apple’s demise. 

The real numbers to pay attention to with Apple is:

(1) What is their share of market share $$$ (and here it’s crushing the competition silly)

(2) Are enough apps making more money on Android that would cause the general App vendors to pivot their top resources to Android vs IOS

until (1) or (2) slip apple owns this baby reach ## be damned. 

Are Unicorns really dangerous to employees?

by Leave a Comment

One of the more intriguing questions, I keep asking myself, is whether the myopic fixation on bubble valuations is reasonable.

At the end of the day, I think what I have convinced myself is that

  1. investors are not investing at ridiculous valuations,
  2. founders and very early employees are cashing out, and yes
  3. some people are going to lose money
  4. it’s not like it was 2001 where retail investors buying on margin were buying theglobe.com

Then who cares? The reason I do is because

  1. I like to understand things
  2. In any new financial instrument someone is assuming more risk and someone less

And I am beginning to conclude that employees are assuming more risk.

In the valley, we’ve created a  system where you have a relatively low-base salary and a very high variable income. The goal of the system is to encourage employees to keep playing lottery with their time in the hopes that they strike it rich.

The problem with this system, from the perspective of the owners of the company, is that the more equity you hand over to the employees, the less equity the owners have.

If a company has 1000 shares, the goal of the owners is to maximize the number they own and minimize the number everyone else has. The problem is that the way employees think about their compensation forces the company to keep giving out shares to new employees, and thus the % of the company owned by owners shrinks over time. The company has to keep issuing new shares, because at some point they have no more shares to give out.  The owner to retain his share of the company, has to keep buying shares and that increases his risk as more and more of his money is put into one company. And one important class of owners, the founders, typically doesn’t have the cash necessary to preserve his share of the company.

Therefore, the goal of the owner is to minimize the number of shares issued for employees. An approach to solving that conundrum is to increase the per-share value. The way you increase per-share value is have investors buy into the company at a high valuation. The problem is that investors don’t want to assume that kind of risk for their investment. And so we have liquidation etc preferences that allow the valuation to be set high but the effective purchase price to be set low.

To keep dilution to a minimum, the founders are able to drive the value of the stock up with investor money and allow the investors to not assume the risk of the high valuation…

The risk of high per-share price is transferred entirely onto the employees for the benefit of the founder and early investors.

Brilliant…

Let’s try that again…

Any half-decent engineer will evaluate their salary like this:

Total Income = Cash + Equity

And stock option equity will be valued like this:

Equity = %of company (Expected Value at time of Cash Out – Current Value)

Any  RSU equity will bed value  like this:

Equity = #RSU * Current Value + #RSU * Expected Value of company when you cash out.

Suppose you are at a company X, your compensation at company X is TotalIncome(X).

When you go to a Unicorn what they will do to make a competitive and attractive offer is the following

TotalIncome(Unicorn) > TotalIncome(X)

Where TotalIncome(Unicorn) = Cash(Unicorn) + Equity(Unicorn)

The way they do it is by saying:

Equity(Unicorn) > Equity(X)

So far so good… Nothing wrong so far.

But remember the value of Equity is very dependent on two parameters:

  1. Current Value
  2. Cash out Value

And here’s where Unicorns can really hurt employees. Unicorns like to offer RSU’s.

  1. Because of the high current value of RSU they can offer a small number
  2. The cash out value – because it’s only common stock – only matters if the company IPO

Giving out a small number really matters to founders and investors who care about dilution. The more shares you give out, the less each share is worth. A high-growth company that is hiring a lot of people prior to the Unicorn phenomenon would keep printing shares to keep hiring employees and that would cause the early investors to get PO’ed. The stock dilution and the employee lockup was a big deal in 2001.

Not so much now.

And here’s how ….

If you are a Unicorn, any time you need to issue more shares or deal with compensation issues, you just artificially increase the value of your company through another round, and hand fewer higher value shares to new employees. This allows you to both simultaneously keep TotalCompensation competitive and keep the number of shares static.

If you are particularly craven as a Unicorn, you can have Cash be lower than your competition with a small number of RSU’s whose value is mythological.

so far so good.

And this is okay if and only if every single company IPO’s and the public markets agree to the private market valuations.

And that still would be okay if everyone was taking on the same risk. I mean everyone, founders, all employees and investors. Except they are not.

The founders benefit the most from the lack of dilution since they own the most shares at the lowest possible value.

The investors are buying into the company at a much lower value. Think about it, you as an employee are buying with your sweat equity valued at 1 billion, and the investor is buying equity valued at 200 million … The guy buying at 1 billion is going to be worse off than the guy buying at 200 million.

At some level, you can argue that I am just describing how start-ups work. And at some level I am…. Except …  the valuations are not being set in the public markets, but in private ones, and the valuation is being set to a billion for reasons other than the actual earnings of the company.

The Unicorn valuations are useful for retention, hiring, advertising, lead generation and ego and a product of a negotiation with little downside for the people doing the negotiation.

In effect, employees are getting less equity based on valuation that has nothing to do with the actual earnings of the company … Instead they are getting paid based on a valuation whose opacity exists by design.

Unfortunately, this kind of shit never ends well…

 

Bubble mania

by 1 Comment

 
My neighborhood was once a dumpy area of the Bay Area. Close to where the old fab plants were this was – despite its proximity to 280, and large lots is a depressed area.

Thanks to Apple landing it’s spaceship near my house and the overall housing bubble things have gotten silly. 

And this flyer from a realtor just made me laugh. They are using Monopoly money and tags to get me to sell my house now now now…

As Takei would say Oh Myyyyy

It’s all about the constants

by Leave a Comment

When I was a student in college, I learned about algorithmic complexity and in particular about the Big O notation.

And what I found fascinating is how the folks who studied algorithms ignored constants.

Over the last 20 or so years, I have begun to realize that I made a career about caring about the constants. That my entire engineering career has been about something that folks who care about algorithms dismiss as irrelevant.

And that got me thinking. As a software engineer, I tend to think that I am making code go faster.

And then I worked at Juniper and the irrelevancy of my efforts on performance became clear.

Physicists make hardware go faster.

Algorithm designers make hardware and software go faster.

The rest of us just sit around trying to tweak the constants. All of our obsession about tight code and efficient code is really a pointless exercise in worrying about constants.

And I got depressed.

And then I remembered that in the world I work in, constants do matter. Because constants map to hardware that people have to buy, and saving a small amount can translate into large amounts of savings.

 

 

Venture Capital has become Private Equity

by Leave a Comment

Private Equity firms are about making money. They buy companies and then extract cash from them at the expense of everyone around them. Nothing wrong about that, it is what it is.

Venture Capital is about a shared risk between the entrepreneur and the venture capital firm. The firm puts the capital, the entrepreneur the sweat and the outcome is either great wealth or middling results or nada.

The emergence of Unicorns and their liquidation preferences and the ability of entrepreneurs to cash out during the investment has transformed the start-up scene considerably into something that feels a lot more like private equity.

Let’s walk you through the numbers.

In a private equity (PE) deal, the company is bought at some value X. Employees are then given signing bonuses to stick around for a period of time while the PE firm tries to extract either a better selling price or enough cash to make a significant return on the initial investment.

In a Unicorn billion dollar investment we have turned VC investment into a PE deal.

VC’s are not investing money they are buying companies with liquidation preferences guaranteeing their X. Why do I say that? Because if the company is sold for X, then the only group that makes any money is the VC firm. Consider a company that gets a 160 million dollar investment at 3 billion dollar valuation. That company, if it gets sold for 160 million dollars, then every cent will go to theVC’s. That’s what I call buying a company for 160 million dollars, not investing 160 million dollars in a company. There is no shared downside risk.

The founders meanwhile are given a one time signing bonus – basically they are allowed to cash out some amount of money and are allowed to keep a certain amount of equity. The bonus acts as a massive salary bump, and the equity should keep them somewhat motivated.

So what do we have:

A VC firm buys a company. Founders get paid to run the company. Sounds like a purchase not an investment.

This is not how VC investment has typically worked, I am told.

The real interesting insight is that because these are really just PE deals masquerading as VC investments is what makes these kind of deals so frigging attractive to investors. If you could buy some of these Unicorns for less than 300 million dollars, wouldn’t you? They are all good bets. The problem isn’t that the investors are making real money, or that the founders are making real money, the problem is that these deals are opaque and the market for them is illiquid and that this can create a serious problem when the market gets somewhat less frothy.

In the nightmare scenario where the market appetite for investments in illiquid early stage startups with excellent growth potential declines, the VC firms might find their customers looking to cash out. Essentially the guys investing in the VC firms may start wanting their money. And then we have this dichotomy between the VC firm that wants the money now and can get a decent return for a sale pre-IPO and the founders who have to sell to make real money. If the firm can IPO, then they will and if they can not then the company will get sold and then the employees will get nada. Because the company is structured in such a way where the VC firm owns the company, but employees owns an option to sell shares in a publicly traded company that is currently private (in other words, worthless stock) they do  not own a piece of the company they are currently working at.

Fun times… Love to learn I am wrong.