wrong tool

You are finite. Zathras is finite. This is wrong tool.

12 architecturalist papers: people write software

by Leave a Comment

When I first became an architect at NetApp, I thought the job was to draw a picture, get the picture approved and then the software would magically be written.

The mental model I had was that there was this massive “power-point to product” compiler and all I had to do was draw the power-point.

To my surprise, it was a little bit more complicated.

People write software, and people are not computers. People have emotions, aspirations, interests, career goals, dislikes, strengths and weaknesses. And those people write software.

How does this influence software system design?

In the first phase, you need to figure out what the right system is. Correctness or appropriateness of a system is independent of human beings.

But then to get it implemented, you need to understand your team.

There will be skills your team has, and there are skills your team needs to acquire and there are skills your team lacks and can’t learn and you need to go find in the marketplace.

And then your job, as a systems architect, is to figure out how to build something with the people you have that adds enough value so you can stay alive.

And sometimes it means you have to wait to hire the people you need.

In many ways, this process feels like being an author of a screenplay who tailors the screenplay to the actors you hired.

One of my projects at Zynga could not start until I hired someone who understood filesystems. And so I lived with data corruption and inconsistency because there was no one who could fix the problem. And when that person was hired, I had to wait for them to ramp up at Zynga. And when they finally ramped out, only then could I actually get them to work on the problem.

But finding the right person to solve a problem is the easy part of the job. Motivating them to solve the problem is the hard thing.

The really hard part is to motivate people to write the software. Remember people have lots of reasons why they do things. And people’s best work is done when they are fully engaged in a problem, when they show up wholly – mind and heart and body.

You don’t want extrinsic motivation, because you don’t get people’s best work.

And that means a bunch of things.

The simplest and most obvious is that people have to feel safe to be themselves. If people don’t feel safe, then they will not be there. They have to feel supported. They have to feel free to be their authentic self.

Screaming at people, dismissing people, being cruel, demonstrating how much smarter than them you are, trashing their work, is how you get something other than their best work. And sadly in my past lives, I had to have a boss explain this very simple thing to me. And I’ve had to be reminded of this on more occasions than I like.

The second is that they have to feel that what they want will happen. And what they want is not what you think it is.

A large part of the job as an architect is to spend time 1×1 with everyone and making sure that they are wholly engaged. And understand what they need. And everyone is different.

For example, a co-worker of mine was trying to re-architect a system, and he was running into flak from his team. And I asked him: Did you talk to everyone to see what they wanted from this effort? And he said, no. And I said: How can you convince people of something if you don’t know what they want?

So he scheduled a bunch of 1×1’s, found out what everyone wanted, and all of a sudden the flak evaporated. It didn’t evaporate because he listened to people, the flak evaporated because he adjusted his plan to meet their wants and needs.

Sometimes I get asked: Why do you spend so much time talking to people? And my answer is: People write software and I want people to be fully invested in a solution because that’s how they do their best work.

Do I always succeed? No. But it’s my North Star.

Godspeed Mark Pincus

by Leave a Comment

Mark, as always, does it his way. His recent announcement to relinquish control of Zynga is quintessential Mark, doing what he thinks the right thing is.

As an ex-Zynga employee, I am well aware of many things that we did that we are not proud of, and there are many things I wish we could have changed. And this is not the moment to rehash everything.

I wish the biggest problem in social media today was that we had too many cows. I really wish the world today had a game that 100 million people were playing. And that game was about a city where everyone was friendly. I wish we had a founder of a social media company who was more concerned about getting people to play together and having real social connections, not just randomly driving growth numbers.

And when we launched Zynga, the amount of hate we got from the gaming industry surprised me.

And after gamergate, I wonder if the problem that Zynga had was that we were the first company to build games that I could play while my kid was awake.

And when one of our games crossed the lines, Mark told them to go back.

For all of our flaws, we did not let our platform be a place you you could stalk or promote hate.

We did not let our games be sexist, misogynistic, etc.

We just wanted people to have fun,

Mark did it his own way.

Godspeed.

11 architecturalist papers: The area under the graph

by Leave a Comment

A year ago, in a promotion meeting, a senior technical leader warned us about promoting someone too soon. And he and I rarely agreed on anything, and I always learned something from our discussion.

His comment is that to be a successful technologist you need to have an area under the curve not just cross the threshold. And that time and experience were how you accumulated that area under the graph.

At NetApp, in my 20’s, I was determined to become a technical director as fast as possible. And I crossed the threshold, and they promoted me, and then it took me one more decade to learn the actual job. I was 33 years old, and I was the second youngest technical director.

And I had to have a lot of failures, and experiences before I actually could do the job properly. And I had to learn a lot from people.

For example, one thing I had to learn and then re-learn, is that in a new market all of your instincts are wrong. And I also had to learn that in all markets some things are always true, there are no shortcuts.

What I realized is that the sentences and verbs never change, but the nouns and adjectives do.

A lot of the job of strategic software architecture is not about technology but pattern matching.

And the job is to understand the details so you can map the right sentence. And that was a lesson I had to learn the hard way.

For example, when I went to Zynga, I sat in meetings, and people were using words that I had never heard before used in ways that made no sense. In my first meeting with the executive team, they said: “We need new IP”. And I was flabbergasted – why would a game company need a new internet protocol? Except they meant “Intellectual property” which meant “new game franchise” which meant “new product” and that the real discussion was about how do we launch a new line of business?

The sentence, in this case, was “How do we launch a new line of business,” the noun was “new IP.” And once you realize that a game company spends all of its time launching new games, it makes you rethink what the purpose of software is.

The solution, again from experience, was that before I could do the job they hired me for, I had to learn about the technology. And so I spent several months taking notes on words and asking questions, and within a few months, I had begun to match patterns.

When I went to Juniper, after Zynga, I took the same process. First, learn the nouns, and adjectives, then do the pattern matching and then start figuring out what needs to be done.

Being able to do that job, the pattern matching, you need the ability and the experience, and so getting promoted fast is great, but it takes time to be able to do the job.

10 architecturalist papers: the 8 year itch

by 1 Comment

The most intriguing part of the tech industry is the interplay between Moore’s law and opportunity.

My rule of thumb is the 8-year rule. The rule says the following; market disruptions happen every eight years because the incumbent software stack can’t be adapted to the new hardware.

As some examples…

My current employer took over the world because of the shift to multi-core servers. Apple took over the world courtesy of mobile processors becoming fast enough to run most useful applications. Microsoft and Linux took over the world when  Pentium closed the performance gap between x86 processors and RISC systems.

But why 8, because we human beings, don’t understand exponentially growing curves.

For small enough values of x, y=x*x is equivalent to y=2x. The two start to diverge when x = 3  or when y = 8.

So?

Let’s assume big tech titan has 80% of the market in year 0. Then in year 2, the new hardware emerges, but it’s not appreciably faster, except for some small use cases, so instead of selling to 8/10 customers, it’s now selling to 8/12. Then in year 4, it’s 8/14, and then in year six it’s 8/18, and you go from being 80% of the market to less than 50%, and there is probably some other tech-titan growing much faster than you were and you are the has-been tech-titan?

Many books and articles cover this topic.

What does this have to do with anything?

When you consider strategic software architecture, the tricky bit is to navigate across that 8-year transition.  And what makes it particularly tricky is that you have to assume that the software you had is going to be your boat-anchor, and simultaneously, your source of funding.

The challenge is that a mature software architecture that’s tuned for one market takes about 8 years. And the reason it takes about 8 years, is that any market takes 8 years to mature. And the reason it takes 8 years, is that it takes 8 years for the hardware to become capable enough to grow the market. And if your software is tuned for one market, it is not tuned for the next one.

How do you solve this?

The short answer is that you have to find the 8-year curve after the current one.

The strategy is the following:

  1.  Grow with the first 8-year cycle
  2.  Preserve market share during the 8-year cycle you missed and figure out what you will do next. Use this time to re-architect your system for the next grow opportunity.
  3.  Grow with the next 8-year cycle.

The tricky bit in my mind is to understand when you are in (2) and to realize that your goal is not to continue to improve what you had but to go build something new that builds on top of the assets you have in place.

Where I have failed in the past is not understanding the importance of (2). That it’s tempting to see some market you missed and try to attack it and repeatedly fail, instead of admitting you blew it and then trying to find the next thing.

So how do you set yourself up for (3)?

You have to think about where the puck is moving to, and then do everything while you are in phase (2) such that it lines up with where you think phase (3) will be.

And that’s the trick, to do the major re-architecting nominally for phase (2), but really for phase (3). And as a strategic software architect, that’s the hardest job, selling the future as an improvement on the present when the entire company is obsessed with a market they already lost, and with a market, they could win at not yet visible.

 

 

 

 

 

09 architecturalist papers: draw me a picture

by 3 Comments

One of the enduring myths about software architecture and in general technology leadership is the degree of control an architect has.

Our advocates believe we walk into a room, draw a picture, everyone listens to us and then code materializes.

And I have worked in places like that. Teams willingly were lead off a cliff.

I remember a time at NetApp where a team just wanted me to draw the picture. And I did and then projects got spun up and engineers got assigned. And then I left because that’s not the way I work.

The best teams don’t work that way.

The best teams draw the picture for you and you evaluate if the picture makes sense.

What you want is people to be passionate and believe in their solution. And only very rarely is what they are proposing wrong. Most of the time it’s good enough.

And so my job is to find out how to nudge them away from potential disasters.

Sometimes it can be exasperating because it’s not the picture you want drawn. And sometimes the picture is a compromise between organizations and not the best software. And there is always someone smart enough to point out the better picture that no one had any passion for.

And then they look at me as a failure, because isn’t my job to build the best possible system and force it down my teams throat?

And the answer is almost never.

The job is to have product that can evolve to be the best product it can be. And to do that you need a committed team. And a committed great team will always produce great software even if the picture isn’t exactly what you would have drawn.

Because drawing the picture – ironically – is almost never the job.

Reflections on Straight to Hell and Leaving Twitter.

by Leave a Comment

The past week I finished “Straight to Hell”.  The book describes the debauchery and decadence of the lives of the bankers in the pre-Lehman’s era. The stories are amusingly written, and horrifying. Although it’s easy to demonize an entire group of people because of a salacious book, the book does force you to ask the question: if you were in a similar environment what would you do?

And this past week, with the discussion of a boycott of Twitter, I thought to myself, maybe I am in such an environment.

Over the last several years, I have been a devoted user of Twitter. I enjoyed the ability to participate in conversations about the Montreal Canadiens. I enjoyed learning in real-time about the chaos of the world.

I found people who were really interesting. Lauren Duca and Teen Vogue. I learned about the Death of Expertise. I learned about how Snowden was a Russian tool from the very beginning.

I learned about Russian counter-espionage.

I learned about Deep Tech, following Matt Ocko.

I saw daily art.

And at the same time, I saw how Twitter refuses to police its own platform. Twitter allows the President of the USA to speak to people in a way that if he was in my house, I would ask him to leave.

Working at VMware, I am made aware of how leadership style and tone are crucial. Pat Gelsinger is an amazing leader and a man of high moral standing. And what I admire most about him, is that he doesn’t tolerate the use of crude language in his presence. And as a result, this percolates down.

VMware is probably the least profane place I have ever worked at.

And that makes VMware special.

And that got me thinking about Twitter and this book I just read.  Donal Trump sets the bar for what is acceptable on Twitter. If the President of the USA can use ethnic slurs (Pocahantas Warren),  then so can Spencer. If David Duke can tell a woman like Lauren Duca to go back into the kitchen, then so can every asshole on the platform.

And I was morally conflicted. On the one hand, Twitter has enriched my life. On the other, I can no longer ignore the tone of Twitter. And when I routinely see Twitter refuse to remove violent language from its feed, it tells me who Twitter wants to hang out with.

And because I hang out on Twitter, Twitter’s decisions reflect on me.

My grandfather who was a very moral person, would have looked at Twitter, and said: I thought I raised my daughter better. He would have been so appalled with my choices, that he would look to my mother for her choices and how she raised me.

And that made me stop and think.

In my life, and I want to work with people like Pat. I want to interact with people who value civility and decency.

Going off Twitter isn’t a permanent thing. There is a lot that I value. And I learn a lot. And I am just one lousy irrelevant person.

But the company I choose to keep says a lot about me. And so I’ll walk away for a while.

 

 

 

The Facebook Conundrum About Russian Spies And Lessons From Zynga and a Scorpion

by 1 Comment

Facebook is one of the most important new developments in human history, providing an efficient mechanism for people to connect with other people and find other people.

Facebook makes it possible for a Santorinian ex-pat like myself to connect with his fellow Santorinians on a regular basis.

I love the platform, I am a devoted consumer of Facebook’s. And I want them to succeed.

However, Facebook’s business model has a problem.

In many ways, their model reminds me of the story of the horse and the scorpion. The scorpion asks the horse to cross the river. The horse says: You will kill me. The scorpion says: If I do we will both die. The horse takes the scorpion on his back, and then halfway across the river, the scorpion poisons the horse. The horse says: But now we will both die. And the scorpion says: It could not be helped, it is in my nature.

The scorpion is people full of hate who want to destroy the people creating Facebook.

There are some who argue that that would be a case of cutting off your nose to spite your face. And as someone who comes from the Balkans, I know a lot of people with no nose.

But enough with analogies.

Facebook makes money from selling ads to companies that want to find specific people.

And there are two kinds of ads: political and commercial.

Political ads come in two flavors, products that are bought, and stories that are designed to go viral and spread a message.

Facebook’s mission is to connect the world. And the business is to profit from that connection by selling those connections.

To be a universal platform, Facebook must allow anyone to connect to anyone.

And here’s where things get messy.

The bad actors use those connections to push their bad product.

Facebook’s default response is that the consumer must protect himself. The danger is that this kind of thinking created The Third Reich.  The consumer can be manipulated and exploited.

The consumer can be manipulated and exploited.

And there is a staggering amount of science that enables bad-actors to do that predictably. At Zynga I met a firm that specialized in exactly that.

And so Facebook must choose to either control the content when it comes in or after the fact.

At Zynga, I watched how I could exploit Facebook channels to get consumers to do whatever I wanted.

Facebook, finally realized that algorithmic approaches to dealing with our exploitation of their channels was impractical and just forcibly shut down our channels.

The problem with shutting down Zynga for Facebook was that it may have alienated some users who only used Facebook to play games. Facebook did a cost-benefit analysis and came to the correct conclusion that the number of people playing games was less than the number of people who would use Facebook and moved on.

Right now they have a similar problem with the Russian Spies.

There is a large audience for hate and lies. And that large audience wants to consume hate and lies. And that hate is being directed to political objectives.

If Facebook were to shut down that content, they may lose users. They would enable the creation of another platform for people who want hate and lies. And their value as a business would decline.

Even if they didn’t lose users, they would lose ad revenue as the spies would use other media to reach people: mail, phone, etc.

On Facebook, I have met an anti-semite who wants all Jews killed. I have seen vile anti-Muslim hate. I have seen comments about Obama and people of color that are appalling.

I will never forget the white female executive unfollowing friend after friend on Facebook because she said: I am not those people.

And the question for Facebook and their team is:

Is growth of the platform worth providing a platform for hate?

And the broader question for the coastal elites creating growth-at-any costs social media platforms:

Are we creating the platform that will destroy us?

And how accountable are we?

In the big media era, I met press barons who said that they would rather have fewer readers and a paper that they could read from cover to cover to their children than have more readers and not be able to read their paper to their children.

I wonder if a universal social media platform is desirable?

And finally, I wonder if this matters at all?

Haters will hate.

People who want to exploit their hate will exist.

And Facebook cannot change that.

 

There will be no live blogging

by Leave a Comment

As is tradition!

08 architecturalist papers: the politics of fear and global warming and product development

by 2 Comments

In 1978, I read a book about the Holocaust. The book was in my elementary school library. A school with a large Jewish population. And I was exposed to horrors that profoundly shake your faith in humanity. Read a book that describes the horrors of Nazi Germany at six, and your world will get warped.

In 1988, a Chemistry teacher at Campion School told my classmates and me that we were dead men walking. The human species was ultimately going to destroy the planet, and our civilization was done. We were 14 years old, and we were dead before our lives had even started.

In 1994, a very senior professor of CS walked into a room of CS majors and told us that our jobs were going to go away. That Indian outsourcing was going to eliminate our jobs. Only 13 people graduated in 1996 with a degree in CS because the rest of my peers took his warning seriously.

From about the mid-1990’s, a profound understanding of global warming made me appreciate that our actions killed our current civilization. Either a gentle transition or a massive collapse would happen. And my understanding of the human condition from my reading of the Holocaust, made me bet on the massive collapse.

Now in 2017, with many of the predictions about global climate coming out true, I look at children and wonder what kind of hellscape they will inherit.

And you think to yourself if you can’t do anything and you are fucked, then you might as well drink the coffee, hug the wife, play with the kid and sing gospels or chant Orthodox prayers.

The fact that this kind of despair has permeated my life makes me wonder why I am still alive, what force has propelled me to keep living?

Only one:  hopelessness is not interesting.

Working in the tech industry, I have learned that you can not motivate people with hopelessness. If you walk into a meeting and tell a bunch of execs that we are going out of business, then they are not interested in what you have to say. If you say to that unless we do X, or Y or Z we are going out of business, they are still not interested.

Why?

Because every company goes out of business.

You are telling your business leaders that the sun will also rise, that the Universe will end and that they will be forgotten. You are giving them no new information.

And the reasons you can go out of business are so broad and varied and complex that this is just one threat in a spectrum of global threats that affect them.

Strategic Software Architects must be about hope. Our job is not to find the millions of reasons we will die; our job is to find the one way we can win.

And what makes the job so very hard is that we have to create the circumstances that allow us to win.

And here’s why I believe that.

In 2006, I was working at NetApp, and I was asked to produce an analysis of data center technology trends and application trends. And I walked into a meeting with my peers and observed that multi-core systems had created a strategic dead end for NetApp. The value of having external storage was to improve the performance of applications. And in particular, to deal with the fact that storage consumed a lot of CPU and Memory. By having external storage, you could improve the overall performance of your system.

Unfortunately utilization was going down on the servers (2-10%), and as a result, it was increasingly obvious that running the storage on the local system made sense. At the time Oracle and Microsoft were pushing hard for clustered file systems and databases that they felt didn’t need external storage arrays.

And I remember, saying in that meeting: We’re fucked, this was a nice company, time for us to look for a job.

A few months later, Tom Georgens asked Dave Kresse and me to study VMware and see what we could do with them. And I came to the same meeting and said: We’re saved! It turns out that VMware has figured out how to make this utilization problem go away.

And then that begat the: how the hell do we sell into EMC accounts NetApp storage?

And I remember sitting in a meeting with every business leader at the company explaining a very me too product strategy. And I remember everyone just staring at their laptops. Ten years later, I would have seen a whole bunch of LinkedIn updates.

And somebody asked me: Hey did you see how this guy saved 90% storage using NetApp dedupe?

And it clicked for us all. We had this feature, called dedup, that allowed us to deduplicate data on primary storage. And VMware had a problem that they needed shared storage to store identical images.

And what was incredible is that dedupe was the feature that we kept trying to kill. Originally imagined as an answer to data domain, or perhaps a generalization of snapshots, for years teams tried to kill it, and somehow it survived. This piece of unwanted technology transformed our company.

We shut down releases, redid roadmaps to take a piece of technology that barely worked and made it the centerpiece of everything we did.

We convinced the world that deduplication on primary storage was the right thing to do. Dedupe was a technology that no one had. Because it was insane. Intentionally introduce fragmentation. Dedupe on primary workloads was a crazy stupid proposition for storage.

And we won and lived for another decade.

The morality play, in my head, was the following that we could have curled up and died, I could have taken that interview at Google or I could have kept looking. And I chose to keep looking.

And because we kept looking, we found something, and we survived and thrived.

If you want to inspire people, don’t tell them they are dead, stare into the abyss and say we will find a portal out of here.

And you know what, you may find a way out of the abyss, and trying to find a way out is always way more interesting.

 

 

 

 

 

 

07 architecturalist papers: how micro-services made picking a programming language different

by Leave a Comment

Once you become an operational or strategic architect, programming languages become an option in the toolbox. And then the question becomes which one to pick.

The most important considerations when I started my career were:

  1. Can the language interface with pre-existing code
  2. How mature and stable is the programming language
  3. How many programmers can you hire that know the language
  4. Does the language have a debugger and a profiler
  5. What tradeoffs does the language impose regarding performance and safety and portability.
  6. What specific libraries and tools and constructs does the language provide for making the project go faster

With large monolithic systems of the 1990’s, #1 forced you to keep the same programming language indefinitely. Unless someone signed up for a rewrite, you had no choice. Even in the case of a rewrite, you always wanted to leverage some of the pre-existing code.

And in the 1990’s the most important piece of code you had to leverage was OS system services.

Microsoft and other programming language vendors attempted to invent program language technologies that allowed applications to call from one another, but the tools didn’t quite work, and they locked you into a specific vendor and OS. The first C compiler I bought from Microsoft in 1987, had a long discussion of how you could get basic and C to work together.

And you still had the whole problem of cross-OS portability.

Attempts at standardizing libraries through things like POSIX didn’t work at all.

What changed in the 2000’s was the movement to multi-process application architecture using databases as a mechanism to exchange data. The database was cross platform, and vendor neutral and language agnostic. And all of a sudden, choice became an option.

And more importantly, C++ was a real option because it was designed to solve #1 and #5.

In 2004, when I had an opportunity to pick a language as the operational architect for the NetApp Performance Advisor, I chose C++.

I agonized over the decision for a month, because – based on my prior experience, this was a once in a decade decision.

And the reasons for C++ were:

  1. C++ could call into all of our C code.
  2. C++ was much more mature than Java at the time
  3. C++ programmers were easy to hire, and C++ programmers could work on the C parts of our system.
  4. Working debuggers and profilers
  5. Allowed us to trade off some performance for safety (string class instead of char*, and reference counted pointers) with no loss of portability across the platforms we care about.

My old school thinking decided that the Java Native Invocation was just too clunky as a mechanism to leverage our existing C code base. And I had spent a lot of time in college writing C++ wrappers and had no time to do that for the huge Data Fabric Manager code base.

Furthermore, I remember thinking that C# would crush Java because it made calling code from C# into C/C++ easier…

Except…

Java, in the end, won, because the service architectures were just a better way to write software, and using the database as a way to get different parts of the system to talk to each other made it easy to add Java to a system.

Applications no longer needed to call into libraries, they could use the database to share information. I believed that this was a dead end architecture because database – increasingly became a bottleneck, and that would lead to large monolithic Java systems and that would lead to a stable Java dominated ecosystem except…

The emergence of SOAP and JSON and REST made it even easier to combine programming languages and circumvented the DB bottleneck.

In the 1990’s picking a language was a once in a decade decision, now picking a language became a pretty standard decision. And that leads to a new problem for strategic software architects, given that operational architects can pick any language at any time, what guidance do you give them?

In short, the basic model still holds, except for one minor tweak:

  1. Can the language interface with pre-existing code
  2. How mature and stable is the programming language
  3. How many programmers can you hire that know the language – and how easily can they move between languages. 
  4. Does the language have a debugger and a profiler
  5. What tradeoffs does the language impose regarding performance and safety and portability?

And that last tweak is hugely important. Each service has a life span, and engineers have to be moved between services as business priorities change. And having different languages creates friction in your ability to move people.

At Zynga, Cadir was adamant that we only support two backend programming languages (C and PHP and Java) because he valued the ability to move engineers easily very highly.

I, personally, loathed PHP and found C to be too low-level and could never quite get over my initial interaction with Java in 1994, but he had a good point and fully supported his decisions. One time, I forced a team to rewrite their Ruby code in PHP because of our policies.

And Cadir’s decision was a huge win because it people could easily move across the company, and sharing code was very easy.

And this leads me to the conclusion that the real list now is:

  1. How mature and stable is the language including debuggers and profilers?
  2. How easily can programmers move between services
  3. What specific tools or constructs does the language provide for the domain problem.

You’ll notice that this, the most important property in 2004, got dropped,

  • Can the language interface with pre-existing code

Because that’s basically a solved problem.