wrong tool

You are finite. Zathras is finite. This is wrong tool.

Regulation or Encryption for Big Data

by Leave a Comment

A few weeks ago I called for a Hippocratic Oath for data scientists.

Now the New York Times is calling for regulation.

We can fight regulation all we want and we will.

But…

Customers will fight us. And they will fight us by moving to vendors that offer privacy and encryption. Where data is wholly owned by the customer.

And the customer will win. Because if we don’t act like their data is a sacred trust they will learn to distrust anything that they cannot control.

And they will control it through encryption.

And then we will have the medical industry where breakthroughs are there if only we could look at data …

Things are getting better with Data Science and our Data

by Leave a Comment

Yesterday I wrote about how the failure of companies to respect the privacy and happiness of their customers posed an existential threat to the entirety of services that relied on big data.

Some folks on twitter remarked that my Data Scientist Hippocratic Oath is what their companies live and breathe.

And that’s great. I think that protecting user-data aligns with being a great company… And I think a great company sometimes may need to be explicit about how it thinks about user data.

Juliet asked how does this apply when the customer isn’t a person? I guess we need to refine the oath to be a little bit more specific – instead of customers we should talk about people.

  1. I will do no intentional harm. I will not knowingly manipulate people to be unhappy or sad or miserable without their explicit clear and obvious consent
  2. I will never use our data in ways that are not aligned with the customer needs of the person whose data this is.
  3. The company is not the customer, and if I must choose the customer needs p person whose data this is over the company I will always choose the person. My job is to protect the user’s data not the company’s survival

And while I called out uber and facebook in my post, it’s only fair to share with folks that Facebook has been working to create a better code for it’s data science efforts described here and that the folks at Uber have hired an outside team to look at their approach to privacy.

I believe the self-interest of companies with vast amounts of data that we want to be kept private will ensure that the data is private because if it isn’t we will have encryption deployed everywhere. And I am delighted to see that happening.

And we’re seeing that. We, as users, need to demand that kind of data protection. The alternative is what happens in medicine where data is so regulated that our ability to fight diseases is being impaired.

 

Physics and Computer Games and Big Data

by Leave a Comment

55794177

Over the last 15 years, there have been two useful heuristics for figuring out where computing is going.

When I want to look at how applications are going to be built, I look at games. After all games are at the forefront of creating new kinds of digital experiences, and the need to push the boundaries of how we entertain ourselves is a crucial to create new revenue and sales opportunities.

When I want to look at how infrastructure is going to change, I look at what people want to do in the super computer space.

Two nights ago, I had the marvelous opportunity to hear a talk that was a discussion of Physics and Big Data. As a software infrastructure guy, at the end of the day I like to think about how to build systems that enable applications, I have been wondering if Big Data was going through a bigger-faster-stronger phase or whether there were new intrinsic problems.

And the answer is yes to both.

Clearly we need systems that can do more analysis faster, store more data at cheaper costs, etc.

What was not as obvious was that exponential increase in transistors coupled with the disruptive trend of 3D printing was going to enable:

  1. A proliferation of very sensitive distributed sensors that need to be calibrated and whose data needs to be collected.
  2. The ability to find even weaker signals in the data.

In effect, we were going to be able collect more data faster and because of that we will be able to find things that we could not find before. And solving 1 and 2 on it’s own are very interesting problems that can keep me busy for the next 10 years …

However there are some new problems that come out of that:

dn1xs

We will need to be able to find new ways to explore data and track our exploration through the data.

dn1ux

 

We will need ways to combine the datasets we create. After all as more and more sensors get created, and sensors get cheaper, the ability to combine data sets will become crucial. And as the scale of the datasets grows, an ETL becomes less realistic.

And then to make this all more interesting, there is some thought that the way we collect data itself may create signals and that meta-analysis of the data will be required. And how you do that is an interesting problem on itself. And how do you create systems can correct for that…

My head has sufficiently exploded. Turns out that just making things go faster isn’t the only problem worth solving…