Data Lake Store

Before I get into the topic at hand, I’d like to bring you up to date on some Microsoft news you may not have heard.  For years now, Amazon Web Services (AWS) have been the gold standard for the Internet as a Service (IaaS) industry.  In fact, most experts have declared them the dominant winner that nobody will catch any time soon.

On July 20th, Gartner released their report on cloud IaaS services offered by AWS, Azure, and Google.  In the Overall Required Category, AWS scored 92%, Azure scored 88%, and Google Cloud scored 70%.  Microsoft surprised more than a few experts as the come-from-behind underdog.  With this new reality in the IaaS industry, there is now talk of Microsoft becoming No. 1 very soon.

This is a testament to Microsoft’s commitment to the Azure cloud.

Now, off to the lake!

The Azure Data Lake Store (DLS) is an Apache Hadoop file system that allows you to store your data in the cloud.  (I’ll bet you were expecting a different meaning for the word “store”.)  The DLS is where you put your lake.

The Azure DLS will ingest virtually any size of data stream that is pointed at it.  This becomes the repository for your big data.  Just like Amazon S3, you can have virtually unlimited storage space.  Unlike S3, Azure DLS allows you to optimize that data for use with integrated analytics.

The DLS includes all the enterprise tools you will need to manage your lake:  Security, tools to manage the data, as well as Azure reliability, and access.

Any type of data streamed into the DLS can be stored in its native format.  If you require specific types of data to be stored in a single file, that can be handled as well.  Azure DLS does not have limits on file sizes.

Of course you are not just storing data for fun.  You will want to make some use out of it.  That is why Azure designed its DLS to optimize your ability to analyze enormous amounts of data.  In addition to being integrated with a variety of Azure services, you are also able to use open source Hadoop tools.

To get started using the Azure DLS, you can choose to start reading here:

https://azure.microsoft.com/en-us/documentation/articles/data-lake-store-get-started-portal

Or you can start watching here:

https://mix.office.com/watch/1k1cycy4l4gen

Overview of Azure Analytics offerings

In June, Microsoft announced that it would pay $26.2 billion to purchase LinkedIn.  They paid a 50% premium over LinkedIn’s share price – a price that had been plummeting due to their substantial losses.  This was the third largest corporate acquisition in history.  Analysts have been scratching their heads trying to justify this purchase.

Despite being the world’s largest software maker, Microsoft’s main focus since Satya Nadella took over as CEO in 2014 has been cloud computing, machine learning, and artificial intelligence.  Acquiring LinkedIn fits well with this focus.  In addition to the synergies Microsoft hopes to leverage, LinkedIn has a well-respected team of data scientists that have been coveted by tech firms.

When it comes to analyzing data in the cloud, Microsoft is going all in.  Money, corporate culture, intellectual capital, and advertising, are some of the indications of how important this is to Microsoft.  The variety of robust products, services, and integration across Microsoft’s various tools and platforms, now on offer at Azure, is the proof that Microsoft is very serious about this.

My keenest interest in Azure lies with the analytics offered on Azure.  Analytics in Azure is actually a substantial group of things that allow you to organize and analyze data.  Depending on the nature of the data and what information you are trying to tease out of it, you will need a different tool.  Microsoft has them all.  Here is an overview of the analytics tools on offer through Azure:

Data Lake Analytics:  A data lake is a very large collection of raw data.  Data lakes are a relatively new phenomenon (2010) that grew, as “Big Data” became a thing.  When you have a steady “stream” of data filling a data lake, analytics will allow you to find the subset(s) of data that point to correlations or trends.

HDInsight:  The “HD” stands for “Hadoop Distribution”.  HDInsight is only available on Azure.  It provides a framework to manage analyze and generate reports using big data.

Machine Learning:  As I already mentioned, ML is one of the main focuses for Microsoft.  ML is used to find hidden insights without having to explicitly tell the computer where to look.  I covered this topic in a series of blogs previously.

Stream Analytics:  (Continuing with the water analogy)  Stream analytics is a high throughput, low latency analytic that allows for immediate understanding of real time data.

Data Factory:  Like any factory, raw materials come in, they are processed, and products (not necessarily finished) come out.  A data factory accepts data and processes it into ready to use data that can be used for consumption, or further analysis.

Event Hubs:  An event hub is a place where millions of data points collected from millions of sensors (welcome to the internet of things) are received, integrated, processed and then shared back to devices that make use of the integrated information.

Data Catalog:  The Azure Data Catalog is a fully managed service that makes it easy to find the data sources you need.  It is a community of data sources.

 Power BI:  Power BI is Microsoft’s suit of BI tools that allow you to set up and use dashboards that will monitor and process your data quickly.  It provides you with visual displays of your data that will give you the big picture on any device.

In my next blog, I will begin to explore each of these analytics tools in more detail.

Crown yourself the Data Analytics Ruler

The Internet of Things is not living up to its hype.  In 2014, as the anticipation of everything being connected was reaching its zenith, Google purchased Nest for $3.2 billion, thinking that this was going to be their foot in the door to everyone’s smart home.  It turns out that people aren’t that interested in taking the time to ensure that their smart light bulb can be turned off when the homeowner is at work.  The promise of Google soaking up terabytes of information from everyone’s home is still relegated to fiction (or fear mongering – depending on your perspective).

That doesn’t mean that more and more data isn’t being collected all the time.  Although the smart home is still a “Jetsons” fantasy, smart cities and smart companies are a reality.  Cities are releasing their data stream to the public so that clever people can analyze it and find trends and other morsels of information that are useful.  Companies that have for years, generated enormous quantities of data, are finally teasing useful information out of that data.

Companies, cities, and even countries are all becoming “smart” because they are starting to use the raw data that they collect.  The science that allows us to transform meaningless data into useful information is Data Analytics (“DA”).  In this era of “Big Data” the Data Analyst is king (or queen?).

Companies use data analytics to make better business decisions.  Scientists use it prove or disprove existing theories and models.  It is astounding how often our “common sense” approach has led to the exact opposite of the correct solution being implemented.  DA has shown us the folly of following our gut on numerous occasions.

Unlike Data miners who are merely searching for patterns, DA is focused on analyzing whether a hypothesis is true or not.

Microsoft Azure has a variety of tools to help you in the game of thrones where you are trying to crown yourself as the DA King or Queen for you employer.  In previous weeks I have posted several blogs about Microsoft’s powerful Machine Learning tools.  That is only one of many tools that will help you analyze data (big or small).

In the coming weeks, I will introduce you to the variety of tools and give you a brief overview of how they can help you.