Microsoft Fabric: A New Era in Data Solutions with Data & AI Director James Frost

Microsoft Fabric: A New Era in Data Solutions.

James Frost – Data and AI Director, Quorum

On May 23rd 2023, Microsoft launched Microsoft Fabric, which Microsoft CEO, Satya Nadella described as:

“The most significant data platform innovation since SQL Server“

Having worked with Fabric over the last six months, my belief as head of Quorum’s Data and AI practice is that Fabric represents a seismic shift in the industry, not just within the Microsoft data space, but will also cause huge ripples throughout the business data community. But what is Fabric, why is it so important, and what will it mean for our customers?

In general, the trend in the wider IT industry is moving towards fully managed solutions, which are focussed on solving business problems, rather than technical offerings which are complex to manage and secure. This trend initially saw Platform as a Service (PaaS) offerings replacing Infrastructure as a Service (IaaS) and now sees Sofware as a Service (SaaS) replacing PaaS.

Microsoft Fabric SaaS, IaaS and PaaS Diagram

But so what? What benefit does this actually deliver? Let’s talk about a high-level example of how this delivers real value with a simple scenario.

Let’s imagine that a business has a number of data sources they want to integrate, do some analysis on, and then present in a simple dashboard. I’ll deliberately stray away from integration with Microsoft Dynamics 365 data, as that feels like cheating (as its going to become so ridiculously easy under Fabric). So for this scenario let’s imagine you have a combination of:

– Salesforce CRM Data,

– excel files stored in an internal file system,

– and website analytics data from a third-party content management solution.

Here’s how the solution would be delivered under the different styles of service:

Under IaaS (SQL Server)

The IaaS version of this is reasonably complex, and involves quite a few steps that will need to be completed by an IT team, and very little that could be done by an end user. It involves installing servers, security patching / monitoring, installing various components, and then building the ETL packages. With a tailwind, a good IT department could probably deliver this within 2-3 weeks, assuming no other workload (which is unlikely to be the case). This then requires ongoing support, patching, maintenance, and alerting to keep functional and make sure the platform remains healthy and secure.

Under PaaS (Azure Synapse)

PaaS requires less management once installed, but there’s still a lot of components to configure, especially networking links and security between the various components. You’ll need to configure storage accounts, Synapse instances, SQL pools, private endpoints, and for a very simple instance you will probably need to get your IT engineers to build something like the diagram below:

Once this is built, data engineers could create Azure Data Factory flows to pull the data from the various sources, transform it and carry out the required analysis in something like Python. If the above environment already exists then you might be looking at a week’s effort, but the initial setup could take 3-4 weeks to get right, and you also need to factor having separate environments for development, test, and production.

Under SaaS (Fabric)

You start by creating a new Fabric Workspace within the portal and then a Lakehouse (or Warehouse) to store your data. That’s a couple of clicks in the portal to create all the infrastructure which is shown in the above PaaS diagram (plus a whole lot more).

Press This

Give It a Name

Done!

Several weeks work saved! Repeat across your development, test and production workspaces.

You then create a new dataflow. This then allows you to connect to Salesforce data directly and pull it into the datalake, making the required transformations as you go. Repeat this process for the Excel and web data.

You can then connect directly to this data in PowerBI, and schedule the dataflows to run automatically. If you want to, you can then do a whole range of data science processes directly on the same data.

With a good understanding of the data, a business user could create the above in a day or so, with some training in Power Query (which is designed as a business data transformation tool) and potentially a little help from IT on the final analysis piece.

Fabric isn’t just a rebranding of the previous set of Microsoft data tools, but is a fundamental reworking of all of these tools into a single unified solution which can be used by IT users, business professionals and data scientists. The key requirement to use these tools is not technical, it’s having a good understanding of the business data and the reporting / analysis requirements. In fact, anyone who has been using the “Get Data” facility within Excel or PowerBI has most of the required skills (Power Query) to get started with pulling data into Fabric and carrying out some pretty complex transformations.

This doesn’t remove the need for good data architecture, analysis and engineering. What it does remove is a lot of the complexity around building the infrastructure to support this, and getting good data solutions into production in a much faster and more agile way.

Over the coming weeks we are going to be talking about the various components of Fabric, what each of these deliver, how you can get started using Fabric, and even the murky subject of Fabric licensing. We’ll also talk about some of the governance and security aspects of the platform so you can make sure you go about deploying Fabric in the right way from the start.

I’m hugely excited to see where Fabric goes next, and hope you will join us on this journey.