All you need to know about Microsoft Data Integration Services

What is Microsoft Data Integration Services 

In this blog we explain you all you need to know about Microsoft Data Integration Services. Microsoft Data Integration Services divides into two categories:  

1. On-premise, (local) variant. You work locally on a server, and you develop the integrations and workflows on this server. We call this SQL Server Integration Services (SSIS).
2. Azure Integration Services (AIS): Here, you work on the cloud platform of Microsoft (Azure), and the Integration service consists of various solutions, which together form the integration solutions of Microsoft. This set of solutions together is called the Microsoft Azure Integration Services. 

  In this blog, we will only focus on the different solutions within the AIS platform. 

What does Microsoft Azure Integration Services?

AIS is mainly intended to move data and is an umbrella name for five solutions contained within it. You can think of transferring data from the various systems of an organization to a data warehouse (data hub) environment. AIS has connectors to make connections to such systems. The data that can be retrieved in different formats: 

• Structured data (data in tables) 
• Semi-structured data (email, PDF, HTML) 
• Unstructured data (video and photo) 

The different solutions within AIS can be divided into two application areas: Integration as a service and Analytics  

Integration as a service (IAAS) 

IAAS is characterized by real-time data shifts between numerous systems and is often the heart of a company. Think of an order placed in a web application and moved to the accounting system and an order management system using an IPAAS (Integration Platform as a Service). This type of data integration is often near real-time and event-based. Also, many of the integrations are based on APIs. The solutions in the Azure Integration Services that are used for this include Logic Apps, Service Bus, API Management, and Event Grid 

Analytics integration

Different systems are made accessible in analytics applications, and this data from the various sources is written into a data hub. You can then use this data in many other systems. Consider, for example, the reporting tool PowerBI or Tableau. The loading and preparation of the data takes place separately in a central environment. This creates consistent results that can be reused. In this case, data (unlike IAAS) usually loads in batches. There is also a more direct connection to source systems, and data refreshes are usually scheduled or triggered. The Azure Integration Services solutions used for this include Logic Apps, Azure Data Factory, and Azure Analysis Services.   

Here is a brief overview of these different solutions from both domains and what they can do: 

• Logic Apps: With Logic Apps, you create transparent workflows (workflows) to connect to different systems using APIs. These workflows run in the predefined order when the trigger is fired.
• Service Bus: Azure Service Bus is a cloud messaging service that connects all applications, devices, and services running in the cloud to other applications or services. 
• API Management: A hybrid multi-cloud management platform for APIs in all environments. In this way, internal and external developers can connect to the back-end systems at any time. 
• Event Grid: Event Grid is an integrated service for managing the routing of all events from any source to any destination (what happens when and where the data should go based on the event). 
• Azure Data Factory: Azure Data Factory is a cloud-based data integration service that allows you to create data-driven workflows in the cloud and automate data movement and data transformation.   

Benefits of Microsoft Data Integration Services 

1. Centrally manageable 
2. Consistent transformations on the data 
3. Clear to work with through process flows 
4. An unprecedented number of connectors to unlock source systems 
5. Easy to make transformations on the date 

Rockfeather & Microsoft Data Integration Services 

With Rockfeather, we use these solutions by creating an intermediate layer when we retrieve the data from systems, whether this is for an IAAS question or more for an Analytics question. 

For example, we provide access to the ERP system, an HR system, and a tool with an API. We can write all this to a data hub. We then use these as a source for the solutions as mentioned above. The advantage of this is that we can load the same results regardless of the solution. In addition, it is efficient and clear to retrieve the data and transform it where necessary. After we have all this down, it will continue to update fully automatically.