Overview of Azure Data Factory
Azure data factory is a fully managed and server less data integrations service that integrates all your data. It easily re-hosts SQL Server Integration Services and builds ETL (extract, transform, and load) and ETL pipelines, ELT (Extract, Load, transform), and data integration without the need for coding.
Data Factory is a system that hosts multiple interconnections and thus data engineers get a complete end-to-end platform. Following are some important technical concepts of data factory crucial for aspiring data engineers and professionals necessary for Azure data certifications.
Connect and collect
Enterprises have a huge amount of various data stored at different locations such as cloud, structured, unstructured, and semi-structured. Data factory allows moving data from both on-premises and cloud source data stores to a centralized server or data store and carries further analysis.
Transform and enrich
Azure data factory uses the ADF mapping data flows to process and transform the data stored in a centralized data store. Moreover, data transformation graphs can be built and managed using mapping data flows. ADF mapping data flows does without the need to understand Spark clusters or Spark programming.
CI/CD and publish
Data factory fully supports CI/CD of your data pipelines. It allows users to develop and deliver ETL processes before publishing the finished product.
Monitor
Once the data integration pipeline is built and deployed, the data factory monitors the scheduled activities and pipelines that provide success and failure rates.
Explore the Building Blocks of Azure Data Factory
Pipeline
A data factory has one or multiple pipelines that contain a logical group of activities performing certain tasks. The pipeline allows managing the activities in a set rather than managing each activity individually. The activities are chained together and can undergo sequencing or even operate independently in parallel.
Mapping data flows
Data factory creates and manages graphs for transforming any size of data. Data engineers can develop graphical data transformations without the need for coding. Data flows within the ADF are executed using scaled-out Azure Databricks clusters.
Activity
Activities in a pipeline are those actions that are performed on the data in the pipeline. Data Factory carries out control activities, data transformation activities, and data movement activities.
Datasets
Datasets retrieve data from various data stores like documents, tables, files, and folders. You can use data from these data stores as inputs or outputs.
Linked services
Linked services are similar to connection strings. Linked services to the data factory provide the connection information necessary to connect it with external sources. Data Factory uses linked services for two purposes such as to represent a data store and to represent a compute resource.
Integration Runtime
In a Data Factory, activity is the action performed in a pipeline whereas linked service is a target store or compute service. Integration runtime in a Data Factory connects activity and linked services. Azure Data factory provides three types of integration runtime namely Azure, Azure-SSIS, and Self-hosted integration runtime.
Triggers
Triggers schedule when a pipeline execution needs to be kicked off. Different types of triggers are used to schedule different types of events.
These concepts are extremely important to pass the DP 203 exam and successfully gain Azure Certification.