In my previous blog I had given an introduction to Azure Data Lake and its offerings. In this blog I am going to show how to create a data lake in Azure using the Azure portal. To create a Data…
Read More Creating Data Lake in Azure
Varun Kumar
Varun works with Microsoft as a Cloud Architect. He comes with 12+ years of experience into Software development, Consultant, Solution Architect, and Delivery Management roles. As an Architect in Microsoft, his job is to design, develop and enahnce enterprise level solutions using Azure, to help organizations to achieve more.
Microsoft Azure Data Lake Store: an introduction
What is Data Lake? A data lake is a storage repository that holds a vast amount of raw data in its original format to apply analytics and run big data analysis. Data lake handles the three Vs of big data (Volume,…
Performance improvement of map reduce through new Hadoop block placement algorithm
HDFS estimates the network bandwidth between two nodes by their distance. The distance from a node to its parent node is assumed to be one. A shorter distance between two nodes means that the greater bandwidth they can utilize to…
Read More Performance improvement of map reduce through new Hadoop block placement algorithm
Synchronizing your database with Azure SQL Data Sync Service
Azure SQL Data Sync is a service built on Azure SQL Database that allows to synchronize data from one or many disparate SQL data sources either on-premises or Azure SQL Databases to a single Azure SQL Database (called as Hub) and…
Read More Synchronizing your database with Azure SQL Data Sync Service
Hadoop Distributed File System (HDFS)
The Hadoop Distributed File System, inspired by GFS from Google, is a distributed filesystem which runs on low cost commodity hardware in a fault tolerant manner to redundantly store Terabyte and larger data sets. The architecture of this filesystem is…
Microsoft Announced Azure Database Migration Service
Microsoft has announced a limited preview of the Azure Database Migration Service and the corresponding Azure SQL Database Managed Instance in Microsoft Build 2017, worldwide, that will help in streamlining the migration process of on-premises databases to Azure. This new…
Read More Microsoft Announced Azure Database Migration Service
HDFS Balancer to balance disk space usage on the cluster
In order to avoid placing new data, HDFS Block placement policy does not take into account DataNode disk space utilization. Hence, data this might cause non-uniform data placement across the cluster. Also, uneveness might occur when new nodes are being…
Read More HDFS Balancer to balance disk space usage on the cluster
HDFS Pipelining to minimize inter-node network traffic
Pipelining is the technique used by HDFS to minimize inter-node network traffic. Whenever the first block replica is written by a to a node, then it’s the responsibility of that node to write the second replica to a random off-rack…
Read More HDFS Pipelining to minimize inter-node network traffic
Apache Hadoop
Apache Hadoop was initially developed by “Doug Cutting” in 2005 because he needed a faster data processing framework for the web crawler project called Nutch. Based on the MapReduce paper which was published by Google in 2004, he replaced the…
Migrating data from different sources to DocumentDB
The DocumentDB Data Migration Tool is an open source tool for importing data to DocumentDB from a variety of sources such as JSON files, CSV files, MongoDB, SQL Server, Azure Table storage, Amazon DynamoDB, HBase etc The import tool can…
Read More Migrating data from different sources to DocumentDB