Microsoft Azure cloud services enable enterprises to manage data at scale in the cloud. That also open massive possibilities for predictive analytics, AI, and real-time applications. Apache Spark has become the platform of choice for building these applications but deploying…
Data Governance refers to the strategy of managing and controlling data throughout the enterprise and thus ensuring to derive more value from that data. It is about enabling and encouraging good behavior regarding data, and limiting behaviors that create risks. It is…
In my previous blog I had given an introduction to Azure Data Lake and its offerings. In this blog I am going to show how to create a data lake in Azure using the Azure portal. To create a Data…
What is Data Lake? A data lake is a storage repository that holds a vast amount of raw data in its original format to apply analytics and run big data analysis. Data lake handles the three Vs of big data (Volume,…
HDFS estimates the network bandwidth between two nodes by their distance. The distance from a node to its parent node is assumed to be one. A shorter distance between two nodes means that the greater bandwidth they can utilize to…
Read More Performance improvement of map reduce through new Hadoop block placement algorithm
The Hadoop Distributed File System, inspired by GFS from Google, is a distributed filesystem which runs on low cost commodity hardware in a fault tolerant manner to redundantly store Terabyte and larger data sets. The architecture of this filesystem is…
In order to avoid placing new data, HDFS Block placement policy does not take into account DataNode disk space utilization. Hence, data this might cause non-uniform data placement across the cluster. Also, uneveness might occur when new nodes are being…
Read More HDFS Balancer to balance disk space usage on the cluster
Pipelining is the technique used by HDFS to minimize inter-node network traffic. Whenever the first block replica is written by a to a node, then it’s the responsibility of that node to write the second replica to a random off-rack…
Read More HDFS Pipelining to minimize inter-node network traffic
Apache Hadoop was initially developed by “Doug Cutting” in 2005 because he needed a faster data processing framework for the web crawler project called Nutch. Based on the MapReduce paper which was published by Google in 2004, he replaced the…
Amazon Kinesis Firehose is a managed service to load real-time streaming data to Amazon Simple Storage Service (S3), Redshift or Elastic search Service (ES). Firehose is part of the Amazon Kinesis streaming data platform, along with Amazon Kinesis Streams and Amazon Kinesis Analytics. There…
Read More Amazon Kinesis – Managed service for real-time data processing
Cloud Avenue 