bdfe

1.	Hadoop Components
1)	Hadoop Distributed File System
The Most common file system used by hadoop. It is based on Google File System and provides a distributed file system that is designed to run on large clusters (thousands of computers) of small computer machine in a reliable, fault tolerant manner.
HDFS consists of a single NameNode that manages the file system metadata and one or more slave DataNodes that store the actual data.
A file in an HDFS namespace is split into several blocks and those blocks are stored in a set of DataNodes. The NameNode determines the mapping of blocks to the DataNodes. The DataNodes takes care of read and write operation with the file system. They also take car of block creation, deletion, and replication based on instruction given by NameNode.
2)	MapReduce
Software framework for easily writing applications which process big amounts of data in-parallel on large clusters (thousand of nodes) of commodity hardware in a reliable, fault-tolerant manner. The name referse to the two different tasks that Hadoop programs perform.
A.	Map Task: Tahkes input data and converts it into a set of data, where individual elements are broken down into tuples (rows or key/value pairs)
B.	Reduce Task: Takes the output of map task as  input and combines those data tuples into a smaller set of tuples. This task always performed after map task
3)	Sqoop
A tool designed for transferring data from a relational database directly into HDFS or into Hive. It automatically generates classes needed to import data into HDFS after analyzing the schema’s table, then reading of tables’ content is a parallel MapReduce job.
4)	Oozie
Oozie is a open-source tool for handling complex pipelines of data processing. Using Oozie, users can define actions and dependencies between them and it will schedule them without any intervention.
5)	Flume
A distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amount of log data. It is designed to import streaming data flows.
6)	HBase
A Distributed, column oriented database management system, modeled on Google’s Big Table that runs on top of HDFS.
7)	RHadoop
RHadoop is a collection of five R packages that allow users to manage and analyze data with Hadoop.
8)	RHIPE
R package that provides a way to use Hadoop from R.
9)	R
Open source programming language and software environment for statistical computing and graphics
10)	Mahout
To build an environment for quickly creating scalable performant machine learning applications. Divided into 4 main groups, collective filtering, categorization, clustering, mining of parallel frequent patterns.
11)	Crunch
A Java library provides a framework for writing, testing, and running MapReduce pipelines. Its goal is to make pipelines that are composed of many user-defined functions simple to write, easy to test, and efficient to run.
12)	Cascading
A Java API for defining complex data flows and integrating those flows with back-end systems, and a query planner for mapping and executing logical flows onto a computing platform.
13)	Pig
Intended to allow people using Hadoop to focus more on analyzing large datasets and thus spend less time having to write mapper and reducer programs.
14)	Hive
A distributed data warehouse. It enables easy data ETL (Extract, Transform, Load) from HDFS or other data storage like HBase or other traditional DBMS. It has the advantage of using a SQL-like syntax, the HiveQL.
15)	Chukwa
A data collection system for monitoring large distributed systems.
16)	Aaaaa
2.	Definition of Smart City
1)	A city well performing in a forward looking way in economy, people, governance, mobility, environment, and living, built on the smart combination of endowments and activities of self-decisive, independent, and aware citizens. The focus of this definition is it views a smart city as a futuristic model of collaborative component.
2)	Smart city is a very broad concept, which includes not only physical infrastructure but also human and social factor.  The focus of this definition is it included the social aspects and agreed that smart city has a broad focus.
3)	A city that monitors and integrates conditions of all of its critical infrastructures, including roads, bridges, tunnels, rails, subways, airports, seaports, communications, water, power, even major buildings, can better optimize its resources, plan its preventive maintenance activities, and monitor security aspects while maximizing services to its citizens. This definition focus on the integration of infrastructure and systems that monitor and control the resources to achieve sustainability as the main aspect of a smart city.
3.	Security part of the building blocks
The security stage of the data life cycle describe the security of data, governance bodies, organization, and agendas. It also clarifies the roles in data stewardship. Therefore,
appropriateness in terms of data type and use must be considered in developing data,systems, tools, policies, and procedures to protect legitimate privacy, confidentiality, and intellectual property.
4.