Data Programming, Libraries & Related

Kaggle – kaggle.com 

World's largest community of data scientists and machine learners, owned by Google, Inc. Kaggle got its start by offering machine learning competitions and now also offers a public data platform, a cloud-based workbench for data science, and short form AI education. 

R (programming language) 

R is a programming language and free software environment for statistical computing and graphics supported by the R Foundation for Statistical Computing. The R language is widely used among statisticians and data miners for developing statistical software and data analysis. Polls, data mining surveys and studies of scholarly literature databases, show substantial increases in popularity in recent years. As of August 2018, R ranks 18th in the TIOBE index, a measure of popularity of programming languages. 

R is an integrated suite of software facilities for data manipulation, calculation and graphical display. It includes: 

• an effective data handling and storage facility, 
• a suite of operators for calculations on arrays, matrices, 
• a large, coherent, integrated collection of intermediate tools for data analysis, 

• graphical facilities for data analysis and display either on-screen or on hardcopy, and 
• a well-developed, simple and effective programming language which includes conditionals, 
• loops, user-defined recursive functions and input and output facilities. 

R Snow 

The snow package provides support for simple parallel computing on a network of workstations using R.  A master R process calls makeCluster to start a cluster of worker processes; the master process then uses functions such as clusterCall and clusterApply to execute R code on the worker processes and collect and return the results on the master. This framework supports many forms of "embarrassingly parallel" computations. 

Snow can use one of four communications mechanisms: sockets, MPI, or NetWorkSpaces (NWS).  NWS support was provided by Steve Weston. MPI clusters use package Rmpi; NWS clusters use package nws.  If MPI the MPI may need to be started externally. If NWS is used, the NetWorkSpaces server must be running.  SOCK clusters are the easiest approach for using snow on a single multi-core computer as they require no additional software. 

R Multicore 

Multicore is an R package that provides functions for parallel execution of R code on machines with multiple cores or CPUs. Unlike other parallel processing methods all jobs share the full state of R when spawned, so no data or code needs to be initialized. The actual spawning is very fast as well since no new R instance needs to be started. 

R Rhadoop 

RHadoop is a collection of five R packages that allow users to manage and analyze data with Hadoop. The packages have been tested (and always before a release) on recent releases of the Cloudera and Hortonworks Hadoop distributions and should have broad compatibility with open source Hadoop and mapR's distribution. We normally test on recent Revolution R/Microsoft R and CentOS releases, but we expect all the RHadoop packages to work on a recent release of open source R and Linux. 

str()// Display data type and content 

seed() 

ctree()Page Break 

Python 

Python is an interpreted high-level programming language for general-purpose programming. Created by Guido van Rossum and first released in 1991, Python has a design philosophy that emphasizes code readability, notably using significant whitespace. It provides constructs that enable clear programming on both small and large scales. In July 2018, Van Rossum stepped down as the leader in the language community after 30 years. 

Python features a dynamic type system and automatic memory management. It supports multiple programming paradigms, including object-oriented, imperative, functional and procedural, and has a large and comprehensive standard library. 

Python interpreters are available for many operating systems. CPython, the reference implementation of Python, is open source software and has a community-based development model, as do nearly all of Python's other implementations. Python and CPython are managed by the non-profit Python Software Foundation. 

Python Numpy (Analysis) 

NumPy is the fundamental package for scientific computing with Python. It contains among other things: 

• a powerful N-dimensional array object 
• sophisticated (broadcasting) functions 
• tools for integrating C/C++ and Fortran code 

• useful linear algebra, Fourier transform, and random number capabilities 

Besides its obvious scientific uses, NumPy can also be used as an efficient multi-dimensional container of generic data. Arbitrary data-types can be defined. This allows NumPy to seamlessly and speedily integrate with a wide variety of databases. 

Python Panda (Wrangling) 

Python has long been great for data munging and preparation, but less so for data analysis and modeling. pandas helps fill this gap, enabling you to carry out your entire data analysis workflow in Python without having to switch to a more domain specific language like R. 

Combined with the excellent IPython toolkit and other libraries, the environment for doing data analysis in Python excels in performance, productivity, and the ability to collaborate. 

pandas does not implement significant modeling functionality outside of linear and panel regression; for this, look to statsmodels and scikit-learn. More work is still needed to make Python a first-class statistical modeling environment, but we are well on our way toward that goal. 

// SciKit 

from sklean import LinearSVC// Support Vector Classifier 

import Cross_Validation 

train_test_split()// Input single dataset and percentage value 

import PCA// Principal Component Analysis 

transform()// Feature extraction 

import SVM// Support Vector Machine 

fit()// Prepare data 

predict() 

import pylab// Scatterplot 

scatter() 

legend() 

title() 

show() 

import metrics 

accuracy_score() 

import linear-model 

LogisticRegression() 

from Sklearn.cluster import KMeans 

labels_// Display results 

cluster_centers_ 

from Sklearn.cluster import DBSCAN 

// Density-Based Spatial Clustering Application with Noise 

Page Break 

Apache Spark (MLLib) 

Apache Spark is an open-source distributed general-purpose cluster-computing framework. Originally developed at the University of California, Berkeley's AMPLab, the Spark codebase was later donated to the Apache Software Foundation, which has maintained it since. Spark provides an interface for programming entire clusters with implicit data parallelism and fault tolerance. 

Run workloads 100x faster. 

Apache Spark achieves high performance for both batch and streaming data, using a state-of-the-art DAG scheduler, a query optimizer, and a physical execution engine. 

Write applications quickly in Java, Scala, Python, R, and SQL. 

Spark offers over 80 high-level operators that make it easy to build parallel apps. And you can use it interactively from the Scala, Python, R, and SQL shells. 

Combine SQL, streaming, and complex analytics. 

Spark powers a stack of libraries including SQL and DataFrames, MLlib for machine learning, GraphX, and Spark Streaming. You can combine these libraries seamlessly in the same application. 

Spark runs on Hadoop, Apache Mesos, Kubernetes, standalone, or in the cloud. It can access diverse data sources. 

You can run Spark using its standalone cluster mode, on EC2, on Hadoop YARN, on Mesos, or on Kubernetes. Access data in HDFS, Alluxio, Apache Cassandra, Apache HBase, Apache Hive, and hundreds of other data sources. 

Apache Storm (Data processing) 

Apache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing. Storm is simple, can be used with any programming language, and is a lot of fun to use! 

Storm has many use cases: realtime analytics, online machine learning, continuous computation, distributed RPC, ETL, and more. Storm is fast: a benchmark clocked it at over a million tuples processed per second per node. It is scalable, fault-tolerant, guarantees your data will be processed, and is easy to set up and operate. 

Storm integrates with the queueing and database technologies you already use. A Storm topology consumes streams of data and processes those streams in arbitrarily complex ways, repartitioning the streams between each stage of the computation however needed. Read more in the tutorial. 

Apache Flink (Data processing) 

Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale. 

Process Unbounded and Bounded Data 

Any kind of data is produced as a stream of events. Credit card transactions, sensor measurements, machine logs, or user interactions on a website or mobile application, all of these data are generated as a stream. 

Data can be processed as unbounded or bounded streams. 

Unbounded streams have a start but no defined end. They do not terminate and provide data as it is generated. Unbounded streams must be continuously processed, i.e., events must be promptly handled after they have been ingested. It is not possible to wait for all input data to arrive because the input is unbounded and will not be complete at any point in time. Processing unbounded data often requires that events are ingested in a specific order, such as the order in which events occurred, to be able to reason about result completeness. 

Bounded streams have a defined start and end. Bounded streams can be processed by ingesting all data before performing any computations. Ordered ingestion is not required to process bounded streams because a bounded data set can always be sorted. Processing of bounded streams is also known as batch processing. 

Apache Hadoop (Library) 

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures. 

• Hadoop Common: The common utilities that support the other Hadoop modules. 
• Hadoop Distributed File System (HDFS™): A distributed file system that provides high-throughput access to application data. 
• Hadoop YARN: A framework for job scheduling and cluster resource management. 
• Hadoop MapReduce: A YARN-based system for parallel processing of large data sets. 

• Hadoop Ozone: An object store for Hadoop. 

Apache Hadoop MapReduce 

MapReduce is the heart of Apache Hadoop. It is a programming paradigm that enables massive scalability across hundreds or thousands of servers in a Hadoop cluster. The MapReduce concept is fairly simple to understand for those who are familiar with clustered scale-out data processing solutions. 

For people new to this topic, it can be somewhat difficult to grasp, because it’s not typically something people have been exposed to previously. If you’re new to Hadoop’s MapReduce jobs, don’t worry: we’re going to describe it in a way that gets you up to speed quickly. 

The term "MapReduce" actually refers to two separate and distinct tasks that Hadoop programs perform. The first is the map job, which takes a set of data and converts it into another set of data, where individual elements are broken down into tuples (key/value pairs). 

The reduce job takes the output from a map as input and combines those data tuples into a smaller set of tuples. As the sequence of the name MapReduce implies, the reduce job is always performed after the map job. 

Apache Mahout 

Produce free implementations of distributed or otherwise scalable machine learning algorithms focused primarily in the areas of collaborative filtering, clustering and classification. 

KMeansMapper, KMeansCominer, KMeansReducer, KMeansDriver 

Apache HBase (database) 

Random, realtime read/write access to your Big Data. This project's goal is the hosting of very large tables -- billions of rows X millions of columns -- atop clusters of commodity hardware. Apache HBase is an open-source, distributed, versioned, non-relational database modeled after Google's Bigtable: A Distributed Storage System for Structured Data by Chang et al. Just as Bigtable leverages the distributed data storage provided by the Google File System, Apache HBase provides Bigtable-like capabilities on top of Hadoop and HDFS. 

Apache Hive (database) 

Data warehouse software facilitates reading, writing, and managing large datasets residing in distributed storage using SQL. Structure can be projected onto data already in storage. A command line tool and JDBC driver are provided to connect users to Hive. 

Statistical Analysis System (SAS) 

A software suite developed by SAS Institute for advanced analytics, multivariate analyses, business intelligence, data management, and predictive analytics.