Autoscaling: Azure HDInsight Cluster

Introduction

Scale out/in of HDInsight Spark cluster is required due to variable workload been executed at specific intervals. This strategy help to optimize the cost and Azure resources. Here in this article we will discuss about various options available to perform autoscaling 

From Azure Portal

Scheduled based HDI Cluster Scale out in is possible from azure portal itself – from HDI settings select “Cluster Size”

The HDI Spark now has “Enable Autoscale” feature available in preview mode – there are 2 options available under this; 1) Load based, and 2) Schedule Based

Load-Based

Simply saying that it will do autoscaling of the cluster based on amount of CPU cores and memory required to complete the pending jobs. If CPU cores and memory required is more than the available CPU core and memory then it will trigger autoscale (up/down) accordingly.

Schedule-Based

With Schedule-based we need to configure the Autoscale schedule as displayed:

One the schedule is configured the – the autoscaling will happen as specified:

Conclusion

This example showcase one particular way to autoscale the HDInsight cluster from Azure portal itself. There are various other custom approaches to achieve similar benefits using Azure CLI or PowerShell, those options we will discuss in next post. 

Docker - mounting an external disk

This short tutorial, In this I will brief about, how we can manage to attach external drive to the docker container. I stumbled on an issue wherein I have already attached and mounted and external drive to the my AWS Linux VM instance. This external drive is supposed to hold all my data (kind of data disk/drive) so that I can keep my OS disk isolated.

The actual issue was that when I tried to run the docker image in the container with current using:
sudo docker run -i -t --rm -v $(pwd):/remote:rw [image_name] /bin/bash

This error was thrown:
docker: Error response from daemon: error while creating mount source path '/remotedata/ws': mkdir /remote: read-only file system
Here, '/data/ws/' is a folder location (with drive mounted on: /remotedata) on my mounted external drive and '/remote' is a path mapped for reference in the container.

Soon I realized that the issue and found that the problem was because I mounted the external disk to different mount point, e.g., /remotedata in my case. So to solve this problem I mounted the external drive again under the hood of  '/mnt/remotedata'. After that, I also had to change the directory permissions to "766" using chmod and lastly made current user to default owner for the folder '/mnt/remotedata'.

Hence, once again running the command as:
sudo docker run -i -t --rm -v $(pwd):/remote:rw [image_name] /bin/bash
Successfully started the container as desired. Here, $(pwd) is '/mnt/remotedata'.

Just to conclude here-
The issue: "docker: Error response from daemon: error while creating mount source path '/data/ws': mkdir /remote: read-only file system" was resolved by mounting the external disk under'/mnt' hood and by modifying the permissions and ownership.

Azure Service Fabric & Azure Kubernetes Service: Comparative Analysis

Hello Folks! in today's blog I will highlight the key main differences between Azure's (of Microsoft's) two widely popular container Orchestration services. Please remember that these differences are analyzed as per today's date. As we know that services on cloud keep evolving and points described here may not be relevant after sometime.
So let's start rolling ---- 
Those who don't know about Azure Service Fabric (ASF) and Azure Kubernetes Service (AKS), I strongly recommend them to look into starting points to get to know more about these services at high level.

Azure Service Fabric from the Microsoft's documentation:

Trust a proven platform for mission-critical applications
Focus on building applications and business logic, and let Azure solve the hard distributed systems problems such as reliability, scalability, management, and latency. Service Fabric is an open source project and it powers core Azure infrastructure as well as other Microsoft services such as Skype for Business, Intune, Azure Event Hubs, Azure Data Factory, Azure Cosmos DB, Azure SQL Database, Dynamics 365, and Cortana. Designed to deliver highly available and durable services at cloud-scale, Azure Service Fabric intrinsically understands the available infrastructure and resource needs of applications, enabling automatic scale, rolling upgrades, and self-healing from faults when they occur.

Choose from a variety of productive programming models and languages including .NET Core 2.0, C#, and Java to build your microservice and container-based applications.

reference

Azure Kubernetes Service from the Microsoft's documentation:

Deploy and manage Kubernetes with ease, scale and run applications with confidence, secure your Kubernetes environment, Accelerate containerized application development.

Brief Overview

Azure Service Fabric - 

• High affinity with Microsoft's (Visual Studio) product and tools.
• It is an orchestration engine and a framework to build micro services using mostly .NET. It also includes and support programming models.
• Application not necessarily to be hosted in containers.
• Unable to support traditional ASP.Net applications for as is deployment.
• It is typically a PaaS offering.
• Support hybrid deployments to run applications hosted on Service Fabric on Azure and onpremise

Azure Kubernetes Service - 

• It is mostly likely a PaaS and an IaaS offering
• The configuration is a bit more complex because you have to define all components in your cluster such as load balancers and endpoints. In Service Fabric more of it is done for you automatically. 
• Store images in Docker Hub or Azure Container Registry and deploy to your preferred targets.
• When you use an AKS cluster the Kubernetes master nodes are hidden from you and are operated by Microsoft, the only VM resources you see are the worker nodes, which are still IaaS and need to be managed and updated as usual. In AKS, you are only paying for the worker nodes, the managed part is free.
• It can easily be used for lift and shift architectural styles and large-scale projects

Key Similarities

Native Cloud Features

• Open source standards of cloud-native foundation
• Powerful native cluster orchestration, cluster management, auto-scaling, self-healing
• Open-source Interoperable (while using Service Fabric with Net-Core)
• Azure Cloud Native: log analytics, managed service identity, encryption

Architectural Patterns

• Micro-service and event-driven architecture
• Multi-tenancy, orchestration, discovery, advanced clustering
• lift and shift architectural styles and large-scale projects
• Suited for Large scale complex prod apps 

DevOps, Language & Tools

• Standard support for widely used DevOps methods and tooling, e.g., Jenkins, HELMS
• Support remote debugging

Comparative Analysis

Following is the gist of overall key differences identified and experienced so far while using these two:

.	Azure Service Fabric	Azure Kubernetes Service
Cloud feature	Native dev framework: "stateful/stateless reactive" & " 12-factor applications"	It’s a container orchestrator
	PaaS: Infrastructure is abstracted	IaaS & PaaS: low level control on infrastructure; resource quota, infra placement.
Legacy Migration	Refactor legacy .Net application; unsupported Nuget Pkg	Rewrite/Refactor/Lift&Shift
Containerization	Containers are just guest executables	Containers are primary need
Language/platform and tools	Primarily Open Source support for .Net Core	Language agnostic
DevOps	Affinity with MS tools and stack, e.g Visual Studio	Integration with wide range of OS tools and extension
Vendor support	Indirect vendor lockin; SF is   not manages service with any other cloud provider	Supported by all leading cloud provider
Community Support	Limited and MS dependent	Extensive and mature
Practical challenges	Remote debugging is tedious
	Inaccessible logs to diagnose errors and issue
	Co-ordination with MS to raise and resolve issues. Service tickets and difficult SLAs
	Inconsistent services upgrade from local to portal

Conclusion

In this article I have tried to explain the key differences and basic similarities between the two container orchestration services (or framework) provided by Microsoft Azure. Hence, this will certainly help decide among these two depending upon the technical and architectural needs of the application. Feel free to provide suggestions in the comment section below.

Quick steps to setup Ubuntu GUI & RDP from Windows Machine

Introduction

The post will discuss mainly about provisioning an Ubutu machine on Amazon AWS, setting up GUI on the machine and finally able to do the RDP from the windows machine.

The approach really helped me to cut my cost of running my Machine Learning experiments from the Ubuntu (Linux) machine rather running them from a Windows machine. This actually reduced the bill by whooping 50%.

Step 1: Provision a Ubuntu machine on AWS

As explained in my earlier post, from the documentation we can acquire the Ubuntu machine on Amazon AWS. The tutorial also help with the process to connect to the new machine using Putty tool.

Step 2: Setup GUI on Ubuntu

Once we are successfully login and connected to the Ubuntu machine, execute the following commands in sequence:-

Command 1: sudo apt update

Command 2: sudo apt upgrade 

Command 3: sudo sed -i 's/^PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config

Command 4: sudo /etc/init.d/ssh restart 

Command 5: sudo passwd ubuntu (this will ask for password reset for user'ubuntu'. Remember it!)

Command 6: sudo apt install xrdp xfce4 xfce4-goodies tightvncserver

Command 7: echo xfce4-session > /home/ubuntu/.xsession 

Command 8: sudo cp /home/ubuntu/.xsession /etc/skel 

Command 9:  sudo sed -i '0,/-1/s//ask-1/' /etc/xrdp/xrdp.ini 

Command 10: sudo service xrdp restart 

Command 11: reboot

Step 3: Configure and Save Connections from Putty

Enable tunneling from the putty to use the localhost port# 8888 (just an e.g.) to the Ubuntu machine RDP port# 3389. Here we also need to use the Private IP of the remote machine and map its port# 3389 to the tunnel. Save the connection for connecting with the machine multiple times later.

Step 4: Connecting Via Windows machine

Again connect to the remote machine by “Load” the above saved putty configuration.

Once connected to machine using putty, use RDP (Run -> mstsc.exe) to connect to the remote machine by using: localhost:8888 in ‘computer’ field. This is the port# which I have used to enable tunneling via local computer.

Once connected – it will prompt for ‘ubuntu’ password which we configured in Step#2, command 5. Upon successful login we will be connected to the Ubuntu GUI from our Windows via RDP.

Conclusion

Once I was able to setup my Ubuntu GUI machine, then I further installed Visual Studio code as an IDE for my experiments and development and later installed my ML Docker image to have a fully functional Python and TensorFlow development environment.

TensorFlow Docker setup on Ubuntu

Introduction

To continue with series of my experiments on setting up TensorFlow development environment, in this post I will cover:

How to setup TensorFlow Docker Development Environment on the Ubuntu machine?

The is one step ahead of what I did to setup Tensorflow on the windows machine. Along side with the robust Visual studio code as IDE for development.

Following are the steps which I used for the quick setup:

STEP 1: ACQUIRE THE UBUNTU MACHINE

I quickly acquire the latest Ubuntu linux machine through my AWS account. One can use this quick tutorial from AWS documentation to “Launch Instance” of type Ubuntu

STEP 2: SETTING UP DOCKER

Once we have the machine, the next step is to quickly do the setup of docker. This require execution of the commands as mentioned in the tutorial: Get Docker CE for Ubuntu 

STEP 3: CREATE A DOCKER IMAGE

At the PWD create a file having name Dockerfile (without any extension), copy the below content and save the file.

FROM python:3.6
 RUN apt-get update -y
 RUN apt-get install -y git
 RUN apt-get install -y unzip
 update pip
 WORKDIR /remote
 VOLUME /remote
 ENV REPO ""
 RUN pip install pip --upgrade
 RUN pip install wheel
 RUN pip install -U pip virtualenv
 RUN virtualenv --system-site-packages -p python ./tensorflow && \
     sh ./tensorflow/bin/activate && \
     pip install --upgrade pip && \
     pip install --upgrade numpy && \
     pip install --upgrade scipy && \
     pip install --upgrade opencv-python && \
     pip install --upgrade matplotlib && \
     pip install --upgrade pandas && \
     pip install --upgrade sklearn && \
     pip install --upgrade scikit-image && \
     pip install --upgrade tensorflow && \
     pip install --upgrade keras && \
     pip list

Refer code in the repo link

STEP 4: Build the Docker Image

Once we have Dockerfile
$ sudo docker build -t di-ubuntu-py3-tf .

STEP 5: RUN THE DOCKER INSIDE THE DOCKER CONTAINER

Till now we have the build the TensorFlow docker environment inside the Ubuntu machine, next is to allocate the computation resources to this image. Hence let’s run this image inside the container using below command:

$ sudo docker run -i -t --rm -v $(pwd):/remote:rw di-py3-tf-base /bin/bash

The above command execute the TensorFlow docker image inside the container and various options used in the command has following capabilities – 

• -t: flag assigns a pseudo-tty or terminal inside the new container
• -i: flag allows you to make an interactive connection by grabbing the standard input (STDIN) of the container
• –rm: flag automatically removes the container when the process exits
• -d: Run the container as the daemon
• -v: is a volume mounting HOST DIRECTORY on the ubuntu machine to the CONTAINER DIRECTORY as defined in docker image.

STEP 6: INSIDE THE CONTAINER

Once the container is started, it enables the fully capable TensorFlow development environment in the Ubuntu (linux) machine. The /remote directory which is also defined as the working directory for the container in Dockerfile is mapped to the /{pwd} on the host machine. This mapped volume will always persist on the host machine even if the container is terminated.

Conclusion

By quickly acquiring an Ubuntu machine  from AWS console and on top of it setting up the docker environment can help to run various Open Source Deep Learning docker images framework on the fly. Here, I demonstrated the process using my own custom configured docker image. This actually gives me lots of flexibility and control on my development environment.

Preparing Visual Studio Code for TensorFlow Development

Introduction

Visual Studio Code is a popular open-source IDE distributed by Microsoft. This is a powerful tool which support wide variety of code development almost across all the platforms. This allows various extensions which can be installed to support entire end-to-end development life-cycle.

Google’s TensorFlow is well known Deep Learning library which was originally available for the Python developers.

This article describe about preparing TensorFlow development environment on Visual Studio Code on Windows VM.

Prerequisites

1. Download and install Visual Studio Code
2. Python: TensorFlow support version 3.4, 3.5, 3.6 (64-bit version)
3. Pip: Which installs as an option with Python
4. VirtualEnv: To setup virtual environment for TF
5. Download and install Visual C++ 2015 Redistributable Update 3 from this URL: https://www.microsoft.com/en-us/download/details.aspx?id=53587

Setup Steps

1. Download and install Visual Studio Code on the Windows 64-bit machine. The the download url and installation instructions are available on VS Code site
2. Open VS Code and create a workspace folder. Also create a dummy python file, call it tf_test.py.
3. Once the file is created – VS Code will prompt for Python installation and Python extensions for VS Code. Install Python 3.6 from official download site. [Ignore if Python is already installed].
4. Allow VSCode to install and enable necessary extensions for Python, e.g. Pylint, etc..
5. TensorFlow installation: refer the [url]
6. Install Virtual env
   PS C:\Program Files\Python36> pip3 install -U pip virtualenv
7. Create new TensorFlow virtual environment:
   virtualenv --system-site-packages -p python ./tensorflow
8. Activate the TenforFlow virtualenv:
   PS C:\Program Files\Python36> ./tensorflow/scripts/activate
   (tensorflow) PS C:\Program Files\Python36>
9. Execute the python script as:
   (tensorflow) PS C:\Program Files\Python36> python <filePpath>\tf_test.py.

Lastly, to deactivate and exit the virtual environment use : deactivate

Conclusion

This simple tutorial for TensorFlow setup on the Windows machine will help developers who are not comfortable to start with training Deep Learning models on Ubuntu (or any Linux) machine. 

Ensemble Approach - Stacking

This is going to be series of tutorials mainly describing about various Ensemble techniques and approaches. This also gives the very high-level idea of the implementation of each of these techniques in Python. Here, few experiments are performed on the famous Iris dataset, and the task is to classify the plant species from its key attributes, namely: length, width, sepal and petal.

At the beginning of this I am going to start with the simple Stacking example- Here i will be using a self-generated random data-set with two input variables X1 and X2, output variable is, Y.

To start with the experiment let's perform the basic steps to setup notebook:

Import the required Python libraries for the experiment

import numpy as npimport pandas as pd

import os

import sklearnfrom sklearn.model_selection import train_test_split

from sklearn.tree import DecisionTreeClassifier

from sklearn.neighbors import KNeighborsClassifier

from sklearn.linear_model import LogisticRegression

Import the data-set, add the columns attributes and shuffle the data-set

#Process on iris dataset
dataset = pd.read_csv("../input/iris-dataset/iris.data.csv"
dataset.columns= ["length","width", "sepal", "petal","class"]
dataset = dataset.sample(frac=1).reset_index(drop=True)
#Process on random dataset
random = pd.read_csv("../input/randomdata/random-data.csv"

Create train and test split

data_x = random.iloc[:,0:3]
data_y = random.iloc[:,3]
train_x, test_x, train_y, test_y = train_test_split(data_x, data_y, test_size = 0.20,random_state = 1001)

Implementing  - Stacking

Stacking is a technique by which we Pick the Model1 as a base model and then create a K-Fold from the training set, then this base model is made to learn from the K-1 part of the training data. Then the prediction is made on the K-th split set of the training data. This process is repeated K-times to fetch next set of (K-1) splits for training and then K-th set for validation purpose.

The approach is better demonstrated from the experiment shown below: The Stacking function is defined with 4 arguments. The stratified K-folds are made to create K-splits for each iteration. *Iteration 1* fit the model on k-1 splits and predict results of the k-th split (which is actually a validation set). Thus repeating the entire process k-times. Along with this, simultaneously the same base model is also fit against the test set.

The entire above process is then repeated for the next base model Model2 resulting in entirely new set of predictions for train set and test set.

Method Definition:

def stacking(model, train, y, test, n_fold):
    folds = sklearn.model_selection.StratifiedKFold(n_splits = n_fold, random_state=1001)
    test_pred = []
    train_pred = []
    for train_indices, val_indices in folds.split(train,y.values):
        x_train, x_val = train.iloc[train_indices], train.iloc[val_indices]
        y_train, y_val = y.iloc[train_indices], y.iloc[val_indices]
        
        model.fit(X=x_train, y=y_train)
        train_pred = np.append(train_pred, model.predict(x_val))
    test_pred = np.append(test_pred, model.predict(test))

    return test_pred, train_pred

Model 1

model1 = DecisionTreeClassifier(random_state=1)
test_pred1,train_pred1 = stacking(model = model1,n_fold = 5,
train = train_x,test = test_x,y = train_y)
train_pred1 = pd.DataFrame(train_pred1).astype(int)
test_pred1 = pd.DataFrame(test_pred1).astype(int)

Model 2

model2 = KNeighborsClassifier()
test_pred2,train_pred2 = stacking(model = model2,n_fold = 5,
train = train_x,test = test_x,y = train_y)
train_pred2 = pd.DataFrame(train_pred2).astype(int)
test_pred2 = pd.DataFrame(test_pred2).astype(int)

Once we have the predictions from the test set, we will use these predictions as the new set of features to create a Model3 (stacking the results from the above 2). Lastly the 3rd Model is used to predict on the test set for the final predictions. Below is the code to implement the same.

df_final_train = pd.concat([train_pred1,train_pred2], axis=1)

df_final_test = pd.concat([test_pred1, test_pred2], axis=1)

Model 3

model3 = DecisionTreeClassifier(random_state=1)

model3.fit(X=df_final_train, y=train_y)

pred = model3.predict(df_final_test.reset_index(drop=True))

model3.score(df_final_test.reset_index(drop=True), test_y.reset_index(drop=True))

Conclusion

Hence the example shows the basic and the simple way to implement the ensemble stacking using simple base models. This, approach helps to combine the predictive power of simple base models to perform better predictions.
In the Next tutorial we will see one more basic ensemble approach which is almost similar to stacking, called blending.