Friday, 20 March 2015

Hadoop The Definitive Guide [Book] - Study Notes

Requirement and adoption in yahoo.
A framwork can scale to the web.
Map and Reduce acitivity and features like data locality.
Can be applied with a variety of algorithms
Huge data processing can beat good algorithms

The Map Java class and Reducer Java class
The Job java class
Jobtracker and tasktracker
Hadoop reduces the input to input splits or just splits
Map tasks write the intermediate output to local disks, so that they can be discarded after use.
Outputs of Reduce tasks are stored in HDFS
Combiner function can be run on map output, and the combiner functions output forms the input to the reduce function
Hadoop streaming proivide hadoop apis in languages other than Java

Fault tolerant solution. Same data written at multiple places.
Filesystems that manage the storage across a network of machines are called distributed filesystems.
Blocks - a block size is the minimum amount of data it can read and write (for hdfs its 64mb by default)
Namenodes and Datanodes - An HDFS cluster has a master-worker pattern: a namenode (master) and number of datanodes(workers). Master has all the meta data and datanode has all the blocks (but not persistent). Its reconstructed at start time.
HDFS federation
HDFS High-availablity
On large clusters the time it takes for a namenode to start from cold can be upto 30 mins
Fencing and failover - When one node fails an entity called 'failover controller' switch to the standby node. But first a ZooKeeper is used to ensure that only one namenode is active.
Graceful failover - triggered by adming
Ungraceful failover - in this case to make sure that the other node has completely stopped running, a mechanism called fencing is done. In worst case it does ' shoot the other node in the head' - force shutdown .
File Operations in HDFS
There are java endpoints to do all operations like create, delete, sync
Use Flume and Sqoop to move data
Copy parallel with distcp
Hadoop archives are compressed blocks that can be used as input to MapReduce

Chap - 4 I/O
Reading compressed data
Serialzation in natively implemented in Hadoop for better perfomance
Apache Avro is a project to do this in an improved way and support multiple languages, diff from Google Protocol Buffer and Thrift

Chapter - 5 - Developing a MapReduce Application

Setting up the Environment
- The Configuration API to read xml resource files etc
- Writing Unit Test with MRUnit
- Running locally on a small data
- Using Tool Interface write a Driver to run our MapReduce Job (Java file)
- Testing the driver
- Run in Cluster
- Package jar
- Launching a Job run the driver
- Debugging a Job
- Running multiple Job in particular flow

Chapter 6 - How MapReduce Works

Tuesday, 3 March 2015

How To Publish An Apple Watch App To The AppStore

The Apple Watch launch is almost nearing at the time of writing this article. I'm all excited and ready to submit my first Apple Watch compatible application to the AppStore. I'll write down my learning experience here so that you can publish your own Apple Watch application to the app store. I'll do this step by step, as the work of my current app progress. This article will be updated over time until I reach the final step to see it live in the AppStore.

Step 1 : Make the iPhone Part of the Apple Watch

An Apple Watch app is not much different from an iPhone app. In fact, it is a sub-part of the main iPhone application running on the iPhone and the Watch App merely acts as the extension of the parent app in the iPhone. So essentially, need an iPhone app anyway. In this scenario, I'm thinking of building an app that will be useful both on iPhone as well as the Apple Watch, instead of solely focusing on the the Apple Watch aspect. The app will be very simple, but all it has will be available on both the devices.

Update: To submit Apple Watch Application now, you should use Xcode 6.2 and not Xcode 6.3. The Xcode 6.3 comes with Swift 1.2 which is currently under beta and not yet supported for AppStore release.

Also, as of now Apple AppStore is not accepting Apple Watch applications . See the screenshot from Apple WatchKit portal below:


Saturday, 31 January 2015

Learn Apple Watch Programming Quickly by Examples - My January Challenge

Apple Watch is soon to release, and being a huge Apple Evangelist, I've been very eager to explore the possibilities of what one can make with these 'Most personal device ever made by Apple'. My hopes are high. I'm in a constant mission to excavate this area at the earliest. And guess what, me and my friends have been working on a couple of interesting 'Apple Watch' things lately:

1. Creating a course - 'Learn Apple Watch Programming Quickly By Examples'
2. Creating a website for Apple WatchKit Tutorials

Both are in beta and will be public soon. I'm trying to get the latest releases of the WatchKit sdk (which is bundled in Xcode 6.2 beta and higher and is required to build Watch Apps) and trying to publish tutorials on new APIs as and when they come.

Following are my objectives with both these online ventures

  • Teach things more by examples and less by theory
  • Convey ideas at the simplest form possible
  • Keep user-interaction at the heart of all materials
  • Make the examples in a way encouraging the user to replicate the same
  • Provide quick support in the discussion forum of the course 

Let me know the feedback/ideas , and yes watch out this space for more details coming soon.

P.S. : If you'd like to get a free coupon for my course, please leave a comment with your email address below.

Saturday, 22 November 2014

I'm Live Blogging : AndroidWear and Good Programming Practices - Google Developer Group Trivandrum DevFest 2014

Android Wear 

Notifications done for android apps will automatically come to the AndroidWear without any extra coding.

Main UI

  • Stacks 
  • Pages - Multiple Glancable Pages
  • Replies - Via Voice Commands etc

Development Option

Make Apps Specifically For AndroidWear

Most apps will have a companion app in wear and main app in phone.

SOLID - Principles of OOP

Robert C. Martin ( Uncle Bob)

Agile Software Development: Principle, Patterns and Practices

Google compute engine 

There was a over view of less heard about technology - Docker. This is massively helpful for sys admins to spin up vm's and deploy quickly. Rapidly reduces the total turn around time. 


Blogs mentioned to read

I'm Live Blogging : Polymer, Webcomponents - Google Developer Group Trivandrum DevFest 2014


Started (by +Arun Shanker Prasad ) with a slight intro and kicked-off with an overview of a data science application that was built by the Qburst (Company) team. And then moved over to Web Components and Polymer.

Parser Combinators for Event Detection

Presenter : +Navin Philip , Associate Architect, Qburst

A beautiful project data-crunching app that utilises the Theory of Computation to verify and quantify different hypothesis and give useful results. Had a quick demo. d

Web Components
Presenter : Shyam, Qburst

Why Web Components?

You could import/export encapsulated custom components. There are 4 main technologies in web components.

Make a Facebook like button without importing javascript.
Include maps in a page.

Polymer , X-tags( (Mozzila), Bosonic (Independent) 


New templating tags in html.

HTML Imports

Loads custom element and resoureces
In node.js
npm  install -g vulcanize

Shadow DOM

This a component that brings in the concept of Encapsulation in html. These elements exist but are not a part of the DOM. Have you ever wondered how the video tag works behind the scene? It uses Shadow DOM.

Scoped CSS!!!+encapsulated markup

<div class=“widget”>Hello, world</div>
var host  = document.querySelector(“.widget”);
host. createShadowRoot();
root.innerHTMLL - “<em> I’m inside yr div!</em>"

Custom Elements

Element prototype must inherit from HTMLElement
there are things like HTMLUnknownElements that swallows the error when u declare undefined errors
Custom Elements vs AngularJS
CE is similar to directives provided by AngualarJS. AngularJS version 2.0 will be using web components.
<tabs>.. </tabs> //invalid elements
<x-tabsx>.. <x-tabs> //Valid elements

Polymer : The future of web development ?
Presenter :Google - +Vijay Dev , Twitter- @vijaydevin
What are Web Components?
Is an umbrella four four different technologies (explained above). 
Have tags like : 
  • <core-toolbar>...</core-toolbar>
  • <paper-tabs>...<paper-tabs>
  • <super-button>
  • <super-textarea>

Implement functionalities like scroll till the end of agreement on clicking 'I agree' button. (Eg; The feature in android playstore). 

Most browsers are yet to catch up with webcomps ( excepet opera, firefox and chrome) and would take a long time. Thats where Polymer comes in. 

Polyfills Web Components with webcomponents.js ( previously platform.js ). 

  • Sugaring: Templates
  • Sugaring: Custom Elements
  • Sugaring: Shadow Elements
  • Sugaring: HTML Imports

Define custom tabs
  • <animated-pages>
  • <menu-button>
  • <page-scaffolde>

Map App Example 
  • Demoed an app like google apps, designed with 
  • <core-toolbar>
  • <core-header-panel flex>

Paper -Elements 
Make interfaces we see in Android Lollipop
  • <paper-elements>
  • <paper-input>

  • ::shadow
  • /deep/
  • With ::shadow and /deep/ you can apply sitewide themes

Similar topics 
Mozilla Brick

Good resources to checkout: 
  • polycasts (Video of what people are doing with polymer
  • YO POLYMER (Auto generate a lot of code)
  • seed-element ( Create basic, your own element)
  • Chrome Dev Editor

Good Web components to checkout
  • <app-router>
  • <page-er>
  • <ajax-form>

Github is using web components. All timestamps are custom elements. 
Rating , accordian

GDG Trivandrum Web :

Sunday, 12 October 2014

World's Most Affordable Telepresence Robot From India - The Making Of Angel

Demo Video: 

Watch us showing a demo of Angel. Our colleague in Australia talking to us and navigating around at the same time. 

Update: We are taking limited pre-orders. If you are interested register here for a call back. 
SlideShare :

Angel -  Is a Telepresence Robot that can be used to do video calling and navigation at the same time. While the least expensive telepresence robot in current market costs at least $2500 we managed to make it for $250.  This huge achievement was a result of long hours spent by a team of 3 members (and a lot of help from our colleagues) , thinking and trying out different possibilities to achieve this dream.

The controller will require two device ( one for making the video call, and one for controlling the device navigation). It's navigation control is done in javascript, so it can be controlled from any device/pc browser. The dashboard sample can be found here . It's advantage from other solutions is the support for multiple video calling apps like Skype, Hangout, Fring, Imo etc.

Navigation signals are passed from a remote user to the smartphone (via internet) mounted on the Angel's navigation unit. The received signals are converted into bluetooths signals by the smartphone and send to the circuit board's that control the wheel movement.

Most Helpful When :

  • Managing geographically distributed workforce
  • Assist people with disabilities when the carer has to work at a distance
  • Doctors can consult patients remotely can assist surgeries  
  • To add emotional expressiveness, engagement and playfulness with video calls

More Behind the Scenes Pics:

Sunday, 17 August 2014

Places To Travel In Sydney - Day 3 - Blue mountains

Blue mountains is the most popular tourist attraction near Sydney. It is a considerably dense and hilly forest area. There are three beautiful mountain peaks standing close by and they are called the Three Sisters. It's the heart of Blue mountains where travelers flock from every part of the world to enjoy the scenic beauty.
However, for this trip we were fortunately accompanied by one of our colleagues who knew the place very well. In fact, in his 4x4 car we got to see several nearby places most of the tourists can't reach. I'll simply stop this post here and put up the photos of  the places in Bluemountains that caught my attention.

Get My Next Post In Your Inbox