Reference : Understand Hadoop in EDW context of large enterprise.

Leave a comment

Organizations have created their EDW ecosystem that they have spend years to develop and implement and people have used the same. Will business allow Hadoop to just come and upset the existing apple cart? Hadoop need not be a big-data-only solution A business with existing EDW system can start using  Hadoop to start and extend the current system.

Business choose better handling of structured data coming from data sources and also processed structured data.  Here a large amount of  data is arriving in the enterprise. To handle large amount of data, the enterprise adds more hardware and/or horsepower to the existing EDW and operational systems or consider alternative ways to manage data. Hence Hadoop (HDFS) can be used as alternate data staging platform to load your EDW and  MapReduce jobs bring the application data into the HDFS, transform it and then send the transformed data on its way to your EDW.

Using Hadoop (HDFS) , we can store both versions of the data in the HDFS: the before application data and the after transformed data. Now the data has been gathered in one place, making it easier to manage, reprocess (if needed) and analyse at a later date. When Hadoop processes the data, EDW resources and operational systems are freed up to  do what they do best, analysis.

Business chooses to leverage structured data sources that have not been integrated into your EDW and unstructured data sources.  Here Hadoop (HDFS) can be leveraged to take advantage of data that’s currently unavailable in your EDW. The current data that is not part of your current EDW might have potential to provide additional insight into your customers, products and services.

You can process and keep the data in Hadoop (HDFS) and, optionally, push relevant data into your EDW to be analysed with existing data. You need not structure all the unstructured data for the EDW. You can then analyse the data using big data apps or BI/analytics tools. Here Hadoop (HDFS) complements the EDW well in terms of storage. any data that EDW cannot handle well can be stored in Hadoop.

Business chooses to  archive all data Use Hadoop to archive all your data on-premises or in the cloud. For the first time, the data need not be destroyed after its regulatory life to save on storage costs. The business analyst or data scientist need not limit his data analysis to the last three, five or seven years. The data also can be stored more easily and cost-effectively.

Enterprise chooses bold step to use Hadoop as the landing platform for all data and exploit the strengths of both the EDW and Hadoop  to enable extracting even more value and insight from one of their greatest strategic assets – data. Here, data captured in Hadoop can be stored in its raw, native state and need not be formatted upfront as with traditional, structured data stores; it can be formatted at the time of the data request. This saves programming efforts by loading data in its native state.

Learning Big Data & analytics

Leave a comment

I started to learn about Big Data & analytics and writing a few blogs to track my own learnings in future.

Big Data simply refers to massive amounts of information that, when combined, exceeds limits of traditional technologies and techniques to make the data useful. Currently businesses face significant challenges when it comes to V3true integration and collaboration. That is a huge missed opportunity.There are three “Vs”.

Volume  refers to the number of data sources and the sheer volume of data put is increasing exponentially, Velocity refers to the coming in extreme faster rate than most of us can imagine and Variety refers the different types of data – both structured and unstructured.

Whatever Vs say, we need to know where the data is and be able to access and understand it anywhere, any-time. Without that capability, you’re always looking in the rear view mirror and talking about what was instead of what can be. Can we have insight in to existing data followed by foresight?

Will Big Data enable us to stop looking at business from what has happened till now (Descriptive) and why it occurred (Diagnostic) and start  looking with a more proactive approach (Predictive) that asks, “What will happen and how can we make it happen?” (Prescriptive). For example Can we have a 360 degree view of your customers with respect to their influencers and behaviours which leads to success business?

Implementation of  analytics’s strategy  involves data, process, people

  • Data generated from business processes. It is difficult to bring data together across different data silos. Will storing more than 90 days’ worth of data not become expensive? 
  • We need to come with  the process for finding value in the data  and the process for adding analytics into the business. it is important that process starts small and fast and build credibility. The process needs to use good visualizations to takes audience on a data journey.
  • The team  needs knowledge of the problem domain and one  data scientist.  Every team member should be able to pull their own data and do some simple analytics. They also should be brought up on speed  on the domain knowledge asap. This team should have a decision maker or C level executive as team member to influence and create needed changes in business processes of the enterprise.

Be prepared to handle these challenges

  1. Machine learning or Data Science cannot replace people.
  2. When you accumulate content,the volume, velocity and variety of information getting collected is challenging people’s ability to manage it.
  3. When engagement with customers become more transparent, the back-end business process needs to be streamlined and optimized.

Does video quality impact user engagement on website?

Leave a comment

I seem to watch more videos on internet compared to 2 years earlier.  I want to create video and these questions arised. How does creator track his video? How to help creator understand the viewer behaviour?    How does enterprise track videos? Which section of the video was cool and which was dumb?   What are the metrics to track the effectiveness of video solution for the business enterprise? A better metric has the potential to drive more video creation.

I see people have packed videos in tablet. One tracks whether student has viewed video or not. Is it possible to verify whether the student has seen each second of the video? Extend the problem to on-line, how is video being  tracked when video is available on-line and streamed?

Today I see sites like YouTube with  features per video similar to likes, dislikes, favourites and so on. This is more close to text content. Are we able to get granular information about the video like tracking each view for each second of the video. This would really generate a large data and demand smart ways to come with insights from the data.

What is alternative to YouTube? When you look at a site like connected with A.R. Rehman, the back-end is YouTube. Some of us publish in vimeo. Should Indian make use of foreign video hosting sites only?

To handle network outage and provide a consistent high-quality streaming experience for enterprise viewers and ensure availability of  videos  without failure, CDN technology has arrived. Today anyone created a video,upload videos to CDN and video is assessed from CDN over internet? What are the other ways other than YouTube and vimeo available and widely used?
As business owner, I am interested to know how the video scored with my visitor experience.? How did the video experience influence engagement with other content in my website? It is important to know exactly how well the videos are giving your audience what they want.

  • How long the video take to start-up?
  • Was the stream available on continuous basis till the end of the video?
  • Did  rebuff-erring occurred causing interruptions while viewing video?
  • Did the same user viewed the video multiple times? Did he finish watching the complete video?

How to track user with respect to video viewing?  Tracking need to happen at client and tracked information is stored in server. Client side tracing needs something like a browser plugin that gets embedded to media player of the  browser and send the usability data to server. Need to find whether such a solution exist? In addition, I am interested to learn how business enterprises manage all about videos.

Is social media one-sided?

1 Comment

In today’s social networking, I  share in front of mixed audience of family, friends, and acquaintances. There are messages that I am not comfortable to share with all of them. I have learnt the option to share selectively and this is not easy when the network size in creasing. Hence we share what we feel very proud and to get ego booster.  Each human being should be supported by technology to express/share/with friends both positive and negative things.  Sharing only positive and not negative have challenges to the behaviour style of the person.

when we fail to share things that we are not proud of, we justify ourself that Facebook does not have Dislike button.  Youngsters are less aware that Facebook has no means to share about one’s sorrow and sadness. Where is the  equivalent to  cry on friend’s shoulders where we need privacy and trust? I am not able to accept the usage of  Like Button on the message that reports the expiry of loved ones or loss of mobile.  We are limited to share what  one is not proud of using social media and continue to make use of phone calls, meetings.

  • Share my opinion with only a limited number of friends.
  • Way to express feelings that cannot be shared with Like. Why not things like Fear?
  • Have a list of friends to whom I can share what I am not proud of myself.

Already we find the practise where business hire third party vendors to boost the  FaceBook Likes without having real users for money.   A person who makes purchases or business decisions  based on Facebook Likes  Count is a  potential pawn in the hands of a rogue business.  Recommendations on LinkedIn can appear of the user profile based on user discretion. The user can delete bad recommendation. Can the LinkedIn endorsement   be influenced by third party vendors?

Business are interested to take feedback from the customer and there is opinion that customer is always right.  Today there are mechanisms to rate the application or article by the customer or reader with ratings from 1 to 5.  Customers always cannot be expected to give the right ratings and we need to be able to identify instances where the customer are not always right. Some of this might be because of wrong understanding of the product by customer and customer exhibits strange behaviours. Some times, customers can be pleasant people or people who cannot be pleased.

Today business and mobile apps have provision for the end user to rate the application and gets tied with the reputation the business to earn as seller, it is time that we need to track the customer reputation? Has customer used the product for a few days? Has customer leveraged a good percentage of the features to make his feedback relevant? Is there a way to explore how the  reputation stands with the other products used by the customer?   Business can decide based on their past reputation whether to make a specific customer eligible for discounts and special benefits on relevant products.

  •  Get the knowledge about the customer, right when the customer service begins?
  •  If multiple customers arrive at service desk at same time, who is the customer to be serviced first

How to measure relevance of the message? Is the message reliable or  true? Is the message relevant to me?  in the area of my interest? Every friend has  updates that are relevant to me and updates not relevant to me. I lose relevant message in the river flow of message that are not relevant to me. Let us also acknowledge that post the first connection acceptance, no message exchange happens between users connected on social networking site except receiving irrelevant  updates.

  • Have you collaborated or connected with each person connected in your professional network and explored some opportunity to work together.
  • If no opportunity was explored with the connected person, should the networking site help you with reminders and  to mark the connection dropped?
  • Can you connect with a person and  continue to get their notification updates and the same time block the person from receiving  your  updates.

I prefer a social networking  where fake connections have little value and  tangible benefit is more  collaboration between people.



Does your business focus on product or customer?

1 Comment

Companies operate in two ways

  • Launch a new product and finds the best fit customers for this product. Here you are changing the target customer segment time to time to suit the product. You mostly have one product and want to sell that product to as many customers as possible [ product centric]
  • Launch a new product and find the best fit customers for this product. Here you identify the most profitable customers and tailor the product to suit the customer needs.  You mostly focus on one customer segment and want to sell that customer as many products as possible. [customer -centric]

When you are focused product centric, you are open to risks and are vulnerable to  fluctuations in the market needs and your customer need to connect with the product. when  product cost increases and the product is positioned at higher price, the customer may not be ready to pay the price, and the company cuts their profit margins.  Another difference is products do not have and customers have memories.  A customer can provide reference to  other customers, based largely on perception of how well they were treated today. Referral or word of mouth is the most effective way of marketing.

Both approaches are mutually exclusive and can be implemented along with each other  in  a company. Every company tries to find the right balance between generating money/customer satisfaction and value creation/ market share growth. Companies that  achieve both becomes the market leaders in the industry. A customer centric approach ensures increase in wallet by referrals, generates  good faith and  leads to growing loyalty base.


The difficulty of getting good at anything.

Leave a comment

Firs you need to Learn to make something. Anything. When you are  learning to make stuff, you’ll want to learn how to make stuff that people actually want. You’ll want to constantly show people your work, no matter how bad it is. You’ll want to try and build an audience, or a customer base. Again, this is a skill in itself and may take years to figure out. You need to find small wins in what you do and this kind of positive feedback will keep you going.

It’s been discussed endlessly in books like Malcolm Gladwell’s Outliers, but it takes years of solid practice to be any good at anything. And then 10,000 hours (roughly 10 years) to become a world-class expert. But even before you sink years into something, you’ll get frustrated much earlier on because you’ll think that you suck at it. There’s a great quote from Ira Glass (of This American Life) about the difficulty of getting good at anything, starting as a beginner:

“What nobody tells people who are beginners — and I really wish someone had told this to me . . . is that all of us who do creative work, we get into it because we have good taste. But there is this gap. For the first couple years you make stuff, and it’s just not that good. It’s trying to be good, it has potential, but it’s not.

But your taste, the thing that got you into the game, is still killer. And your taste is why your work disappoints you. A lot of people never get past this phase. They quit. Most people I know who do interesting, creative work went through years of this. We know our work doesn’t have this special thing that we want it to have. We all go through this. And if you are just starting out or you are still in this phase, you gotta know it’s normal and the most important thing you can do is do a lot of work. Put yourself on a deadline so that every week you will finish one story.

It is only by going through a volume of work that you will close that gap, and your work will be as good as your ambitions. And I took longer to figure out how to do this than anyone I’ve ever met. It’s gonna take awhile. It’s normal to take awhile. You’ve just gotta fight your way through.”


How to bridge the skill gap? – Part 3

Leave a comment

Man jump

Post to Part 1 and Part 2 of the series, we look at the impact of the labour force broken in to  3 sectors based on skills.

  • The organized sector has skills acquisition in formal and recognized approach and the formal training is through professional course or apprenticeship. Example: CBSE,NO.
  • The un-organized Sector has skill acquisition in  informal and unrecognised way and actual training is through informal training or learning on the job. Example are NGO/SHG, Artisans Workshop.
  • There is section that does not have  sufficient skill to earn decent livelihood

Here are the ground realities to be handled in the process  to construct the bridge of skill gap across 3 sectors.(What?)

  • No recognition for prior learning/informal skills
  • Poor vertical/horizontal mobility in jobs
  • Weak linkage of academia with industry- employers
  • No common competency standards across similar jobs.
  • Minimum educational level needed for Formal Training
  • Courses are either supply oriented or they are demand oriented. They are not designed and focused on skills that student needs for a job.

Here are set of things to be thought through to develop skill development ecosystem.

  • How to standardize delivery of quality of education  across rural-urban divide?
  • How to handle individual differences ?
  • How to identify occupation standards across Employers’ preferences ?
  • How to create  Assessors and Certifiers who are well equipped?
  • Does getting certification add to the skill pool ?
  • How to change the mindsets with mass awareness generation and connect vocational  training with the aspirations of youth?
  • How to implement on pilot basis and also perform perform reality check?
  • How to grade competencies?
  • How to remove the —income parity between academic and vocational employment?
  • How to train and scale faculty required for intensive skill development?
  • How to —study of effectiveness of such a system ?

Older Entries

Srinivasan writes

my learnings

confused of calcutta

a blog about information

The Year On Campaign

Take a Break. Take a Year On.

Best Engaging Communities

The personal blog of Mukund Mohan

Ankur Learning Solutions

for holistic growth

Senthil Bhardwaj's Blog

Just another site

Sendhil's Cyber Home

Make a difference

Digital Software Development

Learning's from my digital journey

Steve Blank

Entrepreneurship and Conservation


Get every new post delivered to your Inbox.

Join 480 other followers