dirty basic MapReduce implementation(well not even an implementation)

So just to understand the concept of MapReduce better, I tried to create some Java code which would use the MapReduce 'pattern' to solve the word count problem (i.e counting number of times each unique word occurs in the given set of documents).Of course this has been tested only on a sample of three basic text files ,not error checking whatsoever and was implemented using the the first way that came to mind.

FileKeyValue.java - File name and File value(list of words in the file)
WordKeyValue.java - Word name and word count
MapReducer.java - This does the bulk of the work.

1. It creates a list of FileKeyValue objects for all the input files.
2. Then threads out each FileKeyValue object to be processed in a Mapper.
3. Waits for all the Mappers to finish.
4. Then sorts the output of all the mappers by the outputKey(i.e word) and consolidates the output value from all the Mappers for that output key (i.e. creates the intermediate list).
5. Threads out each unique combination of (outputKey, intermediate list) combination to a Reducer for reduction.
6. Waits for all reducers to complete.
7. Prints out the results

Mapper.java - Breaks the FileKeyValue object into a list of WordKeyValue objects.Each Mapper runs as a separate Thread
Reducer.java - Sums up the intermediate list values for a given word and passes it back to MapReducer class.Each Reducer runs as a separate Thread

Lecture on Cluster Computing and MapReduce from Google

For a while I was curious on what was MapReduce exactly and if even you are curious these lectures from Google will help

http://code.google.com/edu/submissions/mapreduce-minilecture/listing.html

So far I took the first and the second one.Both are approx an hour long.

The first one gives an overview of distributed computing and it's history.

• Difference between parallel computing and distributed computing
• Synchronization primitives and Semaphores
• Condition variables
• Fundamentals of Networking (what is a port, TCP, IP etc)

The second lecture goes into details of Map Reduce

• Overview of Functional Programming
• What is Map and Fold in the context of Functional Programming
• Overview of MapReduce with the example of a word count on a bunch of files Algorithm

I started writing a very basic MapReduce implementation in Java (using regular Threads of course to parallelize the Mappers and Reducers) for the Word Count Algorithm.I have about 50% completed and hopefully will complete it all tomorrow and post the code out here.

T61 RIAA SoundMax

So today I spent a good 4 hours of time trying to record what my speakers play on my Lenovo T61.Looks like recording right off the sound mixer was kind of a standard feature in sound cards even as back as in 2002.On windows XP if you go to Volume Control Applet -> Options ->Properties and then select the Recording radio button and if your sound card/driver support the record what you listen feature, you will see more than just the microphone in the white box below the radio button

Unfortunately looks like on T61 ,Lenovo deliberately prevents "record what you hear" feature from the sound card at a hardware level.So even if you upgrade to the latest version of SoundMax driver it won't help

Here are a few forum posts I found

http://forums.lenovo.com/t5/General-Discussion/Why-has-stereo-Mix-been-disabled-on-thinkpads-amp-when-do-we-get/m-p/38690

http://forum.thinkpads.com/viewtopic.php?t=52527

People think it is probably Recording Industry Association of America(RIAA) to blame.

But if you really want to overcome the hardware restriction you can go for a combination of audacity and Virtual Audio Cable. Worked pretty slick for me, audacity is open source and free whereas VAC is 30 $ per License.They have a trial version too but it keeps saying the word "Trial" every 30 seconds in your audio output.

T61 Bug

My office T61 Lenovo runs Win XP Pro.When I shut the lid of my laptop it automatically hibernates and when I bring that lid back up it automatically wakes up.But the problem is every time the laptop wakes up like this there is no sound and I had to restart all over again

Apparently it is a bug and there are a couple of ways to fix it. 1) Disable/Enable your sound device 2)Uncheck “allow this device to bring the computer out of standby” from the Power Management properties of your software modem 3)If you don't use your software modem at all just disable it.

All I got from http://www.paralaptop.com/lenovo-laptops/fix-sound-bux-on-thinkpad-t61-after-standby.html but just blogged it on my own blog in case I forget this 6 months from now and start looking for it all over again.

The link never explained what exactly the bug was but made it sound it was some sort of a hardware conflict.I did not find any other material about it on the inter tubes.

A couple of useful google map mashups

http://www.propertymaps.com/maps/polylines.php - You give the zip code and it shows the county and also some information (select zip in the return drop down ) about the county for example average income in that county, average house value etc etc.

http://maps.huge.info/zip.htm - This one is really useful.If you are planning a road trip and want to halt somewhere en route.Google maps gives you the route and the map but with this mashup you can get the zip codes along the route as well which will help you in finding hotels easily.

Started on a book on Algorithms

I started reading "Introduction to Algorithms" by Cormen, Leiserson,Rivest and Stein. I was not able to get much past the first chapter which basically talked about how Algorithms in itself was a technology and all the real world places where Algorithms are used.

I did start off with chapter 2 which explains insertion sort and merge sort and does their analysis but my rusty rusty brain got tired very quickly and it is bed time.

Theorizing

I love theorizing.Since I spent so much time day dreaming, my mind tries to generalize everything into a pattern.I thought I can as well write about it here.It's just for fun.period.

So we can start off with "The purpose and meaning of life" , my favorite topic which I use for procrastinating just about anything.

I started off by reading about religion , specifically Hindu philosophy and a little bit of Buddhism and a teeny weeny bit of Christianity.Initially it used to fascinate me - with concepts of Soul, After life, Good vs Evil, Karma, Enlightenment,Existence of a fair and kind God etc.

The problem for me is that I cannot live on blind faith. Eventually Religion to me started seeming like mankind's need to explain the things he perceives around him in a naive way.Religion is just a Pre-Science version of explaining reality.

Religion which opposed Darwin's theory of evolution for so long actually fits into the very theory quite beautifully.Think about it - the gist of almost all religion is "Love" and "Peace", now with "Love" and "Peace" you have a good chance of surviving and passing on your genes into the future.So people are religious because it helps them propogate their species better

Have you seen typically what happens to people who interpret religion to be "Hate" and "War" ? - they get bombed the shit out of them by B52 stealth bombers these days - now unless your God actually strikes down those B52 bombers with a bolt of divine lightening - hard luck not getting your ass roasted :-).Obviously good religion with all that Love and Peace is good for species propagation and bad religion which encourages Hate and War will be weaned out eventually.

In my opinion Life is not even as "special" as we make it out to be.By special I mean it does not need some "magical" touch by the Gods to convert an inanimate thing to something which has Life.We don't have souls to make us alive.

Life for me is just a spectacularly complex arrangement of atoms or quarks or whatever is the fundamental particles of nature which gives all the properties of life as we know it.Simple as that.We basically follow the same laws of nature as everything else which does not have life follows.

But I don't think your or my life is pre-determined in the sense that if you know the initial position and velocity of a ball you can predict where it will be 5 hours from now.I *think* we do have free will, maybe some law of nature does accommodate for free will.If you even read basics of quantum mechanics you realize that everything is a probability wave - so it is almost like the particles have a "free will" on where they want to be at a given point in time.I am not equating human free will with the probability waves - all I am trying to say is it is not "unthinkable" for accounting will within the bounds of nature's laws without resorting to the divine.

The laws and the Good Vs Evil ideas we have been brought up on where everybody is encouraged to be "good", "kind" etc are just how we have evolved since being "Good" helps propagation of the species.By being "Evil" you will eventually end up killing each other, so "Evil" people get weaned out of the society. The corollary of this statement is even if you are the most dastardly cold blooded serial killer and rapist and if you can get away with your crime - you actually won't burn in hell.You are a criminal only if you are caught.Sad but true.

So with this definition of life what do you think could be the purpose of your life.My thoughts are you really don't have any "special" purpose.You are just dust of the universe in a particular form and hence you have "life" and when you die that dust just changes into other form ( for example if you are buried the form is called compost and if you are burned it is ash :-) ).

The only purpose is what you give it.Whatever you like to do is the purpose of your life.

I ,for one find life to be the most amazing thing around, not because we have divinity within us but because it is unbelievably amazing that the dust around you, if given a particular form can ACTUALLY figure itself out and it's surrounding!Tell me what can be more amazing than that!

So that's my theory.Maybe I am wrong and the Gods will strike me down for this blasphemous blog - but even if God does exist I am sure he is smart enough to understand the need of his creation to question it's creator :-)