Hadoop Debugging Notes

Today, when running a mapreduce job on 4 Amazon m1.xlarge nodes. Got the fallowing error.

java.io.IOException: Cannot run program "bash": java.io.IOException: error=12, Cannot allocate memory

java.lang.Throwable: Child Error                                                                                                                                                
        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:532)    



As amazon instance doesn't provide any swap space which means all the process are stay fully in memory all the time. It's very easy to cause memory problems. TO solve the above errors. Simply add swap space in your amazon instance.

Another course could be overload nodes, it's very important to have a proper task numbers, overloaded nodes may lead to memory problems too.