hadoop版本:1.0.3
问题描述:
在研究DataNode启动代码的时候遇到这么一个问题,通过Hadoop工具类ReflectionUtils反射生成NullInstance实例时,NullInstance父类Configured的成员变量conf未正确赋值,由于未正确赋值,在随后的代码中会抛出空指针异常。
问题分析:
Hadoop在ObjectWritable对象的反序列化过程中,对于非基本类型都会通过反射生成对应的类实例,在这个过程中还会根据类是不是接口Configurable的实现类,进行成员变量conf的赋值操作。相关代码如下:
ObjectWritable:
public static Object readObject(DataInput in, ObjectWritable objectWritable, Configuration conf)
throws IOException {
...
Writable writable = WritableFactories.newInstance(instanceClass, conf);
writable.readFields(in);
instance = writable;
...
}
WritableFactories:
public static Writable newInstance(Class<? extends Writable> c, Configuration conf) {
WritableFactory factory = WritableFactories.getFactory(c);
if (factory != null) {
Writable result = factory.newInstance();
if (result instanceof Configurable) {
((Configurable) result).setConf(conf);
}
return result;
} else {
return ReflectionUtils.newInstance(c, conf);
}
}
ReflectionUtils:
public static <T> T newInstance(Class<T> theClass, Configuration conf) {
T result;
try {
Constructor<T> meth = (Constructor<T>) CONSTRUCTOR_CACHE
.get(theClass);
if (meth == null) {
meth = theClass.getDeclaredConstructor(EMPTY_ARRAY);
meth.setAccessible(true);
CONSTRUCTOR_CACHE.put(theClass, meth);
}
result = meth.newInstance();
} catch (Exception e) {
throw new RuntimeException(e);
}
setConf(result, conf);
return result;
}
public static void setConf(Object theObject, Configuration conf) {
if (conf != null) {
if (theObject instanceof Configurable) {
((Configurable) theObject).setConf(conf);
}
setJobConf(theObject, conf);
}
}
回到NullInstance的类定义,由于继承了类Configured,意味着NullInstance需要在实例化的时候,初始化conf对象。如果conf对象未被正确初始化,可能会出现程序异常,如空指针错误,conf为空,在执行下面代码的时候,会抛空指针异常:
NullInstance:
public void readFields(DataInput in) throws IOException {
String className = Text.readString(in);
declaredClass = PRIMITIVE_NAMES.get(className);
if (declaredClass == null) {
try {
//此处调用getConf()可能拿到null
declaredClass = getConf().getClassByName(className);
} catch (ClassNotFoundException e) {
throw new RuntimeException(e.toString());
}
}
}
问题解决:
在ReflectionUtils的方法setConf中增加代码:
if(Configured.class.isAssignableFrom(theObject.getClass())){
((Configured)theObject).setConf(conf);
}