文档章节

Java Magic. Part 4: sun.misc.Unsafe

nifork
 nifork
发布于 2017/09/05 14:15
字数 2452
阅读 5
收藏 0
点赞 0
评论 0

Java is a safe programming language and prevents programmer from doing a lot of stupid mistakes, most of which based on memory management. But, there is a way to do such mistakes intentionally, using Unsafeclass.

This article is a quick overview of sun.misc.Unsafe public API and few interesting cases of its usage.

Unsafe instantiation

Before usage, we need to create instance of Unsafe object. There is no simple way to do it like Unsafe unsafe = new Unsafe(), because Unsafe class has private constructor. It also has static getUnsafe() method, but if you naively try to call Unsafe.getUnsafe() you, probably, get SecurityException. Using this method available only from trusted code.

public static Unsafe getUnsafe() {
    Class cc = sun.reflect.Reflection.getCallerClass(2);
    if (cc.getClassLoader() != null)
        throw new SecurityException("Unsafe");
    return theUnsafe;
}

This is how java validates if code is trusted. It is just checking that our code was loaded with primary classloader.

We can make our code “trusted”. Use option bootclasspath when running your program and specify path to system classes plus your one that will use Unsafe.

java -Xbootclasspath:/usr/jdk1.7.0/jre/lib/rt.jar:. com.mishadoff.magic.UnsafeClient

But it’s too hard.

Unsafe class contains its instance called theUnsafe, which marked as private. We can steal that variable via java reflection.

Field f = Unsafe.class.getDeclaredField("theUnsafe");
f.setAccessible(true);
Unsafe unsafe = (Unsafe) f.get(null);

Note: Ignore your IDE. For example, eclipse show error “Access restriction…” but if you run code, all works just fine. If the error is annoying, ignore errors on Unsafe usage in:

Preferences -> Java -> Compiler -> Errors/Warnings ->
Deprecated and restricted API -> Forbidden reference -> Warning

Unsafe API

Class sun.misc.Unsafe consists of 105 methods. There are, actually, few groups of important methods for manipulating with various entities. Here is some of them:

  • Info. Just returns some low-level memory information.
    • addressSize
    • pageSize
  • Objects. Provides methods for object and its fields manipulation.
    • allocateInstance
    • objectFieldOffset
  • Classes. Provides methods for classes and static fields manipulation.
    • staticFieldOffset
    • defineClass
    • defineAnonymousClass
    • ensureClassInitialized
  • Arrays. Arrays manipulation.
    • arrayBaseOffset
    • arrayIndexScale
  • Synchronization. Low level primitives for synchronization.
    • monitorEnter
    • tryMonitorEnter
    • monitorExit
    • compareAndSwapInt
    • putOrderedInt
  • Memory. Direct memory access methods.
    • allocateMemory
    • copyMemory
    • freeMemory
    • getAddress
    • getInt
    • putInt

Interesting use cases

Avoid initialization

allocateInstance method can be useful when you need to skip object initialization phase or bypass security checks in constructor or you want instance of that class but don’t have any public constructor. Consider following class:

class A {
    private long a; // not initialized value

    public A() {
        this.a = 1; // initialization
    }

    public long a() { return this.a; }
}

Instantiating it using constructor, reflection and unsafe gives different results.

A o1 = new A(); // constructor
o1.a(); // prints 1

A o2 = A.class.newInstance(); // reflection
o2.a(); // prints 1

A o3 = (A) unsafe.allocateInstance(A.class); // unsafe
o3.a(); // prints 0

Just think what happens to all your Singletons.

Memory corruption

This one is usual for every C programmer. By the way, its common technique for security bypass.

Consider some simple class that check access rules:

class Guard {
    private int ACCESS_ALLOWED = 1;

    public boolean giveAccess() {
        return 42 == ACCESS_ALLOWED;
    }
}

The client code is very secure and calls giveAccess() to check access rules. Unfortunately, for clients, it always returns false. Only privileged users somehow can change value of ACCESS_ALLOWED constant and get access.

In fact, it’s not true. Here is the code demostrates it:

Guard guard = new Guard();
guard.giveAccess();   // false, no access

// bypass
Unsafe unsafe = getUnsafe();
Field f = guard.getClass().getDeclaredField("ACCESS_ALLOWED");
unsafe.putInt(guard, unsafe.objectFieldOffset(f), 42); // memory corruption

guard.giveAccess(); // true, access granted

Now all clients will get unlimited access.

Actually, the same functionality can be achieved by reflection. But interesting, that we can modify any object, even ones that we do not have references to.

For example, there is another Guard object in memory located next to current guard object. We can modify its ACCESS_ALLOWED field with the following code

unsafe.putInt(guard, 16 + unsafe.objectFieldOffset(f), 42); // memory corruption

Note, we didn’t use any reference to this object. 16 is size of Guard object in 32 bit architecture. We can calculate it manually or use sizeOf method, that defined… right now.

sizeOf

Using objectFieldOffset method we can implement C-style sizeof function. This implementation returns shallow size of object:

public static long sizeOf(Object o) {
    Unsafe u = getUnsafe();
    HashSet<Field> fields = new HashSet<Field>();
    Class c = o.getClass();
    while (c != Object.class) {
        for (Field f : c.getDeclaredFields()) {
            if ((f.getModifiers() & Modifier.STATIC) == 0) {
                fields.add(f);
            }
        }
        c = c.getSuperclass();
    }

    // get offset
    long maxSize = 0;
    for (Field f : fields) {
        long offset = u.objectFieldOffset(f);
        if (offset > maxSize) {
            maxSize = offset;
        }
    }

    return ((maxSize/8) + 1) * 8;   // padding
}

Algorithm is the following: go through all non-static fields including all superclases, get offset for each field, find maximum and add padding. Probably, I missed something, but idea is clear.

Much simpler sizeOf can be achieved if we just read size value from the class struct for this object, which located with offset 12 in JVM 1.7 32 bit.

public static long sizeOf(Object object){
    return getUnsafe().getAddress(
        normalize(getUnsafe().getInt(object, 4L)) + 12L);
}

normalize is a method for casting signed int to unsigned long, for correct address usage.

private static long normalize(int value) {
    if(value >= 0) return value;
    return (~0L >>> 32) & value;
}

Awesome, this method returns the same result as our previous sizeof function.

In fact, for good, safe and accurate sizeof function better to use java.lang.instrument package, but it requires specifyng agent option in your JVM.

Shallow copy

Having implementation of calculating shallow object size, we can simply add function that copy objects. Standard solution need modify your code with Cloneable, or you can implement custom copy function in your object, but it won’t be multipurpose function.

Shallow copy:

static Object shallowCopy(Object obj) {
    long size = sizeOf(obj);
    long start = toAddress(obj);
    long address = getUnsafe().allocateMemory(size);
    getUnsafe().copyMemory(start, address, size);
    return fromAddress(address);
}

toAddress and fromAddress convert object to its address in memory and vice versa.

static long toAddress(Object obj) {
    Object[] array = new Object[] {obj};
    long baseOffset = getUnsafe().arrayBaseOffset(Object[].class);
    return normalize(getUnsafe().getInt(array, baseOffset));
}

static Object fromAddress(long address) {
    Object[] array = new Object[] {null};
    long baseOffset = getUnsafe().arrayBaseOffset(Object[].class);
    getUnsafe().putLong(array, baseOffset, address);
    return array[0];
}

This copy function can be used to copy object of any type, its size will be calculated dynamically. Note that after copying you need to cast object to specific type.

Hide Password

One more interesting usage of direct memory access in Unsafe is removing unwanted objects from memory.

Most of the APIs for retrieving user’s password, have signature as byte[] or char[]. Why arrays?

It is completely for security reason, because we can nullify array elements after we don’t need them. If we retrieve password as String it can be saved like an object in memory and nullifying that string just perform dereference operation. This object still in memory by the time GC decide to perform cleanup.

This trick creates fake String object with the same size and replace original one in memory:

String password = new String("l00k@myHor$e");
String fake = new String(password.replaceAll(".", "?"));
System.out.println(password); // l00k@myHor$e
System.out.println(fake); // ????????????

getUnsafe().copyMemory(
          fake, 0L, null, toAddress(password), sizeOf(password));

System.out.println(password); // ????????????
System.out.println(fake); // ????????????

Feel safe.

UPDATE: That way is not really safe. For real safety we need to nullify backed char array via reflection:

Field stringValue = String.class.getDeclaredField("value");
stringValue.setAccessible(true);
char[] mem = (char[]) stringValue.get(password);
for (int i=0; i < mem.length; i++) {
  mem[i] = '?';
}

Thanks to Peter Verhas for pointing out that.

Multiple Inheritance

There is no multiple inheritance in java.

Correct, except we can cast every type to every another one, if we want.

long intClassAddress = normalize(getUnsafe().getInt(new Integer(0), 4L));
long strClassAddress = normalize(getUnsafe().getInt("", 4L));
getUnsafe().putAddress(intClassAddress + 36, strClassAddress);

This snippet adds String class to Integer superclasses, so we can cast without runtime exception.

(String) (Object) (new Integer(666))

One problem that we must do it with pre-casting to object. To cheat compiler.

Dynamic classes

We can create classes in runtime, for example from compiled .class file. To perform that read class contents to byte array and pass it properly to defineClass method.

byte[] classContents = getClassContent();
Class c = getUnsafe().defineClass(
              null, classContents, 0, classContents.length);
    c.getMethod("a").invoke(c.newInstance(), null); // 1

And reading from file defined as:

private static byte[] getClassContent() throws Exception {
    File f = new File("/home/mishadoff/tmp/A.class");
    FileInputStream input = new FileInputStream(f);
    byte[] content = new byte[(int)f.length()];
    input.read(content);
    input.close();
    return content;
}

This can be useful, when you must create classes dynamically, some proxies or aspects for existing code.

Throw an Exception

Don’t like checked exceptions? Not a problem.

getUnsafe().throwException(new IOException());

This method throws checked exception, but your code not forced to catch or rethrow it. Just like runtime exception.

Fast Serialization

This one is more practical.

Everyone knows that standard java Serializable capability to perform serialization is very slow. It also require class to have public non-argument constructor.

Externalizable is better, but it needs to define schema for class to be serialized.

Popular high-performance libraries, like kryo have dependencies, which can be unacceptable with low-memory requirements.

But full serialization cycle can be easily achieved with unsafe class.

Serialization:

  • Build schema for object using reflection. It can be done once for class.
  • Use Unsafe methods getLonggetIntgetObject, etc. to retrieve actual field values.
  • Add class identifier to have capability restore this object.
  • Write them to the file or any output.

You can also add compression to save space.

Deserialization:

  • Create instance of serialized class. allocateInstance helps, because does not require any constructor.
  • Build schema. The same as 1 step in serialization.
  • Read all fields from file or any input.
  • Use Unsafe methods putLongputIntputObject, etc. to fill the object.

Actually, there are much more details in correct inplementation, but intuition is clear.

This serialization will be really fast.

By the way, there are some attempts in kryo to use Unsafe http://code.google.com/p/kryo/issues/detail?id=75

Big Arrays

As you know Integer.MAX_VALUE constant is a max size of java array. Using direct memory allocation we can create arrays with size limited by only heap size.

Here is SuperArray implementation:

class SuperArray {
    private final static int BYTE = 1;

    private long size;
    private long address;

    public SuperArray(long size) {
        this.size = size;
        address = getUnsafe().allocateMemory(size * BYTE);
    }

    public void set(long i, byte value) {
        getUnsafe().putByte(address + i * BYTE, value);
    }

    public int get(long idx) {
        return getUnsafe().getByte(address + idx * BYTE);
    }

    public long size() {
        return size;
    }
}

And sample usage:

long SUPER_SIZE = (long)Integer.MAX_VALUE * 2;
SuperArray array = new SuperArray(SUPER_SIZE);
System.out.println("Array size:" + array.size()); // 4294967294
for (int i = 0; i < 100; i++) {
    array.set((long)Integer.MAX_VALUE + i, (byte)3);
    sum += array.get((long)Integer.MAX_VALUE + i);
}
System.out.println("Sum of 100 elements:" + sum);  // 300

In fact, this technique uses off-heap memory and partially available in java.nio package.

Memory allocated this way not located in the heap and not under GC management, so take care of it using Unsafe.freeMemory(). It also does not perform any boundary checks, so any illegal access may cause JVM crash.

It can be useful for math computations, where code can operate with large arrays of data. Also, it can be interesting for realtime programmers, where GC delays on large arrays can break the limits.

Concurrency

And few words about concurrency with UnsafecompareAndSwap methods are atomic and can be used to implement high-performance lock-free data structures.

For example, consider the problem to increment value in the shared object using lot of threads.

First we define simple interface Counter:

interface Counter {
    void increment();
    long getCounter();
}

Then we define worker thread CounterClient, that uses Counter:

class CounterClient implements Runnable {
    private Counter c;
    private int num;

    public CounterClient(Counter c, int num) {
        this.c = c;
        this.num = num;
    }

    @Override
    public void run() {
        for (int i = 0; i < num; i++) {
            c.increment();
        }
    }
}

And this is testing code:

int NUM_OF_THREADS = 1000;
int NUM_OF_INCREMENTS = 100000;
ExecutorService service = Executors.newFixedThreadPool(NUM_OF_THREADS);
Counter counter = ... // creating instance of specific counter
long before = System.currentTimeMillis();
for (int i = 0; i < NUM_OF_THREADS; i++) {
    service.submit(new CounterClient(counter, NUM_OF_INCREMENTS));
}
service.shutdown();
service.awaitTermination(1, TimeUnit.MINUTES);
long after = System.currentTimeMillis();
System.out.println("Counter result: " + c.getCounter());
System.out.println("Time passed in ms:" + (after - before));

First implementation is not-synchronized counter:

class StupidCounter implements Counter {
    private long counter = 0;

    @Override
    public void increment() {
        counter++;
    }

    @Override
    public long getCounter() {
        return counter;
    }
}

Output:

Counter result: 99542945
Time passed in ms: 679

Working fast, but no threads management at all, so result is inaccurate. Second attempt, add easiest java-way synchronization:

class SyncCounter implements Counter {
    private long counter = 0;

    @Override
    public synchronized void increment() {
        counter++;
    }

    @Override
    public long getCounter() {
        return counter;
    }
}

Output:

Counter result: 100000000
Time passed in ms: 10136

Radical synchronization always work. But timings is awful. Let’s try ReentrantReadWriteLock:

class LockCounter implements Counter {
    private long counter = 0;
    private WriteLock lock = new ReentrantReadWriteLock().writeLock();

    @Override
    public void increment() {
        lock.lock();
        counter++;
        lock.unlock();
    }

    @Override
    public long getCounter() {
        return counter;
    }
}

Output:

Counter result: 100000000
Time passed in ms: 8065

Still correct, and timings are better. What about atomics?

class AtomicCounter implements Counter {
    AtomicLong counter = new AtomicLong(0);

    @Override
    public void increment() {
        counter.incrementAndGet();
    }

    @Override
    public long getCounter() {
        return counter.get();
    }
}

Output:

Counter result: 100000000
Time passed in ms: 6552

AtomicCounter is even better. Finally, try Unsafe primitive compareAndSwapLong to see if it is really privilegy to use it.

class CASCounter implements Counter {
    private volatile long counter = 0;
    private Unsafe unsafe;
    private long offset;

    public CASCounter() throws Exception {
        unsafe = getUnsafe();
        offset = unsafe.objectFieldOffset(CASCounter.class.getDeclaredField("counter"));
    }

    @Override
    public void increment() {
        long before = counter;
        while (!unsafe.compareAndSwapLong(this, offset, before, before + 1)) {
            before = counter;
        }
    }

    @Override
    public long getCounter() {
        return counter;
    }

Output:

Counter result: 100000000
Time passed in ms: 6454

Hmm, seems equal to atomics. Maybe atomics use Unsafe? (YES)

In fact this example is easy enough, but it shows some power of Unsafe.

As I said, CAS primitive can be used to implement lock-free data structures. The intuition behind this is simple:

  • Have some state
  • Create a copy of it
  • Modify it
  • Perform CAS
  • Repeat if it fails

Actually, in real it is more hard than you can imagine. There are a lot of problems like ABA Problem, instructions reordering, etc.

If you really interested, you can refer to the awesome presentation about lock-free HashMap

UPDATE: Added volatile keyword to counter variable to avoid risk of infinite loop.

Kudos to Nitsan Wakart

Bonus

Documentation for park method from Unsafe class contains longest English sentence I’ve ever seen:

Block current thread, returning when a balancing unpark occurs, or a balancing unpark has already occurred, or the thread is interrupted, or, if not absolute and time is not zero, the given time nanoseconds have elapsed, or if absolute, the given deadline in milliseconds since Epoch has passed, or spuriously (i.e., returning for no “reason”). Note: This operation is in the Unsafe class only because unpark is, so it would be strange to place it elsewhere.

Conclusion

Although, Unsafe has a bunch of useful applications, never use it.

© 著作权归作者所有

共有 人打赏支持
nifork
粉丝 0
博文 27
码字总数 13098
作品 0

暂无相关文章

聊聊spring cloud的RequestRateLimiterGatewayFilter

序 本文主要研究一下spring cloud的RequestRateLimiterGatewayFilter GatewayAutoConfiguration @Configuration@ConditionalOnProperty(name = "spring.cloud.gateway.enabled", matchIfMi......

go4it ⋅ 48分钟前 ⋅ 0

Spring JavaConfig 注解

JavaConfig注解允许开发者将Bean的定义和配置放在Java类中。它是除使用XML文件定义和配置Bean外的另一种方案。 配置: 如一个Bean如果在XML文件可以这样配置: <bean id="helloBean" class="...

霍淇滨 ⋅ 55分钟前 ⋅ 0

Spring clound 组件

Spring Cloud技术应用从场景上可以分为两大类:润物无声类和独挑大梁类。 润物无声,融合在每个微服务中、依赖其它组件并为其提供服务。 Ribbon,客户端负载均衡,特性有区域亲和、重试机制。...

英雄有梦没死就别停 ⋅ 57分钟前 ⋅ 0

Confluence 6 重新获得站点备份文件

Confluence 将会创建备份,同时压缩 XML 文件后存储熬你的 <home-directory>/backups> 目录中。你需要自己访问你安装的 Confluence 服务器,并且从服务器上获得这个文件。 运行从 Confluence...

honeymose ⋅ 今天 ⋅ 0

informix的常用SQL语句

1、创建数据库 eg1. 创建不记录日志的库testdb,参考语句如下: CREATE DATABASE testdb; eg2. 创建带缓冲式的记录日志的数据库testdb(SQL语句不一定在事务之中,拥有者名字不被用于对象的解...

wangxuwei ⋅ 今天 ⋅ 0

matplotlib画图

最简单的入门是从类 MATLAB API 开始,它被设计成兼容 MATLAB 绘图函数。 from pylab import *from numpy import *x = linspace(0, 5, 10)y = x ** 2figure()plot(x, y, 'r')...

Dr_hu ⋅ 今天 ⋅ 0

RabbitMQ学习以及与Spring的集成(三)

本文介绍RabbitMQ与Spring的简单集成以及消息的发送和接收。 在RabbitMQ的Spring配置文件中,首先需要增加命名空间。 xmlns:rabbit="http://www.springframework.org/schema/rabbit" 其次是模...

onedotdot ⋅ 今天 ⋅ 0

JAVA实现仿微信红包分配规则

最近过年发红包拜年成为一种新的潮流,作为程序猿对算法的好奇远远要大于对红包的好奇,这里介绍一种自己想到的一种随机红包分配策略,还请大家多多指教。 算法介绍 一、红包金额限制 对于微...

小致dad ⋅ 今天 ⋅ 0

Python 数电表格格式化 xlutils xlwt xlrd的使用

需要安装 xlutils xlwt xlrd 格式化前 格式化后 代码 先copy读取的表格,然后按照一定的规则修改,将昵称中的学号提取出来替换昵称即可 from xlrd import open_workbookfrom xlutils.copy ...

阿豪boy ⋅ 今天 ⋅ 0

面试题:使用rand5()生成rand7()

前言 读研究生这3 年,思维与本科相比变化挺大的,这几年除了看论文、设计方案,更重要的是学会注重先思考、再实现,感觉更加成熟吧,不再像个小P孩,人年轻时总会心高气傲。有1 道面试题:给...

初雪之音 ⋅ 今天 ⋅ 0

没有更多内容

加载失败,请刷新页面

加载更多

下一页

返回顶部
顶部