Java 自动扩容Mmap数据写入性能测试

原创
2023/04/10 14:18
阅读数 117

背景:

低配设备I/O优化,利用mmap实现日志的管理。

 

一、测试代码:

public class MmapWriter {

    private static final int BUF_SIZE = 4096;
    private  File mFile;
    private FileChannel mChannel;
    private RandomAccessFile raf;
    private long GROVE_SPACE = 1 * 1024 * 1024L;
    private MappedByteBuffer buffer;
    private long writeSize = 0L;

    public MmapWriter(String path) {  // avg = 42612619225 /100 ns
        mFile = new File(path);
        try {
            if(mFile.exists()){
                mFile.delete();
            }
            raf = new RandomAccessFile(mFile, "rws");
            mChannel = raf.getChannel();
        } catch (FileNotFoundException e) {
            e.printStackTrace();
        }
    }


    public static void main(String[] args) {
        long millis = System.currentTimeMillis();
        System.out.println("--开始写入--" );
        try {
            for (int i = 0; i < 100; i++) {
                fisTest();  //buf=8192 (33355+33123)/200   | buf=2048 41929  | buf=4096
              //   mmapTest(); //buf=8192  (26417 + 25735)/200     |buf=2048  41685 |buf = 4096 31874
            }
            //mmap 相比普通的读写,快70ms左右
        } catch (Exception e) {
            e.printStackTrace();
        }finally {
            System.out.println("--写入完成-- cost="+(System.currentTimeMillis() - millis));
        }
    }

    private static void fisTest(){
        File file = new File("/Users/dagege/Downloads/FFmpeg从入门到精通.pdf");
        FileInputStream fis = null;
        try {
            fis = new FileInputStream(file);
            File outFile = new File("FFmpeg从入门到精通.pdf");
            if(outFile.exists()){
                outFile.delete();
            }
            BufferedOutputStream bos = new BufferedOutputStream(new FileOutputStream(outFile));
            byte[] buf = new byte[BUF_SIZE];
            int len;
            while ((len = fis.read(buf,0,buf.length))!=-1){
                bos.write(buf,0,len);
            }
            bos.close();
        } catch (FileNotFoundException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }

    }

    private static void mmapTest() throws IOException {
        MmapWriter writer = new MmapWriter("FFmpeg从入门到精通.pdf");
        File file = new File("/Users/dagege/Downloads/FFmpeg从入门到精通.pdf");
        FileInputStream fis = new FileInputStream(file);
        byte[] buf = new byte[BUF_SIZE];
        int len;
        while ((len = fis.read(buf,0,buf.length))!=-1){
            writer.writeBuffer(buf,0,len);
        }
        writer.stopRecord();
    }

    public void writeBuffer(byte[] frame, int pos, int bufferSize) {
        RandomAccessFile raf = this.raf;
        if (mChannel == null || frame == null || bufferSize <= 0) {
            return;
        }
        try {

            int n = 1;
            while ((buffer == null) || (writeSize + bufferSize) >= mChannel.size()) {
                buffer = mChannel.map(FileChannel.MapMode.READ_WRITE, writeSize, GROVE_SPACE * n);
//mmap 需要扩容,不然会写入失败
                n++;
            }

            buffer.put(frame, pos, bufferSize);
            writeSize += bufferSize;

        } catch (IOException e) {
            e.printStackTrace();
        }catch (BufferOverflowException e){
            e.printStackTrace();
        }
    }

    @Override
    protected void finalize() throws Throwable {
        super.finalize();
        stopRecord();
    }

    public void stopRecord() {
        RandomAccessFile raf = this.raf;
        if (raf != null) {
            try {
                if (mChannel != null) {
                    mChannel.truncate(writeSize);
                }
                raf.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
        this.raf = null;
    }


}

结论:

buffer越大,mmap写入越明显

 // fisTest();  //buf=8192 (33355+33123)/200         | buf=2048 41929  | buf=4096 40110
//  mmapTest();  //buf=8192  (26417 + 25735)/200     |buf=2048  41685  | buf=4096 31874

 

mmap进行日志管理的优缺点:

优点:

断电保护,写入速度相对较快

缺点:

【1】需要经常扩容,如果写入前crash,容易造成空文件,造成磁盘空间浪费(当然可以改善,就是每次写入时记录写入的位置,下次从写入位置开始查找空数据)

【2】write次数过少的情况,效果并不明显,甚至会劣化

 

场景:

mmap 其实并不是所有场景都适合,而是适合长时间多频次I/O写入(多次write),如果写入频次只有1-2次,性能反而劣于普通I/O写入,主要原因是mmap的过程本身也是相当浪费资源的。

 

二、附加内容: 扩容与文件指针

2.1 为什么mmap需要扩容

主要原因是避免过多的资源浪费,也符合磁盘管理的理念。

 

2.2 文件指针如何保证不出现问题?

保证文件指针不小于0即可

 

2.3文件指针位置可以超过文件长度么?

 

private static void testFileChannel() throws IOException {

    File file = new File("/Users/dagege/Downloads/FFmpeg从入门到精通.pdf");
    RandomAccessFile raf = new RandomAccessFile(file, "rws");
    FileChannel channel = raf.getChannel();

  //  raf.setLength(startPosition);
  //  raf.seek(startPosition);//指针头超过长度,不会触发异常
    
    channel.truncate(file.length());
    channel.position(file.length()+1); //指针头超过长度,不会触发异常

    ByteBuffer byteBuffer = ByteBuffer.allocate(BUF_SIZE);
    byteBuffer.put(new byte[1024]);

    channel.write(byteBuffer);
    System.out.println(channel); //171709472
    raf.close();
}

【1】文件指针只要保证为正数、且状态正常,是可以超过文件长度的,但这种设置也非法的设置,并不会真正增大长度,反而会引发读取时EOF异常

 

展开阅读全文
加载中
点击引领话题📣 发布并加入讨论🔥
打赏
0 评论
0 收藏
0
分享
返回顶部
顶部