kafka Map failed Caused by: java.lang.OutOfMemoryError: Map failed

原创
2022/06/09 18:15
阅读数 776
  • 问题信息:

公司服务器,kafka启动中发现有以下错误信:

java.io.IOException: Map failed
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:938)
        at kafka.log.AbstractIndex.<init>(AbstractIndex.scala:124)
        at kafka.log.TimeIndex.<init>(TimeIndex.scala:54)
        at kafka.log.LazyIndex$.$anonfun$forTime$1(LazyIndex.scala:109)
        at kafka.log.LazyIndex.$anonfun$get$1(LazyIndex.scala:63)
        at kafka.log.LazyIndex.get(LazyIndex.scala:60)
        at kafka.log.LogSegment.timeIndex(LogSegment.scala:66)
        at kafka.log.LogSegment.resizeIndexes(LogSegment.scala:77)
        at kafka.log.LogLoader$.load(LogLoader.scala:164)
        at kafka.log.Log$.apply(Log.scala:2087)
        at kafka.log.LogManager.$anonfun$getOrCreateLog$1(LogManager.scala:861)
        at scala.Option.getOrElse(Option.scala:201)
        at kafka.log.LogManager.getOrCreateLog(LogManager.scala:813)
        at kafka.cluster.Partition.createLog(Partition.scala:349)
        at kafka.cluster.Partition.maybeCreate$1(Partition.scala:324)
        at kafka.cluster.Partition.createLogIfNotExists(Partition.scala:331)
        at kafka.cluster.Partition.$anonfun$makeLeader$1(Partition.scala:536)
        at kafka.cluster.Partition.makeLeader(Partition.scala:520)
        at kafka.server.ReplicaManager.$anonfun$makeLeaders$5(ReplicaManager.scala:1577)
        at kafka.utils.Implicits$MapExtensionMethods$.$anonfun$forKeyValue$1(Implicits.scala:62)
        at scala.collection.mutable.HashMap$Node.foreachEntry(HashMap.scala:633)
        at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:499)
        at kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:1575)
        at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:266)
        at kafka.server.KafkaApis.handle(KafkaApis.scala:173)
        at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:75)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Map failed
        at sun.nio.ch.FileChannelImpl.map0(Native Method)
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:935)
        ... 26 more
[2022-06-08 11:17:20,609] ERROR [Broker id=0] Error while processing LeaderAndIsr request correlationId 1 received from controller 0 epoch 4 for partition notify-topic-prod-inte-14 (state.change.logger)
java.io.IOException: Map failed
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:938)
        at kafka.log.AbstractIndex.<init>(AbstractIndex.scala:124)
        at kafka.log.TimeIndex.<init>(TimeIndex.scala:54)
        at kafka.log.LazyIndex$.$anonfun$forTime$1(LazyIndex.scala:109)
        at kafka.log.LazyIndex.$anonfun$get$1(LazyIndex.scala:63)
        at kafka.log.LazyIndex.get(LazyIndex.scala:60)
        at kafka.log.LogSegment.timeIndex(LogSegment.scala:66)
        at kafka.log.LogSegment.resizeIndexes(LogSegment.scala:77)
        at kafka.log.LogLoader$.load(LogLoader.scala:164)
        at kafka.log.Log$.apply(Log.scala:2087)
        at kafka.log.LogManager.$anonfun$getOrCreateLog$1(LogManager.scala:861)
        at scala.Option.getOrElse(Option.scala:201)
        at kafka.log.LogManager.getOrCreateLog(LogManager.scala:813)
        at kafka.cluster.Partition.createLog(Partition.scala:349)
        at kafka.cluster.Partition.maybeCreate$1(Partition.scala:324)
        at kafka.cluster.Partition.createLogIfNotExists(Partition.scala:331)
        at kafka.cluster.Partition.$anonfun$makeLeader$1(Partition.scala:536)
        at kafka.cluster.Partition.makeLeader(Partition.scala:520)
        at kafka.server.ReplicaManager.$anonfun$makeLeaders$5(ReplicaManager.scala:1577)
        at kafka.utils.Implicits$MapExtensionMethods$.$anonfun$forKeyValue$1(Implicits.scala:62)
        at scala.collection.mutable.HashMap$Node.foreachEntry(HashMap.scala:633)
        at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:499)
        at kafka.server.ReplicaManager.becomeLeaderOrFollower(ReplicaManager.scala:1575)
        at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:266)
        at kafka.server.KafkaApis.handle(KafkaApis.scala:173)
        at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:75)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Map failed
        at sun.nio.ch.FileChannelImpl.map0(Native Method)
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:935)
        ... 26 more

 

  • 修改方式:
  1. 修改虚拟内存使用数量
     sysctl -w vm.max_map_count=262144 #设置数量
     sysctl vm.max_map_count  #查看当前数量
  2. 修改 kafka-server-start 启动时配置 缓存
  3. 查看 jdk 版本是否为64位

以上信息为,stackoverflow 中查询的,并没有解决 我的问题

4.重装系统,由于采用的公司的虚拟机,就选择重装了一下系统,就可以了。

  • 问题原因 :

jdk在调用 c 进行内存申请时,无法申请到更多内存,就抛出异常。

我用的公司的服务器,内存还有3个G,正常来说不是不会出现这个问题。也不清楚为什么没有申请下来。

对系统函数的调用不清楚了。后续,我在查一下继续更新

mmap64  使用说明:

https://www.ibm.com/docs/en/i/7.2?topic=ssw_ibm_i_72/apis/mmap64.htm

说明介绍:

https://www.zhihu.com/question/48161206

展开阅读全文
加载中
点击加入讨论🔥(2) 发布并加入讨论🔥
2 评论
0 收藏
0
分享
返回顶部
顶部