java.lang.StackOverflowError: null at org.elasticsearch.common.io.stream.BytesStreamOutput.writeByte(BytesStreamOutput.java:72)
报错信息:
[2021-02-24T07:00:11,057][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [vwop-es-node2] fatal error in thread [elasticsearch[vwop-es-node2][generic][T#89]], exiting
java.lang.StackOverflowError: null
at org.elasticsearch.common.io.stream.BytesStreamOutput.writeByte(BytesStreamOutput.java:72) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.common.io.stream.StreamOutput.write(StreamOutput.java:455) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.transport.CompressibleBytesOutputStream.writeByte(CompressibleBytesOutputStream.java:80) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.common.io.stream.StreamOutput.writeBoolean(StreamOutput.java:428) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.common.io.stream.StreamOutput.writeException(StreamOutput.java:829) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.ElasticsearchException.writeStackTraces(ElasticsearchException.java:743) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.common.io.stream.StreamOutput.writeException(StreamOutput.java:952) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.common.io.stream.StreamOutput.writeException(StreamOutput.java:950) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.ElasticsearchException.writeStackTraces(ElasticsearchException.java:743) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.common.io.stream.StreamOutput.writeException(StreamOutput.java:952) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.common.io.stream.StreamOutput.writeException(StreamOutput.java:950) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.ElasticsearchException.writeStackTraces(ElasticsearchException.java:743) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.common.io.stream.StreamOutput.writeException(StreamOutput.java:952) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.common.io.stream.StreamOutput.writeException(StreamOutput.java:950) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.ElasticsearchException.writeStackTraces(ElasticsearchException.java:7
现stackOverFlow的经查是es以前版本的bug,触发条件是在index很多,且分配数量超多的情况, fix链接为:
https://github.com/elastic/elasticsearch/pull/29125
解决方案:
1、先屏蔽所有有问题的index
2、将业务组所需要的index,先恢复
3、通过shell脚本串行尝试恢复不能恢复的index(4分钟超时如果没恢复 直接close)
4、 将无法恢复的index 手动恢复(两种情况:1、主分片不可用的:强制指定空索引
2、副本不可用的:先将副本数调整为0 然后再调整为1)