一次zipkin接口down问题的排查
一次zipkin接口down问题的排查
路飞Luffy 发表于1个月前
一次zipkin接口down问题的排查
  • 发表于 1个月前
  • 阅读 53
  • 收藏 0
  • 点赞 0
  • 评论 0

腾讯云 十分钟定制你的第一个小程序>>>   

线上的zipkin老是出问题,之前做的人已经离职了,所以我来看看咋回事。

---------------------------------------

1)登录机器,取出jar包

drwxr-xr-x  2 root root     4096 Sep  1 13:48 .
drwxr-xr-x 13 root root     4096 Jul  4 10:21 ..
-rw-r--r--  1 root root      841 May 31 17:31 dependency-output
-rw-r--r--  1 root root      523 Mar 16 15:49 docker-compose.yml
-rw-r--r--  1 root root    15514 Sep  5 10:39 output
-rwxrwxrwx  1 root root      209 Feb  3  2017 start-dependency.sh
-rwxr-xr-x  1 root root      414 Sep  1 13:48 start.sh
-rw-r--r--  1 root root 89911534 Feb  2  2017 zipkin-dependencies.jar
-rw-r--r--  1 root root 69280615 Nov 30  2016 zipkin.jar

可以看到有2个jar包,确定是zipkin.jar

上传到我自己的机器上,启动一把,看看web-url有哪些映射存在!

java -jar zipkin.jar

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v1.4.1.RELEASE)

2017-09-07 09:56:00.551  INFO 15002 --- [           main] zipkin.server.ZipkinServer               : Starting ZipkinServer on iZuf6aaey7a7ooq2nczewcZ with PID 15002 (/root/zipkin/zipkin.jar started by root in /root/zipkin)
2017-09-07 09:56:00.561  INFO 15002 --- [           main] zipkin.server.ZipkinServer               : The following profiles are active: shared
2017-09-07 09:56:00.741  INFO 15002 --- [           main] ationConfigEmbeddedWebApplicationContext : Refreshing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@7c16905e: startup date [Thu Sep 07 09:56:00 CST 2017]; root of context hierarchy
2017-09-07 09:56:03.769  INFO 15002 --- [           main] o.s.b.f.s.DefaultListableBeanFactory     : Overriding bean definition for bean 'characterEncodingFilter' with a different definition: replacing [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=org.springframework.boot.autoconfigure.web.HttpEncodingAutoConfiguration; factoryMethodName=characterEncodingFilter; initMethodName=null; destroyMethodName=(inferred); defined in class path resource [org/springframework/boot/autoconfigure/web/HttpEncodingAutoConfiguration.class]] with [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=zipkin.autoconfigure.ui.ZipkinUiAutoConfiguration; factoryMethodName=characterEncodingFilter; initMethodName=null; destroyMethodName=(inferred); defined in class path resource [zipkin/autoconfigure/ui/ZipkinUiAutoConfiguration.class]]
2017-09-07 09:56:04.290  INFO 15002 --- [           main] f.a.AutowiredAnnotationBeanPostProcessor : JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
2017-09-07 09:56:05.500  INFO 15002 --- [           main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat initialized with port(s): 9411 (http)
2017-09-07 09:56:05.794  INFO 15002 --- [           main] o.apache.catalina.core.StandardService   : Starting service Tomcat
2017-09-07 09:56:05.796  INFO 15002 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet Engine: Apache Tomcat/8.5.5
2017-09-07 09:56:06.077  INFO 15002 --- [ost-startStop-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2017-09-07 09:56:06.081  INFO 15002 --- [ost-startStop-1] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 5349 ms
2017-09-07 09:56:06.728  INFO 15002 --- [ost-startStop-1] o.s.b.w.servlet.ServletRegistrationBean  : Mapping servlet: 'dispatcherServlet' to [/]
2017-09-07 09:56:06.738  INFO 15002 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'metricsFilter' to: [/*]
2017-09-07 09:56:06.739  INFO 15002 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'hiddenHttpMethodFilter' to: [/*]
2017-09-07 09:56:06.739  INFO 15002 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'httpPutFormContentFilter' to: [/*]
2017-09-07 09:56:06.739  INFO 15002 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'requestContextFilter' to: [/*]
2017-09-07 09:56:06.739  INFO 15002 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'webRequestLoggingFilter' to: [/*]
2017-09-07 09:56:06.740  INFO 15002 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'applicationContextIdFilter' to: [/*]
2017-09-07 09:56:06.740  INFO 15002 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'characterEncodingFilter' to: [/*]
2017-09-07 09:56:07.721  INFO 15002 --- [           main] s.w.s.m.m.a.RequestMappingHandlerAdapter : Looking for @ControllerAdvice: org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@7c16905e: startup date [Thu Sep 07 09:56:00 CST 2017]; root of context hierarchy
2017-09-07 09:56:07.904  INFO 15002 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/api/v1/traces],methods=[GET],produces=[application/json]}" onto public java.lang.String zipkin.server.ZipkinQueryApiV1.getTraces(java.lang.String,java.lang.String,java.lang.String,java.lang.Long,java.lang.Long,java.lang.Long,java.lang.Long,java.lang.Integer)
2017-09-07 09:56:07.910  INFO 15002 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/api/v1/trace/{traceIdHex}],methods=[GET],produces=[application/json]}" onto public java.lang.String zipkin.server.ZipkinQueryApiV1.getTrace(java.lang.String,org.springframework.web.context.request.WebRequest)
2017-09-07 09:56:07.911  INFO 15002 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/api/v1/dependencies],methods=[GET],produces=[application/json]}" onto public byte[] zipkin.server.ZipkinQueryApiV1.getDependencies(long,java.lang.Long)
2017-09-07 09:56:07.913  INFO 15002 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/api/v1/services],methods=[GET]}" onto public org.springframework.http.ResponseEntity<java.util.List<java.lang.String>> zipkin.server.ZipkinQueryApiV1.getServiceNames()
2017-09-07 09:56:07.913  INFO 15002 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/api/v1/spans],methods=[GET]}" onto public org.springframework.http.ResponseEntity<java.util.List<java.lang.String>> zipkin.server.ZipkinQueryApiV1.getSpanNames(java.lang.String)
2017-09-07 09:56:07.916  INFO 15002 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/api/v1/spans],methods=[POST]}" onto public org.springframework.util.concurrent.ListenableFuture<org.springframework.http.ResponseEntity<?>> zipkin.server.ZipkinHttpCollector.uploadSpansJson(java.lang.String,byte[])
2017-09-07 09:56:07.920  INFO 15002 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/api/v1/spans],methods=[POST],consumes=[application/x-thrift]}" onto public org.springframework.util.concurrent.ListenableFuture<org.springframework.http.ResponseEntity<?>> zipkin.server.ZipkinHttpCollector.uploadSpansThrift(java.lang.String,byte[])
2017-09-07 09:56:07.924  INFO 15002 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error],produces=[text/html]}" onto public org.springframework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)
2017-09-07 09:56:07.928  INFO 15002 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error]}" onto public org.springframework.http.ResponseEntity<java.util.Map<java.lang.String, java.lang.Object>> org.springframework.boot.autoconfigure.web.BasicErrorController.error(javax.servlet.http.HttpServletRequest)
2017-09-07 09:56:07.931  INFO 15002 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/prometheus],methods=[GET]}" onto public org.springframework.http.ResponseEntity<java.lang.String> zipkin.autoconfigure.metrics.PrometheusMetricsAutoConfiguration.prometheusMetrics()
2017-09-07 09:56:07.933  INFO 15002 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/config.json],methods=[GET],produces=[application/json]}" onto public org.springframework.http.ResponseEntity<zipkin.autoconfigure.ui.ZipkinUiProperties> zipkin.autoconfigure.ui.ZipkinUiAutoConfiguration.serveUiConfig()
2017-09-07 09:56:07.937  INFO 15002 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/index.html],methods=[GET]}" onto public org.springframework.http.ResponseEntity<org.springframework.core.io.Resource> zipkin.autoconfigure.ui.ZipkinUiAutoConfiguration.serveIndex()
2017-09-07 09:56:07.939  INFO 15002 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/ || /traces/{id} || /dependency],methods=[GET]}" onto public org.springframework.web.servlet.ModelAndView zipkin.autoconfigure.ui.ZipkinUiAutoConfiguration.forwardUiEndpoints(org.springframework.ui.ModelMap)
2017-09-07 09:56:08.023  INFO 15002 --- [           main] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped URL path [/webjars/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2017-09-07 09:56:08.027  INFO 15002 --- [           main] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2017-09-07 09:56:08.760  INFO 15002 --- [           main] com.datastax.driver.core.ClockFactory    : Using native clock to generate timestamps.
2017-09-07 09:56:09.691  INFO 15002 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/env/{name:.*}],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EnvironmentMvcEndpoint.value(java.lang.String)
2017-09-07 09:56:09.692  INFO 15002 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/env || /env.json],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2017-09-07 09:56:09.694  INFO 15002 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/mappings || /mappings.json],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2017-09-07 09:56:09.695  INFO 15002 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/autoconfig || /autoconfig.json],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2017-09-07 09:56:09.696  INFO 15002 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/health || /health.json],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.invoke(java.security.Principal)
2017-09-07 09:56:09.704  INFO 15002 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/metrics/{name:.*}],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.MetricsMvcEndpoint.value(java.lang.String)
2017-09-07 09:56:09.709  INFO 15002 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/metrics || /metrics.json],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2017-09-07 09:56:09.710  INFO 15002 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/info || /info.json],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2017-09-07 09:56:09.711  INFO 15002 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/heapdump || /heapdump.json],methods=[GET],produces=[application/octet-stream]}" onto public void org.springframework.boot.actuate.endpoint.mvc.HeapdumpMvcEndpoint.invoke(boolean,javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException,javax.servlet.ServletException
2017-09-07 09:56:09.711  INFO 15002 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/beans || /beans.json],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2017-09-07 09:56:09.715  INFO 15002 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/configprops || /configprops.json],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2017-09-07 09:56:09.719  INFO 15002 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/trace || /trace.json],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2017-09-07 09:56:09.720  INFO 15002 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/dump || /dump.json],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2017-09-07 09:56:10.314  INFO 15002 --- [           main] c.f.nifty.core.NettyServerTransport      : started transport thrift:9410
2017-09-07 09:56:10.422  INFO 15002 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Registering beans for JMX exposure on startup
2017-09-07 09:56:10.441  INFO 15002 --- [           main] o.s.c.support.DefaultLifecycleProcessor  : Starting beans in phase 0
2017-09-07 09:56:10.700  INFO 15002 --- [           main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 9411 (http)
2017-09-07 09:56:10.714  INFO 15002 --- [           main] zipkin.server.ZipkinServer               : Started ZipkinServer in 11.727 seconds (JVM running for 12.847)

确实有health代码的存在

2017-09-07 09:56:09.696  INFO 15002 --- [           main] 
o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{
[
/health 
|| 
/health.json],
produces=[application/json]}" 
onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.invoke(java.security.Principal)

手动触发下看看

curl http://localhost:9411/health
{"status":"UP","zipkin":{"status":"UP","InMemoryStorage":{"status":"UP"},"ScribeCollector":{"status":"UP"}},"diskSpace":{"status":"UP","total":42139451392,"free":4717076480,"threshold":10485760}}

说明是好的,确实可以访问,而且显示了是内存存储

访问另外1个接口 

 curl http://localhost:9411/health.json
{"status":"UP","zipkin":{"status":"UP","InMemoryStorage":{"status":"UP"},"ScribeCollector":{"status":"UP"}},"diskSpace":{"status":"UP","total":42139451392,"free":4717047808,"threshold":10485760}}

访问以下生产环境的看看

curl http://localhost:9411/health
{"status":"UP","zipkin":{"status":"UP","ScribeCollector":{"status":"UP"},"ElasticsearchStorage":{"status":"UP"}},"diskSpace":{"status":"UP","total":42140499968,"free":31591333888,"threshold":10485760}}

显示了是es作为存储,然后我就想,如果es有问题,是不是就不行了。

带着这个疑问,让我们去debug源码吧。

 

2)源码debug准备

上面的映射类.方法是

org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.invoke

版本1.4.1,具体源码见

http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22org.springframework.boot%22%20AND%20a%3A%22spring-boot-actuator%22

断点位置

jdb org.springframework.boot.loader.JarLauncher

jdb -DSTORAGE_TYPE=elasticsearch -DES_CLUSTER=elasticsearch -DES_HOSTS=http://127.0.0.1:9200 org.springframework.boot.loader.JarLauncher

stop in org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.invoke

stop in org.springframework.boot.actuate.endpoint.HealthEndpoint.invoke

stop in org.springframework.boot.actuate.health.CompositeHealthIndicator.health

然后触发curl 

 curl http://localhost:9411/health.json

顺利触发断点

 

3)正式debug源码

看了下代码,关键的位置在于

  private Health getHealth(Principal principal) {
    long accessTime = System.currentTimeMillis();
    if (isCacheStale(accessTime)) {
      this.lastAccess = accessTime;
      this.cached = ((HealthEndpoint)getDelegate()).invoke();
    }
    if (exposeHealthDetails(principal)) {
      return this.cached;
    }
    return Health.status(this.cached.getStatus()).build();
  }
Step completed: "thread=http-nio-9411-exec-3", org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.getHealth(), line=166 bci=17
166    			this.cached = getDelegate().invoke();

http-nio-9411-exec-3[1] step
> 
Step completed: "thread=http-nio-9411-exec-3", org.springframework.boot.actuate.endpoint.mvc.AbstractEndpointMvcAdapter.getDelegate(), line=60 bci=0
60    		return this.delegate;

http-nio-9411-exec-3[1] print this.delegate
 this.delegate = "org.springframework.boot.actuate.endpoint.HealthEndpoint@79767781"

关键在于这个HealthEndpoint.invoke()方法,我感觉,于是进去跟踪

最后找到这么1个东西

http-nio-9411-exec-3[1] print indicators
 indicators = "{zipkin=zipkin.server.ZipkinHealthIndicator@55f3c410, diskSpace=org.springframework.boot.actuate.health.DiskSpaceHealthIndicator@522b2631}"
http-nio-9411-exec-3[1] print this.indicators.entrySet()
 this.indicators.entrySet() = "[zipkin=zipkin.server.ZipkinHealthIndicator@55f3c410, diskSpace=org.springframework.boot.actuate.health.DiskSpaceHealthIndicator@522b2631]"
http-nio-9411-exec-3[1] 

那么,如果这里有es的话,应该更多一个es的,我们看这里是直接拿结果,还是说会触发一个真实的health

感觉应该会触发一个真实的health

jdb org.springframework.boot.loader.JarLauncher

jdb -DSTORAGE_TYPE=elasticsearch -DES_CLUSTER=elasticsearch -DES_HOSTS=http://127.0.0.1:9200 org.springframework.boot.loader.JarLauncher

stop in org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.invoke

stop in org.springframework.boot.actuate.endpoint.HealthEndpoint.invoke

stop in org.springframework.boot.actuate.health.CompositeHealthIndicator.health

3.1)ScribeCollector的health

stop in zipkin.server.ZipkinHealthIndicator$ComponentHealthIndicator.health

准备源码

http://search.maven.org/#search%7Cgav%7C2%7Cg%3A%22io.zipkin.java%22%20AND%20a%3A%22zipkin-server%22

准备好,继续debug

关于zipkin scriber本身的health结果如下

Breakpoint hit: "thread=http-nio-9411-exec-3", org.springframework.boot.actuate.health.CompositeHealthIndicator.health(), line=66 bci=0
66    		Map<String, Health> healths = new LinkedHashMap<String, Health>();

http-nio-9411-exec-3[1] cont
> 
Breakpoint hit: "thread=http-nio-9411-exec-3", zipkin.server.ZipkinHealthIndicator$ComponentHealthIndicator.health(), line=41 bci=0
41          Component.CheckResult result = component.check();

http-nio-9411-exec-3[1] step
> 
Step completed: "thread=http-nio-9411-exec-3", zipkin.collector.scribe.ScribeCollector.check(), line=100 bci=0
100          checkState(server.isRunning(), "server not running");

http-nio-9411-exec-3[1] print server
 server = "com.facebook.swift.service.ThriftServer@50b7df25"
http-nio-9411-exec-3[1] step
> 
Step completed: "thread=http-nio-9411-exec-3", com.facebook.swift.service.ThriftServer.isRunning(), line=286 bci=0

http-nio-9411-exec-3[1] next
> 
Step completed: "thread=http-nio-9411-exec-3", zipkin.collector.scribe.ScribeCollector.check(), line=100 bci=7
100          checkState(server.isRunning(), "server not running");

http-nio-9411-exec-3[1] step
> 
Step completed: "thread=http-nio-9411-exec-3", com.google.common.base.Preconditions.checkState(), line=172 bci=0

http-nio-9411-exec-3[1] next
> 
Step completed: "thread=http-nio-9411-exec-3", com.google.common.base.Preconditions.checkState(), line=175 bci=16

http-nio-9411-exec-3[1] next
> 
Step completed: "thread=http-nio-9411-exec-3", zipkin.collector.scribe.ScribeCollector.check(), line=103 bci=12
103        }

http-nio-9411-exec-3[1] print server.isRunning()
 server.isRunning() = true
http-nio-9411-exec-3[1] step
> 
Step completed: "thread=http-nio-9411-exec-3", zipkin.collector.scribe.ScribeCollector.check(), line=104 bci=21
104        return CheckResult.OK;

这个简单,只要server存活就可以了

 

3.2)ElasticsearchStorage的health

这个很可能就是影响我们线上的问题

  public Component.CheckResult check()
  {
    try {
      client().ensureClusterReady(this.indexNameFormatter.catchAll());
    } catch (UncheckedExecutionException e) {
      return Component.CheckResult.failed((Exception)e.getCause());
    } catch (Exception e) {
      return Component.CheckResult.failed(e);
    }
    return Component.CheckResult.OK;
  }

可以看到,这里是发送了1个HTTP请求

client().ensureClusterReady(indexNameFormatter.catchAll());

断点位于

stop in zipkin.storage.elasticsearch.http.HttpClient.ensureClusterReady

最后抓包如下

说明确实会发送一个http请求到es.

 

如果我把es停掉的话,就会报错

curl http://localhost:9411/health
{"status":"DOWN",--- 看这里!!!
"zipkin":
{"status":"DOWN","ScribeCollector":{"status":"UP"},
"ElasticsearchStorage":{"status":"DOWN","error":"java.net.ConnectException: Failed to connect to /127.0.0.1:9200"}},
"diskSpace":{"status":"UP","total":42139451392,"free":4623486976,"threshold":10485760}}

 

 

---上面解释为啥运维发现zipkin的health接口有问题,因为es有问题

 

同时,还发现zipkin报错

先来研究下这段代码的出处,然后看了下上面的,读超时

说明已经建立了连接到es,但是没有响应,这就要去看es的日志了!

 

标签: Zipkin
共有 人打赏支持
路飞Luffy
粉丝 811
博文 699
码字总数 697592
作品 7
评论 (0)
×
路飞Luffy
如果觉得我的文章对您有用,请随意打赏。您的支持将鼓励我继续创作!
* 金额(元)
¥1 ¥5 ¥10 ¥20 其他金额
打赏人
留言
* 支付类型
微信扫码支付
打赏金额:
已支付成功
打赏金额: