文档章节

一次zipkin接口down问题的排查

强子哥哥
 强子哥哥
发布于 2017/09/07 09:59
字数 2304
阅读 716
收藏 0
点赞 0
评论 0

线上的zipkin老是出问题,之前做的人已经离职了,所以我来看看咋回事。

---------------------------------------

1)登录机器,取出jar包

drwxr-xr-x  2 root root     4096 Sep  1 13:48 .
drwxr-xr-x 13 root root     4096 Jul  4 10:21 ..
-rw-r--r--  1 root root      841 May 31 17:31 dependency-output
-rw-r--r--  1 root root      523 Mar 16 15:49 docker-compose.yml
-rw-r--r--  1 root root    15514 Sep  5 10:39 output
-rwxrwxrwx  1 root root      209 Feb  3  2017 start-dependency.sh
-rwxr-xr-x  1 root root      414 Sep  1 13:48 start.sh
-rw-r--r--  1 root root 89911534 Feb  2  2017 zipkin-dependencies.jar
-rw-r--r--  1 root root 69280615 Nov 30  2016 zipkin.jar

可以看到有2个jar包,确定是zipkin.jar

上传到我自己的机器上,启动一把,看看web-url有哪些映射存在!

java -jar zipkin.jar

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v1.4.1.RELEASE)

2017-09-07 09:56:00.551  INFO 15002 --- [           main] zipkin.server.ZipkinServer               : Starting ZipkinServer on iZuf6aaey7a7ooq2nczewcZ with PID 15002 (/root/zipkin/zipkin.jar started by root in /root/zipkin)
2017-09-07 09:56:00.561  INFO 15002 --- [           main] zipkin.server.ZipkinServer               : The following profiles are active: shared
2017-09-07 09:56:00.741  INFO 15002 --- [           main] ationConfigEmbeddedWebApplicationContext : Refreshing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@7c16905e: startup date [Thu Sep 07 09:56:00 CST 2017]; root of context hierarchy
2017-09-07 09:56:03.769  INFO 15002 --- [           main] o.s.b.f.s.DefaultListableBeanFactory     : Overriding bean definition for bean 'characterEncodingFilter' with a different definition: replacing [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=org.springframework.boot.autoconfigure.web.HttpEncodingAutoConfiguration; factoryMethodName=characterEncodingFilter; initMethodName=null; destroyMethodName=(inferred); defined in class path resource [org/springframework/boot/autoconfigure/web/HttpEncodingAutoConfiguration.class]] with [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=zipkin.autoconfigure.ui.ZipkinUiAutoConfiguration; factoryMethodName=characterEncodingFilter; initMethodName=null; destroyMethodName=(inferred); defined in class path resource [zipkin/autoconfigure/ui/ZipkinUiAutoConfiguration.class]]
2017-09-07 09:56:04.290  INFO 15002 --- [           main] f.a.AutowiredAnnotationBeanPostProcessor : JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
2017-09-07 09:56:05.500  INFO 15002 --- [           main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat initialized with port(s): 9411 (http)
2017-09-07 09:56:05.794  INFO 15002 --- [           main] o.apache.catalina.core.StandardService   : Starting service Tomcat
2017-09-07 09:56:05.796  INFO 15002 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet Engine: Apache Tomcat/8.5.5
2017-09-07 09:56:06.077  INFO 15002 --- [ost-startStop-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2017-09-07 09:56:06.081  INFO 15002 --- [ost-startStop-1] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 5349 ms
2017-09-07 09:56:06.728  INFO 15002 --- [ost-startStop-1] o.s.b.w.servlet.ServletRegistrationBean  : Mapping servlet: 'dispatcherServlet' to [/]
2017-09-07 09:56:06.738  INFO 15002 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'metricsFilter' to: [/*]
2017-09-07 09:56:06.739  INFO 15002 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'hiddenHttpMethodFilter' to: [/*]
2017-09-07 09:56:06.739  INFO 15002 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'httpPutFormContentFilter' to: [/*]
2017-09-07 09:56:06.739  INFO 15002 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'requestContextFilter' to: [/*]
2017-09-07 09:56:06.739  INFO 15002 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'webRequestLoggingFilter' to: [/*]
2017-09-07 09:56:06.740  INFO 15002 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'applicationContextIdFilter' to: [/*]
2017-09-07 09:56:06.740  INFO 15002 --- [ost-startStop-1] o.s.b.w.servlet.FilterRegistrationBean   : Mapping filter: 'characterEncodingFilter' to: [/*]
2017-09-07 09:56:07.721  INFO 15002 --- [           main] s.w.s.m.m.a.RequestMappingHandlerAdapter : Looking for @ControllerAdvice: org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@7c16905e: startup date [Thu Sep 07 09:56:00 CST 2017]; root of context hierarchy
2017-09-07 09:56:07.904  INFO 15002 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/api/v1/traces],methods=[GET],produces=[application/json]}" onto public java.lang.String zipkin.server.ZipkinQueryApiV1.getTraces(java.lang.String,java.lang.String,java.lang.String,java.lang.Long,java.lang.Long,java.lang.Long,java.lang.Long,java.lang.Integer)
2017-09-07 09:56:07.910  INFO 15002 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/api/v1/trace/{traceIdHex}],methods=[GET],produces=[application/json]}" onto public java.lang.String zipkin.server.ZipkinQueryApiV1.getTrace(java.lang.String,org.springframework.web.context.request.WebRequest)
2017-09-07 09:56:07.911  INFO 15002 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/api/v1/dependencies],methods=[GET],produces=[application/json]}" onto public byte[] zipkin.server.ZipkinQueryApiV1.getDependencies(long,java.lang.Long)
2017-09-07 09:56:07.913  INFO 15002 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/api/v1/services],methods=[GET]}" onto public org.springframework.http.ResponseEntity<java.util.List<java.lang.String>> zipkin.server.ZipkinQueryApiV1.getServiceNames()
2017-09-07 09:56:07.913  INFO 15002 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/api/v1/spans],methods=[GET]}" onto public org.springframework.http.ResponseEntity<java.util.List<java.lang.String>> zipkin.server.ZipkinQueryApiV1.getSpanNames(java.lang.String)
2017-09-07 09:56:07.916  INFO 15002 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/api/v1/spans],methods=[POST]}" onto public org.springframework.util.concurrent.ListenableFuture<org.springframework.http.ResponseEntity<?>> zipkin.server.ZipkinHttpCollector.uploadSpansJson(java.lang.String,byte[])
2017-09-07 09:56:07.920  INFO 15002 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/api/v1/spans],methods=[POST],consumes=[application/x-thrift]}" onto public org.springframework.util.concurrent.ListenableFuture<org.springframework.http.ResponseEntity<?>> zipkin.server.ZipkinHttpCollector.uploadSpansThrift(java.lang.String,byte[])
2017-09-07 09:56:07.924  INFO 15002 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error],produces=[text/html]}" onto public org.springframework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)
2017-09-07 09:56:07.928  INFO 15002 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/error]}" onto public org.springframework.http.ResponseEntity<java.util.Map<java.lang.String, java.lang.Object>> org.springframework.boot.autoconfigure.web.BasicErrorController.error(javax.servlet.http.HttpServletRequest)
2017-09-07 09:56:07.931  INFO 15002 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/prometheus],methods=[GET]}" onto public org.springframework.http.ResponseEntity<java.lang.String> zipkin.autoconfigure.metrics.PrometheusMetricsAutoConfiguration.prometheusMetrics()
2017-09-07 09:56:07.933  INFO 15002 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/config.json],methods=[GET],produces=[application/json]}" onto public org.springframework.http.ResponseEntity<zipkin.autoconfigure.ui.ZipkinUiProperties> zipkin.autoconfigure.ui.ZipkinUiAutoConfiguration.serveUiConfig()
2017-09-07 09:56:07.937  INFO 15002 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/index.html],methods=[GET]}" onto public org.springframework.http.ResponseEntity<org.springframework.core.io.Resource> zipkin.autoconfigure.ui.ZipkinUiAutoConfiguration.serveIndex()
2017-09-07 09:56:07.939  INFO 15002 --- [           main] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped "{[/ || /traces/{id} || /dependency],methods=[GET]}" onto public org.springframework.web.servlet.ModelAndView zipkin.autoconfigure.ui.ZipkinUiAutoConfiguration.forwardUiEndpoints(org.springframework.ui.ModelMap)
2017-09-07 09:56:08.023  INFO 15002 --- [           main] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped URL path [/webjars/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2017-09-07 09:56:08.027  INFO 15002 --- [           main] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2017-09-07 09:56:08.760  INFO 15002 --- [           main] com.datastax.driver.core.ClockFactory    : Using native clock to generate timestamps.
2017-09-07 09:56:09.691  INFO 15002 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/env/{name:.*}],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EnvironmentMvcEndpoint.value(java.lang.String)
2017-09-07 09:56:09.692  INFO 15002 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/env || /env.json],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2017-09-07 09:56:09.694  INFO 15002 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/mappings || /mappings.json],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2017-09-07 09:56:09.695  INFO 15002 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/autoconfig || /autoconfig.json],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2017-09-07 09:56:09.696  INFO 15002 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/health || /health.json],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.invoke(java.security.Principal)
2017-09-07 09:56:09.704  INFO 15002 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/metrics/{name:.*}],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.MetricsMvcEndpoint.value(java.lang.String)
2017-09-07 09:56:09.709  INFO 15002 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/metrics || /metrics.json],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2017-09-07 09:56:09.710  INFO 15002 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/info || /info.json],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2017-09-07 09:56:09.711  INFO 15002 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/heapdump || /heapdump.json],methods=[GET],produces=[application/octet-stream]}" onto public void org.springframework.boot.actuate.endpoint.mvc.HeapdumpMvcEndpoint.invoke(boolean,javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException,javax.servlet.ServletException
2017-09-07 09:56:09.711  INFO 15002 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/beans || /beans.json],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2017-09-07 09:56:09.715  INFO 15002 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/configprops || /configprops.json],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2017-09-07 09:56:09.719  INFO 15002 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/trace || /trace.json],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2017-09-07 09:56:09.720  INFO 15002 --- [           main] o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{[/dump || /dump.json],methods=[GET],produces=[application/json]}" onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.EndpointMvcAdapter.invoke()
2017-09-07 09:56:10.314  INFO 15002 --- [           main] c.f.nifty.core.NettyServerTransport      : started transport thrift:9410
2017-09-07 09:56:10.422  INFO 15002 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Registering beans for JMX exposure on startup
2017-09-07 09:56:10.441  INFO 15002 --- [           main] o.s.c.support.DefaultLifecycleProcessor  : Starting beans in phase 0
2017-09-07 09:56:10.700  INFO 15002 --- [           main] s.b.c.e.t.TomcatEmbeddedServletContainer : Tomcat started on port(s): 9411 (http)
2017-09-07 09:56:10.714  INFO 15002 --- [           main] zipkin.server.ZipkinServer               : Started ZipkinServer in 11.727 seconds (JVM running for 12.847)

确实有health代码的存在

2017-09-07 09:56:09.696  INFO 15002 --- [           main] 
o.s.b.a.e.mvc.EndpointHandlerMapping     : Mapped "{
[
/health 
|| 
/health.json],
produces=[application/json]}" 
onto public java.lang.Object org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.invoke(java.security.Principal)

手动触发下看看

curl http://localhost:9411/health
{"status":"UP","zipkin":{"status":"UP","InMemoryStorage":{"status":"UP"},"ScribeCollector":{"status":"UP"}},"diskSpace":{"status":"UP","total":42139451392,"free":4717076480,"threshold":10485760}}

说明是好的,确实可以访问,而且显示了是内存存储

访问另外1个接口 

 curl http://localhost:9411/health.json
{"status":"UP","zipkin":{"status":"UP","InMemoryStorage":{"status":"UP"},"ScribeCollector":{"status":"UP"}},"diskSpace":{"status":"UP","total":42139451392,"free":4717047808,"threshold":10485760}}

访问以下生产环境的看看

curl http://localhost:9411/health
{"status":"UP","zipkin":{"status":"UP","ScribeCollector":{"status":"UP"},"ElasticsearchStorage":{"status":"UP"}},"diskSpace":{"status":"UP","total":42140499968,"free":31591333888,"threshold":10485760}}

显示了是es作为存储,然后我就想,如果es有问题,是不是就不行了。

带着这个疑问,让我们去debug源码吧。

 

2)源码debug准备

上面的映射类.方法是

org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.invoke

版本1.4.1,具体源码见

http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22org.springframework.boot%22%20AND%20a%3A%22spring-boot-actuator%22

断点位置

jdb org.springframework.boot.loader.JarLauncher

jdb -DSTORAGE_TYPE=elasticsearch -DES_CLUSTER=elasticsearch -DES_HOSTS=http://127.0.0.1:9200 org.springframework.boot.loader.JarLauncher

stop in org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.invoke

stop in org.springframework.boot.actuate.endpoint.HealthEndpoint.invoke

stop in org.springframework.boot.actuate.health.CompositeHealthIndicator.health

然后触发curl 

 curl http://localhost:9411/health.json

顺利触发断点

 

3)正式debug源码

看了下代码,关键的位置在于

  private Health getHealth(Principal principal) {
    long accessTime = System.currentTimeMillis();
    if (isCacheStale(accessTime)) {
      this.lastAccess = accessTime;
      this.cached = ((HealthEndpoint)getDelegate()).invoke();
    }
    if (exposeHealthDetails(principal)) {
      return this.cached;
    }
    return Health.status(this.cached.getStatus()).build();
  }
Step completed: "thread=http-nio-9411-exec-3", org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.getHealth(), line=166 bci=17
166    			this.cached = getDelegate().invoke();

http-nio-9411-exec-3[1] step
> 
Step completed: "thread=http-nio-9411-exec-3", org.springframework.boot.actuate.endpoint.mvc.AbstractEndpointMvcAdapter.getDelegate(), line=60 bci=0
60    		return this.delegate;

http-nio-9411-exec-3[1] print this.delegate
 this.delegate = "org.springframework.boot.actuate.endpoint.HealthEndpoint@79767781"

关键在于这个HealthEndpoint.invoke()方法,我感觉,于是进去跟踪

最后找到这么1个东西

http-nio-9411-exec-3[1] print indicators
 indicators = "{zipkin=zipkin.server.ZipkinHealthIndicator@55f3c410, diskSpace=org.springframework.boot.actuate.health.DiskSpaceHealthIndicator@522b2631}"
http-nio-9411-exec-3[1] print this.indicators.entrySet()
 this.indicators.entrySet() = "[zipkin=zipkin.server.ZipkinHealthIndicator@55f3c410, diskSpace=org.springframework.boot.actuate.health.DiskSpaceHealthIndicator@522b2631]"
http-nio-9411-exec-3[1] 

那么,如果这里有es的话,应该更多一个es的,我们看这里是直接拿结果,还是说会触发一个真实的health

感觉应该会触发一个真实的health

jdb org.springframework.boot.loader.JarLauncher

jdb -DSTORAGE_TYPE=elasticsearch -DES_CLUSTER=elasticsearch -DES_HOSTS=http://127.0.0.1:9200 org.springframework.boot.loader.JarLauncher

stop in org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.invoke

stop in org.springframework.boot.actuate.endpoint.HealthEndpoint.invoke

stop in org.springframework.boot.actuate.health.CompositeHealthIndicator.health

3.1)ScribeCollector的health

stop in zipkin.server.ZipkinHealthIndicator$ComponentHealthIndicator.health

准备源码

http://search.maven.org/#search%7Cgav%7C2%7Cg%3A%22io.zipkin.java%22%20AND%20a%3A%22zipkin-server%22

准备好,继续debug

关于zipkin scriber本身的health结果如下

Breakpoint hit: "thread=http-nio-9411-exec-3", org.springframework.boot.actuate.health.CompositeHealthIndicator.health(), line=66 bci=0
66    		Map<String, Health> healths = new LinkedHashMap<String, Health>();

http-nio-9411-exec-3[1] cont
> 
Breakpoint hit: "thread=http-nio-9411-exec-3", zipkin.server.ZipkinHealthIndicator$ComponentHealthIndicator.health(), line=41 bci=0
41          Component.CheckResult result = component.check();

http-nio-9411-exec-3[1] step
> 
Step completed: "thread=http-nio-9411-exec-3", zipkin.collector.scribe.ScribeCollector.check(), line=100 bci=0
100          checkState(server.isRunning(), "server not running");

http-nio-9411-exec-3[1] print server
 server = "com.facebook.swift.service.ThriftServer@50b7df25"
http-nio-9411-exec-3[1] step
> 
Step completed: "thread=http-nio-9411-exec-3", com.facebook.swift.service.ThriftServer.isRunning(), line=286 bci=0

http-nio-9411-exec-3[1] next
> 
Step completed: "thread=http-nio-9411-exec-3", zipkin.collector.scribe.ScribeCollector.check(), line=100 bci=7
100          checkState(server.isRunning(), "server not running");

http-nio-9411-exec-3[1] step
> 
Step completed: "thread=http-nio-9411-exec-3", com.google.common.base.Preconditions.checkState(), line=172 bci=0

http-nio-9411-exec-3[1] next
> 
Step completed: "thread=http-nio-9411-exec-3", com.google.common.base.Preconditions.checkState(), line=175 bci=16

http-nio-9411-exec-3[1] next
> 
Step completed: "thread=http-nio-9411-exec-3", zipkin.collector.scribe.ScribeCollector.check(), line=103 bci=12
103        }

http-nio-9411-exec-3[1] print server.isRunning()
 server.isRunning() = true
http-nio-9411-exec-3[1] step
> 
Step completed: "thread=http-nio-9411-exec-3", zipkin.collector.scribe.ScribeCollector.check(), line=104 bci=21
104        return CheckResult.OK;

这个简单,只要server存活就可以了

 

3.2)ElasticsearchStorage的health

这个很可能就是影响我们线上的问题

  public Component.CheckResult check()
  {
    try {
      client().ensureClusterReady(this.indexNameFormatter.catchAll());
    } catch (UncheckedExecutionException e) {
      return Component.CheckResult.failed((Exception)e.getCause());
    } catch (Exception e) {
      return Component.CheckResult.failed(e);
    }
    return Component.CheckResult.OK;
  }

可以看到,这里是发送了1个HTTP请求

client().ensureClusterReady(indexNameFormatter.catchAll());

断点位于

stop in zipkin.storage.elasticsearch.http.HttpClient.ensureClusterReady

最后抓包如下

说明确实会发送一个http请求到es.

 

如果我把es停掉的话,就会报错

curl http://localhost:9411/health
{"status":"DOWN",--- 看这里!!!
"zipkin":
{"status":"DOWN","ScribeCollector":{"status":"UP"},
"ElasticsearchStorage":{"status":"DOWN","error":"java.net.ConnectException: Failed to connect to /127.0.0.1:9200"}},
"diskSpace":{"status":"UP","total":42139451392,"free":4623486976,"threshold":10485760}}

 

 

---上面解释为啥运维发现zipkin的health接口有问题,因为es有问题

 

同时,还发现zipkin报错

先来研究下这段代码的出处,然后看了下上面的,读超时

说明已经建立了连接到es,但是没有响应,这就要去看es的日志了!

 

© 著作权归作者所有

共有 人打赏支持
强子哥哥

强子哥哥

粉丝 859
博文 926
码字总数 640946
作品 8
南京
架构师
SpringCloud学习之sleuth&zipkin

一。调用链跟踪的必要性   首先我们简单来看一下下单到支付的过程,别的不多说,在业务复杂的时候往往服务会一层接一层的调用,当某一服务环节出现响应缓慢时会影响整个服务的响应速度,由...

java~nick
03/21
0
0
SpringCloud 微服务 (十六) 服务追踪 Zipkin

问题 在服务中,有一个接口,该A接口中又调用了其他服务的B、C、D接口,出现一个请求耗时大的问题,这时候并不知道该B、C、D接口中哪个接口造成的耗时量,然后比如确定C服务接口出现的耗时量大,但...

___大侠
昨天
0
0
APM入门与实战

篇幅一:APM基础篇 1、什么是APM? APM,全称:Application Performance Management ,目前市面的系统基本都是参考Google的Dapper(大规模分布式系统的跟踪系统)来做的,翻译传送门《google的...

jooben
2017/12/22
0
0
回到网易后开源APM技术选型与实战

篇幅一:APM基础篇 1、什么是APM? APM,全称:Application Performance Management ,目前市面的系统基本都是参考Google的Dapper(大规模分布式系统的跟踪系统)来做的,翻译传送门《google的...

李乐
2017/12/22
0
0
分布式跟踪调研与设计

背景 公司业务由数以百计的分布式服务沟通,每一个请求路由过来后,会经过多个业务系统并留下足迹,并产生对各种缓存或者DB的访问,但是这些分散的数据对于问题排查,或者流程优化比较有限。...

6776jkjk
2017/10/15
0
0
Spring Cloud 分布式链路跟踪 Sleuth + Zipkin + Elasticsearch【Finchley 版】

随着业务越来越复杂,系统也随之进行各种拆分,特别是随着微服务架构的兴起,看似一个简单的应用,后台可能很多服务在支撑;一个请求可能需要多个服务的调用;当请求迟缓或不可用时,无法得知...

Jreey
07/19
0
0
Twitter zipkin 分布式跟踪系统的设计与实现

概述 Twitter的zipkin是一个致力于收集Twitter所有的分布式服务的时间数据的分布式跟踪系统。它提供了收集数据,和查询数据两大服务。系统的理论模型来自于Google Dapper 论文。Dapper这篇论...

tqyin
2016/09/28
3.8K
2
各大厂分布式链路跟踪系统架构对比

随着互联网架构的扩张,分布式系统变得日趋复杂,越来越多的组件开始走向分布式化,如微服务、消息收发、分布式数据库、分布式缓存、分布式对象存储、跨域调用,这些组件共同构成了繁杂的分布...

james8888
04/19
0
0
部署Zipkin分布式性能追踪日志系统的操作记录

Zipkin是Twitter的一个开源项目,是一个致力于收集Twitter所有服务的监控数据的分布式跟踪系统,它提供了收集数据,和查询数据两大接口服务。 部署Zipkin环境的操作记录: 部署Zipkin,比较麻...

konglongaa
2017/02/27
0
0
疯狂Spring Cloud连载(29)微服务跟踪概述

本文节选自《疯狂Spring Cloud微服务架构实战》 京东购买地址:https://item.jd.com/12256011.html 当当网购买地址:http://product.dangdang.com/25201393.html Spring Cloud教学视频:htt...

杨大仙的程序空间
01/09
116
0

没有更多内容

加载失败,请刷新页面

加载更多

下一页

CVE-2013-0077 堆溢出分析

找了很久才发现这个环境比较容易搭建分析... 环境: 系统---Win XP SP3 漏洞程序:QQPlayer 3.7.892.400 出错DLL:quartz.dll 6.5.2600.5512 调试工具:x32db+gflag.exe 过程: 首先gflag设置...

Explorer0
8分钟前
1
0
python上传文件

//注意 <form action="/login/" method="post" enctype="multipart/form-data"> f=request.FILES.get('fafa') ff=open(f.name,mode='wb') for i in f.chunks(): ff.write(i) ff.close()......

南桥北木
20分钟前
0
0
CISCO VPN Client Reason 442 WIN8/10错误解决方案

http://jdkleo.iteye.com/blog/2163493 引用 http://my.oschina.net/cloudcoder/blog/220391?p={{currentPage 1}} 在使用cisco VPN 客户端登录时,产生Reason 442:Failedto enable Virtual......

chenfj_fer
24分钟前
0
0
信号量有没有容量限制?

之前一直误以为信号量初始化的时候那个初始化的值是信号量的“容量”,昨天同事指出了我的错误,最初我是不相信的,经过以下代码实践,证明了我的错误: Java版: import java.util.concurr...

锟斤拷烫烫烫
28分钟前
0
0
【RocketMQ】Message存储笔记

概述 消息中间件存储分为三种,一是保存在内存中,速度快但会因为系统宕机等因素造成消息丢失;二是保存在内存中,同时定时将消息写入DB中,好处是持久化消息,如何读写DB是MQ的瓶颈;三是内...

SaintTinyBoy
39分钟前
0
0
Android应用Context详解及源码解析

Android应用Context详解及源码解析 本文定位:优质文章收集 本文转载 1 背景 今天突然想起之前在上家公司(做TV与BOX盒子)时有好几个人问过我关于Android的Context到底是啥的问题,所以就马...

lichuangnk
今天
0
0
PostgreSQL的昨天今天和明天

PostgreSQL 是一种非常复杂的对象-关系型数据库管理系统(ORDBMS), 也是目前功能最强大,特性最丰富和最复杂的自由软件数据库系统。有些特性甚至连商业数据库都不具备。 这个起源于伯克利(...

闻术苑
今天
1
0
Mysql对自增主键ID进行重新排序

1,删除原有主键: ALTER TABLE `table_name` DROP `id`; 2,添加新主键字段: ALTER TABLE `table_name` ADD `id` MEDIUMINT( 8 ) NOT NULL FIRST; 3,设置新主键: ALTER TABLE `table_nam......

niithub
今天
0
0
福利篇:免费csdn vip账号分享

分享一个发布免费csdn vip账号的网站:啰嗦vip www.lostvip.com , 各种软件开发类的视频教程:慕课网、动脑学院、黑马各大培训机构VIP视频教程,非常不错!

在水一方发盐人
今天
1
0
Nginx+Tomcat搭建高性能负载均衡集群

一、 工具   nginx-1.8.0   apache-tomcat-6.0.33 二、 目标   实现高性能负载均衡的Tomcat集群:    三、 步骤   1、首先下载Nginx,要下载稳定版:      2、然后解压两个Tom...

码代码的小司机
今天
0
0

没有更多内容

加载失败,请刷新页面

加载更多

下一页

返回顶部
顶部