Python 性能分析入门指南
Python 性能分析入门指南
异类深呼吸 发表于3年前
Python 性能分析入门指南
  • 发表于 3年前
  • 阅读 189
  • 收藏 0
  • 点赞 0
  • 评论 0

标题:腾讯云 新注册用户域名抢购1元起>>>   

摘要: 摘抄自http://blog.segmentfault.com/yexiaobai/1190000000616798 一篇不错的python性能分析的文章

<h1 class="post-title" style="box-sizing: border-box; font-size: 22px; margin: 22px 0px 20px; font-family: 'Droid Serif', Georgia, 'Times New Roman', Times, STHeiti, SimSun, serif; line-height: 1.125; color: #333333;"><a style="box-sizing: border-box; color: #017e66; text-decoration: none; background: transparent;" href="http://blog.segmentfault.com/yexiaobai/1190000000616798">Python 性能分析入门指南</a></h1> <div class="post-content fmt" style="box-sizing: border-box; font-size: 14px; word-wrap: break-word; color: #333333; font-family: 'Droid Serif', Georgia, 'Times New Roman', Times, STHeiti, SimSun, serif; line-height: 22.399999618530273px;"> <blockquote style="box-sizing: border-box; padding: 11px 22px; margin: 20px 0px; font-size: 1em; border-left-width: 5px; border-color: #017e66; color: #555555;"> <p style="box-sizing: border-box; margin: 0px; line-height: 1.6;">注: 本文的原作者是 <a style="box-sizing: border-box; color: #017e66; text-decoration: none; background: transparent;" href="http://www.huyng.com/" rel="nofollow" target="_blank">Huy Nguyen</a> ,原文地址为 <a style="box-sizing: border-box; color: #017e66; text-decoration: none; background: transparent;" href="http://www.huyng.com/posts/python-performance-analysis/" rel="nofollow" target="_blank">A guide to analyzing Python performance</a></p> </blockquote> <p style="box-sizing: border-box; margin: 20px 0px;">虽然并非你编写的每个 Python 程序都要求一个严格的性能分析,但是让人放心的是,当问题发生的时候,Python 生态圈有各种各样的工具可以处理这类问题。</p> <p style="box-sizing: border-box; margin: 20px 0px;">分析程序的性能可以归结为回答四个基本问题:</p> <ol style="box-sizing: border-box; margin: 20px 0px;"> <li style="box-sizing: border-box; margin: 5px 0px;">正运行的多快</li> <li style="box-sizing: border-box; margin: 5px 0px;">速度瓶颈在哪里</li> <li style="box-sizing: border-box; margin: 5px 0px;">内存使用率是多少</li> <li style="box-sizing: border-box; margin: 5px 0px;">内存泄露在哪里</li> </ol> <p style="box-sizing: border-box; margin: 20px 0px;">下面,我们将用一些神奇的工具深入到这些问题的答案中去。</p> <h1 style="box-sizing: border-box; font-size: 1.57143em; margin: 20px 0px; font-family: inherit; line-height: 1.125; color: inherit; padding-bottom: 10px; border-bottom-width: 1px; border-bottom-style: solid; border-bottom-color: #dddddd;">用 <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">time</code> 粗粒度的计算时间</h1> <p style="box-sizing: border-box; margin: 20px 0px;">让我们开始通过使用一个快速和粗暴的方法计算我们的代码:传统的 unix <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">time</code> 工具。</p> <pre class="matlab" style="box-sizing: border-box; overflow: auto; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 10px; margin: 20px 0px; line-height: 1.3; word-break: break-all; word-wrap: break-word; border: 0px; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f5f5f5;"> $ time python <span class="transposed_variable" style="box-sizing: border-box;">yourprogram.</span>py <span class="built_in" style="box-sizing: border-box; color: #0086b3;">real</span> <span class="number" style="box-sizing: border-box; color: #009999;">0</span><span class="transposed_variable" style="box-sizing: border-box;">m1.</span><span class="number" style="box-sizing: border-box; color: #009999;">028</span>s user <span class="number" style="box-sizing: border-box; color: #009999;">0</span><span class="transposed_variable" style="box-sizing: border-box;">m0.</span><span class="number" style="box-sizing: border-box; color: #009999;">001</span>s sys <span class="number" style="box-sizing: border-box; color: #009999;">0</span><span class="transposed_variable" style="box-sizing: border-box;">m0.</span><span class="number" style="box-sizing: border-box; color: #009999;">003</span>s </pre> <p style="box-sizing: border-box; margin: 20px 0px;">三个输出测量值之间的详细意义在这里 <a style="box-sizing: border-box; color: #017e66; text-decoration: none; background: transparent;" href="http://stackoverflow.com/questions/556405/what-do-real-user-and-sys-mean-in-the-output-of-time1" rel="nofollow" target="_blank">stackoverflow article</a>,但简介在这:</p> <ul style="box-sizing: border-box; margin: 20px 0px;"> <li style="box-sizing: border-box; margin: 5px 0px;">real -- 指的是实际耗时</li> <li style="box-sizing: border-box; margin: 5px 0px;">user -- 指的是内核之外的 CPU 耗时</li> <li style="box-sizing: border-box; margin: 5px 0px;">sys -- 指的是花费在内核特定函数的 CPU 耗时</li> </ul> <p style="box-sizing: border-box; margin: 20px 0px;">你会有你的应用程序用完了多少 CPU 周期的即视感,不管系统上其他运行的程序添加的系统和用户时间。</p> <p style="box-sizing: border-box; margin: 20px 0px;">如果 sys 和 user 时间之和小于 real 时间,然后你可以猜测到大多数程序的性能问题最有可能与 <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">IO wait</code> 相关。</p> <h1 style="box-sizing: border-box; font-size: 1.57143em; margin: 20px 0px; font-family: inherit; line-height: 1.125; color: inherit; padding-bottom: 10px; border-bottom-width: 1px; border-bottom-style: solid; border-bottom-color: #dddddd;">用 <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">timing context</code> 管理器细粒度的计算时间</h1> <p style="box-sizing: border-box; margin: 20px 0px;">我们下一步的技术包括直接嵌入代码来获取细粒度的计时信息。下面是我进行时间测量的代码的一个小片段</p> <p style="box-sizing: border-box; margin: 20px 0px;">timer.py</p> <pre class="python" style="box-sizing: border-box; overflow: auto; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 10px; margin: 20px 0px; line-height: 1.3; word-break: break-all; word-wrap: break-word; border: 0px; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f5f5f5;"><span class="keyword" style="box-sizing: border-box; font-weight: bold;">import</span> time

<span class="class" style="box-sizing: border-box; color: #445588; font-weight: bold;"><span class="keyword" style="box-sizing: border-box; color: #333333;">class</span> <span class="title" style="box-sizing: border-box;">Timer</span><span class="params" style="box-sizing: border-box;">(object)</span>:</span> <span class="function" style="box-sizing: border-box;"><span class="keyword" style="box-sizing: border-box; font-weight: bold;">def</span> <span class="title" style="box-sizing: border-box; color: #990000; font-weight: bold;">init</span><span class="params" style="box-sizing: border-box;">(self, verbose=False)</span>:</span> self.verbose = verbose

<span class="function" style="box-sizing: border-box;"><span class="keyword" style="box-sizing: border-box; font-weight: bold;">def</span> <span class="title" style="box-sizing: border-box; color: #990000; font-weight: bold;">__enter__</span><span class="params" style="box-sizing: border-box;">(self)</span>:</span>
    self.start = time.time()
    <span class="keyword" style="box-sizing: border-box; font-weight: bold;">return</span> self

<span class="function" style="box-sizing: border-box;"><span class="keyword" style="box-sizing: border-box; font-weight: bold;">def</span> <span class="title" style="box-sizing: border-box; color: #990000; font-weight: bold;">__exit__</span><span class="params" style="box-sizing: border-box;">(self, *args)</span>:</span>
    self.end = time.time()
    self.secs = self.end - self.start
    self.msecs = self.secs * <span class="number" style="box-sizing: border-box; color: #009999;">1000</span>  <span class="comment" style="box-sizing: border-box; color: #999988; font-style: italic;"># millisecs</span>
    <span class="keyword" style="box-sizing: border-box; font-weight: bold;">if</span> self.verbose:
        <span class="keyword" style="box-sizing: border-box; font-weight: bold;">print</span> <span class="string" style="box-sizing: border-box; color: #dd1144;">'elapsed time: %f ms'</span> % self.msecs

</pre> <p style="box-sizing: border-box; margin: 20px 0px;">为了使用它,使用 Python 的 <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">with</code> 关键字和 <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">Timer</code> 上下文管理器来包装你想计算的代码。当您的代码块开始执行,它将照顾启动计时器,当你的代码块结束的时候,它将停止计时器。</p> <p style="box-sizing: border-box; margin: 20px 0px;">这个代码片段示例:</p> <pre class="haskell" style="box-sizing: border-box; overflow: auto; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 10px; margin: 20px 0px; line-height: 1.3; word-break: break-all; word-wrap: break-word; border: 0px; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f5f5f5;"><span class="title" style="box-sizing: border-box; color: #990000; font-weight: bold;">from</span> timer <span class="import" style="box-sizing: border-box;"><span class="keyword" style="box-sizing: border-box; font-weight: bold;">import</span> Timer</span> <span class="title" style="box-sizing: border-box; color: #990000; font-weight: bold;">from</span> redis <span class="import" style="box-sizing: border-box;"><span class="keyword" style="box-sizing: border-box; font-weight: bold;">import</span> Redis</span> <span class="title" style="box-sizing: border-box; color: #990000; font-weight: bold;">rdb</span> = <span class="type" style="box-sizing: border-box;">Redis</span>()

<span class="title" style="box-sizing: border-box; color: #990000; font-weight: bold;">with</span> <span class="type" style="box-sizing: border-box;">Timer</span>() <span class="keyword" style="box-sizing: border-box; font-weight: bold;">as</span> t: rdb.lpush(<span class="string" style="box-sizing: border-box; color: #dd1144;">"foo"</span>, <span class="string" style="box-sizing: border-box; color: #dd1144;">"bar"</span>) <span class="title" style="box-sizing: border-box; color: #990000; font-weight: bold;">print</span> <span class="string" style="box-sizing: border-box; color: #dd1144;">"=> elasped lpush: %s s"</span> % t.secs

<span class="title" style="box-sizing: border-box; color: #990000; font-weight: bold;">with</span> <span class="type" style="box-sizing: border-box;">Timer</span>() <span class="keyword" style="box-sizing: border-box; font-weight: bold;">as</span> t: rdb.lpop(<span class="string" style="box-sizing: border-box; color: #dd1144;">"foo"</span>) <span class="title" style="box-sizing: border-box; color: #990000; font-weight: bold;">print</span> <span class="string" style="box-sizing: border-box; color: #dd1144;">"=> elasped lpop: %s s"</span> % t.secs </pre>

<p style="box-sizing: border-box; margin: 20px 0px;">为了看看我的程序的性能随着时间的演化的趋势,我常常记录这些定时器的输出到一个文件中。</p> <h1 style="box-sizing: border-box; font-size: 1.57143em; margin: 20px 0px; font-family: inherit; line-height: 1.125; color: inherit; padding-bottom: 10px; border-bottom-width: 1px; border-bottom-style: solid; border-bottom-color: #dddddd;">使用 <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">profiler</code> 逐行计时和分析执行的频率</h1> <p style="box-sizing: border-box; margin: 20px 0px;">罗伯特·克恩有一个不错的项目称为 <a style="box-sizing: border-box; color: #017e66; text-decoration: none; background: transparent;" href="http://packages.python.org/line_profiler/" rel="nofollow" target="_blank">line_profiler</a> , 我经常使用它来分析我的脚本有多快,以及每行代码执行的频率:</p> <p style="box-sizing: border-box; margin: 20px 0px;">为了使用它,你可以通过使用 <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">pip</code> 来安装它:</p> <pre class="nginx" style="box-sizing: border-box; overflow: auto; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 10px; margin: 20px 0px; line-height: 1.3; word-break: break-all; word-wrap: break-word; border: 0px; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f5f5f5;"><span class="title" style="box-sizing: border-box; color: #990000; font-weight: bold;">pip</span> install line_profiler </pre> <p style="box-sizing: border-box; margin: 20px 0px;">安装完成后,你将获得一个新模块称为 <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">line_profiler</code> 和 <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">kernprof.py</code> 可执行脚本。</p> <p style="box-sizing: border-box; margin: 20px 0px;">为了使用这个工具,首先在你想测量的函数上设置 <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">@profile</code> 修饰符。不用担心,为了这个修饰符,你不需要引入任何东西。<code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">kernprof.py</code> 脚本会在运行时自动注入你的脚本。</p> <p style="box-sizing: border-box; margin: 20px 0px;">primes.py</p> <pre class="xml" style="box-sizing: border-box; overflow: auto; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 10px; margin: 20px 0px; line-height: 1.3; word-break: break-all; word-wrap: break-word; border: 0px; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f5f5f5;">@profile def primes(n): if n==2: return [2] elif n<span class="tag" style="box-sizing: border-box; color: #000080;">&lt;<span class="title" style="box-sizing: border-box;">2: </span> <span class="attribute" style="box-sizing: border-box; color: teal;">return</span> [] <span class="attribute" style="box-sizing: border-box; color: teal;">s</span>=<span class="value" style="box-sizing: border-box; color: #dd1144;">range(3,n+1,2)</span> <span class="attribute" style="box-sizing: border-box; color: teal;">mroot</span> = <span class="attribute" style="box-sizing: border-box; color: teal;">n</span> ** <span class="attribute" style="box-sizing: border-box; color: teal;">0.5</span> <span class="attribute" style="box-sizing: border-box; color: teal;">half</span>=<span class="value" style="box-sizing: border-box; color: #dd1144;">(n+1)</span>/<span class="attribute" style="box-sizing: border-box; color: teal;">2-1</span> <span class="attribute" style="box-sizing: border-box; color: teal;">i</span>=<span class="value" style="box-sizing: border-box; color: #dd1144;">0</span> <span class="attribute" style="box-sizing: border-box; color: teal;">m</span>=<span class="value" style="box-sizing: border-box; color: #dd1144;">3</span> <span class="attribute" style="box-sizing: border-box; color: teal;">while</span> <span class="attribute" style="box-sizing: border-box; color: teal;">m</span> &lt;= <span class="attribute" style="box-sizing: border-box; color: teal;">mroot:</span> <span class="attribute" style="box-sizing: border-box; color: teal;">if</span> <span class="attribute" style="box-sizing: border-box; color: teal;">s</span>[<span class="attribute" style="box-sizing: border-box; color: teal;">i</span>]<span class="attribute" style="box-sizing: border-box; color: teal;">:</span> <span class="attribute" style="box-sizing: border-box; color: teal;">j</span>=<span class="value" style="box-sizing: border-box; color: #dd1144;">(m*m-3)</span>/<span class="attribute" style="box-sizing: border-box; color: teal;">2</span> <span class="attribute" style="box-sizing: border-box; color: teal;">s</span>[<span class="attribute" style="box-sizing: border-box; color: teal;">j</span>]=<span class="value" style="box-sizing: border-box; color: #dd1144;">0</span> <span class="attribute" style="box-sizing: border-box; color: teal;">while</span> <span class="attribute" style="box-sizing: border-box; color: teal;">j</span>&lt;<span class="attribute" style="box-sizing: border-box; color: teal;">half:</span> <span class="attribute" style="box-sizing: border-box; color: teal;">s</span>[<span class="attribute" style="box-sizing: border-box; color: teal;">j</span>]=<span class="value" style="box-sizing: border-box; color: #dd1144;">0</span> <span class="attribute" style="box-sizing: border-box; color: teal;">j</span>+=<span class="value" style="box-sizing: border-box; color: #dd1144;">m</span> <span class="attribute" style="box-sizing: border-box; color: teal;">i</span>=<span class="value" style="box-sizing: border-box; color: #dd1144;">i+1</span> <span class="attribute" style="box-sizing: border-box; color: teal;">m</span>=<span class="value" style="box-sizing: border-box; color: #dd1144;">2*i+3</span> <span class="attribute" style="box-sizing: border-box; color: teal;">return</span> [<span class="attribute" style="box-sizing: border-box; color: teal;">2</span>]+[<span class="attribute" style="box-sizing: border-box; color: teal;">x</span> <span class="attribute" style="box-sizing: border-box; color: teal;">for</span> <span class="attribute" style="box-sizing: border-box; color: teal;">x</span> <span class="attribute" style="box-sizing: border-box; color: teal;">in</span> <span class="attribute" style="box-sizing: border-box; color: teal;">s</span> <span class="attribute" style="box-sizing: border-box; color: teal;">if</span> <span class="attribute" style="box-sizing: border-box; color: teal;">x</span>] <span class="attribute" style="box-sizing: border-box; color: teal;">primes</span>(<span class="attribute" style="box-sizing: border-box; color: teal;">100</span>) </span></pre> <p style="box-sizing: border-box; margin: 20px 0px;">一旦你得到了你的设置了修饰符 <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">@profile</code> 的代码,使用 <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">kernprof.py</code> 运行这个脚本。</p> <pre class="objectivec" style="box-sizing: border-box; overflow: auto; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 10px; margin: 20px 0px; line-height: 1.3; word-break: break-all; word-wrap: break-word; border: 0px; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f5f5f5;">kernprof<span class="variable" style="box-sizing: border-box; color: teal;">.py</span> -l -v fib<span class="variable" style="box-sizing: border-box; color: teal;">.py</span> </pre> <p style="box-sizing: border-box; margin: 20px 0px;"><code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">-l</code> 选项告诉 <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">kernprof</code> 把修饰符 <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">@profile</code> 注入你的脚本,<code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">-v</code> 选项告诉 <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">kernprof</code> 一旦你的脚本完成后,展示计时信息。这是一个以上脚本的类似输出:</p> <pre class="xml" style="box-sizing: border-box; overflow: auto; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 10px; margin: 20px 0px; line-height: 1.3; word-break: break-all; word-wrap: break-word; border: 0px; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f5f5f5;">Wrote profile results to primes.py.lprof Timer unit: 1e-06 s

File: primes.py Function: primes at line 2 Total time: 0.00019 s

Line # Hits Time Per Hit % Time Line Contents

 2                                           @profile
 3                                           def primes(n): 
 4         1            2      2.0      1.1      if n==2:
 5                                                   return [2]
 6         1            1      1.0      0.5      elif n<span class="tag" style="box-sizing: border-box; color: #000080;">&lt;<span class="title" style="box-sizing: border-box;">2: </span> <span class="attribute" style="box-sizing: border-box; color: teal;">7</span> <span class="attribute" style="box-sizing: border-box; color: teal;">return</span> [] <span class="attribute" style="box-sizing: border-box; color: teal;">8</span> <span class="attribute" style="box-sizing: border-box; color: teal;">1</span> <span class="attribute" style="box-sizing: border-box; color: teal;">4</span> <span class="attribute" style="box-sizing: border-box; color: teal;">4.0</span> <span class="attribute" style="box-sizing: border-box; color: teal;">2.1</span> <span class="attribute" style="box-sizing: border-box; color: teal;">s</span>=<span class="value" style="box-sizing: border-box; color: #dd1144;">range(3,n+1,2)</span> <span class="attribute" style="box-sizing: border-box; color: teal;">9</span> <span class="attribute" style="box-sizing: border-box; color: teal;">1</span> <span class="attribute" style="box-sizing: border-box; color: teal;">10</span> <span class="attribute" style="box-sizing: border-box; color: teal;">10.0</span> <span class="attribute" style="box-sizing: border-box; color: teal;">5.3</span> <span class="attribute" style="box-sizing: border-box; color: teal;">mroot</span> = <span class="attribute" style="box-sizing: border-box; color: teal;">n</span> ** <span class="attribute" style="box-sizing: border-box; color: teal;">0.5</span> <span class="attribute" style="box-sizing: border-box; color: teal;">10</span> <span class="attribute" style="box-sizing: border-box; color: teal;">1</span> <span class="attribute" style="box-sizing: border-box; color: teal;">2</span> <span class="attribute" style="box-sizing: border-box; color: teal;">2.0</span> <span class="attribute" style="box-sizing: border-box; color: teal;">1.1</span> <span class="attribute" style="box-sizing: border-box; color: teal;">half</span>=<span class="value" style="box-sizing: border-box; color: #dd1144;">(n+1)</span>/<span class="attribute" style="box-sizing: border-box; color: teal;">2-1</span> <span class="attribute" style="box-sizing: border-box; color: teal;">11</span> <span class="attribute" style="box-sizing: border-box; color: teal;">1</span> <span class="attribute" style="box-sizing: border-box; color: teal;">1</span> <span class="attribute" style="box-sizing: border-box; color: teal;">1.0</span> <span class="attribute" style="box-sizing: border-box; color: teal;">0.5</span> <span class="attribute" style="box-sizing: border-box; color: teal;">i</span>=<span class="value" style="box-sizing: border-box; color: #dd1144;">0</span> <span class="attribute" style="box-sizing: border-box; color: teal;">12</span> <span class="attribute" style="box-sizing: border-box; color: teal;">1</span> <span class="attribute" style="box-sizing: border-box; color: teal;">1</span> <span class="attribute" style="box-sizing: border-box; color: teal;">1.0</span> <span class="attribute" style="box-sizing: border-box; color: teal;">0.5</span> <span class="attribute" style="box-sizing: border-box; color: teal;">m</span>=<span class="value" style="box-sizing: border-box; color: #dd1144;">3</span> <span class="attribute" style="box-sizing: border-box; color: teal;">13</span> <span class="attribute" style="box-sizing: border-box; color: teal;">5</span> <span class="attribute" style="box-sizing: border-box; color: teal;">7</span> <span class="attribute" style="box-sizing: border-box; color: teal;">1.4</span> <span class="attribute" style="box-sizing: border-box; color: teal;">3.7</span> <span class="attribute" style="box-sizing: border-box; color: teal;">while</span> <span class="attribute" style="box-sizing: border-box; color: teal;">m</span> &lt;= <span class="attribute" style="box-sizing: border-box; color: teal;">mroot:</span> <span class="attribute" style="box-sizing: border-box; color: teal;">14</span> <span class="attribute" style="box-sizing: border-box; color: teal;">4</span> <span class="attribute" style="box-sizing: border-box; color: teal;">4</span> <span class="attribute" style="box-sizing: border-box; color: teal;">1.0</span> <span class="attribute" style="box-sizing: border-box; color: teal;">2.1</span> <span class="attribute" style="box-sizing: border-box; color: teal;">if</span> <span class="attribute" style="box-sizing: border-box; color: teal;">s</span>[<span class="attribute" style="box-sizing: border-box; color: teal;">i</span>]<span class="attribute" style="box-sizing: border-box; color: teal;">:</span> <span class="attribute" style="box-sizing: border-box; color: teal;">15</span> <span class="attribute" style="box-sizing: border-box; color: teal;">3</span> <span class="attribute" style="box-sizing: border-box; color: teal;">4</span> <span class="attribute" style="box-sizing: border-box; color: teal;">1.3</span> <span class="attribute" style="box-sizing: border-box; color: teal;">2.1</span> <span class="attribute" style="box-sizing: border-box; color: teal;">j</span>=<span class="value" style="box-sizing: border-box; color: #dd1144;">(m*m-3)</span>/<span class="attribute" style="box-sizing: border-box; color: teal;">2</span> <span class="attribute" style="box-sizing: border-box; color: teal;">16</span> <span class="attribute" style="box-sizing: border-box; color: teal;">3</span> <span class="attribute" style="box-sizing: border-box; color: teal;">4</span> <span class="attribute" style="box-sizing: border-box; color: teal;">1.3</span> <span class="attribute" style="box-sizing: border-box; color: teal;">2.1</span> <span class="attribute" style="box-sizing: border-box; color: teal;">s</span>[<span class="attribute" style="box-sizing: border-box; color: teal;">j</span>]=<span class="value" style="box-sizing: border-box; color: #dd1144;">0</span> <span class="attribute" style="box-sizing: border-box; color: teal;">17</span> <span class="attribute" style="box-sizing: border-box; color: teal;">31</span> <span class="attribute" style="box-sizing: border-box; color: teal;">31</span> <span class="attribute" style="box-sizing: border-box; color: teal;">1.0</span> <span class="attribute" style="box-sizing: border-box; color: teal;">16.3</span> <span class="attribute" style="box-sizing: border-box; color: teal;">while</span> <span class="attribute" style="box-sizing: border-box; color: teal;">j</span>&lt;<span class="attribute" style="box-sizing: border-box; color: teal;">half:</span> <span class="attribute" style="box-sizing: border-box; color: teal;">18</span> <span class="attribute" style="box-sizing: border-box; color: teal;">28</span> <span class="attribute" style="box-sizing: border-box; color: teal;">28</span> <span class="attribute" style="box-sizing: border-box; color: teal;">1.0</span> <span class="attribute" style="box-sizing: border-box; color: teal;">14.7</span> <span class="attribute" style="box-sizing: border-box; color: teal;">s</span>[<span class="attribute" style="box-sizing: border-box; color: teal;">j</span>]=<span class="value" style="box-sizing: border-box; color: #dd1144;">0</span> <span class="attribute" style="box-sizing: border-box; color: teal;">19</span> <span class="attribute" style="box-sizing: border-box; color: teal;">28</span> <span class="attribute" style="box-sizing: border-box; color: teal;">29</span> <span class="attribute" style="box-sizing: border-box; color: teal;">1.0</span> <span class="attribute" style="box-sizing: border-box; color: teal;">15.3</span> <span class="attribute" style="box-sizing: border-box; color: teal;">j</span>+=<span class="value" style="box-sizing: border-box; color: #dd1144;">m</span> <span class="attribute" style="box-sizing: border-box; color: teal;">20</span> <span class="attribute" style="box-sizing: border-box; color: teal;">4</span> <span class="attribute" style="box-sizing: border-box; color: teal;">4</span> <span class="attribute" style="box-sizing: border-box; color: teal;">1.0</span> <span class="attribute" style="box-sizing: border-box; color: teal;">2.1</span> <span class="attribute" style="box-sizing: border-box; color: teal;">i</span>=<span class="value" style="box-sizing: border-box; color: #dd1144;">i+1</span> <span class="attribute" style="box-sizing: border-box; color: teal;">21</span> <span class="attribute" style="box-sizing: border-box; color: teal;">4</span> <span class="attribute" style="box-sizing: border-box; color: teal;">4</span> <span class="attribute" style="box-sizing: border-box; color: teal;">1.0</span> <span class="attribute" style="box-sizing: border-box; color: teal;">2.1</span> <span class="attribute" style="box-sizing: border-box; color: teal;">m</span>=<span class="value" style="box-sizing: border-box; color: #dd1144;">2*i+3</span> <span class="attribute" style="box-sizing: border-box; color: teal;">22</span> <span class="attribute" style="box-sizing: border-box; color: teal;">50</span> <span class="attribute" style="box-sizing: border-box; color: teal;">54</span> <span class="attribute" style="box-sizing: border-box; color: teal;">1.1</span> <span class="attribute" style="box-sizing: border-box; color: teal;">28.4</span> <span class="attribute" style="box-sizing: border-box; color: teal;">return</span> [<span class="attribute" style="box-sizing: border-box; color: teal;">2</span>]+[<span class="attribute" style="box-sizing: border-box; color: teal;">x</span> <span class="attribute" style="box-sizing: border-box; color: teal;">for</span> <span class="attribute" style="box-sizing: border-box; color: teal;">x</span> </span></pre>

<p style="box-sizing: border-box; margin: 20px 0px;">寻找 <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">hits</code> 值比较高的行或是一个高时间间隔。这些地方有最大的优化改进空间。</p> <h1 style="box-sizing: border-box; font-size: 1.57143em; margin: 20px 0px; font-family: inherit; line-height: 1.125; color: inherit; padding-bottom: 10px; border-bottom-width: 1px; border-bottom-style: solid; border-bottom-color: #dddddd;">它使用了多少内存?</h1> <p style="box-sizing: border-box; margin: 20px 0px;">现在我们掌握了很好我们代码的计时信息,让我们继续找出我们的程序使用了多少内存。我们真是非常幸运, Fabian Pedregosa 仿照 Robert Kern 的 <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">line_profiler</code> 实现了一个很好的内存分析器 <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">[memory profiler][5]</code>。</p> <p style="box-sizing: border-box; margin: 20px 0px;">首先通过 pip 安装它:</p> <pre class="ruby" style="box-sizing: border-box; overflow: auto; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 10px; margin: 20px 0px; line-height: 1.3; word-break: break-all; word-wrap: break-word; border: 0px; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f5f5f5;"><span class="variable" style="box-sizing: border-box; color: teal;">$ </span>pip install -<span class="constant" style="box-sizing: border-box;">U</span> memory_profiler <span class="variable" style="box-sizing: border-box; color: teal;">$ </span>pip install psutil </pre> <p style="box-sizing: border-box; margin: 20px 0px;">在这里建议安装 <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">psutil</code> 是因为该包能提升 <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">memory_profiler</code> 的性能。</p> <p style="box-sizing: border-box; margin: 20px 0px;">想 <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">line_profiler</code> 一样, <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">memory_profiler</code> 要求在你设置 <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">@profile</code> 来修饰你的函数:</p> <pre class="r" style="box-sizing: border-box; overflow: auto; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 10px; margin: 20px 0px; line-height: 1.3; word-break: break-all; word-wrap: break-word; border: 0px; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f5f5f5;">@profile def primes(n): <span class="keyword" style="box-sizing: border-box; font-weight: bold;">...</span> <span class="keyword" style="box-sizing: border-box; font-weight: bold;">...</span> </pre> <p style="box-sizing: border-box; margin: 20px 0px;">运行如下命令来显示你的函数使用了多少内存:</p> <pre class="ruby" style="box-sizing: border-box; overflow: auto; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 10px; margin: 20px 0px; line-height: 1.3; word-break: break-all; word-wrap: break-word; border: 0px; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f5f5f5;"><span class="variable" style="box-sizing: border-box; color: teal;">$ </span>python -m memory_profiler primes.py </pre> <p style="box-sizing: border-box; margin: 20px 0px;">一旦你的程序退出,你应该可以看到这样的输出:</p> <pre class="http" style="box-sizing: border-box; overflow: auto; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 10px; margin: 20px 0px; line-height: 1.3; word-break: break-all; word-wrap: break-word; border: 0px; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f5f5f5;"><span class="attribute" style="box-sizing: border-box; color: teal;">Filename</span>: <span class="string" style="box-sizing: border-box; color: #dd1144;">primes.py</span>

<span class="python" style="box-sizing: border-box;">Line <span class="comment" style="box-sizing: border-box; color: #999988; font-style: italic;"># Mem usage Increment Line Contents</span> ============================================== <span class="number" style="box-sizing: border-box; color: #009999;">2</span> <span class="decorator" style="box-sizing: border-box;">@profile</span> <span class="number" style="box-sizing: border-box; color: #009999;">3</span> <span class="number" style="box-sizing: border-box; color: #009999;">7.9219</span> MB <span class="number" style="box-sizing: border-box; color: #009999;">0.0000</span> MB <span class="function" style="box-sizing: border-box;"><span class="keyword" style="box-sizing: border-box; font-weight: bold;">def</span> <span class="title" style="box-sizing: border-box; color: #990000; font-weight: bold;">primes</span><span class="params" style="box-sizing: border-box;">(n)</span>:</span> <span class="number" style="box-sizing: border-box; color: #009999;">4</span> <span class="number" style="box-sizing: border-box; color: #009999;">7.9219</span> MB <span class="number" style="box-sizing: border-box; color: #009999;">0.0000</span> MB <span class="keyword" style="box-sizing: border-box; font-weight: bold;">if</span> n==<span class="number" style="box-sizing: border-box; color: #009999;">2</span>: <span class="number" style="box-sizing: border-box; color: #009999;">5</span> <span class="keyword" style="box-sizing: border-box; font-weight: bold;">return</span> [<span class="number" style="box-sizing: border-box; color: #009999;">2</span>] <span class="number" style="box-sizing: border-box; color: #009999;">6</span> <span class="number" style="box-sizing: border-box; color: #009999;">7.9219</span> MB <span class="number" style="box-sizing: border-box; color: #009999;">0.0000</span> MB <span class="keyword" style="box-sizing: border-box; font-weight: bold;">elif</span> n<<span class="number" style="box-sizing: border-box; color: #009999;">2</span>: <span class="number" style="box-sizing: border-box; color: #009999;">7</span> <span class="keyword" style="box-sizing: border-box; font-weight: bold;">return</span> [] <span class="number" style="box-sizing: border-box; color: #009999;">8</span> <span class="number" style="box-sizing: border-box; color: #009999;">7.9219</span> MB <span class="number" style="box-sizing: border-box; color: #009999;">0.0000</span> MB s=range(<span class="number" style="box-sizing: border-box; color: #009999;">3</span>,n+<span class="number" style="box-sizing: border-box; color: #009999;">1</span>,<span class="number" style="box-sizing: border-box; color: #009999;">2</span>) <span class="number" style="box-sizing: border-box; color: #009999;">9</span> <span class="number" style="box-sizing: border-box; color: #009999;">7.9258</span> MB <span class="number" style="box-sizing: border-box; color: #009999;">0.0039</span> MB mroot = n ** <span class="number" style="box-sizing: border-box; color: #009999;">0.5</span> <span class="number" style="box-sizing: border-box; color: #009999;">10</span> <span class="number" style="box-sizing: border-box; color: #009999;">7.9258</span> MB <span class="number" style="box-sizing: border-box; color: #009999;">0.0000</span> MB half=(n+<span class="number" style="box-sizing: border-box; color: #009999;">1</span>)/<span class="number" style="box-sizing: border-box; color: #009999;">2</span>-<span class="number" style="box-sizing: border-box; color: #009999;">1</span> <span class="number" style="box-sizing: border-box; color: #009999;">11</span> <span class="number" style="box-sizing: border-box; color: #009999;">7.9258</span> MB <span class="number" style="box-sizing: border-box; color: #009999;">0.0000</span> MB i=<span class="number" style="box-sizing: border-box; color: #009999;">0</span> <span class="number" style="box-sizing: border-box; color: #009999;">12</span> <span class="number" style="box-sizing: border-box; color: #009999;">7.9258</span> MB <span class="number" style="box-sizing: border-box; color: #009999;">0.0000</span> MB m=<span class="number" style="box-sizing: border-box; color: #009999;">3</span> <span class="number" style="box-sizing: border-box; color: #009999;">13</span> <span class="number" style="box-sizing: border-box; color: #009999;">7.9297</span> MB <span class="number" style="box-sizing: border-box; color: #009999;">0.0039</span> MB <span class="keyword" style="box-sizing: border-box; font-weight: bold;">while</span> m <= mroot: <span class="number" style="box-sizing: border-box; color: #009999;">14</span> <span class="number" style="box-sizing: border-box; color: #009999;">7.9297</span> MB <span class="number" style="box-sizing: border-box; color: #009999;">0.0000</span> MB <span class="keyword" style="box-sizing: border-box; font-weight: bold;">if</span> s[i]: <span class="number" style="box-sizing: border-box; color: #009999;">15</span> <span class="number" style="box-sizing: border-box; color: #009999;">7.9297</span> MB <span class="number" style="box-sizing: border-box; color: #009999;">0.0000</span> MB j=(m*m-<span class="number" style="box-sizing: border-box; color: #009999;">3</span>)/<span class="number" style="box-sizing: border-box; color: #009999;">2</span> <span class="number" style="box-sizing: border-box; color: #009999;">16</span> <span class="number" style="box-sizing: border-box; color: #009999;">7.9258</span> MB -<span class="number" style="box-sizing: border-box; color: #009999;">0.0039</span> MB s[j]=<span class="number" style="box-sizing: border-box; color: #009999;">0</span> <span class="number" style="box-sizing: border-box; color: #009999;">17</span> <span class="number" style="box-sizing: border-box; color: #009999;">7.9297</span> MB <span class="number" style="box-sizing: border-box; color: #009999;">0.0039</span> MB <span class="keyword" style="box-sizing: border-box; font-weight: bold;">while</span> j<half: <span class="number" style="box-sizing: border-box; color: #009999;">18</span> <span class="number" style="box-sizing: border-box; color: #009999;">7.9297</span> MB <span class="number" style="box-sizing: border-box; color: #009999;">0.0000</span> MB s[j]=<span class="number" style="box-sizing: border-box; color: #009999;">0</span> <span class="number" style="box-sizing: border-box; color: #009999;">19</span> <span class="number" style="box-sizing: border-box; color: #009999;">7.9297</span> MB <span class="number" style="box-sizing: border-box; color: #009999;">0.0000</span> MB j+=m <span class="number" style="box-sizing: border-box; color: #009999;">20</span> <span class="number" style="box-sizing: border-box; color: #009999;">7.9297</span> MB <span class="number" style="box-sizing: border-box; color: #009999;">0.0000</span> MB i=i+<span class="number" style="box-sizing: border-box; color: #009999;">1</span> <span class="number" style="box-sizing: border-box; color: #009999;">21</span> <span class="number" style="box-sizing: border-box; color: #009999;">7.9297</span> MB <span class="number" style="box-sizing: border-box; color: #009999;">0.0000</span> MB m=<span class="number" style="box-sizing: border-box; color: #009999;">2</span>*i+<span class="number" style="box-sizing: border-box; color: #009999;">3</span> <span class="number" style="box-sizing: border-box; color: #009999;">22</span> <span class="number" style="box-sizing: border-box; color: #009999;">7.9297</span> MB <span class="number" style="box-sizing: border-box; color: #009999;">0.0000</span> MB <span class="keyword" style="box-sizing: border-box; font-weight: bold;">return</span> [<span class="number" style="box-sizing: border-box; color: #009999;">2</span>]+[x <span class="keyword" style="box-sizing: border-box; font-weight: bold;">for</span> x <span class="keyword" style="box-sizing: border-box; font-weight: bold;">in</span> s <span class="keyword" style="box-sizing: border-box; font-weight: bold;">if</span> x] </span></pre>

<h1 style="box-sizing: border-box; font-size: 1.57143em; margin: 20px 0px; font-family: inherit; line-height: 1.125; color: inherit; padding-bottom: 10px; border-bottom-width: 1px; border-bottom-style: solid; border-bottom-color: #dddddd;"><strong style="box-sizing: border-box;"><code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">line_profiler</code></strong> 和 <strong style="box-sizing: border-box;"><code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">memory_profiler</code></strong> 的 IPython 快捷命令</h1> <p style="box-sizing: border-box; margin: 20px 0px;"><code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">line_profiler</code> 和 <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">memory_profiler</code> 一个鲜为人知的特性就是在 IPython 上都有快捷命令。你所能做的就是在 IPython 上键入以下命令:</p> <pre class="perl" style="box-sizing: border-box; overflow: auto; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 10px; margin: 20px 0px; line-height: 1.3; word-break: break-all; word-wrap: break-word; border: 0px; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f5f5f5;"><span class="variable" style="box-sizing: border-box; color: teal;">%load_ext</span> memory_profiler <span class="variable" style="box-sizing: border-box; color: teal;">%load_ext</span> line_profiler </pre> <p style="box-sizing: border-box; margin: 20px 0px;">这样做了以后,你就可以使用魔法命令 <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">%lprun</code> 和 <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">%mprun</code> 了,它们表现的像它们命令行的副本,最主要的不同就是你不需要给你需要分析的函数设置 <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">@profile</code> 修饰符。直接在你的 IPython 会话上继续分析吧。</p> <pre class="matlab" style="box-sizing: border-box; overflow: auto; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 10px; margin: 20px 0px; line-height: 1.3; word-break: break-all; word-wrap: break-word; border: 0px; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f5f5f5;">In <span class="matrix" style="box-sizing: border-box;">[<span class="number" style="box-sizing: border-box; color: #009999;">1</span>]</span>: from <span class="built_in" style="box-sizing: border-box; color: #0086b3;">primes</span> import <span class="built_in" style="box-sizing: border-box; color: #0086b3;">primes</span> In <span class="matrix" style="box-sizing: border-box;">[<span class="number" style="box-sizing: border-box; color: #009999;">2</span>]</span>: <span class="comment" style="box-sizing: border-box; color: #999988; font-style: italic;">%mprun -f primes primes(1000)</span> In <span class="matrix" style="box-sizing: border-box;">[<span class="number" style="box-sizing: border-box; color: #009999;">3</span>]</span>: <span class="comment" style="box-sizing: border-box; color: #999988; font-style: italic;">%lprun -f primes primes(1000)</span> </pre> <p style="box-sizing: border-box; margin: 20px 0px;">这可以节省你大量的时间和精力,因为使用这些分析命令,你不需要修改你的源代码。</p> <h1 style="box-sizing: border-box; font-size: 1.57143em; margin: 20px 0px; font-family: inherit; line-height: 1.125; color: inherit; padding-bottom: 10px; border-bottom-width: 1px; border-bottom-style: solid; border-bottom-color: #dddddd;">哪里内存溢出了?</h1> <p style="box-sizing: border-box; margin: 20px 0px;">cPython的解释器使用引用计数来作为它跟踪内存的主要方法。这意味着每个对象持有一个计数器,当增加某个对象的引用存储的时候,计数器就会增加,当一个引用被删除的时候,计数器就是减少。当计数器达到0, cPython 解释器就知道该对象不再使用,因此解释器将删除这个对象,并且释放该对象持有的内存。</p> <p style="box-sizing: border-box; margin: 20px 0px;">内存泄漏往往发生在即使该对象不再使用的时候,你的程序还持有对该对象的引用。</p> <p style="box-sizing: border-box; margin: 20px 0px;">最快速发现内存泄漏的方式就是使用一个由 Marius Gedminas 编写的非常好的称为 <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">[objgraph][6]</code> 的工具。<br style="box-sizing: border-box;" />这个工具可以让你看到在内存中对象的数量,也定位在代码中所有不同的地方,对这些对象的引用。</p> <p style="box-sizing: border-box; margin: 20px 0px;">开始,我们首先安装 <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">objgraph</code></p> <pre class="nginx" style="box-sizing: border-box; overflow: auto; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 10px; margin: 20px 0px; line-height: 1.3; word-break: break-all; word-wrap: break-word; border: 0px; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f5f5f5;"><span class="title" style="box-sizing: border-box; color: #990000; font-weight: bold;">pip</span> install objgraph </pre> <p style="box-sizing: border-box; margin: 20px 0px;">一旦你安装了这个工具,在你的代码中插入一个调用调试器的声明。</p> <pre class="haskell" style="box-sizing: border-box; overflow: auto; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 10px; margin: 20px 0px; line-height: 1.3; word-break: break-all; word-wrap: break-word; border: 0px; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f5f5f5;"><span class="import" style="box-sizing: border-box;"><span class="keyword" style="box-sizing: border-box; font-weight: bold;">import</span> pdb;</span> pdb.set_trace() </pre> <h2 style="box-sizing: border-box; font-family: inherit; line-height: 1.125; color: inherit; margin: 20px 0px; font-size: 1.42857em;">哪个对象最常见</h2> <p style="box-sizing: border-box; margin: 20px 0px;">在运行时,你可以检查在运行在你的程序中的前20名最普遍的对象</p> <pre class="haskell" style="box-sizing: border-box; overflow: auto; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 10px; margin: 20px 0px; line-height: 1.3; word-break: break-all; word-wrap: break-word; border: 0px; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f5f5f5;"><span class="title" style="box-sizing: border-box; color: #990000; font-weight: bold;">pdb</span>) <span class="import" style="box-sizing: border-box;"><span class="keyword" style="box-sizing: border-box; font-weight: bold;">import</span> objgraph</span> (pdb) objgraph.show_most_common_types()

<span class="type" style="box-sizing: border-box;">MyBigFatObject</span> <span class="number" style="box-sizing: border-box; color: #009999;">20000</span> <span class="title" style="box-sizing: border-box; color: #990000; font-weight: bold;">tuple</span> <span class="number" style="box-sizing: border-box; color: #009999;">16938</span> <span class="title" style="box-sizing: border-box; color: #990000; font-weight: bold;">function</span> <span class="number" style="box-sizing: border-box; color: #009999;">4310</span> <span class="title" style="box-sizing: border-box; color: #990000; font-weight: bold;">dict</span> <span class="number" style="box-sizing: border-box; color: #009999;">2790</span> <span class="title" style="box-sizing: border-box; color: #990000; font-weight: bold;">wrapper_descriptor</span> <span class="number" style="box-sizing: border-box; color: #009999;">1181</span> <span class="title" style="box-sizing: border-box; color: #990000; font-weight: bold;">builtin_function_or_method</span> <span class="number" style="box-sizing: border-box; color: #009999;">934</span> <span class="title" style="box-sizing: border-box; color: #990000; font-weight: bold;">weakref</span> <span class="number" style="box-sizing: border-box; color: #009999;">764</span> <span class="title" style="box-sizing: border-box; color: #990000; font-weight: bold;">list</span> <span class="number" style="box-sizing: border-box; color: #009999;">634</span> <span class="title" style="box-sizing: border-box; color: #990000; font-weight: bold;">method_descriptor</span> <span class="number" style="box-sizing: border-box; color: #009999;">507</span> <span class="title" style="box-sizing: border-box; color: #990000; font-weight: bold;">getset_descriptor</span> <span class="number" style="box-sizing: border-box; color: #009999;">451</span> <span class="typedef" style="box-sizing: border-box;"><span class="keyword" style="box-sizing: border-box; font-weight: bold;">type</span> 439</span> </pre>

<h2 style="box-sizing: border-box; font-family: inherit; line-height: 1.125; color: inherit; margin: 20px 0px; font-size: 1.42857em;">哪个对象被增加或是删除了?</h2> <p style="box-sizing: border-box; margin: 20px 0px;">我们能在两个时间点之间看到哪些对象被增加或是删除了。</p> <pre class="haskell" style="box-sizing: border-box; overflow: auto; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 10px; margin: 20px 0px; line-height: 1.3; word-break: break-all; word-wrap: break-word; border: 0px; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f5f5f5;">(pdb) <span class="import" style="box-sizing: border-box;"><span class="keyword" style="box-sizing: border-box; font-weight: bold;">import</span> objgraph</span> (pdb) objgraph.show_growth() . . . (pdb) objgraph.show_growth() # this only shows objects that has been added or deleted since last show_growth() call

<span class="title" style="box-sizing: border-box; color: #990000; font-weight: bold;">traceback</span> <span class="number" style="box-sizing: border-box; color: #009999;">4</span> +<span class="number" style="box-sizing: border-box; color: #009999;">2</span> <span class="type" style="box-sizing: border-box;">KeyboardInterrupt</span> <span class="number" style="box-sizing: border-box; color: #009999;">1</span> +<span class="number" style="box-sizing: border-box; color: #009999;">1</span> <span class="title" style="box-sizing: border-box; color: #990000; font-weight: bold;">frame</span> <span class="number" style="box-sizing: border-box; color: #009999;">24</span> +<span class="number" style="box-sizing: border-box; color: #009999;">1</span> <span class="title" style="box-sizing: border-box; color: #990000; font-weight: bold;">list</span> <span class="number" style="box-sizing: border-box; color: #009999;">667</span> +<span class="number" style="box-sizing: border-box; color: #009999;">1</span> <span class="title" style="box-sizing: border-box; color: #990000; font-weight: bold;">tuple</span> <span class="number" style="box-sizing: border-box; color: #009999;">16969</span> +<span class="number" style="box-sizing: border-box; color: #009999;">1</span> </pre>

<h2 style="box-sizing: border-box; font-family: inherit; line-height: 1.125; color: inherit; margin: 20px 0px; font-size: 1.42857em;">这个泄漏对象的引用是什么?</h2> <p style="box-sizing: border-box; margin: 20px 0px;">继续下去,我们还可以看到任何给定对象的引用在什么地方。让我们以下面这个简单的程序举个例子。</p> <pre class="haskell" style="box-sizing: border-box; overflow: auto; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 10px; margin: 20px 0px; line-height: 1.3; word-break: break-all; word-wrap: break-word; border: 0px; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f5f5f5;"><span class="title" style="box-sizing: border-box; color: #990000; font-weight: bold;">x</span> = [<span class="number" style="box-sizing: border-box; color: #009999;">1</span>] <span class="title" style="box-sizing: border-box; color: #990000; font-weight: bold;">y</span> = [x, [x], {<span class="string" style="box-sizing: border-box; color: #dd1144;">"a"</span>:x}] <span class="import" style="box-sizing: border-box;"><span class="keyword" style="box-sizing: border-box; font-weight: bold;">import</span> pdb;</span> pdb.set_trace() </pre> <p style="box-sizing: border-box; margin: 20px 0px;">为了看到持有变量 X 的引用是什么,运行 <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">objgraph.show_backref()</code> 函数:</p> <pre class="haskell" style="box-sizing: border-box; overflow: auto; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 10px; margin: 20px 0px; line-height: 1.3; word-break: break-all; word-wrap: break-word; border: 0px; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f5f5f5;">(pdb) <span class="import" style="box-sizing: border-box;"><span class="keyword" style="box-sizing: border-box; font-weight: bold;">import</span> objgraph</span> (pdb) objgraph.show_backref([x], filename=<span class="string" style="box-sizing: border-box; color: #dd1144;">"/tmp/backrefs.png"</span>) </pre> <p style="box-sizing: border-box; margin: 20px 0px;">该命令的输出是一个 PNG 图片,被存储在 <code style="box-sizing: border-box; font-family: Menlo, Monaco, Consolas, 'Courier New', monospace; font-size: 0.92857em; padding: 2px 4px; color: #c7254e; white-space: pre; border-top-left-radius: 2px; border-top-right-radius: 2px; border-bottom-right-radius: 2px; border-bottom-left-radius: 2px; background-color: #f9f2f4;">/tmp/backrefs.png</code>,它应该看起来像这样:</p> <p style="box-sizing: border-box; margin: 20px 0px;"><img style="box-sizing: border-box; vertical-align: middle; max-width: 100%;" src="http://www.huyng.com/media/3011/backrefs.png" alt="此处输入图片的描述" /></p> <p style="box-sizing: border-box; margin: 20px 0px;">盒子底部有红色字体就是我们感兴趣的对象,我们可以看到它被符号 x 引用了一次,被列表 y 引用了三次。如果 x 这个对象引起了内存泄漏,我们可以使用这种方法来追踪它的所有引用,以便看到为什么它没有被自动被收回。</p> <p style="box-sizing: border-box; margin: 20px 0px;">回顾一遍,objgraph 允许我们:</p> <ul style="box-sizing: border-box; margin: 20px 0px;"> <li style="box-sizing: border-box; margin: 5px 0px;">显示占用 Python 程序内存的前 N 个对象</li> <li style="box-sizing: border-box; margin: 5px 0px;">显示在一段时期内哪些对象被增加了,哪些对象被删除了</li> <li style="box-sizing: border-box; margin: 5px 0px;">显示我们脚本中获得的所有引用</li> </ul> <h1 style="box-sizing: border-box; font-size: 1.57143em; margin: 20px 0px; font-family: inherit; line-height: 1.125; color: inherit; padding-bottom: 10px; border-bottom-width: 1px; border-bottom-style: solid; border-bottom-color: #dddddd;">Effort vs precision</h1> <p style="box-sizing: border-box; margin: 20px 0px;">在这篇文章中,我展示了如何使用一些工具来分析一个python程序的性能。通过这些工具和技术的武装,你应该可以获取所有要求追踪大多数内存泄漏以及在Python程序快速识别瓶颈的信息。</p> <p style="box-sizing: border-box; margin: 20px 0px;">和许多其他主题一样,运行性能分析意味着要在付出和精度之间的平衡做取舍。当有疑问是,用最简单的方案,满足你当前的需求。</p> <p style="box-sizing: border-box; margin: 20px 0px;">相关阅读:</p> <ul style="box-sizing: border-box; margin: 20px 0px;"> <li style="box-sizing: border-box; margin: 5px 0px;"><a style="box-sizing: border-box; color: #017e66; text-decoration: none; background: transparent;" href="http://stackoverflow.com/questions/556405/what-do-real-user-and-sys-mean-in-the-output-of-time1" rel="nofollow" target="_blank">stack overflow - time explained</a></li> <li style="box-sizing: border-box; margin: 5px 0px;"><a style="box-sizing: border-box; color: #017e66; text-decoration: none; background: transparent;" href="http://packages.python.org/line_profiler/" rel="nofollow" target="_blank">line_profiler</a></li> <li style="box-sizing: border-box; margin: 5px 0px;"><a style="box-sizing: border-box; color: #017e66; text-decoration: none; background: transparent;" href="https://github.com/fabianp/memory_profiler" rel="nofollow" target="_blank">memory_profiler</a></li> <li style="box-sizing: border-box; margin: 5px 0px;"><a style="box-sizing: border-box; color: #017e66; text-decoration: none; background: transparent;" href="http://mg.pov.lt/objgraph/" rel="nofollow" target="_blank">objgraph</a></li> </ul> </div>

共有 人打赏支持
粉丝 5
博文 46
码字总数 42368
×
异类深呼吸
如果觉得我的文章对您有用,请随意打赏。您的支持将鼓励我继续创作!
* 金额(元)
¥1 ¥5 ¥10 ¥20 其他金额
打赏人
留言
* 支付类型
微信扫码支付
打赏金额:
已支付成功
打赏金额: