Jekyll2020-07-14T05:36:50+00:00https://rohanvarma.me/feed.xmlRohan Varma📋machine learning, math, and other random thoughtsHow to Tax-Loss Harvest during the COVID-19 Market Downturn2020-06-29T00:00:00+00:002020-06-29T00:00:00+00:00https://rohanvarma.me/TLH<p>This blog post is a deviation from the technical content that I usually post, since I thought it may be useful to share my experience with my first time venturing into tax-loss harvesting, especially given the market conditions in response to COVID-19. This post is not meant to be a comprehensive guide for when to tax loss harvest, but more of a series of steps that would need to take when deciding to go through with it, and a rough estimation of the actual benefits. Now for the obligatory disclaimer: this post isn’t intended to provide investment or tax advice!</p>
<h5 id="what-is-tax-loss-harvesting">What is Tax-Loss Harvesting?</h5>
<p>Tax-Loss Harvesting (TLH) refers to the process of selling your investments at a loss in order to deduct this loss, up to a limit, from your capital gain or other income for the year, thus offsetting your total tax burden for the year. This “buy high, sell low” approach is of course contrary to typical investment approaches, but through a series of steps described below, you can actually remain fully invested in the markets for the long term, reduce your cost basis, and save on taxes in the event of a market downturn. Many robo-advisor firms such as Wealthfront and Betterment, as well as traditional portfolio managers use this method to improve an investor’s overall return, but it can of course be done yourself. Let’s look at how.</p>
<p><strong>First things first: Why Tax-Loss Harvest?</strong></p>
<p>When the market declines harvesting your losses could provide you with a tax benefit, provided that you approach it correctly and avoid all of the different catches, such as the wash sale rule (to be explained later). When tax-loss harvesting, you take a loss of the value of your investment when you purchased it minus the value of the investment when you sold it. This loss can be used to offset any capital gains that you incurred in the same year at a 1:1 ratio - i.e. every dollar of loss will cancel out one dollar of capital gains.</p>
<p>If your loss exceeds your year’s capital gain, you can then use it to deduct up to $3000 from your taxable income that year. If you still have more tax losses, you can carry those forward to future years. In those future years, the tax losses must be used to offset any capital gains you incur in those years, and then can be applied to deduct up to $3000 from your taxable income that year. This means that a $30,000 loss one year can reduce your taxable income by $3000 every year for 10 years, saving you $3000 * your marginal rate every year for 10 years.</p>
<p>Based on the above, we can see that it’s quite beneficial to tax loss harvest if you have significant capital gains in the year that you are tax loss harvesting, as if your loss is large enough, it can be used to offset the entire capital gain, and you’ll pay zero capital gains tax. It would also be beneficial if you are planning to book a large loss and anticipate large capital gains in the future, as the losses can be carried forward and applied when you realize that large capital gain.</p>
<p><strong>How to tax loss harvest while staying invested for the long term</strong></p>
<p>It’s clear that tax loss harvesting has its benefits, but if you just stop at selling the investment at a loss and booking the loss on your taxes, then you’re missing out on potential future gains in the stock market by leaving that money uninvested. If you subscribe to the philosophy that the market goes up in the long-term and would like to stay fully invested for decades to come, there is a way to do this while still booking tax losses.</p>
<p>Let’s go through the steps in order to harvest investment losses. Here, I will use two Vanguard funds, VTSAX and VFIAX, as an example. Let’s say that you purchased $100,000 worth of VTSAX, but by March 2020, this has declined to a value of $73,000.</p>
<p><strong>Step 0: Make sure your cost basis is set correctly.</strong> More on this below, but the TLDR is that you want to be able to choose the shares of the fund/stock that you are selling if you own many shares of the same fund purchased at different times and prices. In order to do this, you want to change the default cost basis to a cost basis known as <a href="https://www.schwab.com/resource-center/insights/content/save-on-taxes-know-your-cost-basis">Specific Identification</a>.</p>
<p><strong>Step 1: Wait for a market decline.</strong> Clearly, the market needs to decline such that you own some shares that you can sell at a loss as a prerequisite. How much the market needs to decline is largely up to you. If your investment only declines by $100, it is likely not worth the trouble to go through the process to only save taxes on $100. I would recommend setting an amount at which it is worth it for you to tax loss harvest - for example, only starting the process when you have a loss of $5000, so that you can book at least that much in losses.</p>
<p><strong>Step 2: Find tax loss harvesting partners to stay invested while avoiding wash sales.</strong> It would be nice if you could sell VTSAX for a loss and then buy it back the same day in order to stay fully invested, but unfortunatately, the IRS will consider this a <a href="https://www.investopedia.com/terms/w/washsalerule.asp">wash sale</a> and thus not allow you to benefit from the loss incurred.</p>
<p>A wash sale is loosely defined as selling and then repurchasing the same or a “substantially identical” asset within a 30 day period before or after selling the asset. A simple example would be selling and then immediately repurchasing VTSAX. More interesting examples include selling VTSAX for a loss and then purchasing FZROX, Fidelity’s total stock market index fund, within a 30 way window - since these funds track the same index it is likely that this will be considered a wash sale even though they’re managed by different companies. Note that the wash sale rule specifies that you must not purchase a “substantially identical” asset 30 days before and after executing the sale which resulted in the loss that you want to harvest, so there is a 61 day period where you will not be able to repurchase the same/substantially identical asset.</p>
<p>A good pair of “tax loss harvesting partners” is thus two funds that are relatively similar so that they allow you to mostly maintain your target allocation, but also track slightly different indices (this seems to be enough to avoid the wash sale rule). Many tax loss harvesting posts online recommend pairs such as VTSAX & VFIAX (tracking US CRSP Index vs S&P 500 index) or VTSAX & VLCAX (the latter tracking large caps). Let’s say that you go with VTSAX & VFIAX.</p>
<p><strong>Step 3: Execute the trade.</strong> This part is simple, since Vanguard allows you to simply exchange one mutual fund for another. Ensure that your cost basis is set appropriately (more on this below), and pick the lot with the largest loss that you want to sell. Exchange this for VFIAX, and you’ve realized your losses.</p>
<p><strong>Step 4: Optionally wait 31 days to re-exchange for your preferred fund.</strong> If you really prefer VTSAX over VFIAX or want to keep your number of funds low, then after waiting for the wash sale period you can simply conduct the above trade in the reverse order. Be aware that you may have to pay short-term capital gains tax however if there is a significant market rebound in those 31 days.</p>
<p><strong>Step 5: Report the losses on your tax return.</strong> You’ll get a form 1099-B which indicates the short-term and long-term losses that you incurred.</p>
<p>With the above steps, we are able to book a $27,000 paper loss, while still remaining invested with basically the same asset allocation (we exchanged VTSAX for VFIAX, which are largely similar in that they track the broad US stock market, although VFIAX has a bent towards larger companies). Since we exchanged our funds, our investment does not sit outside of the market for even a day, and thus has potential to benefit from a market rebound (or booking even more tax losses) in the future. We could have of course avoided the headache of having to navigate around the wash sale rule by just leaving the money out of the market for the next 31 days, although with the exchange we ensure the money never mises a second in the market.</p>
<p>This $27,000 paper loss will first be used to offset any capital gains in the current year. Assuming that you are investing for the long term (i.e. retirement) and thus don’t have significant capital gains in a working year, what will actually happen is that you will be taxed on $3000 less of income for this year, thus saving you your marginal rate on $3000. Let’s assume that you live in California and have a marginal federal tax rate of 32%. This means that the total tax you’d pay on $3000 at your marginal rate would be 41.3% (32 + 9.3) or $1239. This means that over 9 years, you’ll save $11,151 assuming you have no capital gains for the following 9 years.</p>
<p>There is a catch to this which makes it so that you don’t actually save that much, although. Since you repurchased similar shares when the market was down, provided that the market eventually does recover in the long term, with your reduced cost basis you will be liable for paying additional capital gains tax than if you hadn’t tax loss harvested. In this example, since your cost basis has effectively been reduced by $27,000, you’ll have to pay capital gains tax on $27,000 you’d otherwise not have to pay. This means you’ll end up paying $6561 in federal and CA income taxes, still netting you a savings of $4590.</p>
<p>The above wrinkle means that tax loss harvesting is not necessarily something that is always beneficial to do during a market downturn. For example, if you fall into the 0% capital gains tax bracket this year, then it is not useful to write off capital gains with capital losses. If you anticipate high income (federal or state) in future years, or move from a no income tax state to a state where there is one, the benefit may be less due to being in a larger capital gains or state income tax bracket.</p>
<p>In the case where the income taxes you save every year ends up being less than the capital gains and state taxes you pay on the extra gain due to the reduced basis, the TLH will actually end up having a negative value (your income in the year the capital gain is realized would have to be much higher than what it is in the year you realize the loss, for this to practically apply).</p>
<p><strong>A Quick Primer on Cost Basis and its relation to tax-loss harvesting</strong></p>
<p>If you don’t understand cost basis and proceed with the sale of assets that have lost value, you may actually end up with a capital gain if your broker used the default FIFO or average cost method for determining your cost basis. To understand how this can happen, let’s take a look at the different types of cost bases that exist.</p>
<p>A cost basis a computation for what you paid for an investment, accounting for any transaction/brokerage fees. In the simple case where you buy 1 share of company A for $100, and then sell it, your cost basis (what you paid) is $100. Now let’s say that you bought another share of company A a year later, for $200, and a year after that, wanted to sell 1 of your 2 shares. Let’s assume that a single share of company A was worth $300 at that point. Assuming a taxable brokerage account, you’ll be taxed on the $300 minus whatever you paid to acquire the share.</p>
<p>You might intuitively think of the following ways to compute what you paid to acquire the share:</p>
<ol>
<li>I’ll sell the first share that I bought 2 years ago, so I paid $100 (known as FIFO cost basis)</li>
<li>I’ll sell the last share that I bought, so I paid $200 (known as LIFO cost basis).</li>
<li>I’ll sell the “average share”, so it’ll be like I paid $150 (known as average cost basis).</li>
<li>I’ll pick the specific share I want to sell when I sell it (known as specific identification cost basis).</li>
</ol>
<p>We can see that the cost basis we pick in this simple case affects the taxes that we will pay on our gain, and this also applies to deducting taxes when incurring a loss. For example, let’s consider the following scenario:</p>
<ol>
<li>In 2015, you bought 100 shares of VTSAX for $100.</li>
<li>In 2019, you bought 50 shares of VTSAX for $200.</li>
<li>In 2020, you decide to sell 30 shares of VTSAX at $180.</li>
</ol>
<p>Clearly despite selling some shares at $180 you’re at a net (paper) gain here, but can you generate a paper loss that allows you to offset other capital gains or lower your taxable income for 2020? It turns out you can, depending on your cost basis. First, let’s assume that you stuck with Vanguard’s default cost basis for mutual funds, the <a href="https://www.investopedia.com/terms/a/averagecostbasismethod.asp">average cost basis</a>.</p>
<p><strong>Average cost basis</strong></p>
<p>Your average cost per share is (100 * 100 + 50 * 200)/150 = $133.33, which means you’d book a capital gain of $1400 as a result of your sale. The question of how this capital gain is taxed - short or long-term - naturally arises: how do you even compute the age of the “average” share?</p>
<p>It turns out that there are two types of average cost basis: single category and double category. In the single-category average cost basis, the cost basis is computed with the average method but then the shares are matched in FIFO order. With double category, it’s a little more complicated: the average cost of short term and long-term shares are computed separately and taxed accordingly. Let’s assume that we are using single category average cost basis, in which the 30 shares would map to the 100 shares purchased in 2015, thus booking a long-term capital gain of $1400. We would pay 1400 * .15 = $210 in federal tax.</p>
<p><strong>FIFO cost basis</strong></p>
<p>Next, let’s assume that you went with the FIFO cost basis (the default cost basis for many brokerages, especially for individual stocks). This means that the 30 shares that you sold will come out of the 100 that you bought in 2015, so your capital gain will actually be calculated as 180 * 30 - 100 * 30 = $2400. You’re left paying more taxes than average cost basis. (Generally as the market trends upwards, FIFO would end up causing you to pay more tax than average-cost, ignoring wrinkles caused by short vs long-term tax differences.)</p>
<p><strong>Specific identification</strong></p>
<p>Finally, let’s assume that you used specific identification for your cost basis and thus picked the lot that you wanted to sell. A lot just refers to a fixed quantity of shares purchased at the same price on the same date. Clearly, you want to choose your lots in descending order of their price (ignoring edge cases where the short and long-term tax rates make a difference). So you pick 30 shares from the 50 you purchased in 2019. Now your cost basis is $200 per share, so your capital gain will be (180 - 200) * 30 = -600, i.e., a capital loss of $600. In this exercise, picking the best cost basis over the worst one reduces your tax burden by $3000, a potential savings of $1239 assuming our above federal and state tax rates. Of course this reduction only increases with larger sums invested.</p>
<p>The TLDR of all this is that you generally want to pick specific identification as your cost basis, which will allow you to manually select the lots that you want to sell. It is more work and a little more complicated but the tax savings can be immense.</p>
<p>Note that there is a <strong>major downside</strong> in using the average cost basis method: once you sell a security using this cost basis method you cannot use any other cost basis method while liquidating your holdings in that security. This makes sense, as it would be easy to abuse if you had a bunch of shares with a low average cost, sold using the average cost method, then purchased the shares at a much more expensive price, and then wanted to sell using LIFO to minimize your taxes. Thus, it’s probably better to stick with specific identification and choose each lot that you want to sell in order to minimize your tax burden.</p>
<h5 id="a-catch-automatically-reinvesting-dividends-and-automatic-investments-in-401k-or-ira-accounts">A Catch: Automatically reinvesting dividends and automatic investments in 401(k) or IRA accounts</h5>
<p>Many investors have automatic dividend reinvestment turned on in their brokerage accounts, which is very convenient since the dividends that you receive are automatically reinvested and don’t spend a minute sitting outside of the market. However, this incurs an additional complication with tax loss harvesting.</p>
<p>In the above example, if you sell VTSAX in order to realize your paper losses, but then are paid a dividend which is reinvested into VTSAX (for example, from other VTSAX holdings you have) within 30 days before or after your sale, then you’ll have accidentally triggered a wash sale. However, note that this isn’t the end of the world because the dividend reinvestment will likely only purchase a small amount of shares. In the above cost basis example, if you sell 30 shares but your dividend reinvestment purchases 5 shares, you can still reap the tax loss harvesting benefits on 25 shares.</p>
<p>An easy way to avoid this is to not automatically reinvest dividends, of course. Vanguard funds mostly pay out dividends four times a year, so it’s not too much work to log in and reinvest them every couple of months.</p>
<p>Another catch is the fact that many people have automatic purchases of funds in their IRA/401(k). The IRS has <a href="https://www.irs.gov/irb/2008-03_IRB">clearly stated</a> that if a security is sold in a taxable brokerage account but then a substantially similar asset is purchased in an IRA account (30 days before or after) then it is a wash sale, and thus you can’t claim the losses on your tax returns. It is an open question of whether this applies to 401(k) investments as well, as the IRS has not clearly issued any guidance (the above statement only specifically mentioned IRAs) on this and forums such as <a href="https://www.bogleheads.org/forum/viewtopic.php?t=172622">Bogleheads</a> are divided on the issue. It’s better to be safe than sorry though and avoid possible wash sales incurred in IRAs or 401(k) accounts.</p>
<p><strong>The TLDR</strong></p>
<p>Tax loss harvesting is a sequence of steps that you can take to potentially offset your capital gains or taxed income for many years, depending on the size of the loss, while staying fully invested in the market. Care must be taken to be aware of the tax tradeoffs (extra future capital gains), choose the right cost basis to incur the loss, and avoid wash sales.</p>
<p><strong>Sources</strong></p>
<ol>
<li><a href="https://investor.vanguard.com/taxes/cost-basis/">Vanguard’s explanation of cost basis</a></li>
<li><a href="https://www.schwab.com/resource-center/insights/content/save-on-taxes-know-your-cost-basis">Schwab’s explanation of cost basis</a></li>
<li><a href="https://www.physicianonfire.com/tax-loss-harvesting-vanguard/">Physician on Fire’s Guide to TLH</a></li>
<li><a href="https://www.madfientist.com/tax-loss-harvesting/">Mad Fientist’s guide to TLH</a></li>
<li><a href="https://www.investopedia.com/articles/retirement/09/ira-wash-sale-rule.asp">Investopedia IRS wash sale rule</a></li>
<li><a href="https://www.irs.gov/irb/2008-03_IRB">IRS 2008-3</a></li>
</ol>This blog post is a deviation from the technical content that I usually post, since I thought it may be useful to share my experience with my first time venturing into tax-loss harvesting, especially given the market conditions in response to COVID-19. This post is not meant to be a comprehensive guide for when to tax loss harvest, but more of a series of steps that would need to take when deciding to go through with it, and a rough estimation of the actual benefits. Now for the obligatory disclaimer: this post isn’t intended to provide investment or tax advice!Efficient Matrix Operations through Diagonalizability2019-07-21T00:00:00+00:002019-07-21T00:00:00+00:00https://rohanvarma.me/Diagonalizability<p>In this blog post, I’ll talk about <em>diagonalizability</em>, what it is, and why it may be useful to diagonalize matrices (when they can be) to efficiently compute operations on matrices. I won’t go into detail <em>when</em> a matrix is diagonalizable, but it will be briefly mentioned in an example.</p>
<p>If <script type="math/tex">A</script> is similar to a diagonal matrix <script type="math/tex">D</script>, then <script type="math/tex">A = QDQ^{-1}</script> for some invertible matrix <script type="math/tex">Q</script> and some diagonal matrix <script type="math/tex">% <![CDATA[
D = \begin{bmatrix} \lambda_1 & 0 &… & 0 \\ 0 & \lambda_2 & … & 0 \\ … & … & … & … \\ 0 & 0 & … & \lambda_n \end{bmatrix} %]]></script>, and <script type="math/tex">A</script> is <em>diagonalizable</em>. Only square matrices <script type="math/tex">A \in \mathbb{M_{nxn}}</script> are (possibly) diagonalizable. The matrix <script type="math/tex">Q</script> has columns given by (distinct, linearly independent) eigenvectors of the linear transformation given by <script type="math/tex">A</script>, and <script type="math/tex">D</script> is a diagonal matrix whose diagonal entries contain eigenvalues corresponding to the eigenvectors of the linear transformation given by <script type="math/tex">A</script>.</p>
<p>Being able to diagonalize matrices this way is useful, since it allows for easy and fast computation of functions of <script type="math/tex">A</script>.</p>
<p>For example, if we wanted to compute <script type="math/tex">A^2</script>, then the naive matrix multiplication algorithm will take <script type="math/tex">O(n^3)</script> time, and the best known algorithm is still <script type="math/tex">O(n^{2.3})</script>. However, if we know that <script type="math/tex">A</script> is diagonalizable, then we have:</p>
<script type="math/tex; mode=display">A^2 = QDQ^{-1}QDQ^{-1} \rightarrow{} A^2 = QD^2Q^{-1}</script>
<p>An induction on the power <script type="math/tex">k</script> we raise <script type="math/tex">A</script> to shows that in general <script type="math/tex">A^k = QD^kQ^{-1}</script>.</p>
<p>Next, we show that this makes our life easier, since <script type="math/tex">% <![CDATA[
D^k = \begin{bmatrix} \lambda_1^k & 0 &… & 0 \\ 0 & \lambda_2^k & … & 0 \\ … & … & … & … \\ 0 & 0 & … & \lambda_n^k \end{bmatrix} %]]></script>.</p>
<p>First, let’s consider <script type="math/tex">D^2</script>. We have <script type="math/tex">% <![CDATA[
D^2 = DD = \begin{bmatrix} \lambda_1 & 0 &… & 0 \\ 0 & \lambda_2 & … & 0 \\ … & … & … & … \\ 0 & 0 & … & \lambda_n \end{bmatrix} \begin{bmatrix} \lambda_1 & 0 &… & 0 \\ 0 & \lambda_2 & … & 0 \\ … & … & … & … \\ 0 & 0 & … & \lambda_n \end{bmatrix} %]]></script>.</p>
<p>In general, for a matrix product <script type="math/tex">C = AB</script> we have <script type="math/tex">C_{ij} = \sum_{k=1}^{m} A_{ik}B_{kj}</script> - the <script type="math/tex">(i,j)</script> entry of <script type="math/tex">C</script> is the dot product of the <script type="math/tex">i</script>th row of <script type="math/tex">A</script> and the <script type="math/tex">j</script>th column of <script type="math/tex">B</script>. Therefore, we have <script type="math/tex">D^2_{i,j} = [0, …, \lambda_i, …][0, …, \lambda_j, …]^T</script> so if <script type="math/tex">i \neq j</script> then <script type="math/tex">D^2_{ij} = 0</script>, otherwise <script type="math/tex">D^2_{ij} = \lambda_i^2 = \lambda_j^2</script>.</p>
<p>Thus, we have <script type="math/tex">% <![CDATA[
D^2 = \begin{bmatrix} \lambda_1 & 0 &… & 0 \\ 0 & \lambda_2^2 & … & 0 \\ … & … & … & … \\ 0 & 0 & … & \lambda_n^2 \end{bmatrix} %]]></script> and an inductive argument gives our above result for <script type="math/tex">D^k</script>.</p>
<p>This drastically reduces the complexity of computing <script type="math/tex">A^k</script>: instead of <script type="math/tex">\log(k)</script> matrix multiplications, taking <script type="math/tex">O(\log(k)n^{2.3})</script> with a good algorithm, we have an <script type="math/tex">O(n)</script> pass to exponentiate across the diagonal, followed by two matrix multiplications (the first matrix multiplication with the diagonal matrix is also more efficient, as the rows of <script type="math/tex">Q^{-1}</script> are just scaled), for a complexity of <script type="math/tex">O(n^{2.3})</script> - and the latter complexity does not grow with <script type="math/tex">k</script>.</p>
<p>We can actually make a similar argument for computing any function <script type="math/tex">f</script> with a matrix argument: <script type="math/tex">% <![CDATA[
f(A) = Qf(D)Q^{-1} =Q \begin{bmatrix} f(\lambda_1) & 0 &… & 0 \\ 0 & f(\lambda_2) & … & 0 \\ … & … & … & … \\ 0 & 0 & … & f(\lambda_n) \end{bmatrix}Q^{-1} %]]></script>, when <script type="math/tex">A</script> is diagonalizable.</p>
<p>To show why this is the case, let’s consider the taylor series expansion of <script type="math/tex">f(x)</script>, where <script type="math/tex">x \in \mathbb{R}</script>, around <script type="math/tex">0</script> (we are assuming here that <script type="math/tex">f</script> is infinitely differentiable around <script type="math/tex">0</script>):</p>
<script type="math/tex; mode=display">f(x) = f(0) + f'(0)(x) + \frac{1}{2}f''(0)x^2 + \frac{1}{3!}f'''(0)x^3 + … = \sum_{i=0}^{\infty}\frac{f^{(n)}(0)x^n}{n!}</script>
<p>To bring in the matrix, we can substitute <script type="math/tex">A</script> in place of <script type="math/tex">x</script>, and sums now become matrix sums and products scale the matrix:</p>
<script type="math/tex; mode=display">f(A) = f(0)I_{n} + f'(0)(A) + \frac{1}{2}f''(0)A^2 + \frac{1}{3!}f'''(0)A^3 + …</script>
<p>Next, since <script type="math/tex">A</script> is diagonalizable and <script type="math/tex">A = QDQ^{-1}</script>, we can subsitute:</p>
<script type="math/tex; mode=display">f(A) = f(QDQ^{-1}) = f(0)I_{n} + f'(0)(QDQ^{-1}) + \frac{1}{2}f''(0)(QDQ^{-1})^2 + \frac{1}{3!}f'''(0)(QDQ^{-1})^3 + …</script>
<p>Writing the above expression as a sum yields the following:</p>
<script type="math/tex; mode=display">f(A) = \sum_{i=0}^{\infty} \frac{f^i(0)(QDQ^{-1})^i}{i!}</script>
<p>The term <script type="math/tex">(QDQ^{-1})^i</script> should be familiar - we know from above that that is just <script type="math/tex">QD^iQ^{-1}</script>. Thus, we’re left with <script type="math/tex">f(QDQ^{-1}) = \sum_{i=0}^{\infty} Q\frac{f^i(0)D^i}{i!}Q^{-1} = Q(\sum_{i=0}^{\infty} \frac{f^i(0)D^i}{i!})Q^{-1}</script>.</p>
<p>But <script type="math/tex">f(D) = \sum_{i=0}^{\infty} \frac{f^i(0)D^i}{i!}</script>, so <script type="math/tex">f(QDQ^{-1}) = Qf(D)Q^{-1}</script>.</p>
<p>Next, all that’s left to show is that <script type="math/tex">% <![CDATA[
f(D) = \begin{bmatrix} f(\lambda_1) & 0 &… & 0 \\ 0 & f(\lambda_2) & … & 0 \\ … & … & … & … \\ 0 & 0 & … & f(\lambda_n) \end{bmatrix} %]]></script>. We use a similar approach with the Taylor series expansion around <script type="math/tex">0</script>:</p>
<script type="math/tex; mode=display">f(D) = f(0)I_{n} + f'(0)D + \frac{1}{2}f''(0)D^2 + \frac{1}{3!}f'''(0)D^3 + …</script>
<script type="math/tex; mode=display">% <![CDATA[
= f(0) \begin{bmatrix} 1 & 0 &… & 0 \\ 0 & 1 & … & 0 \\ … & … & … & … \\ 0 & 0 & … & 1 \end{bmatrix} + f'(0)\begin{bmatrix} \lambda_1 & 0 &… & 0 \\ 0 & \lambda_2 & … & 0 \\ … & … & … & … \\ 0 & 0 & … & \lambda_n \end{bmatrix} + \frac{1}{2}f''(0)\begin{bmatrix} \lambda_1^2 & 0 &… & 0 \\ 0 & \lambda_2^2 & … & 0 \\ … & … & … & … \\ 0 & 0 & … & \lambda_n^2 \end{bmatrix} + ... %]]></script>
<p>This is just a bunch of matrix scaling and summation, so let’s move some things inside the matrices:</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{bmatrix} f(0) & 0 &… & 0 \\ 0 & f(0) & … & 0 \\ … & … & … & … \\ 0 & 0 & … & f(0) \end{bmatrix} + \begin{bmatrix} f'(0) \lambda_1 & 0 &… & 0 \\ 0 & f'(0)\lambda_2 & … & 0 \\ … & … & … & … \\ 0 & 0 & … & f'(0)\lambda_n \end{bmatrix} + \begin{bmatrix} \frac{1}{2}f''(0)\lambda_1^2 & 0 &… & 0 \\ 0 & \frac{1}{2}f''(0)\lambda_2^2 & … & 0 \\ … & … & … & … \\ 0 & 0 & … & \frac{1}{2}f''(0)\lambda_n^2 \end{bmatrix} + … %]]></script>
<p>Now, adding up the matrices, we get:</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{bmatrix} \sum_{i=0}^{\infty} \frac{f^i(0)\lambda_1^i}{i!}& 0 &… & 0 \\ 0 & \sum_{i=0}^{\infty}\frac{f^i(0)\lambda_2^i}{i!} & … & 0 \\ … & … & … & … \\ 0 & 0 & … & \sum_{i=0}^{\infty}\frac{f^i(0)\lambda_n^i}{i!} \end{bmatrix} %]]></script>
<p>But each nonzero term is the Taylor series expansion for the corresponding <script type="math/tex">\lambda_i</script>, so this is just</p>
<p><script type="math/tex">% <![CDATA[
f(D) = \begin{bmatrix} f(\lambda_1) & 0 &… & 0 \\ 0 & f(\lambda_2) & … & 0 \\ … & … & … & … \\ 0 & 0 & … & f(\lambda_n) \end{bmatrix} %]]></script>, so the proof is complete.</p>
<p>This is useful since it allows us to compute functions over matrices much more easily, as we saw above with exponentiation. Consider for example <script type="math/tex">f(A) = \exp(A)</script>. A valid notion of exponentiating matrices can be given by defining <script type="math/tex">\exp(A)</script> similar to <script type="math/tex">\exp(x), x \in \mathbb{R}</script>:</p>
<script type="math/tex; mode=display">\exp(A) = A^0 + A^1 + \frac{A^2}{2} + \frac{A^3}{3!} + … = \sum_{i=0}^{\infty} \frac{A^i}{i!}</script>
<p>(As an aside, it turns out that the above sum is a much more natural way to think about the <script type="math/tex">\exp</script> function, instead of thinking of it as raising <script type="math/tex">e</script> to some power. Read <a href="https://www.quora.com/Is-it-just-a-coincidence-that-the-derivative-of-e-x-is-also-e-x-or-is-there-some-feature-in-the-function-due-to-which-this-happens/answer/Alon-Amit">this fascinating Quora answer</a> to find out why.)</p>
<p>Clearly, computing matrix powers like this to approximate <script type="math/tex">\exp(A)</script> is very expensive. But if <script type="math/tex">A</script> is diagonalizable, we have a much less expensive approach to computing <script type="math/tex">\exp(A)</script>:</p>
<script type="math/tex; mode=display">% <![CDATA[
\exp(A) = \exp(QDQ^{-1}) = Q\exp(D)Q^{-1} = Q \begin{bmatrix} \exp(\lambda_1) & 0 &… & 0 \\ 0 & \exp(\lambda_2) & … & 0 \\ … & … & … & … \\ 0 & 0 & … & \exp(\lambda_n) \end{bmatrix}Q^{-1} %]]></script>
<p><strong>An application: predicting the growth of a portfolio through solving linear differential equations</strong></p>
<p>Consider a simple portfolio given by dollars invested into two investments (say IBM stock and a money market fund). Then, this portfolio lives in a two-dimensional vector space <script type="math/tex">P = \begin{bmatrix} Stock \\ Money Market \end{bmatrix}</script>.</p>
<p>Assume further that we expect IBM stock to have a long-term growth rate of 3% (indeed an egregnious assumption for any individual stock), and to issue a 2% dividend every year. We also will assume that the money market fund will provide a consistent 2% return on investment. Let’s say that we initially have $100 invested in IBM stock, and 500 invested in the money market fund.</p>
<p>With these assumptions, the growth of our portfolio every year can be given by a linear transformation <script type="math/tex">T</script> who’s matrix is given by <script type="math/tex">A = [T(v_1), T(v_2)]</script> where <script type="math/tex">v_1 = (1,0), v_2 = (0,1)</script> are the standard basis vectors.</p>
<p>We have <script type="math/tex">T(v_1) = [1.03, 0.02]^T</script> since one dollar of stock returns 1.03 in the stock and 0.02 as a dividend, and <script type="math/tex">T(v_2)= [0, 1.02]^T</script> since one dollar in the money market fund does not return anything in the stock, and returns 1.02 in the fund.</p>
<p>Thus, the matrix of the linear transformation is given by <script type="math/tex">% <![CDATA[
A = \begin{bmatrix} 1.03 & 0 \\ 0.02 & 1.02 \end{bmatrix} %]]></script>, and we can model the growth of our portfolio as a first-order linear differential equation:</p>
<script type="math/tex; mode=display">x'(t) = Ax(t)</script>
<p>If <script type="math/tex">A</script> were diagonalizable, we’d have a simple, exact solution, so we attempt to diagonalize <script type="math/tex">A</script>. Since <script type="math/tex">A</script> is upper triangular, we have that the eigenvalues for <script type="math/tex">A</script> are its diagonal entries <script type="math/tex">\lambda_1 = 1.0, \lambda_2 = 1.02</script>. Since <script type="math/tex">A</script>’s eigenvalues both have algebraic multiplicity one, they must have geometric multiplicity one, and thus <script type="math/tex">A</script> is diagonalizable.</p>
<p>We compute <script type="math/tex">ker(A - 1.03I_2)</script> and <script type="math/tex">ker(A - 1.02I_2)</script> to see that the eigenspace corresponding to <script type="math/tex">\lambda_1</script> and <script type="math/tex">\lambda_2</script> are <script type="math/tex">E_{\lambda_1} = Span(0.5,1)</script> and <script type="math/tex">E_{\lambda_2} = Span(0,1)</script>, respectively.</p>
<p>Thus, we have <script type="math/tex">A = QDQ^{-1}</script> with <script type="math/tex">% <![CDATA[
Q = \begin{bmatrix} 0.5 & 0 \\ 1 & 1 \end{bmatrix}, D = \begin{bmatrix} 1.0 3& 0 \\ 0 & 1.0 2\end{bmatrix} %]]></script>, and <script type="math/tex">x'(t) = QDQ^{-1}x(t)</script>.</p>
<p>To simplify things, we can multiply both sides by <script type="math/tex">Q^{-1}</script> to get <script type="math/tex">(Q^{-1}x(t))' = DQ^{-1}x(t)</script>. Now, we let <script type="math/tex">y(t) = Q^{-1}x(t)</script> so that <script type="math/tex">y'(t) = Dy(t)</script>. Expanding this, we have</p>
<p><script type="math/tex">% <![CDATA[
\begin{bmatrix} y_1'(t) \\ y_2'(t) \end{bmatrix} = \begin{bmatrix} 1.03 & 0 \\ 0 & 1.02 \end{bmatrix}\begin{bmatrix} y_1(t) \\ y_2(t) \end{bmatrix} %]]></script> , and we get the uncoupled differential equations</p>
<p><script type="math/tex">y_1'(t) = 1.03y_1(t)</script>, <script type="math/tex">y_2'(t) = 1.0y2_2(t)</script> which we solve separately to get <script type="math/tex">y_1(t) = C_1e^{1.03t}, y_2(t) = C_2e^{1.02t}</script>, for some constants <script type="math/tex">C_1, C_2</script>. Now, since <script type="math/tex">x(t) = Qy(t)</script>, we recover <script type="math/tex">x(t)</script> by applying <script type="math/tex">Q</script>:</p>
<p><script type="math/tex">% <![CDATA[
x(t) = \begin{bmatrix} 0.5 & 0 \\ 1 & 1 \end{bmatrix} \begin{bmatrix} C_1e^{1.03t} \\ C_2e^{1.02t} \end{bmatrix} %]]></script>. Multiplying, we have that</p>
<p><script type="math/tex">x_1(t) = \frac{1}{2}C_1e^{1.03t}</script> and <script type="math/tex">x_2(t) = C_1e^{1.03t} + C_2e^{1.02t}</script>. Taking into account our initial conditions, we have that <script type="math/tex">x_1(0) = \frac{1}{2}C_1 = 100</script> and <script type="math/tex">x_2(0) = C_1 + C_2 = 500</script> , so we have that <script type="math/tex">C_1 = 200, C_2 = 300</script>. Thus, our final solution is <script type="math/tex">x_1(t) = 100e^{1.03t}</script> and <script type="math/tex">x_2(t) = 200e^{1.03t} + 300e^{1.02t}</script>, in which we can plug in a value for <script type="math/tex">t</script> to estimate the value of our portfolio at that time.</p>In this blog post, I’ll talk about diagonalizability, what it is, and why it may be useful to diagonalize matrices (when they can be) to efficiently compute operations on matrices. I won’t go into detail when a matrix is diagonalizable, but it will be briefly mentioned in an example.Hessians - A tool for debugging neural network optimization2019-03-03T00:00:00+00:002019-03-03T00:00:00+00:00https://rohanvarma.me/Optimization<p>Optimizing deep neural networks has long followed a general tried-and-true template. Generally, we randomly initialize our weights, which can be thought of as randomly picking a place on the “hill” which is the optimization landscape. There are some tricks we can do to achieve better initialization schemes, such as the He or Xavier initialization.</p>
<p>Then, we follow the gradient and update our parameters until we’ve met some stopping criterion. This is known as gradient descent. More commonly, stochastic gradient descent is used to add noise (via approximating the gradient instead of computing the exact gradient) to the optimization process and speed up training time. Additionally, vanilla SGD is less common now, with practitioners instead opting for methods that add in momentum or adaptive learning rate techniques.</p>
<p><img src="https://raw.githubusercontent.com/ucla-labx/deeplearningbook-notes/master/images/along_the_ravine.png" alt="" /></p>
<p>It’s natural to question theoretical - and practical - aspects of this common technique. What guarantees does it provide, and what are some conditions in which it could fail? Here, I’ll attempt to dissect some of these questions so we can better understand what to watch out for when optimizing deep networks.</p>
<h5 id="precursor-convexity-and-the-hessian">Precursor: Convexity and the Hessian</h5>
<p>Say we want to optimize <script type="math/tex">f(x)</script> where <script type="math/tex">x</script> are our parameters that we want to learn. The question of convexity arises. Is <script type="math/tex">f</script> convex, meaning that any local minimum we achieve is the optimal solution to our problem? Or is <script type="math/tex">f</script> nonconvex, in which case local minima can be larger than global minima? It turns out that the Hessian, or the matrix of <script type="math/tex">f</script>’s second derivatives, can tell us a lot about this.</p>
<p>The Hessian is real and symmetric, since in general we assume that the second derivatives exist and <script type="math/tex">\frac{dy}{dx_1 x_2} = \frac{dy}{dx_2 x_1}</script>for the functions that we are considering (Schwarz’s theorem provides the conditions that need to be true for this to hold). For example, the Hessian for a function <script type="math/tex">y = f(x_1, x_2)</script> could be expressed as</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{bmatrix} \frac{dy}{dx_1x_1} & \frac{dy}{dx_1x_2} \\ \frac{dy}{dx_2x_1} & \frac{dy}{dx_2x_2}\end{bmatrix} %]]></script>
<p>Real symmetric matrices have nice properties:</p>
<ul>
<li>All eigenvalues are real and distinct eigenvalues correspond to distinct eigenvectors</li>
<li>The eigenvectors of distinct values are orthogonal, and therefore form a basis for <script type="math/tex">R^n</script>, where <script type="math/tex">n</script> is the dimension of the row/column space of the matrix.</li>
<li>Thus, the matrix is diagonalizable, i.e. <script type="math/tex">Q^{-1} H Q = D</script>, where <script type="math/tex">D</script> is a diagonal matrix with the eigenvalues on the diagonal, and <script type="math/tex">Q</script>’s column vectors form an orthonormal basis for <script type="math/tex">R^n</script>. This is a result of the <a href="http://www.math.lsa.umich.edu/~speyer/417/SpectralTheorem.pdf">Spectral Theorem</a>.</li>
</ul>
<p>Next, a <em>positive definite</em> matrix is a symmetric matrix that has all positive eigenvalues. One way to determine if a function is <em>convex</em> is to check if its Hessian is positive definite.</p>
<p>To show this, it is enough to show that <script type="math/tex">z^T H z > 0</script> for any real vector <script type="math/tex">z</script>. To see why all positive eigenvalues imply this, first let’s consider the case where <script type="math/tex">z</script> is an eigenvector of <script type="math/tex">H</script>. Since <script type="math/tex">Hz = \lambda z</script> we have</p>
<p><script type="math/tex">z^T H z = z^T\lambda z = \lambda z^Tz = \lambda \vert \vert z\vert \vert^2 > 0</script> since <script type="math/tex">\lambda >0</script></p>
<p>To prove this for an arbitrary vector <script type="math/tex">z</script>, we first note that we can diagonalize <script type="math/tex">H</script> as follows:</p>
<script type="math/tex; mode=display">z^T H z = z^T Q \Lambda Q^{-1}z</script>
<p>Where <script type="math/tex">Q</script> is a matrix whose columns are (distinct) eigenvectors of <script type="math/tex">H</script> and <script type="math/tex">\Lambda</script> is a diagonal matrix with the corresponding eigenvalues on its diagonal. We know that this diagonalization is possible since <script type="math/tex">H</script> is real and symmetric.</p>
<p>As mentioned, the eigenvectors are orthogonal. Since <script type="math/tex">Q</script> is a matrix whose columns are the eigenvectors, <script type="math/tex">Q</script> is an orthogonal matrix, so we have <script type="math/tex">Q^{-1} = Q^T</script>, giving us:</p>
<script type="math/tex; mode=display">z^T Q \Lambda Q^Tz >0</script>
<p>Let’s define <script type="math/tex">s = Q^T z</script>, so we now have <script type="math/tex">s^T \Lambda s > 0</script>. Taking</p>
<script type="math/tex; mode=display">s = \begin{bmatrix} s_1 \\ … \\ s_n \end{bmatrix}</script>
<p>and</p>
<script type="math/tex; mode=display">% <![CDATA[
\Lambda = \begin{bmatrix} \lambda_1 & 0 &… & 0 \\ 0 & \lambda_2 & … & 0 \\ … & … & … & … \\ 0 & 0 & … & \lambda_n \end{bmatrix} %]]></script>
<p>We now have</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{bmatrix} s_1 & … & s_n \end{bmatrix} \begin{bmatrix} \lambda_1 & 0 &… & 0 \\ 0 & \lambda_2 & … & 0 \\ … & … & … & … \\ 0 & 0 & … & \lambda_n \end{bmatrix} \begin{bmatrix} s_1 \\ … \\ s_n \end{bmatrix} = \begin{bmatrix} s_1 & … & s_n \end{bmatrix} \begin{bmatrix} \lambda_1 s_1 \\ … \\ \lambda_ns_n \end{bmatrix} = \sum_{i=1}^{N}\lambda_is_i^2 > 0 %]]></script>
<p>Which is true since all the eigenvalues are positive.</p>
<p>As an example of using this analysis to prove the convexity of a machine learning problem, we can take the loss function of the (l2-regularized) SVM:</p>
<script type="math/tex; mode=display">L(w) = \lambda \sum_n w_n^2 + \sum_n \max (0, 1 - y_n(w^Tx_n))</script>
<p>The derivatives for each <script type="math/tex">w_m</script> are <script type="math/tex">% <![CDATA[
\frac{dL}{dw_m} = \lambda w_m + \sum \textbf{1} (y_n w^T x_n < 1)(-y_nx_{nm}) %]]></script>, where <script type="math/tex">\textbf{1}</script> denotes the indicator function that returns the second argument if its first argument is true, and <script type="math/tex">x_{nm}</script> denotes the <script type="math/tex">m</script>th feature of the <script type="math/tex">n</script>th feature vector.</p>
<p>The second derivatives can be characterized as <script type="math/tex">\frac{dL}{dw_m w_k}, k != m</script> and <script type="math/tex">\frac{dL}{dw_m^2}</script>. The latter derivatives will appear as the diagonal entries of the Hessian.</p>
<p>For the first case, since our expression <script type="math/tex">\frac{dL}{dw_m}</script> is constant with respect to <script type="math/tex">w_k</script> the derivative is simply <script type="math/tex">0</script>. For the second case, the derivative is simply <script type="math/tex">\lambda</script>. Therefore, our Hessian is the following diagonal matrix:</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{bmatrix} \lambda & 0 &… & 0 \\ 0 & \lambda & … & 0 \\ … & … & … & … \\ 0 & 0 & … & \lambda \end{bmatrix} %]]></script>
<p>Since this is a diagonal matrix, the eigenvalues are simply the entries on the diagonal. Since the regularization constant <script type="math/tex">\lambda > 0</script>, the Hessian is positive definite, therefore the above formulation of the hinge loss is a convex problem. We could have alternatively shown that <script type="math/tex">z^T H z > 0</script> as well, since <script type="math/tex">H z</script> simply scales <script type="math/tex">z</script> by a positive scalar.</p>
<h5 id="okbut-what-about-neural-networks">Ok…But what about Neural Networks?</h5>
<p>The optimization problem for neural networks are generally highly nonconvex, making optimizing deep neural networks much tougher. Fortunately, there are still some applications of the Hessian that we could use.</p>
<h6 id="local-minima">Local Minima</h6>
<p>First, it may be worth it to know if we are at a local minimum at any point during our optimization process. In some cases, if this occurs early in our training process or at a high value for our objective function, then we could consider restarting our training/initialization process or randomly update our objective to “kick” our parameters out of the bad local minima.</p>
<p>Even though the Hessian may not be positive definite at any given point, we can check if we’re at a local minimum by examining the Hessian. This is basically the second derivative test in single-variable calculus. Consider the Taylor Series expansion of our objective <script type="math/tex">f</script> around <script type="math/tex">x_0</script>:</p>
<script type="math/tex; mode=display">f(x)\approx f(x_0) + (x-x_0)\nabla_xf(x_0) + \frac{1}{2}(x-x_0)^T\textbf{H}(x-x_0)</script>
<p>If we’re at a critical point, it is a potential local minimum and we have <script type="math/tex">\nabla_x f(x) = 0</script>. Considering an SGD update <script type="math/tex">x = x_0 - \epsilon u</script>, we have</p>
<script type="math/tex; mode=display">f(x_0-\epsilon u) \approx f(x_0) + \frac{1}{2}(x_0 - \epsilon u - x_0)^T\textbf{H}(x_0 - \epsilon u - x_0)</script>
<script type="math/tex; mode=display">f(x_0-\epsilon u) \approx f(x_0) + \frac{1}{2}\epsilon^2 u^T\textbf{H} u</script>
<p>If the Hessian is positive definite at <script type="math/tex">x_0</script>, then it tells us that any direction <script type="math/tex">u</script> in which we choose to travel will result an increased value of the objective function (since the term <script type="math/tex">\frac{1}{2}\epsilon^2 u^T\textbf{H} u > 0</script>), so we are at a local minimum. We can similarly use the idea of negative definiteness to see if we are at a local maximum.</p>
<p><strong>Are local minima really that big of a problem?</strong></p>
<p>Arguably, the goal of fitting deep neural networks may not be to reach the global minimum, as this could result in overfitting, but rather find an acceptable local minima that obains good performance on the test set. In <a href="https://arxiv.org/pdf/1412.0233.pdf">this paper</a>, the authors state that “while local minima are numerous, they are relatively easy to find, and they are all more or less equivalent in terms of performance on the test set”. In the <a href="http://www.deeplearningbook.org/">Deep Learning Book</a>, Goodfellow et al. state the early convergence to a high value for the objective is generally <em>not</em> due to local minima, and this can be verified by plotting the norm of the gradient throughout the training process. The gradient at these supposed local minima exhibit healthy norms, indicating that the point is not actually a local minima.</p>
<h6 id="a-poorly-conditioned-hessian">A Poorly Conditioned Hessian</h6>
<p>There are some cases where the regular optimization process - a small step in the direction of the gradient - could actually fail, and increase the value of our objective function - couteractive to our goal. This happens due to the <em>curvature</em> of the local region of our objective function, and can be investigated with the Hessian as well.</p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/rohan-blog/gh-pages/curvature.png" alt="" /></p>
<p>How much will a gradient descent update change the value of our objective? First, let’s again look at the Taylor series expansion for an update <script type="math/tex">x \leftarrow x_0 - \epsilon \textbf{g}</script> where <script type="math/tex">\textbf{g} = \nabla f(x_0)</script>:</p>
<script type="math/tex; mode=display">f(x_0 - \epsilon\textbf{g})\approx f(x_0) + (x_0 - \epsilon \textbf{g}-x_0)\textbf{g} + \frac{1}{2}(x_0 - \epsilon \textbf{g}-x_0)^T\textbf{H}(x_0 - \epsilon \textbf{g}-x_0)</script>
<script type="math/tex; mode=display">f(x_0 - \epsilon\textbf{g})\approx f(x_0) -\epsilon\textbf{g}^T\textbf{g} + \frac{1}{2}\epsilon^2\textbf{g}^T\textbf{H}\textbf{g}</script>
<p>We can see that the decrease in the objective is related to 2nd order information, specifically the term <script type="math/tex">g^T H g</script>. If this term is negative, then the decrease in our objective is greater than our (scaled) squared gradient norm, if it is positive, then the decrease is less, if it is zero, the decrease is completely given by first-order information.</p>
<p>“Poor Conditioning” refers to the Hessian at the point <script type="math/tex">x_0</script> being such that it results in an <em>increased</em> value of our objective function. This can happen if the term that indicates our decrease in our objective is actually greater than <script type="math/tex">0</script>:</p>
<p><script type="math/tex">-\epsilon g^Tg + \frac{1}{2}\epsilon^2g^T Hg > 0</script> which corresponds to <script type="math/tex">% <![CDATA[
\epsilon g^T g < \frac{1}{2}\epsilon^2g^THg %]]></script></p>
<p>This can happen if the Hessian is very large, in which case we’d want to use a smaller learning rate than <script type="math/tex">\epsilon</script> to offset some of the influence of the curvature. An intuitive justification for this is that due to the large magnitude of curvature, we’re more “unsure” of our gradient updates, so we want to take a smaller step, and therefore have a smaller learning rate.</p>
<h6 id="saddle-points">Saddle Points</h6>
<p>Getting stuck at a <em>saddle point</em> is a very real issue for optimizing deep neural networks, and arguably a more important issue than getting stuck at a local minima (see <a href="http://www.offconvex.org/2016/03/22/saddlepoints/">http://www.offconvex.org/2016/03/22/saddlepoints/</a>). One explanation of this is that it’s much more likely to arrive at a saddle point than a local minimum or maximum: the probability of a given point in an <script type="math/tex">n</script> dimensional optimization space being a local minimum or maximum is just <script type="math/tex">\frac{1}{2^n}</script>.</p>
<p>A saddle point is defined as a point with <script type="math/tex">0</script> gradient, but the Hessian is neither positive definite or negative definite. It is possible that learning would stop if we are relying only on first-order information (i.e. since <script type="math/tex">\nabla_x f(x) = 0</script>, the weights will not be updated) but in practice, using techniques such as momentum reduce the chance of this.</p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/rohan-blog/gh-pages/saddle.png" alt="" /></p>
<p>Around a saddle point, the expansion of our objective is similar to the expansion at a local minimum:</p>
<script type="math/tex; mode=display">f(x_0-\epsilon u) \approx f(x_0) + \frac{1}{2}\epsilon^2 u^T\textbf{H} u</script>
<p>Here, <script type="math/tex">\textbf{H}</script> is not positive definite, so we may be able to pick certain directions <script type="math/tex">u</script> that increase or decrease the value of our objective. Concretely, if we pick <script type="math/tex">u</script> such that <script type="math/tex">% <![CDATA[
u^T \textbf{H}u < 0 %]]></script>, then <script type="math/tex">u</script> is a direction that would decrease the value of our objective (since the term <script type="math/tex">% <![CDATA[
\frac{1}{2}\epsilon^2 u^T\textbf{H} u < 0 %]]></script> decreases the value of our objective) , and we can update our parameters with <script type="math/tex">u</script>. Ideally we’d like to find a <script type="math/tex">u</script> such that <script type="math/tex">u^T\textbf{H}u</script> is significantly less than <script type="math/tex">0</script>, so that we have a steeper direction of descent to travel in.</p>
<p>How do we know what functions have saddle points that are “well-behaved” like this? Ge et al. in their <a href="https://arxiv.org/abs/1503.02101">paper</a> introduced the notion of “strict saddle” functions. One of the properties of these “strict saddle” functions is that at all saddle points <script type="math/tex">x</script>, the Hessian has a (significant) negative eigenvalue, which means that we can pick the eigenvector corresponding to this eigenvalue as our direction to travel in if we are at a saddle point.</p>
<p>However, it is often infeasible to compute the Hessian while training deep learning models, since it is computationally expensive. How can we escape saddle points using only first-order information?</p>
<p>Get et al. in their paper also describe a variant of stochastic gradient descent which they call “noisy stochastic gradient descent”. The only variant is that some random noise is added to the gradient:</p>
<script type="math/tex; mode=display">\textbf{g} = \nabla_\theta \frac{1}{m}\sum_{i=1}^{m} L(f(x^i; \theta), y^i)</script>
<script type="math/tex; mode=display">\theta_{t+1} \leftarrow{} \theta_{t} - \epsilon (\textbf{g} + \nu)</script>
<p>where <script type="math/tex">\nu</script> is random noise sampled from the unit sphere. This ensures that there is noise in every direction, allowing this variant of SGD to explore the region around the saddle point. The authors show that this noise can help escape from saddle points for strict-saddle functions.</p>
<p>The authors mention in the <a href="http://www.offconvex.org/2016/03/22/saddlepoints/">corresponding blog post</a> that it is useful to think of a saddle point as <em>unstable</em>, so slight perturbations can be effective in order to kick your weights out of the saddle point. To me this indicates that any perturbation in addition to the regular SGD update could be beneficial to continue optimization at a saddle point, so techniques such as momentum would be helpful as well.</p>
<h6 id="computing-the-hessian">Computing the Hessian</h6>
<p>Most of these methods discussed involve the need to compute the Hessian, which is often unrealistic in practice due to the aformented reasons. However, we can use <em>finite difference methods</em> in order to approximate the Hessian, which may be enough to do our job. To understand this, first we can consider the limit definition of the derivative in the single-variable case:</p>
<script type="math/tex; mode=display">\frac{d}{dx} f(x) = \lim_{h \rightarrow{} 0} \frac{f(x + h) - f(x)}{h}</script>
<p>This means an approximation for <script type="math/tex">\frac{d}{dx}f(x)</script> can be given by</p>
<script type="math/tex; mode=display">\frac{f(x + h) - f(x)}{h}</script>
<p>where our approximation improves with a smaller <script type="math/tex">h</script> but we run into numerical stability issues if we take <script type="math/tex">h</script> to be too small. It turns out that a slightly better approximation would be to use the centered version of the above (see the explanation on <a href="http://cs231n.github.io/neural-networks-3/#gradcheck">CS 231n</a> for details):</p>
<script type="math/tex; mode=display">\frac{d}{dx}f(x) \approx \frac{f(x + h) - f(x-h)}{2h}</script>
<p>For vector-valued functions, we can generalize this idea to several dimensions:</p>
<script type="math/tex; mode=display">\frac{d}{dx_i}f(x) \approx \frac{f(x + h s_i) - f(x - h s_i)}{2h}</script>
<p>where <script type="math/tex">s_i</script> is a vector that has <script type="math/tex">1</script> as its <script type="math/tex">i</script>th entry, and is zero everywhere else. We’re basically taking the unit vector pointing in that direction, scaling it so that it is small, and adding that to our setting of parameters given by the vector <script type="math/tex">x</script>.</p>
<p>It turns out that this identity can be further generalized, to approximate the gradient-vector product (i.e. the product of the gradient times a given vector). We simply remove the constraint that <script type="math/tex">s_i</script> is the unit vector pointing in a particular direction, and instead let it be any vector <script type="math/tex">u</script>. This lets us approximate the gradient vector product:</p>
<script type="math/tex; mode=display">\nabla_x f(x)^T u \approx \frac{f(x + h u) - f(x - h u)}{2h}</script>
<p>It’s quite simple to extend this to computing the Hessian-vector product: given the gradient, the gradient of the gradient is the Hessian. This means that we can replace the gradient on the left hand side of the above equation with the Hessian, and the function on the right hand side with the gradient:</p>
<script type="math/tex; mode=display">H u \approx \frac{\nabla_xf(x + h u) - \nabla_xf(x - h u)}{2h}</script>
<p>which gives us the Hessian-vector product.</p>
<h6 id="takeaways">Takeaways</h6>
<p>The Hessian can give us useful second order information when optimizing machine learning algorithms, though it is computationally tough to compute in practice. By analyzing the Hessian, we may be able to get information regarding the convex nature of our problem, and it can also help us determine local minima or “debug” gradient descent when it actually fails to reduce our cost function or gets stuck at a saddle point.</p>
<h6 id="sources">Sources</h6>
<ol>
<li><a href="http://www.offconvex.org/2016/03/22/saddlepoints/">Escaping from Saddle Points</a></li>
<li><a href="https://arxiv.org/abs/1503.02101">Paper: Escaping from Saddle Points - Online SGD for Tensor Decomposition</a></li>
<li><a href="http://www.deeplearningbook.org/">Deep Learning Book Ch. 8</a></li>
<li><a href="https://timvieira.github.io/blog/post/2014/02/10/gradient-vector-product/">Explanation of approximating gradient-vector products</a></li>
</ol>
<h5 id="notes">Notes</h5>
<p>[4/7/19] - Added further explanation for the claim that “The eigenvectors of distinct values are orthogonal, and therefore form a basis for <script type="math/tex">R^n</script>”</p>
<p>[4/7/19] - Added a note about the indicator function notation that I used when explaining the L2-regularized SVM.</p>
<p>[5/7/19] - Fixed an incorrect equation and added some clarifications.</p>
<p>[5/30/19] - Fixed the gradient in the SVM example.</p>
<p>[6/7/19] - Added a note about the Spectral Theorem.</p>Optimizing deep neural networks has long followed a general tried-and-true template. Generally, we randomly initialize our weights, which can be thought of as randomly picking a place on the “hill” which is the optimization landscape. There are some tricks we can do to achieve better initialization schemes, such as the He or Xavier initialization.ResNets2019-02-17T00:00:00+00:002019-02-17T00:00:00+00:00https://rohanvarma.me/ResNetCIFAR<p><img src="https://raw.githubusercontent.com/rohan-varma/resnet-implementation/master/images/verydeep_network.png" alt="" /></p>
<p>These are some notes that I took while reading the paper <a href="https://arxiv.org/abs/1512.03385">Deep Residual Learning for Image Recognition</a>, the paper that introduced modern ResNets. A mock implementation of the network described for the CIFAR-10 portion of the paper is available <a href="https://github.com/rohan-varma/resnet-implementation">here</a>.</p>
<h5 id="1-introduction">1. Introduction</h5>
<ul>
<li>
<p>Main motivation: very deep neural networks are harder to fit</p>
<ul>
<li>
<p>Have higher training error on CIFAR 10 - so learning is not as simple as stacking more layers as it was once thought</p>
</li>
<li>
<p>Degredation problem is because it is difficult to fit very deep networks (despite batch normalization and He/Xavier initialization methods), they don’t just overfit, they actually have worse training error</p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/resnet-implementation/master/images/verydeep_network.png" alt="" /></p>
</li>
</ul>
</li>
<li>
<p>Intuitively, deep networks should’t be “harder” to fit. If there is a certain number of layers <script type="math/tex">N</script> that achieve optimal accuracy on a dataset, then the layers after N could just learn the identity mapping (i.e. each layer computes their mapping as <script type="math/tex">H(x) = x</script> where <script type="math/tex">H(x)</script> is the mapping of the layer to be learned), and then the network will effectively have their final output at layer <script type="math/tex">N</script></p>
</li>
<li>
<p>However, it is not “easy”for weights to be pushed in such ways that they exactly produce the identity mapping</p>
</li>
<li>
<p>Authors introduce the idea of <strong>residual learning</strong> - instead of directly approximating the underlying mapping we want, <script type="math/tex">H(x)</script>, we instead learn a residual function <script type="math/tex">H(x) - x</script>. This is done by making the output of a stack of layers be <script type="math/tex">y = F(x) + x</script>, where <script type="math/tex">F(x)</script> is the output of the layers (before the ReLU of the last layer) and then the original input <script type="math/tex">x</script> is element-wise added:</p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/resnet-implementation/master/images/residual_learning_block.png" alt="" /></p>
</li>
<li>
<p>Therefore, if our underlying mapping is still <script type="math/tex">y = H(x)</script> that we want to learn, then <script type="math/tex">F(x) = H(x) - x</script> so that <script type="math/tex">y = F(x) + x = H(x) - x + x = H(x)</script> .</p>
<ul>
<li>The idea of learning identity mappings is now easier, since we just need to set all weights to <script type="math/tex">0</script>, so that <script type="math/tex">H(x) = 0</script> and <script type="math/tex">F(x) = -x</script>, so <script type="math/tex">y = x</script> is learned</li>
</ul>
</li>
<li>
<p>Ensemble of ResNets attained <strong>3.57%</strong> top-5 error rate on ImageNet dataset</p>
<ul>
<li>Six total ResNets of different dimension, 2 152-layer ResNets are used</li>
</ul>
</li>
</ul>
<h5 id="2-related-work">2. Related Work</h5>
<ul>
<li>Auxiliary classifiers inserted at early layers of a deep network to send back stronger gradient signals to deal w/vanishing gradient problems are similar</li>
<li>Inception network which uses concatenations of different operations includes a shortcut connection</li>
</ul>
<h5 id="3-deep-residual-learning">3. Deep Residual Learning</h5>
<ul>
<li>
<p>Say we want to learn <script type="math/tex">y = H(x)</script>. We can either cdirectly learn this or try to learn <script type="math/tex">F(x)</script> where <script type="math/tex">F(x)= H(x)-x</script> and formulate our output as <script type="math/tex">y = F(x) + x</script>. So by addign the identity in a so-called “residual block” we force the nework to learn a residual mapping <script type="math/tex">F(x) = H(x) -x</script>.</p>
</li>
<li>
<p>A <em>building block</em> in ResNet is defined as <script type="math/tex">y = F(x_i, {W_i}) + x</script> where <script type="math/tex">F</script> can be multiple layers. For example, the above figure has <script type="math/tex">F = W_2(\sigma (W_1 x))</script></p>
<ul>
<li>The <script type="math/tex">+</script> operation is performed by a shortcut operation and element-wise addition</li>
</ul>
</li>
<li>
<p>Described like this, the shortcut connection introduces no new parameters in a network, so training the network in this way doesn’t introduce an increase in training time due to the numbers of parameters that must be trained. But this isn’t possible when dimensionalities are different - for example, the 2 layers of conv/relu above may result in the output before the addition having different dimension than that of <script type="math/tex">x</script>.</p>
</li>
<li>
<p>To handle this, we can use a projection matrix <script type="math/tex">W_s</script> that projects <script type="math/tex">x</script> to the same space as <script type="math/tex">F(x)</script>. We have <script type="math/tex">y = F(x, {W_i}) + W_sx</script>, but this introduces more parameters into the network.</p>
</li>
<li>
<p>These functions are just as applicable to convlutional layers as they are to FC layers. For examples, <script type="math/tex">F</script> can represent multiple conv layers, and the element-wise addition is performed on the two feature maps, going channel by channel (so the dims must be the same)</p>
</li>
<li>
<p><strong>Residual Network details</strong></p>
<ul>
<li>
<p>Based off a plain 34-layer network that has a <script type="math/tex">7 * 7</script> conv, then a series of <script type="math/tex">3*3</script> convs gradually increasing the channel size, followed by a global average pooling layer, followed by a <script type="math/tex">1000</script> way FC layer + softmax at the end that represents <script type="math/tex">p(y \vert{} x)</script></p>
</li>
<li>
<p>The residual network is similar , but shortcut connectins are added every 2 layers, and the network looks as follows:</p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/resnet-implementation/master/images/resnet_layout.png" alt="" /></p>
</li>
<li>
<p>Two options when dimensions don’t map:</p>
<ul>
<li>Projection matrix (as mentioned above), or just padding extra zeros to increase dimension (doesn’t increase number of parameters) are both tried out</li>
</ul>
</li>
<li>
<p>Downsampling is directly performed with the stride size, which is <script type="math/tex">2</script> in all of the conv layers.</p>
</li>
<li>
<p>Design rules: If the output of a conv layer has the same feature map size, the layers have the same number of filters, but if the size of the feature map is halved, then the number of filters is doubled, so as to preserve the time complexity per layer.</p>
</li>
<li>
<p>Implementation details:</p>
<ul>
<li>224 x 224 crops sampled from ImageNet dataset, with per-pixel mean subtracted</li>
<li>Data augmentation: images are flipped to increase dataset size</li>
<li>Batch norm is used, the pattern is conv-bn-relu, so before the activation</li>
<li><em>He</em> initialization of weights is used, namely weights are initialized from sampling from a Gaussian with mean <script type="math/tex">0</script> and standard deviation <script type="math/tex">\sqrt{\frac{2}{n_l}}</script>. Biases are initialized to be <script type="math/tex">0</script>.</li>
<li>SGD with minibatch size 256 is used</li>
<li>Learning rate starts off as <script type="math/tex">0.1</script> and is then decreased by dividing by <script type="math/tex">10</script> when the error plateus.</li>
<li><script type="math/tex">60 * 10^4</script> total iterations</li>
<li>Weight decay of <script type="math/tex">0.0001</script> and momentum of <script type="math/tex">0.9</script> is used</li>
<li>Dropout is <em>not</em> used, in favor of only batchnorm.</li>
</ul>
</li>
</ul>
</li>
</ul>
<h5 id="4-training-and-approach">4. Training and Approach</h5>
<ul>
<li>
<p>Trained 18 and 34 layer plain networks, along with 18 and 34 layer ResNets</p>
<ul>
<li>It was shown that 34 layer plain nets have higher training error than 18 layer nets, and it was argued that this was not due to vanishing gradients, because: 1) proper initialization was used, 2) BN was used, 3) it was ensured that gradients have healthy norms throughout training</li>
<li>Speculated that deep networks have exponentially lowe convergence rates (i.e. need to be trained for much longer to achieve same results compared to shallower networks)</li>
</ul>
</li>
<li>
<p>For 18 and 24 layer ResNets, simple element-wise addition shortcut additions were used, so there were no new parameters in the network</p>
<ul>
<li>34 layer ResNet did better compared to 18 layer, indicating that the degredation problem observed in shallow nets was not evident here</li>
</ul>
</li>
<li>
<p><strong>Identity vs projection shortcuts</strong></p>
<ul>
<li>3 types:
<ul>
<li>A: zero-padding shortcuts used when dimensions do not match, all shortcuts are parameter free</li>
<li>B: Projection shortcuts used when dimensions do not match, and other shortcuts are regular element-wise addition</li>
<li>C: All shorcuts are projections (meaning that a square matrix is used even when the dimensions match)</li>
</ul>
</li>
<li>It was shown that B is slightly better than A and C was slightly better than B, but C introduces more parameters and increases the time/memory complexity of the network, so B was used overall (projections when dimensions do not match, otherwise regular identity and element-wise addition)</li>
</ul>
</li>
<li>
<p><strong>Bottleneck architecture</strong></p>
<ul>
<li>
<p>For every residual function <script type="math/tex">F</script>, 3 layers instead of 2 are used: first layer is a 1x1 conv, then a 3x3 conv, then a 1x1 conv</p>
<ul>
<li>The 1x1 layers reduce and increase dimensionality, and the 3x3 conv operates on a smaller dimensional space</li>
</ul>
</li>
<li>
<p>Exampe: in the following figure, a <script type="math/tex">256</script> dimensional (256 channels) input is fed into a 1x1 which maps it to 64 channels, then a 3x3 which maps it to 64 channels, and then 1 x 1 that maps it back to the original dimensionality of 256 channels.</p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/resnet-implementation/master/images/bottleneck.png" alt="" /></p>
</li>
<li>
<p>Parameter-free shortcuts here are particularly important, the time complexity and model size are doubled if identity shortcuts are replaced with projection</p>
</li>
<li>
<p>This architecture is used to create 50/101/152 layer ResNets, which all had improved accuracy compared to the 34 layer ResNets, and the degredation problem is not observed</p>
<ul>
<li>152-layer ResNet performed the best</li>
</ul>
</li>
</ul>
</li>
</ul>
<h5 id="resnets-on-cifar-10">ResNets on CIFAR-10</h5>
<ul>
<li>Network inputs are 32 * 32 with per-pixel mean subtracted</li>
<li>First layer: 3 x 3 conv, then stack of <script type="math/tex">6n</script> layers with <script type="math/tex">3*3</script> convolutions with feature map sizes of 32, 16, and 8. Each feature map size has <script type="math/tex">2n</script> layers for <script type="math/tex">6n</script> total layers.
<ul>
<li>This means that the output feature map size is 32 twice, then 16 twice, etc</li>
</ul>
</li>
<li>Number of filters are 16, 32, 64, respectively. Subsampling is done with conv layers of stride 2 instead of max/average pooling throughout the network (which is the traditional way of downsampling)</li>
<li>Global average pooling after all the conv layers, and then a 10-way fully connected layer + softmax at the end</li>
<li>Identity shortcuts used in all cases</li>
<li>Weight decay: <script type="math/tex">0.0001</script>, momentum of <script type="math/tex">0.9</script>, with He init, BN, and no dropout, with a batch size of <script type="math/tex">128</script>.</li>
<li>Learning rate of <script type="math/tex">0.1</script> which is divided by <script type="math/tex">10</script> at 32k and 48k iterations, and training is terminated at 64k iterations</li>
<li>110 layer network achieved <script type="math/tex">6.43</script>% error, which is state of the art</li>
<li>Noticed that deeper ResNets have a smaller magnitute of responses, where a response is the standard deviation of layer responses for each layer (i.e. the responses in layers of the ResNets generally have lower standard deviations compared to plain networks)</li>
<li>1202 layer network did not work well (had similar training error, but higher testing error, indicatign overfitting)
<ul>
<li>Not much regularization was used in these ResNets (i.e. no maxout or dropout), regularization is just imposed by the architecture of the design</li>
</ul>
</li>
</ul>The Python GIL2019-02-06T00:00:00+00:002019-02-06T00:00:00+00:00https://rohanvarma.me/GIL<h4 id="a-story">A story</h4>
<p>Let’s imagine that you’re trying to optimize some code that computes statistics about how much time users spend on your website. You have a list of objects for each user - where each object itself is a list representing the amount of time a user spent on your website across several sessions. Your psuedocode looks something like this:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">get_time_spent</span><span class="p">(</span><span class="n">users</span><span class="p">):</span>
<span class="n">user_to_time</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">for</span> <span class="p">(</span><span class="n">user_id</span><span class="p">,</span> <span class="n">user_sessions</span><span class="p">)</span> <span class="ow">in</span> <span class="n">users</span><span class="p">:</span>
<span class="n">total_time_spent</span> <span class="o">=</span> <span class="nb">sum</span><span class="p">(</span><span class="n">user_sessions</span><span class="p">)</span>
<span class="n">user_to_time</span><span class="p">[</span><span class="n">user_id</span><span class="p">]</span> <span class="o">=</span> <span class="n">total_time_spent</span>
<span class="k">return</span> <span class="n">user_to_time</span>
</code></pre></div></div>
<p>This code works, but you’re concerned about the time it takes, since the list of <code class="language-plaintext highlighter-rouge">users</code> and <code class="language-plaintext highlighter-rouge">user_session</code> is very large. You remember threads and know that this code will run on an 8-core machine, so you decide to split up the <code class="language-plaintext highlighter-rouge">users</code> list into 8 chunks and spin up 8 different threads. You quickly conclude that your code will now run about 8x faster, save for any overhead taken up in chunking the list and thread management - you don’t have to worry about things like locking since each thread is working on its own component of the data.</p>
<p>All you have to do is create the threads, and tell each one to run the above <code class="language-plaintext highlighter-rouge">get_time_spent</code> function on their distinct list of <code class="language-plaintext highlighter-rouge">users</code>:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">threaded_time_spent</span><span class="p">(</span><span class="n">users</span><span class="p">):</span>
<span class="n">user_chunks</span> <span class="o">=</span> <span class="n">chunk_list</span><span class="p">(</span><span class="n">users</span><span class="p">,</span> <span class="mi">8</span><span class="p">)</span>
<span class="n">threads</span> <span class="o">=</span> <span class="p">[</span><span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">get_time_spent</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">chunk</span><span class="p">,))</span> <span class="k">for</span> <span class="n">chunk</span> <span class="ow">in</span> <span class="n">user_chunks</span><span class="p">]</span>
<span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="n">threads</span><span class="p">:</span>
<span class="n">t</span><span class="p">.</span><span class="n">start</span><span class="p">()</span>
<span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="n">threads</span><span class="p">:</span>
<span class="n">t</span><span class="p">.</span><span class="n">join</span><span class="p">()</span>
</code></pre></div></div>
<p>After coding this up, you run some basic performance tests and realize that your new code actually takes <em>longer</em> to execute than your old code - a far cry from the 8x performance gain you had anticipated.</p>
<p>You’re confused, and begin to look for a bug in your threading implementation, but find nothing. After doing some research, you come across the Global Interpreter Lock, and realize that it effectively serializes your code, even though you wanted to be able to take advantage of multiple cores.</p>
<p>What the hell is that, and why does it even exist in Python?</p>
<h4 id="what-does-the-gil-do">What does the GIL do?</h4>
<p>Simply put, the GIL is a lock around the interpreter. Any thread wishing to execute Python bytecode must hold the GIL in order to do so. This means that at most one thread can be executing Python bytecode at any given moment. This effectively serializes portions of multithreaded programs where each thread is executing Python bytecode. To allow other threads to run, the thread holding the GIL releases it periodically - both voluntarily when it no longer needs it and involuntarily after a certain interval.</p>
<h4 id="why-does-python-have-a-gil">Why does Python have a GIL?</h4>
<p>First, as an aside, Python doesn’t technically have a GIL - the reference implementation, CPython, has one. Other implementations of Python - such as Jython and IronPython - don’t have a GIL, and have different tradeoffs than CPython due to it. In this post, I’ll mostly be focusing on the GIL as it is implemented in CPython.</p>
<p>The GIL was originally introduced as part of effort to support multithreaded programming in Python. Python uses automatic memory management via garbage collection, implemented with a technique called reference counting. Python internally manages a data structure containing all object references that can be accessed by a program, and when an object has zero references, it can be freed.</p>
<p>However, race conditions in multithreaded programming made it so that the count of these references could be updated incorrectly, making it so that objects could be erroneously freed or never freed at all. One way to solve this problem is with more granular locking, such as around every shared object, but this would create issues such as increased overhead due to a lot of lock acquire/release requests, as well as increase the possibility of deadlock. The Python developers instead chose to solve this problem by placing a lock around the entire interpreter, making each thread acquire this lock when it runs Python bytecode. This avoids a lot of the performance issues around excessive locking, but effectively serializes bytecode execution.</p>
<h4 id="how-does-the-gil-impact-performance">How does the GIL impact performance?</h4>
<p>Let’s consider two examples to illustrate the difference in performance for CPU-bound and I/O bound threads. First, we’ll create a dummy CPU bound task that just counts down to zero from an input. We’ll also define two implementations that call this function twice - one that just makes two successive calls, and another that spawns two threads that run this function, and then <code class="language-plaintext highlighter-rouge">join</code>s them:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">count</span><span class="p">(</span><span class="n">n</span><span class="p">):</span>
<span class="k">while</span> <span class="n">n</span> <span class="o">></span> <span class="mi">0</span><span class="p">:</span>
<span class="n">n</span><span class="o">-=</span><span class="mi">1</span>
<span class="o">@</span><span class="n">report_time</span>
<span class="k">def</span> <span class="nf">run_threaded</span><span class="p">():</span>
<span class="n">t1</span> <span class="o">=</span> <span class="n">threading</span><span class="p">.</span><span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">count</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="mi">10000000</span><span class="p">,))</span>
<span class="n">t2</span> <span class="o">=</span> <span class="n">threading</span><span class="p">.</span><span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">count</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="mi">10000000</span><span class="p">,))</span>
<span class="n">t1</span><span class="p">.</span><span class="n">start</span><span class="p">()</span>
<span class="n">t2</span><span class="p">.</span><span class="n">start</span><span class="p">()</span>
<span class="n">t1</span><span class="p">.</span><span class="n">join</span><span class="p">()</span>
<span class="n">t2</span><span class="p">.</span><span class="n">join</span><span class="p">()</span>
<span class="o">@</span><span class="n">report_time</span>
<span class="k">def</span> <span class="nf">run_sequential</span><span class="p">():</span>
<span class="n">count</span><span class="p">(</span><span class="mi">10000000</span><span class="p">)</span>
<span class="n">count</span><span class="p">(</span><span class="mi">10000000</span><span class="p">)</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">report_time</code> <a href="https://github.com/rohan-varma/python-gil/blob/master/time_decorator.py">decorator</a> is a simple decorator that uses the <code class="language-plaintext highlighter-rouge">time</code> module to report how long the function took to execute. Running this script 10 times and averaging the result gave me an average of 1.53 seconds for sequential execution, and 1.57 seconds for threaded execution (see <a href="https://github.com/rohan-varma/python-gil/blob/master/output.txt">results</a>) - meaning that despite having a 4-core machine, threading here did not help at all, and in fact marginally worsened performance.</p>
<p>Now let’s consider two I/O bound threads instead. The following <a href="https://github.com/rohan-varma/python-gil/blob/master/gil_test_io_bound.py">code</a> runs the <code class="language-plaintext highlighter-rouge">select</code> function on empty lists of file descriptors, and times out after 2 seconds:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def run_select():
a, b, c = select.select([], [], [], 2)
@report_time
def run_threaded():
t1 = threading.Thread(target=run_select)
t2 = threading.Thread(target=run_select)
t1.start()
t2.start()
t1.join()
t2.join()
@report_time
def run_sequential():
run_select()
run_select()
_, threaded_time = run_threaded()
_, seq_time = run_sequential()
print(f'{threaded_time} with threading, {seq_time} sequentially')
</code></pre></div></div>
<p>As expected, the sequential execution takes 4 seconds, and the threaded execution takes 2 seconds. In the threaded model, the I/O can be awaited in parallel. When the first thread runs, it grabs the GIL, loads in the <code class="language-plaintext highlighter-rouge">select</code> name and the empty lists along with the constant <code class="language-plaintext highlighter-rouge">2</code>, and then makes an I/O request. Python’s implementation of blocking I/O operations drops the GIL and then reacquires the GIL when the thread runs again after the I/O completes, so another thread is free to execute. In this case, the other thread can run almost instantly, making a call to <code class="language-plaintext highlighter-rouge">select</code> as well. This process of yielding the GIL during blocking I/O operations is similar to <em>cooperative multitasking</em>.</p>
<p>On the other hand, CPU-bound threads don’t voluntarily yield the CPU to other threads, and instead have to be forcibly switched out to allow other threads to run. In Python 2, there was a concept of “interpreter ticks”, and the GIL is forcibly dropped every 100 ticks if the thread hasn’t voluntarily given it up. In Python 3, this was switched (and the GIL was in general revamped) to be a time interval, which is by default 5 milliseconds. Both intervals can be reset with <code class="language-plaintext highlighter-rouge">sys.setcheckinterval()</code> and <code class="language-plaintext highlighter-rouge">sys.setswitchinterval()</code> respectively.</p>
<p>The main takeaway is that the GIL only inhibits performence when you have several CPU-bound threads that you’d like to execute across multiple cores, since the GIL serializes them. Parellizing I/O bound work via multiple threads is still a good bet.</p>
<h4 id="working-around-pythons-gil">Working around Python’s GIL</h4>
<p>There are still a few things that we can do to potentially optimize performance of CPU-bound Python code across multiple threads, despite the GIL. The first thing to consider is using the <code class="language-plaintext highlighter-rouge">Process</code> module. <code class="language-plaintext highlighter-rouge">Process</code> has an API similar to that of <code class="language-plaintext highlighter-rouge">thread</code>, but it spawns an entirely separate process to run your function. The good news is that the new process has its own interpreter, so you can take advantage of multiple cores on your machine. <a href="https://github.com/rohan-varma/python-gil/blob/master/gil_test_multiprocessing.py">Porting the above threaded code</a> to use <code class="language-plaintext highlighter-rouge">Process</code> shows the expected ~2x performance gain. However, processes are much heavier than threads, which brings in efficiency concerns if you’re creating a nontrivial number of separate processes. In addition, there’s more overhead involved in using IPC mechanisms rather than simply communicating with shared variables with threads.</p>
<p>As a more involved task, it may be worth considering porting your CPU-bound code to C, and then writing a C extension to bridge your Python code to the C code. This can provide significant performance advantages, and several scientific computing libraries such as <code class="language-plaintext highlighter-rouge">numpy</code> and <code class="language-plaintext highlighter-rouge">hashlib</code> release the GIL in their C extensions.</p>
<p>By default, code that runs as part of a call to a C extension is still subject to the GIL being held, but it can be manually released. This is common for blocking I/O operations as well as processor intensive computations. Python makes this easy to do with two macros: <code class="language-plaintext highlighter-rouge">Py_BEGIN_ALLOW_THREADS</code> and <code class="language-plaintext highlighter-rouge">Py_END_ALLOW_THREADS</code>, which save the thread state and drop the GIL, and restore the thread state and reacquire the GIL, respectively.</p>
<h4 id="details-around-the-gil-implementation">Details around the GIL implementation</h4>
<p>The GIL is implemented in <a href="https://github.com/python/cpython/blob/27e2d1f21975dfb8c0ddcb192fa0f45a51b7977e/Python/ceval_gil.h#L12">ceval_gil.h</a> and used by the interpreter in <a href="https://github.com/python/cpython/blob/master/Python/ceval.c">ceval.c</a>. A thread waiting for the GIL will do a timed wait on the GIL, with a <a href="https://github.com/python/cpython/blob/27e2d1f21975dfb8c0ddcb192fa0f45a51b7977e/Python/ceval_gil.h#L12">preset interval</a> that can be modified with <code class="language-plaintext highlighter-rouge">sys.setswitchinterval</code>. If the GIL hasn’t been released at all during that interval, then a <a href="https://github.com/python/cpython/blob/27e2d1f21975dfb8c0ddcb192fa0f45a51b7977e/Python/ceval_gil.h#L216">drop request</a> will be sent to the current running thread (which has the GIL). This is done via <code class="language-plaintext highlighter-rouge">COND_TIMED_WAIT</code> in the <a href="https://github.com/python/cpython/blob/27e2d1f21975dfb8c0ddcb192fa0f45a51b7977e/Python/ceval_gil.h#L209">source code</a>, which sets a <code class="language-plaintext highlighter-rouge">timed_out</code> <a href="https://github.com/python/cpython/blob/27e2d1f21975dfb8c0ddcb192fa0f45a51b7977e/Python/ceval_gil.h#L89">variable</a> that indicates if the wait has timed out.</p>
<p>The running thread will finish the instruction that it’s on, drop the GIL, and <a href="https://github.com/python/cpython/blob/27e2d1f21975dfb8c0ddcb192fa0f45a51b7977e/Python/ceval_gil.h#L166">signal on a condition variable that the GIL is available</a>. This is encapsulated by the <code class="language-plaintext highlighter-rouge">drop_gil</code> <a href="https://github.com/python/cpython/blob/master/Python/ceval.c#L1030">function</a>.</p>
<p>Importantly, it will also <a href="https://github.com/python/cpython/blob/27e2d1f21975dfb8c0ddcb192fa0f45a51b7977e/Python/ceval_gil.h#L173-L187">wait for a signal</a> that another thread was able to get the GIL and run. This is done by checking if the last holder of the GIL was the thread itself, and if so, resetting the GIL drop request and waiting on a condition variable that signals that the GIL has been switched.</p>
<p>This wasn’t the case in Python 2, where a thread that just dropped the GIL could potentially compete for it again. This would often result in starvation of certain threads, due to how the OS would schedule these threads. For example, if you had two cores, then the thread dropping the GIL (let’s call this t1) could still be running on one core, and the thread attempting to acquire the GIL (let’s call this t2) could be scheduled on the 2nd core. What could happen is that since t1 is still running, it could re-acquire the GIL before t2 even gets a chance to wake up and see that it can acquire the GIL, so t2 will continue to block since it wasn’t able to acquire the GIL, and then wake up and try again repeatedly. This would result in something dubbed the GIL battle:</p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/python-gil/master/gil_battle.png" alt="" /><em>The GIL battle (<a href="https://www.dabeaz.com/python/NewGIL.pdf">source</a>)</em></p>
<p>This would frequently happen for CPU-bound threads left running on a core in Python 2. This could result in I/O bound threads being starved and lots of extra signalling on condition variables, reducing performance. <a href="https://www.dabeaz.com/python/GIL.pdf">These slides</a> have some more details about the Python 2 GIL.</p>
<p>There’s one important subtelty in the case of multiple threads. Since Python doesn’t have its own thread scheduling and wraps POSIX/Windows threads, scheduling of threads is left up to the OS. Therefore, when multiple threads are competing to run, the thread that issued the GIL <code class="language-plaintext highlighter-rouge">drop_request</code> may not actually be the thread that acquires the GIL (since a context switch could occur, another waiting thread could see that the GIL is available, and acquire it). <a href="https://www.dabeaz.com/python/NewGIL.pdf">These slides</a> have some more details about this.</p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/python-gil/master/new_gil_multiple_threads.png" alt="" /><em>GIL behavior with multiple threads (<a href="https://www.dabeaz.com/python/NewGIL.pdf">source</a>)</em></p>
<p>What could (but doesn’t) happen is that the thread that was unable to acquire the GIL, but still timed out on waiting for it, could continue to issue the <code class="language-plaintext highlighter-rouge">drop_request</code> and attempt to re-acquire the GIL. This would essentially be like a spin lock - the thread would keep polling for the GIL and demanding for it to be released, using up CPU to accomplish nothing.</p>
<p>Instead, on a time out, a check is also done to see if the GIL has switched in that time interval (i.e. to another thread). If so, then this thread simply goes to sleep waiting for the GIL again. While this does in a sense reduce “fairness” (in the sense that the thread that requested the GIL should get it next), it also dramatically reduces GIL contention, compared to Python 2, and is somewhat of a necessary tradeoff since Python doesn’t control when its threads run.</p>
<p>One criticism about the new GIL relates to the above point of the “most deserving” thread (i.e. the thread that sent the <code class="language-plaintext highlighter-rouge">drop_request</code>) getting the GIL. This is largely because thread scheduling is entirely up to the OS, but also has some repercussions for I/O operations that complete very quickly. Since any I/O operation will release the GIL, CPU-bound threads will restart and use their entire time slice before yielding back, and the I/O bound thread has to go through the entire timeout process to re-acquire the GIL. This can create somewhat of a convoy effect of quick-running I/O operations having to queue up to acquire the GIL:</p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/python-gil/master/newgil_convoy.png" alt="" /><em>Convoy effect with the new GIL (<a href="https://www.dabeaz.com/python/UnderstandingGIL.pdf">source</a>)</em></p>
<h4 id="summary">Summary</h4>
<p>The GIL is an interesting part of Python, and it’s cool to see the different tradeoffs and optimizations that were done in both Python 2 and Python 3 to improve performance as it relates to the GIL. The seemingly small changes to Python 3’s GIL (such as the time-based, as opposed to tick interval and reduction of GIL contention) emphasizes just how important and nuanced issues such as lock contention and thread switching are, and how hard they are to get right.</p>
<h3 id="sources">Sources</h3>
<ol>
<li><a href="https://www.dabeaz.com/python/NewGIL.pdf">Talk on the new GIL</a></li>
<li><a href="https://www.dabeaz.com/python/GIL.pdf">Talk on the old GIL</a></li>
<li><a href="https://www.dabeaz.com/python/UnderstandingGIL.pdf">Another talk on the new GIL</a></li>
<li><a href="https://opensource.com/article/17/4/grok-gil">Grokking the GIL</a></li>
<li><a href="https://realpython.com/python-gil/">About the Python GIL</a></li>
</ol>A storyHow Much do I use my Phone?2019-01-02T00:00:00+00:002019-01-02T00:00:00+00:00https://rohanvarma.me/Phone-Usage<p>Towards the end of 2017, I started using an iOS app called <a href="https://inthemoment.io/">Moment</a>, which tracks how much time you spent on your phone each day and how many times you pick it up. Through using this application for the year of 2018 and poking around in the app for a way to export my day-by-day data, I was able to obtain a <a href="https://github.com/rohan-varma/phone-usage-tracking/blob/master/data/moment.json">JSON file</a> consisting of my phone usage time and number of pickups for every day of the year.</p>
<p>I decided to do some exploring to figure out just how much I’ve been using my phone on a daily basis, and see if there are any daily, weekly, or monthly differences - i.e. did I use my phone more on the weekends or on the weekdays? What follows is a Jupyter notebook that I created for analyzing this data and coming up with some plots, as well as a bit of analysis.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># imports
</span><span class="kn">import</span> <span class="nn">json</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="n">np</span>
<span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="n">plt</span>
<span class="o">%</span><span class="n">matplotlib</span> <span class="n">inline</span>
<span class="c1"># a class to manage the phone usage data for a particular day
</span><span class="k">class</span> <span class="nc">Day</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">day_dict</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">minutes</span> <span class="o">=</span> <span class="n">day_dict</span><span class="p">[</span><span class="s">'minuteCount'</span><span class="p">]</span>
<span class="bp">self</span><span class="p">.</span><span class="n">pickups</span> <span class="o">=</span> <span class="n">day_dict</span><span class="p">[</span><span class="s">'pickupCount'</span><span class="p">]</span>
<span class="c1"># get the date and save if it is a weekday or not
</span> <span class="bp">self</span><span class="p">.</span><span class="n">date</span> <span class="o">=</span> <span class="n">day_dict</span><span class="p">[</span><span class="s">'date'</span><span class="p">].</span><span class="n">split</span><span class="p">(</span><span class="s">'T'</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span>
<span class="bp">self</span><span class="p">.</span><span class="n">is_weekday</span> <span class="o">=</span> <span class="n">datetime</span><span class="p">.</span><span class="n">strptime</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">date</span><span class="p">,</span> <span class="s">'%Y-%M-%d'</span><span class="p">).</span><span class="n">weekday</span><span class="p">()</span> <span class="o"><</span> <span class="mi">5</span>
<span class="k">def</span> <span class="nf">__repr__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="s">'minutes: {0}, pickups: {1}, date: {2}'</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">minutes</span><span class="p">,</span> <span class="bp">self</span><span class="p">.</span><span class="n">pickups</span><span class="p">,</span> <span class="bp">self</span><span class="p">.</span><span class="n">date</span><span class="p">)</span>
<span class="c1"># open and deserialize json, convert into day objects
</span><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">'data/moment.json'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">json</span><span class="p">.</span><span class="n">load</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="n">day_data</span> <span class="o">=</span> <span class="nb">next</span><span class="p">(</span><span class="nb">iter</span><span class="p">(</span><span class="n">data</span><span class="p">.</span><span class="n">values</span><span class="p">()))</span>
<span class="n">days</span> <span class="o">=</span> <span class="p">[</span><span class="n">Day</span><span class="p">(</span><span class="n">d</span><span class="p">)</span> <span class="k">for</span> <span class="n">d</span> <span class="ow">in</span> <span class="n">day_data</span><span class="p">]</span>
<span class="c1"># filter out non 2018
</span><span class="n">days</span> <span class="o">=</span> <span class="p">[</span><span class="n">d</span> <span class="k">for</span> <span class="n">d</span> <span class="ow">in</span> <span class="n">days</span> <span class="k">if</span> <span class="s">'2018'</span> <span class="ow">in</span> <span class="n">d</span><span class="p">.</span><span class="n">date</span><span class="p">]</span>
</code></pre></div></div>
<p><br />
Here is what some of the raw JSON data coming from the Moment app looks like:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">print</span><span class="p">(</span><span class="n">day_data</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{'pickupCount': 69, 'pickups': [], 'date': '2018-12-30T00:00:00+11:00',
'minuteCount': 181, 'appUsages': [], 'sessions': []}
</code></pre></div></div>
<p>To attempt to understand the overall data, we can find the mean and standard deviation of how many minutes per day I used my phone, as well as plot a histogram.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">minute_data</span> <span class="o">=</span> <span class="p">[</span><span class="n">d</span><span class="p">.</span><span class="n">minutes</span> <span class="k">for</span> <span class="n">d</span> <span class="ow">in</span> <span class="n">days</span><span class="p">]</span>
<span class="n">mean_time</span><span class="p">,</span> <span class="n">time_std</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">mean</span><span class="p">(</span><span class="n">minute_data</span><span class="p">),</span> <span class="n">np</span><span class="p">.</span><span class="n">std</span><span class="p">(</span><span class="n">minute_data</span><span class="p">)</span>
<span class="c1"># hourly bins
</span><span class="n">bins</span> <span class="o">=</span> <span class="p">[</span><span class="n">i</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="nb">max</span><span class="p">(</span><span class="n">minute_data</span><span class="p">)</span> <span class="o">+</span> <span class="mi">60</span><span class="p">,</span> <span class="mi">60</span><span class="p">)]</span>
<span class="c1"># plot overall usage
</span><span class="n">n</span><span class="p">,</span> <span class="n">bins</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="n">plt</span><span class="p">.</span><span class="n">hist</span><span class="p">([</span><span class="n">minute_data</span><span class="p">],</span> <span class="n">bins</span><span class="o">=</span><span class="n">bins</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s">'Minutes of Phone Usage'</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">xticks</span><span class="p">(</span><span class="n">bins</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s">'Frequency (# of Days)'</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">title</span><span class="p">(</span><span class="s">'Histogram of Phone Usage Time'</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">text</span><span class="p">(</span><span class="mi">300</span><span class="p">,</span> <span class="mi">50</span><span class="p">,</span> <span class="s">r'$\mu={0:.2f},\ \sigma={1:.2f}$'</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">mean_time</span><span class="p">,</span> <span class="n">time_std</span><span class="p">))</span>
<span class="n">plt</span><span class="p">.</span><span class="n">grid</span><span class="p">(</span><span class="bp">True</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">show</span><span class="p">()</span>
<span class="c1"># hour-by-hour data
</span><span class="n">bin_ranges</span> <span class="o">=</span> <span class="p">[(</span><span class="n">bins</span><span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="n">bins</span><span class="p">[</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">])</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">bins</span><span class="p">)</span><span class="o">-</span><span class="mi">1</span><span class="p">)]</span>
<span class="n">hours_to_num_days</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span><span class="nb">zip</span><span class="p">(</span><span class="n">bin_ranges</span><span class="p">,</span> <span class="n">n</span><span class="p">))</span>
<span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">hours_to_num_days</span><span class="p">.</span><span class="n">items</span><span class="p">()):</span>
<span class="k">print</span><span class="p">(</span><span class="s">'Between {0} and {1} hours of usage: {2} days'</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">k</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">/</span><span class="mi">60</span><span class="p">,</span> <span class="n">k</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">/</span><span class="mi">60</span><span class="p">,</span> <span class="nb">int</span><span class="p">(</span><span class="n">v</span><span class="p">)))</span>
</code></pre></div></div>
<p><img src="https://raw.githubusercontent.com/rohan-varma/phone-usage-tracking/master/How%20Much%20do%20I%20use%20my%20Phone%3F_files/How%20Much%20do%20I%20use%20my%20Phone%3F_6_0.png" alt="png" /></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Between 0.0 and 1.0 hours of usage: 33 days
Between 1.0 and 2.0 hours of usage: 154 days
Between 2.0 and 3.0 hours of usage: 121 days
Between 3.0 and 4.0 hours of usage: 42 days
Between 4.0 and 5.0 hours of usage: 8 days
Between 5.0 and 6.0 hours of usage: 3 days
Between 6.0 and 7.0 hours of usage: 1 days
Between 7.0 and 8.0 hours of usage: 1 days
Between 8.0 and 9.0 hours of usage: 1 days
</code></pre></div></div>
<p>It looks like I spent an average of about 2 hours and 6 minutes on my phone each day, with a standard deviation of 1 hour and 2 minutes. This is slightly lower than the <a href="https://hackernoon.com/how-much-time-do-people-spend-on-their-mobile-phones-in-2017-e5f90a0b10a6">average time per day</a> spent on their phones by American adults, which comes in at 2 hours and 51 minutes.</p>
<p>In other words, I spent about 8.75% of my entire day on my phone. If you only consider waking hours and assume 8 hours of sleep per day, then I spent about 13% of my waking hours using my phone each day. Translated to a year, this means I spent a whopping 766.25 hours on my phone in 2018, or 31.93 days - more than an entire month!</p>
<p>Another interesting thing to look at is the variability in my phone usage. Most days, I was around one to three hours of phone usage per day - this accounts for about 75% of all days of the year. However, there were a couple days with more than 6+ hours of phone usage per day, which increased the variability in my phone usage. Looking back, I think that this makes sense, as I do use my phone a lot on days when I’m traveling or on a road trip, or if I’m just really bored that day and don’t feel like doing anything else.</p>
<p>Let’s look at some more data, such as whether there’s a difference between weekdays and weekends.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># separate weekdays and weekends, and plot each
</span><span class="n">weekdays</span> <span class="o">=</span> <span class="p">[</span><span class="n">d</span><span class="p">.</span><span class="n">minutes</span> <span class="k">for</span> <span class="n">d</span> <span class="ow">in</span> <span class="n">days</span> <span class="k">if</span> <span class="n">d</span><span class="p">.</span><span class="n">is_weekday</span><span class="p">]</span>
<span class="n">weekends</span> <span class="o">=</span> <span class="p">[</span><span class="n">d</span><span class="p">.</span><span class="n">minutes</span> <span class="k">for</span> <span class="n">d</span> <span class="ow">in</span> <span class="n">days</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">d</span><span class="p">.</span><span class="n">is_weekday</span><span class="p">]</span>
<span class="n">weekday_mean</span><span class="p">,</span> <span class="n">weekend_mean</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">mean</span><span class="p">(</span><span class="n">weekdays</span><span class="p">),</span> <span class="n">np</span><span class="p">.</span><span class="n">mean</span><span class="p">(</span><span class="n">weekends</span><span class="p">)</span>
<span class="n">weekday_std</span><span class="p">,</span> <span class="n">weekend_std</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">std</span><span class="p">(</span><span class="n">weekdays</span><span class="p">),</span> <span class="n">np</span><span class="p">.</span><span class="n">std</span><span class="p">(</span><span class="n">weekends</span><span class="p">)</span>
<span class="n">n</span><span class="p">,</span> <span class="n">bins</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="n">plt</span><span class="p">.</span><span class="n">hist</span><span class="p">(</span><span class="n">weekdays</span><span class="p">,</span> <span class="n">bins</span><span class="o">=</span><span class="n">bins</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s">'Minutes of Phone Usage on Weekdays'</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">xticks</span><span class="p">(</span><span class="n">bins</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s">'Frequency (# of Days)'</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">title</span><span class="p">(</span><span class="s">'Weekday Phone Usage Time'</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">text</span><span class="p">(</span><span class="mi">300</span><span class="p">,</span> <span class="mi">50</span><span class="p">,</span> <span class="s">r'$\mu={0:.2f},\ \sigma={1:.2f}$'</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">weekday_mean</span><span class="p">,</span> <span class="n">weekday_std</span><span class="p">))</span>
<span class="n">plt</span><span class="p">.</span><span class="n">grid</span><span class="p">(</span><span class="bp">True</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">show</span><span class="p">()</span>
<span class="n">n</span><span class="p">,</span> <span class="n">bins</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="n">plt</span><span class="p">.</span><span class="n">hist</span><span class="p">(</span><span class="n">weekends</span><span class="p">,</span> <span class="n">bins</span><span class="o">=</span><span class="n">bins</span><span class="p">,</span> <span class="n">facecolor</span><span class="o">=</span><span class="s">'orange'</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s">'Minutes of Phone Usage on Weekends'</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">xticks</span><span class="p">(</span><span class="n">bins</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s">'Frequency (# of Days)'</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">title</span><span class="p">(</span><span class="s">'Weekend Phone Usage Time'</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">text</span><span class="p">(</span><span class="mi">300</span><span class="p">,</span> <span class="mi">12</span><span class="p">,</span> <span class="s">r'$\mu={0:.2f},\ \sigma={1:.2f}$'</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">weekend_mean</span><span class="p">,</span> <span class="n">weekend_std</span><span class="p">))</span>
<span class="n">plt</span><span class="p">.</span><span class="n">grid</span><span class="p">(</span><span class="bp">True</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">show</span><span class="p">()</span>
</code></pre></div></div>
<p><img src="https://raw.githubusercontent.com/rohan-varma/phone-usage-tracking/master/How%20Much%20do%20I%20use%20my%20Phone%3F_files/How%20Much%20do%20I%20use%20my%20Phone%3F_8_0.png" alt="png" /></p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/phone-usage-tracking/master/How%20Much%20do%20I%20use%20my%20Phone%3F_files/How%20Much%20do%20I%20use%20my%20Phone%3F_8_1.png" alt="png" /></p>
<p>This was really interesting to me - the distributions for my weekday and weekend phone usage are quite similar, indicating that there’s little difference in my phone usage on a weekend or weekday. This ran counter to my hypothesis that I’d use my phone a lot more on weekends, given that I don’t have class or work. Next, lets see if there’s any difference in phone usage on different days of the week, different weeks, and different months.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># separate each day of the week, and plot each.
</span><span class="n">mon</span> <span class="o">=</span> <span class="p">[</span><span class="n">d</span><span class="p">.</span><span class="n">minutes</span> <span class="k">for</span> <span class="n">d</span> <span class="ow">in</span> <span class="n">days</span> <span class="k">if</span> <span class="n">datetime</span><span class="p">.</span><span class="n">strptime</span><span class="p">(</span><span class="n">d</span><span class="p">.</span><span class="n">date</span><span class="p">,</span> <span class="s">'%Y-%M-%d'</span><span class="p">).</span><span class="n">weekday</span><span class="p">()</span> <span class="o">==</span> <span class="mi">0</span><span class="p">]</span>
<span class="n">tues</span> <span class="o">=</span> <span class="p">[</span><span class="n">d</span><span class="p">.</span><span class="n">minutes</span> <span class="k">for</span> <span class="n">d</span> <span class="ow">in</span> <span class="n">days</span> <span class="k">if</span> <span class="n">datetime</span><span class="p">.</span><span class="n">strptime</span><span class="p">(</span><span class="n">d</span><span class="p">.</span><span class="n">date</span><span class="p">,</span> <span class="s">'%Y-%M-%d'</span><span class="p">).</span><span class="n">weekday</span><span class="p">()</span> <span class="o">==</span> <span class="mi">1</span><span class="p">]</span>
<span class="n">wed</span> <span class="o">=</span> <span class="p">[</span><span class="n">d</span><span class="p">.</span><span class="n">minutes</span> <span class="k">for</span> <span class="n">d</span> <span class="ow">in</span> <span class="n">days</span> <span class="k">if</span> <span class="n">datetime</span><span class="p">.</span><span class="n">strptime</span><span class="p">(</span><span class="n">d</span><span class="p">.</span><span class="n">date</span><span class="p">,</span> <span class="s">'%Y-%M-%d'</span><span class="p">).</span><span class="n">weekday</span><span class="p">()</span> <span class="o">==</span> <span class="mi">2</span><span class="p">]</span>
<span class="n">thurs</span> <span class="o">=</span> <span class="p">[</span><span class="n">d</span><span class="p">.</span><span class="n">minutes</span> <span class="k">for</span> <span class="n">d</span> <span class="ow">in</span> <span class="n">days</span> <span class="k">if</span> <span class="n">datetime</span><span class="p">.</span><span class="n">strptime</span><span class="p">(</span><span class="n">d</span><span class="p">.</span><span class="n">date</span><span class="p">,</span> <span class="s">'%Y-%M-%d'</span><span class="p">).</span><span class="n">weekday</span><span class="p">()</span> <span class="o">==</span> <span class="mi">3</span><span class="p">]</span>
<span class="n">fri</span> <span class="o">=</span> <span class="p">[</span><span class="n">d</span><span class="p">.</span><span class="n">minutes</span> <span class="k">for</span> <span class="n">d</span> <span class="ow">in</span> <span class="n">days</span> <span class="k">if</span> <span class="n">datetime</span><span class="p">.</span><span class="n">strptime</span><span class="p">(</span><span class="n">d</span><span class="p">.</span><span class="n">date</span><span class="p">,</span> <span class="s">'%Y-%M-%d'</span><span class="p">).</span><span class="n">weekday</span><span class="p">()</span> <span class="o">==</span> <span class="mi">4</span><span class="p">]</span>
<span class="n">sat</span> <span class="o">=</span> <span class="p">[</span><span class="n">d</span><span class="p">.</span><span class="n">minutes</span> <span class="k">for</span> <span class="n">d</span> <span class="ow">in</span> <span class="n">days</span> <span class="k">if</span> <span class="n">datetime</span><span class="p">.</span><span class="n">strptime</span><span class="p">(</span><span class="n">d</span><span class="p">.</span><span class="n">date</span><span class="p">,</span> <span class="s">'%Y-%M-%d'</span><span class="p">).</span><span class="n">weekday</span><span class="p">()</span> <span class="o">==</span> <span class="mi">5</span><span class="p">]</span>
<span class="n">sun</span> <span class="o">=</span> <span class="p">[</span><span class="n">d</span><span class="p">.</span><span class="n">minutes</span> <span class="k">for</span> <span class="n">d</span> <span class="ow">in</span> <span class="n">days</span> <span class="k">if</span> <span class="n">datetime</span><span class="p">.</span><span class="n">strptime</span><span class="p">(</span><span class="n">d</span><span class="p">.</span><span class="n">date</span><span class="p">,</span> <span class="s">'%Y-%M-%d'</span><span class="p">).</span><span class="n">weekday</span><span class="p">()</span> <span class="o">==</span> <span class="mi">6</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">plot</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">title</span><span class="p">):</span>
<span class="k">global</span> <span class="n">bins</span>
<span class="n">n</span><span class="p">,</span> <span class="n">bins</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="n">plt</span><span class="p">.</span><span class="n">hist</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">bins</span><span class="o">=</span><span class="n">bins</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">title</span><span class="p">(</span><span class="n">title</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">xticks</span><span class="p">(</span><span class="n">bins</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s">'Frequency (# of Days)'</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s">'Phone Usage Time'</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">text</span><span class="p">(</span><span class="mi">300</span><span class="p">,</span> <span class="mi">12</span><span class="p">,</span> <span class="s">r'$\mu={0:.2f},\ \sigma={1:.2f}$'</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">np</span><span class="p">.</span><span class="n">mean</span><span class="p">(</span><span class="n">data</span><span class="p">),</span> <span class="n">np</span><span class="p">.</span><span class="n">std</span><span class="p">(</span><span class="n">data</span><span class="p">)))</span>
<span class="n">plt</span><span class="p">.</span><span class="n">grid</span><span class="p">(</span><span class="bp">True</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">show</span><span class="p">()</span>
<span class="n">plot</span><span class="p">(</span><span class="n">data</span><span class="o">=</span><span class="n">mon</span><span class="p">,</span> <span class="n">title</span><span class="o">=</span><span class="s">'Minutes of Phone Usage on Monday'</span><span class="p">)</span>
<span class="n">plot</span><span class="p">(</span><span class="n">data</span><span class="o">=</span><span class="n">tues</span><span class="p">,</span> <span class="n">title</span><span class="o">=</span><span class="s">'Minutes of Phone Usage on Tuesday'</span><span class="p">)</span>
<span class="n">plot</span><span class="p">(</span><span class="n">data</span><span class="o">=</span><span class="n">wed</span><span class="p">,</span> <span class="n">title</span><span class="o">=</span><span class="s">'Minutes of Phone Usage on Wednesday'</span><span class="p">)</span>
<span class="n">plot</span><span class="p">(</span><span class="n">data</span><span class="o">=</span><span class="n">thurs</span><span class="p">,</span> <span class="n">title</span><span class="o">=</span><span class="s">'Minutes of Phone Usage on Thursday'</span><span class="p">)</span>
<span class="n">plot</span><span class="p">(</span><span class="n">data</span><span class="o">=</span><span class="n">fri</span><span class="p">,</span> <span class="n">title</span><span class="o">=</span><span class="s">'Minutes of Phone Usage on Friday'</span><span class="p">)</span>
<span class="n">plot</span><span class="p">(</span><span class="n">data</span><span class="o">=</span><span class="n">sat</span><span class="p">,</span> <span class="n">title</span><span class="o">=</span><span class="s">'Minutes of Phone Usage on Saturday'</span><span class="p">)</span>
<span class="n">plot</span><span class="p">(</span><span class="n">data</span><span class="o">=</span><span class="n">sun</span><span class="p">,</span> <span class="n">title</span><span class="o">=</span><span class="s">'Minutes of Phone Usage on Sunday'</span><span class="p">)</span>
<span class="c1"># plot overall for mean comparison
</span><span class="n">means</span> <span class="o">=</span> <span class="p">[</span><span class="n">np</span><span class="p">.</span><span class="n">mean</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="p">[</span><span class="n">mon</span><span class="p">,</span> <span class="n">tues</span><span class="p">,</span> <span class="n">wed</span><span class="p">,</span> <span class="n">thurs</span><span class="p">,</span> <span class="n">fri</span><span class="p">,</span> <span class="n">sat</span><span class="p">,</span> <span class="n">sun</span><span class="p">]]</span>
<span class="k">print</span><span class="p">(</span><span class="n">means</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">bar</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="mi">7</span><span class="p">),</span> <span class="n">means</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">title</span><span class="p">(</span><span class="s">'Average Phone Usage for Day of Week'</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s">'Average Phone Usage'</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s">'Day of Week'</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">show</span><span class="p">()</span>
</code></pre></div></div>
<p><img src="https://raw.githubusercontent.com/rohan-varma/phone-usage-tracking/master/How%20Much%20do%20I%20use%20my%20Phone%3F_files/How%20Much%20do%20I%20use%20my%20Phone%3F_10_0.png" alt="png" /></p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/phone-usage-tracking/master/How%20Much%20do%20I%20use%20my%20Phone%3F_files/How%20Much%20do%20I%20use%20my%20Phone%3F_10_1.png" alt="png" /></p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/phone-usage-tracking/master/How%20Much%20do%20I%20use%20my%20Phone%3F_files/How%20Much%20do%20I%20use%20my%20Phone%3F_10_2.png" alt="png" /></p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/phone-usage-tracking/master/How%20Much%20do%20I%20use%20my%20Phone%3F_files/How%20Much%20do%20I%20use%20my%20Phone%3F_10_3.png" alt="png" /></p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/phone-usage-tracking/master/How%20Much%20do%20I%20use%20my%20Phone%3F_files/How%20Much%20do%20I%20use%20my%20Phone%3F_10_4.png" alt="png" /></p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/phone-usage-tracking/master/How%20Much%20do%20I%20use%20my%20Phone%3F_files/How%20Much%20do%20I%20use%20my%20Phone%3F_10_5.png" alt="png" /></p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/phone-usage-tracking/master/How%20Much%20do%20I%20use%20my%20Phone%3F_files/How%20Much%20do%20I%20use%20my%20Phone%3F_10_6.png" alt="png" /></p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/phone-usage-tracking/master/How%20Much%20do%20I%20use%20my%20Phone%3F_files/How%20Much%20do%20I%20use%20my%20Phone%3F_10_8.png" alt="png" /></p>
<p>We can see that there’s a lot of similarity between the days of the weeks, though it looks like on average, I use my phone less on Thursdays, Fridays, and Sundays, while I use it comparatively more on Tuesdays, Wednesdays, and Saturdays. Overall, we can see that each day’s distribution is similar, taking on a mean of around two hours and a standard deviaton of around an hour. Let’s examine weekly usage now.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># extract weeks from the year by sorting days and going by sevens
</span><span class="n">ordered_days</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="nb">reversed</span><span class="p">(</span><span class="n">days</span><span class="p">))</span>
<span class="n">weeks</span> <span class="o">=</span> <span class="p">[</span><span class="n">ordered_days</span><span class="p">[</span><span class="n">i</span><span class="p">:</span><span class="n">i</span><span class="o">+</span><span class="mi">7</span><span class="p">]</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">ordered_days</span><span class="p">),</span> <span class="mi">7</span><span class="p">)]</span>
<span class="n">weekly_usages</span> <span class="o">=</span> <span class="p">[</span><span class="nb">sum</span><span class="p">(</span><span class="n">d</span><span class="p">.</span><span class="n">minutes</span> <span class="k">for</span> <span class="n">d</span> <span class="ow">in</span> <span class="n">week</span><span class="p">)</span> <span class="k">for</span> <span class="n">week</span> <span class="ow">in</span> <span class="n">weeks</span><span class="p">]</span>
<span class="c1"># plot each week's usage in a bar graph
</span><span class="n">plt</span><span class="p">.</span><span class="n">bar</span><span class="p">([</span><span class="n">i</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">weekly_usages</span><span class="p">))],</span><span class="n">weekly_usages</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s">'Week of the Year'</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s">'Phone Usage Minutes'</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">title</span><span class="p">(</span><span class="s">'Week-by-Week Phone Usage Minutes'</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">show</span><span class="p">()</span>
<span class="c1"># plot weekly usage histogram
</span><span class="n">n</span><span class="p">,</span> <span class="n">bins</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="n">plt</span><span class="p">.</span><span class="n">hist</span><span class="p">(</span><span class="n">weekly_usages</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">title</span><span class="p">(</span><span class="s">'Weekly Phone Usage Distribution'</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">xticks</span><span class="p">(</span><span class="n">bins</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s">'Frequency (# of Weeks)'</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s">'Weekly Phone Usage Time'</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">text</span><span class="p">(</span><span class="mi">950</span><span class="p">,</span> <span class="mi">12</span><span class="p">,</span> <span class="s">r'$\mu={0:.2f},\ \sigma={1:.2f}$'</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span>
<span class="n">np</span><span class="p">.</span><span class="n">mean</span><span class="p">(</span><span class="n">weekly_usages</span><span class="p">),</span> <span class="n">np</span><span class="p">.</span><span class="n">std</span><span class="p">(</span><span class="n">weekly_usages</span><span class="p">)))</span>
<span class="n">plt</span><span class="p">.</span><span class="n">grid</span><span class="p">(</span><span class="bp">True</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">show</span><span class="p">()</span>
<span class="n">max_weekly</span><span class="p">,</span> <span class="n">min_weekly</span> <span class="o">=</span> <span class="nb">max</span><span class="p">(</span><span class="n">weekly_usages</span><span class="p">),</span> <span class="nb">min</span><span class="p">(</span><span class="n">weekly_usages</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="s">'{} minutes in highest-usage week, {} minutes in lowest-usage week'</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span>
<span class="n">max_weekly</span><span class="p">,</span> <span class="n">min_weekly</span><span class="p">))</span>
</code></pre></div></div>
<p><img src="https://raw.githubusercontent.com/rohan-varma/phone-usage-tracking/master/How%20Much%20do%20I%20use%20my%20Phone%3F_files/How%20Much%20do%20I%20use%20my%20Phone%3F_12_0.png" alt="png" /></p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/phone-usage-tracking/master/How%20Much%20do%20I%20use%20my%20Phone%3F_files/How%20Much%20do%20I%20use%20my%20Phone%3F_12_1.png" alt="png" /></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1685 minutes in highest-usage week, 580 minutes in lowest-usage week
</code></pre></div></div>
<p>This is pretty interesting - it looks like my phone usage clustered around the 700-900 minute range for many weeks, with frequent spikes up to the 1100+ minute range in a couple of the weeks. My highest-usage week was a whopping 1685 minutes, or over 28 hours, or more than an entire day of the week spent solely on my phone. Finally, let’s move on to monthly usage.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># parse months out of dates, and get those days corresponding to the month
</span><span class="n">months</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="nb">set</span><span class="p">([</span><span class="s">"-"</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">day</span><span class="p">.</span><span class="n">date</span><span class="p">.</span><span class="n">split</span><span class="p">(</span><span class="s">"-"</span><span class="p">)[</span><span class="mi">0</span><span class="p">:</span><span class="mi">2</span><span class="p">])</span> <span class="k">for</span> <span class="n">day</span> <span class="ow">in</span> <span class="n">ordered_days</span><span class="p">]))</span>
<span class="n">month_to_days</span> <span class="o">=</span> <span class="p">{</span><span class="nb">int</span><span class="p">(</span><span class="n">month</span><span class="p">.</span><span class="n">split</span><span class="p">(</span><span class="s">"-"</span><span class="p">)[</span><span class="mi">1</span><span class="p">]):</span>
<span class="p">[</span><span class="n">day</span> <span class="k">for</span> <span class="n">day</span> <span class="ow">in</span> <span class="n">ordered_days</span> <span class="k">if</span> <span class="n">month</span> <span class="ow">in</span> <span class="n">day</span><span class="p">.</span><span class="n">date</span><span class="p">]</span> <span class="k">for</span> <span class="n">month</span> <span class="ow">in</span> <span class="n">months</span><span class="p">}</span>
<span class="c1"># plot bar graph of monthly means
</span><span class="n">monthly_means</span> <span class="o">=</span> <span class="p">[</span><span class="n">np</span><span class="p">.</span><span class="n">mean</span><span class="p">([</span><span class="n">day</span><span class="p">.</span><span class="n">minutes</span> <span class="k">for</span> <span class="n">day</span> <span class="ow">in</span> <span class="n">li</span><span class="p">])</span> <span class="k">for</span> <span class="n">li</span> <span class="ow">in</span> <span class="nb">list</span><span class="p">(</span><span class="n">month_to_days</span><span class="p">.</span><span class="n">values</span><span class="p">())]</span>
<span class="n">plt</span><span class="p">.</span><span class="n">bar</span><span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="n">month_to_days</span><span class="p">.</span><span class="n">keys</span><span class="p">()),</span> <span class="n">monthly_means</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s">'Month (1 = Jan)'</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s">'Average daily minutes of phone usage'</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">title</span><span class="p">(</span><span class="s">'Average daily minutes of phone usage per month'</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">xticks</span><span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="mi">13</span><span class="p">)))</span>
<span class="n">plt</span><span class="p">.</span><span class="n">show</span><span class="p">()</span>
<span class="c1"># plot histograms of most and least used months.
</span><span class="n">most_use_month</span><span class="p">,</span> <span class="n">least_use_month</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">argmax</span><span class="p">(</span><span class="n">monthly_means</span><span class="p">)</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span> <span class="n">np</span><span class="p">.</span><span class="n">argmin</span><span class="p">(</span><span class="n">monthly_means</span><span class="p">)</span> <span class="o">+</span> <span class="mi">1</span>
<span class="n">most_use_days</span><span class="p">,</span> <span class="n">least_use_days</span> <span class="o">=</span> <span class="n">month_to_days</span><span class="p">[</span><span class="n">most_use_month</span><span class="p">],</span> <span class="n">month_to_days</span><span class="p">[</span><span class="n">least_use_month</span><span class="p">]</span>
<span class="n">most_use_mins</span> <span class="o">=</span> <span class="p">[</span><span class="n">day</span><span class="p">.</span><span class="n">minutes</span> <span class="k">for</span> <span class="n">day</span> <span class="ow">in</span> <span class="n">most_use_days</span><span class="p">]</span>
<span class="n">least_use_mins</span> <span class="o">=</span> <span class="p">[</span><span class="n">day</span><span class="p">.</span><span class="n">minutes</span> <span class="k">for</span> <span class="n">day</span> <span class="ow">in</span> <span class="n">least_use_days</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">plot</span><span class="p">(</span><span class="n">month</span><span class="p">,</span> <span class="n">mins</span><span class="p">):</span>
<span class="n">bins</span> <span class="o">=</span> <span class="p">[</span><span class="n">i</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="nb">max</span><span class="p">(</span><span class="n">mins</span><span class="p">)</span> <span class="o">+</span> <span class="mi">60</span><span class="p">,</span> <span class="mi">60</span><span class="p">)]</span>
<span class="n">n</span><span class="p">,</span> <span class="n">bins</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="n">plt</span><span class="p">.</span><span class="n">hist</span><span class="p">(</span><span class="n">mins</span><span class="p">,</span> <span class="n">bins</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s">'Minutes of Phone Usage'</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">xticks</span><span class="p">(</span><span class="n">bins</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s">'Frequency (# of Days)'</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">title</span><span class="p">(</span><span class="s">'Histogram of Phone Usage Time: Month {0:02d}'</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">month</span><span class="p">))</span>
<span class="n">plt</span><span class="p">.</span><span class="n">text</span><span class="p">(</span><span class="mi">300</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="s">r'$\mu={0:.2f},\ \sigma={1:.2f}$'</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">np</span><span class="p">.</span><span class="n">mean</span><span class="p">(</span><span class="n">mins</span><span class="p">),</span> <span class="n">np</span><span class="p">.</span><span class="n">std</span><span class="p">(</span><span class="n">mins</span><span class="p">)))</span>
<span class="n">plt</span><span class="p">.</span><span class="n">grid</span><span class="p">(</span><span class="bp">True</span><span class="p">)</span>
<span class="n">plt</span><span class="p">.</span><span class="n">show</span><span class="p">()</span>
<span class="n">plot</span><span class="p">(</span><span class="n">most_use_month</span><span class="p">,</span> <span class="n">most_use_mins</span><span class="p">)</span>
<span class="n">plot</span><span class="p">(</span><span class="n">least_use_month</span><span class="p">,</span> <span class="n">least_use_mins</span><span class="p">)</span>
</code></pre></div></div>
<p><img src="https://raw.githubusercontent.com/rohan-varma/phone-usage-tracking/master/How%20Much%20do%20I%20use%20my%20Phone%3F_files/How%20Much%20do%20I%20use%20my%20Phone%3F_14_0.png" alt="png" /></p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/phone-usage-tracking/master/How%20Much%20do%20I%20use%20my%20Phone%3F_files/How%20Much%20do%20I%20use%20my%20Phone%3F_14_1.png" alt="png" /></p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/phone-usage-tracking/master/How%20Much%20do%20I%20use%20my%20Phone%3F_files/How%20Much%20do%20I%20use%20my%20Phone%3F_14_2.png" alt="png" /></p>
<p>It looks like my monthly phone usage was consistent, hovering around the slightly above two hour mark, with a dip during the summer months and an increase during April and December. Over April, I used my phone for slightly over three hours a day on average (nearly 19% of my waking hours!) while using my phone for 1 hour and 43 minutes, or 10.7% of my waking hours, in the least-used month of August.</p>
<p>To be fair, the month of April had a large amount of variability, so the mean of 3 hours doesn’t really reflect my usual usage that month: April contained all three of the outliers in the entire year, where I used my phone for more than 6 hours. Honestly, I’m not too sure what may have happened, I either left my phone on accidently at some points during the month or more realistically just used my phone a lot those couple of days.</p>
<h4 id="concluding-remarks">Concluding Remarks</h4>
<p>The data indicates that my phone occupies a pretty significant chunk of my waking hours on an average day. I knew, like many people, that I used my phone a lot, but I didn’t quite understand how much until I actually looked at the data. However, this data doesn’t capture more granular information of whether I’m using my phone for more “useful” purposes such as necessary communication, homework/work-related stuff, calling a lyft/uber, getting directions, or talking on the phone, versus more typical time wasters (randomly checking social media/email for the 10000th+ time or just browsing around).</p>
<p>The overall takeaway for me is to think of my phone more as a tool, instead of as a distraction for when I’m bored. Phones and applications can be incredibly useful in keeping us connected with our friends and family, getting from place to place, learning new things, or capturing incredible moments, but can also take away from the present moment.</p>
<p>In 2019, I’m going to make a conscious effort to simply note when I use my phone immediately when boredom presents itself, such as during a long car ride, waiting for an elevator, or even just walking from place to place. Hopefully, this will make me more mindful when I use my phone to distract myself from the present moment, and in time, I can learn to turn off this deeply ingrained habit. Here’s to being more present in 2019.</p>Towards the end of 2017, I started using an iOS app called Moment, which tracks how much time you spent on your phone each day and how many times you pick it up. Through using this application for the year of 2018 and poking around in the app for a way to export my day-by-day data, I was able to obtain a JSON file consisting of my phone usage time and number of pickups for every day of the year.Training very deep networks with Batchnorm2018-02-19T00:00:00+00:002018-02-19T00:00:00+00:00https://rohanvarma.me/Batch-Norm<p><img src="https://raw.githubusercontent.com/rohan-varma/nn-init-demo/master/plots/batchnorm_grad_first_layer.png" alt="grad" /></p>
<p>Training very deep neural networks is hard. It turns out one significant issue with deep neural networks is that the activations of each layer tend to converge to 0 in the later layers, and therefore the gradients vanish as they backpropagate throughout the network.</p>
<p>A lot of this has to do with the sheer size of the network - obviously as you multiply numbers less than zero together over and over, they’ll converge to zero, and that’s partially why network architectures such as InceptionV3 insert auxiliary classifiers after layers earlier on in their network, so there’s a stronger gradient signal back propagated during the first few epochs of training.</p>
<p>However, there’s also a more subtle issue that leads to this problem of vanishing activations and gradients. It has to do with the initialization of the weights in each layer of our network, and the subsequent distributions of the activations in our network. Understanding this issue is key to understanding why batch normalization is now a staple in training deep networks.</p>
<p>First, we can write some code to generate some random data, and forward it through a dummy deep neural network:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">n_examples</span><span class="p">,</span> <span class="n">hidden_layer_dim</span> <span class="o">=</span> <span class="mi">100</span><span class="p">,</span> <span class="mi">100</span>
<span class="n">input_dim</span> <span class="o">=</span> <span class="mi">1000</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">randn</span><span class="p">(</span><span class="n">n_examples</span><span class="p">,</span> <span class="n">input_dim</span><span class="p">)</span> <span class="c1"># 100 examples of 1000 points
</span><span class="n">n_layers</span> <span class="o">=</span> <span class="mi">20</span>
<span class="n">layer_dim</span> <span class="o">=</span> <span class="p">[</span><span class="n">hidden_layer_dim</span><span class="p">]</span> <span class="o">*</span> <span class="n">n_layers</span> <span class="c1"># each one has 100 neurons
</span>
<span class="n">hs</span> <span class="o">=</span> <span class="p">[</span><span class="n">X</span><span class="p">]</span> <span class="c1"># stores the hidden layer activations
</span><span class="n">zs</span> <span class="o">=</span> <span class="p">[</span><span class="n">X</span><span class="p">]</span> <span class="c1"># stores the affine transforms in each layer, used for backprop
</span><span class="n">ws</span> <span class="o">=</span> <span class="p">[]</span> <span class="c1"># stores the weights
</span>
<span class="c1"># the forward pass
</span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">np</span><span class="p">.</span><span class="n">arange</span><span class="p">(</span><span class="n">n_layers</span><span class="p">):</span>
<span class="n">h</span> <span class="o">=</span> <span class="n">hs</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="c1"># get the input into this hidden layer
</span> <span class="c1">#W = np.random.randn(h.shape[0], layer_dim[i]) * np.sqrt(2)/(np.sqrt(200) * np.sqrt(3))
</span> <span class="c1">#W = np.random.uniform(-np.sqrt(6)/(200), np.sqrt(6)/200, size = (h.shape[0], layer_dim[i]))
</span> <span class="n">W</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">normal</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">np</span><span class="p">.</span><span class="n">sqrt</span><span class="p">(</span><span class="mi">2</span><span class="o">/</span><span class="p">(</span><span class="n">h</span><span class="p">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">+</span> <span class="n">layer_dim</span><span class="p">[</span><span class="n">i</span><span class="p">])),</span> <span class="n">size</span> <span class="o">=</span> <span class="p">(</span><span class="n">layer_dim</span><span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="n">h</span><span class="p">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]))</span>
<span class="c1">#W = np.random.normal(0, np.sqrt(2/(h.shape[0] + layer_dim[i])), size = (layer_dim[i], h.shape[0])) * 0.01
</span> <span class="n">z</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">dot</span><span class="p">(</span><span class="n">W</span><span class="p">,</span> <span class="n">h</span><span class="p">)</span>
<span class="n">h_out</span> <span class="o">=</span> <span class="n">z</span> <span class="o">*</span> <span class="p">(</span><span class="n">z</span> <span class="o">></span> <span class="mi">0</span><span class="p">)</span>
<span class="n">ws</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">W</span><span class="p">)</span>
<span class="n">zs</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">z</span><span class="p">)</span>
<span class="n">hs</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">h_out</span><span class="p">)</span>
</code></pre></div></div>
<p>Now that we have a list of each layer’s hidden activations stored in <strong>hs</strong>, we can go ahead and plot the activations to see what their distribution looks like. Here, I’ve included plots of the activations at the first and final hidden layers in our 20 layer network:</p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/nn-init-demo/master/plots/activation_0.png" alt="act0" /></p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/nn-init-demo/master/plots/activation_19.png" alt="act19" /></p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/nn-init-demo/master/plots/activation_20.png" alt="act20" /></p>
<p>What’s important to notice is that in later layers, <em>nearly all of the activations are zero</em> (just look at the scale of the axes). If we look at the distributions of these activations, it’s clear that they differ significantly with respect to each other - the first activation takes on a clear Gaussian shape around 0, while successive hidden layers have most of their activations at 0, with rapidly decreasing variance. This is what the <a href="https://arxiv.org/pdf/1502.03167.pdf">batch normalization paper</a> refers to as <em>internal covariate shift</em> - it basically means that the distributions of activations differ with respect to each other.</p>
<p><strong>Why does this matter, and why is this bad?</strong></p>
<p>This is bad mostly due to the small, and decreasing variance in the distributions of our activations across layers. Having zero activations is fine, unless nearly all your activations are zero. To understand why this is bad, we need to look at the backwards pass of our network, which is responsible for computing each gradient <script type="math/tex">\frac{dL}{dW_i}</script> across each hidden layer in our network. Given the following formulation of an arbitrary layer in our network: <script type="math/tex">h_i=relu(W_ih_{i−1}+b_i)</script> where <script type="math/tex">h_i</script> denotes the activations of the <em>i</em>th layer in our network, we can construct the local gradient <script type="math/tex">\frac{dL}{dW_i}</script>. Given an upstream gradient into this layer <script type="math/tex">\frac{dL}{dh_i}</script>, we can compute the local gradient with the chain rule:</p>
<script type="math/tex; mode=display">\frac{dL}{dW_i} = \frac{dh_i}{dW_i} * \frac{dL}{dh_i}</script>
<p>Applying the derivatives, we obtain:</p>
<script type="math/tex; mode=display">\frac{dL}{dW_i} = [\mathbb{1}(W_ih_{i-1} + b > 0) \odot \frac{dL}{dh_i}]h_{i-1}^T</script>
<p>Concretely, we can take our loss function for a single point to be given by the squared error, i.e. <script type="math/tex">L_i = \frac{1}{2}(y-t)^2</script>, and if we were at the last layer of our network (i.e. <script type="math/tex">h_i = y</script>), our upstream gradient would be <script type="math/tex">\frac{dL}{dh_i} = h_i - t</script>. This would give us a gradient of</p>
<script type="math/tex; mode=display">\frac{dL}{dW_i} = [\mathbb{1}(W_ih_{i-1} + b > 0) \odot (h_i - t)]h_{i-1}^T</script>
<p>in the final layer of our network.</p>
<p><strong>What does this tell us about our gradients for our weights?</strong></p>
<p>The expression for the gradient of our weights is intuitive: for every element in the incoming gradient matrix, pass the gradient through if this layer’s linear transformation would activate the relu neuron at that element, and scale the gradient by our input into this layer. Otherwise, zero out the gradient.</p>
<p>This means that if the incoming gradient at a certain element wasn’t already zero, it will be scaled by the input into this layer. The input in this layer is just the activations from the previous layer in our network. And as we discussed above, nearly all of those activations were zero.</p>
<p>Therefore, nearly all of the gradients backpropagated through our network will be zero, and few weight updates, if any, will occur. In the final few layers of our network, this isn’t as much of a problem. We have a strong gradient signal (i.e. <script type="math/tex">h_i - t</script> in the example above) coming from the gradient of our loss function with respect to the outputs of our network (since it is early in learning, and our predictions are inaccurate). However, after we backpropagate this signal even a few layers, chances that the gradient is zeroed out become extremely high.</p>
<p>In order to see if this is actually true, we can write out the backwards pass of our 20 layer network, and plot the gradients as we did with our activations. The following code computes the gradients using the expression given above, for all layers in our network:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">dLdh</span> <span class="o">=</span> <span class="mi">100</span> <span class="o">*</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">randn</span><span class="p">(</span><span class="n">hidden_layer_dim</span><span class="p">,</span> <span class="n">input_dim</span><span class="p">)</span> <span class="c1"># random incoming grad into our last layer
</span><span class="n">h_grads</span> <span class="o">=</span> <span class="p">[</span><span class="n">dLdh</span><span class="p">]</span> <span class="c1"># store the incoming grads into each layer
</span><span class="n">w_grads</span> <span class="o">=</span> <span class="p">[]</span> <span class="c1"># store dL/dw for each layer
</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">np</span><span class="p">.</span><span class="n">flip</span><span class="p">(</span><span class="n">np</span><span class="p">.</span><span class="n">arange</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">n_layers</span><span class="p">),</span> <span class="n">axis</span> <span class="o">=</span> <span class="mi">0</span><span class="p">):</span>
<span class="c1"># get the incoming gradient
</span> <span class="n">incoming_loss_grad</span> <span class="o">=</span> <span class="n">h_grads</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
<span class="c1"># backprop through the relu
</span> <span class="n">dLdz</span> <span class="o">=</span> <span class="n">incoming_loss_grad</span> <span class="o">*</span> <span class="p">(</span><span class="n">zs</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">></span> <span class="mi">0</span><span class="p">)</span> <span class="c1"># zs was the result of Wx + b
</span> <span class="c1"># get the gradient dL/dh_{i-1}, this will be the incoming grad into the next layer
</span> <span class="n">h_grad</span> <span class="o">=</span> <span class="n">ws</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">].</span><span class="n">T</span><span class="p">.</span><span class="n">dot</span><span class="p">(</span><span class="n">dLdz</span><span class="p">)</span> <span class="c1"># ws[i-1] are our weights at this layer
</span> <span class="c1"># get the gradient of the weights of this layer (dL/dw)
</span> <span class="n">weight_grad</span> <span class="o">=</span> <span class="n">dLdz</span><span class="p">.</span><span class="n">dot</span><span class="p">(</span><span class="n">hs</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">].</span><span class="n">T</span><span class="p">)</span> <span class="c1"># hs[i-1] was our input into this layer
</span> <span class="n">h_grads</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">h_grad</span><span class="p">)</span>
<span class="n">w_grads</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">weight_grad</span><span class="p">)</span>
</code></pre></div></div>
<p>Now, we can plot our gradients for our earlier layers and see if our hypothesis was true:</p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/nn-init-demo/master/plots/grad_layer2.png" alt="grad1" /></p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/nn-init-demo/master/plots/grad_layer_3.png" alt="grad3" /></p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/nn-init-demo/master/plots/grad_layer_4.png" alt="grad4" /></p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/nn-init-demo/master/plots/grads_layer_20.png" alt="grad20" /></p>
<p>As we can see, for the final layer vanishing gradients aren’t an issue, but they are for earlier layers - in fact, after a few layers nearly all of the gradients are zero). This will result in extremely slow learning (if at all).</p>
<p><strong>Ok, but what does batch normalization have to do any of this?</strong></p>
<p>Batch normalization is a way to fix the root cause of our issue of zero activations and vanishing gradients: reducing internal covariate shift. We want to ensure that the variances of our activations do not differ too much from each other. Batch normalization does this by normalizing each activation in a batch:</p>
<script type="math/tex; mode=display">x_k = \frac{x_k - \mu_B}{\sqrt{\sigma^2_B + \epsilon}}</script>
<p>Here, we denote<script type="math/tex">x_k</script> to be a certain activation, and <script type="math/tex">\mu_B</script>, <script type="math/tex">\sigma^2_B</script> to be the mean and variance across the minibatch for that activation. A small constant <script type="math/tex">\epsilon</script> is added to ensure that we don’t divide by zero.</p>
<p>This constrains all hidden layer activations to have zero mean and unit variance, so the variances in our hidden layer activations should not differ too much from each other, and therefore we shouldn’t have nearly all our activations be zero.</p>
<p>It’s important to note here that batch normalization doesn’t <em>force</em> the network activations to rigidly follow this distribution at all times, because the above result is scaled and shifted by some parameters before being passed as input into the next layer:</p>
<script type="math/tex; mode=display">y_k = \gamma \hat{x_i} + \beta</script>
<p>This allows the network to “undo” the previous normalization procedure if it wants to, such as if <script type="math/tex">y_k</script> was an input into a sigmoid neuron, we may not want to normalize at all, because doing so may constrain the expressivity of the sigmoid neuron.</p>
<p><strong>Does normalizing our inputs into the next layer actually work?</strong></p>
<p>With batch normalization, we can be confident that the distributions of our activations across hidden layers are reasonably similar. If this is true, then we know that the gradients should have a wider distribution, and not be nearly all zero, following the same scaling logic described above.</p>
<p>Let’s add batch normalization to our forward pass to see if the activations have reasonable variances. Our forward pass changes in only a few lines:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">n_examples</span><span class="p">,</span> <span class="n">hidden_layer_dim</span> <span class="o">=</span> <span class="mi">100</span><span class="p">,</span> <span class="mi">100</span>
<span class="n">input_dim</span> <span class="o">=</span> <span class="mi">1000</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">randn</span><span class="p">(</span><span class="n">n_examples</span><span class="p">,</span> <span class="n">input_dim</span><span class="p">)</span> <span class="c1"># 100 examples of 1000 points
</span><span class="n">n_layers</span> <span class="o">=</span> <span class="mi">20</span>
<span class="n">layer_dim</span> <span class="o">=</span> <span class="p">[</span><span class="n">hidden_layer_dim</span><span class="p">]</span> <span class="o">*</span> <span class="n">n_layers</span> <span class="c1"># each one has 100 neurons
</span>
<span class="n">hs</span> <span class="o">=</span> <span class="p">[</span><span class="n">X</span><span class="p">]</span> <span class="c1"># save hidden states
</span><span class="n">hs_not_batchnormed</span> <span class="o">=</span> <span class="p">[</span><span class="n">X</span><span class="p">]</span> <span class="c1"># saves the results before we do batchnorm, because we need this in the backward pass.
</span><span class="n">zs</span> <span class="o">=</span> <span class="p">[</span><span class="n">X</span><span class="p">]</span> <span class="c1"># save affine transforms for backprop
</span><span class="n">ws</span> <span class="o">=</span> <span class="p">[]</span> <span class="c1"># save the weights
</span><span class="n">gamma</span><span class="p">,</span> <span class="n">beta</span> <span class="o">=</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">0</span>
<span class="c1"># the forward pass
</span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">np</span><span class="p">.</span><span class="n">arange</span><span class="p">(</span><span class="n">n_layers</span><span class="p">):</span>
<span class="n">h</span> <span class="o">=</span> <span class="n">hs</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="c1"># get the input into this hidden layer
</span> <span class="n">W</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">normal</span><span class="p">(</span><span class="n">size</span> <span class="o">=</span> <span class="p">(</span><span class="n">layer_dim</span><span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="n">h</span><span class="p">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]))</span> <span class="o">*</span> <span class="mf">0.01</span> <span class="c1"># weight init: gaussian around 0
</span> <span class="n">z</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">dot</span><span class="p">(</span><span class="n">W</span><span class="p">,</span> <span class="n">h</span><span class="p">)</span>
<span class="n">h_out</span> <span class="o">=</span> <span class="n">z</span> <span class="o">*</span> <span class="p">(</span><span class="n">z</span> <span class="o">></span> <span class="mi">0</span><span class="p">)</span>
<span class="c1"># save the not batchnormmed part for backprop
</span> <span class="n">hs_not_batchnormed</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">h_out</span><span class="p">)</span>
<span class="c1"># apply batch normalization
</span> <span class="n">h_out</span> <span class="o">=</span> <span class="p">(</span><span class="n">h_out</span> <span class="o">-</span> <span class="n">np</span><span class="p">.</span><span class="n">mean</span><span class="p">(</span><span class="n">h_out</span><span class="p">,</span> <span class="n">axis</span> <span class="o">=</span> <span class="mi">0</span><span class="p">))</span> <span class="o">/</span> <span class="n">np</span><span class="p">.</span><span class="n">std</span><span class="p">(</span><span class="n">h_out</span><span class="p">,</span> <span class="n">axis</span> <span class="o">=</span> <span class="mi">0</span><span class="p">)</span>
<span class="c1"># scale and shift
</span> <span class="n">h_out</span> <span class="o">=</span> <span class="n">gamma</span> <span class="o">*</span> <span class="n">h_out</span> <span class="o">+</span> <span class="n">beta</span>
<span class="n">ws</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">W</span><span class="p">)</span>
<span class="n">zs</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">z</span><span class="p">)</span>
<span class="n">hs</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">h_out</span><span class="p">)</span>
</code></pre></div></div>
<p>Using the results of this forward pass (again stored in <strong>hs</strong>), we can plot a few of the activations:</p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/nn-init-demo/master/plots/batchnorm_activation_4.png" alt="act4" /></p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/nn-init-demo/master/plots/batchnorm_activation_19.png" alt="act20" /></p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/nn-init-demo/master/plots/batchnorm_activation_20.png" alt="act20" /></p>
<p>This is great! Our later activations now have a much more reasonable distribution compared to previously, where they were all almost zero - just compare the scales of the axes on the batchnorm graphs against the non-original graphs.</p>
<p>Let’s see if this makes any difference in our gradients. First, we have to rewrite our original backwards pass to accommodate the gradients for the batchnorm operation. The gradients I used in the batchnorm layer are the ones given by the <a href="https://arxiv.org/pdf/1502.03167.pdf">original paper</a>. Our backwards pass now becomes:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">dLdh</span> <span class="o">=</span> <span class="mf">0.01</span> <span class="o">*</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">randn</span><span class="p">(</span><span class="n">hidden_layer_dim</span><span class="p">,</span> <span class="n">input_dim</span><span class="p">)</span> <span class="c1"># random incoming grad into our last layer
</span><span class="n">h_grads</span> <span class="o">=</span> <span class="p">[</span><span class="n">dLdh</span><span class="p">]</span> <span class="c1"># incoming grads into each layer
</span><span class="n">w_grads</span> <span class="o">=</span> <span class="p">[]</span> <span class="c1"># will hold dL/dw_i for each layer
</span>
<span class="c1"># the backwards pass
</span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">np</span><span class="p">.</span><span class="n">flip</span><span class="p">(</span><span class="n">np</span><span class="p">.</span><span class="n">arange</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">n_layers</span><span class="p">),</span> <span class="n">axis</span> <span class="o">=</span> <span class="mi">0</span><span class="p">):</span>
<span class="c1"># get the incoming gradient
</span> <span class="n">incoming_loss_grad</span> <span class="o">=</span> <span class="n">h_grads</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
<span class="c1"># backprop through the batchnorm layer
</span> <span class="c1">#the y_i is the restult of batch norm, so h_out or hs[i]
</span> <span class="n">dldx_hat</span> <span class="o">=</span> <span class="n">incoming_loss_grad</span> <span class="o">*</span> <span class="n">gamma</span>
<span class="n">dldvar</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="nb">sum</span><span class="p">(</span><span class="n">dldx_hat</span> <span class="o">*</span> <span class="p">(</span><span class="n">hs_not_batchnormed</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">-</span> <span class="n">np</span><span class="p">.</span><span class="n">mean</span><span class="p">(</span><span class="n">hs_not_batchnormed</span><span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="n">axis</span> <span class="o">=</span> <span class="mi">0</span><span class="p">))</span> <span class="o">*</span> <span class="o">-</span><span class="p">.</span><span class="mi">5</span> <span class="o">*</span> <span class="n">np</span><span class="p">.</span><span class="n">power</span><span class="p">(</span><span class="n">np</span><span class="p">.</span><span class="n">var</span><span class="p">(</span><span class="n">hs_not_batchnormed</span><span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="n">axis</span> <span class="o">=</span> <span class="mi">0</span><span class="p">),</span> <span class="o">-</span><span class="mf">1.5</span><span class="p">),</span> <span class="n">axis</span> <span class="o">=</span> <span class="mi">0</span><span class="p">)</span>
<span class="n">dldmean</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="nb">sum</span><span class="p">(</span><span class="n">dldx_hat</span> <span class="o">*</span> <span class="o">-</span><span class="mi">1</span><span class="o">/</span><span class="n">np</span><span class="p">.</span><span class="n">std</span><span class="p">(</span><span class="n">hs_not_batchnormed</span><span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="n">axis</span> <span class="o">=</span> <span class="mi">0</span><span class="p">),</span> <span class="n">axis</span> <span class="o">=</span> <span class="mi">0</span><span class="p">)</span> <span class="o">+</span> <span class="n">dldvar</span> <span class="o">*</span> <span class="o">-</span><span class="mi">2</span> <span class="o">*</span> <span class="p">(</span><span class="n">hs_not_batchnormed</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">-</span> <span class="n">np</span><span class="p">.</span><span class="n">mean</span><span class="p">(</span><span class="n">hs_not_batchnormed</span><span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="n">axis</span> <span class="o">=</span> <span class="mi">0</span><span class="p">))</span><span class="o">/</span><span class="n">hs_not_batchnormed</span><span class="p">[</span><span class="n">i</span><span class="p">].</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="c1"># the following is dL/hs_not_batchnormmed[i] (aka dL/dx_i) in the paper!
</span> <span class="n">dldx</span> <span class="o">=</span> <span class="n">dldx_hat</span> <span class="o">*</span> <span class="mi">1</span><span class="o">/</span><span class="n">np</span><span class="p">.</span><span class="n">std</span><span class="p">(</span><span class="n">hs_not_batchnormed</span><span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="n">axis</span> <span class="o">=</span> <span class="mi">0</span><span class="p">)</span> <span class="o">+</span> <span class="n">dldvar</span> <span class="o">*</span> <span class="o">-</span><span class="mi">2</span> <span class="o">*</span> <span class="p">(</span><span class="n">hs_not_batchnormed</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">-</span> <span class="n">np</span><span class="p">.</span><span class="n">mean</span><span class="p">(</span><span class="n">hs_not_batchnormed</span><span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="n">axis</span> <span class="o">=</span> <span class="mi">0</span><span class="p">))</span><span class="o">/</span><span class="n">hs_not_batchnormed</span><span class="p">[</span><span class="n">i</span><span class="p">].</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">+</span> <span class="n">dldmean</span><span class="o">/</span><span class="n">hs_not_batchnormed</span><span class="p">[</span><span class="n">i</span><span class="p">].</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="c1"># although we don't need it for this demo, for completeness we also compute the derivatives with respect to gamma and beta.
</span> <span class="n">dldgamma</span> <span class="o">=</span> <span class="n">incoming_loss_grad</span> <span class="o">*</span> <span class="n">hs</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
<span class="n">dldbeta</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="nb">sum</span><span class="p">(</span><span class="n">incoming_loss_grad</span><span class="p">)</span>
<span class="c1"># now incoming_loss_grad should be replaced by that backpropped result
</span> <span class="n">incoming_loss_grad</span> <span class="o">=</span> <span class="n">dldx</span>
<span class="c1"># backprop through the relu
</span> <span class="k">print</span><span class="p">(</span><span class="n">incoming_loss_grad</span><span class="p">.</span><span class="n">shape</span><span class="p">)</span>
<span class="n">dLdz</span> <span class="o">=</span> <span class="n">incoming_loss_grad</span> <span class="o">*</span> <span class="p">(</span><span class="n">zs</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">></span> <span class="mi">0</span><span class="p">)</span>
<span class="c1"># get the gradient dL/dh_{i-1}, this will be the incoming grad into the next layer
</span> <span class="n">h_grad</span> <span class="o">=</span> <span class="n">ws</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">].</span><span class="n">T</span><span class="p">.</span><span class="n">dot</span><span class="p">(</span><span class="n">dLdz</span><span class="p">)</span>
<span class="c1"># get the gradient of the weights of this layer (dL/dw)
</span> <span class="n">weight_grad</span> <span class="o">=</span> <span class="n">dLdz</span><span class="p">.</span><span class="n">dot</span><span class="p">(</span><span class="n">hs</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">].</span><span class="n">T</span><span class="p">)</span>
<span class="n">h_grads</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">h_grad</span><span class="p">)</span>
<span class="n">w_grads</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">weight_grad</span><span class="p">)</span>
</code></pre></div></div>
<p>Using this backwards pass, we can now plot our gradients. We expect them to no longer be nearly all zero, which will mean that avoiding internal covariate shift fixed our vanishing gradients problem:</p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/nn-init-demo/master/plots/batchnorm_grad_first_layer.png" alt="bngrad1" /></p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/nn-init-demo/master/plots/batchnorm_grad_second_layer.png" alt="bngrad3" /></p>
<p>Awesome! Looking at our gradients early in the network, we can see that they follow a roughly normal distribution with plenty of non-zero, large-magnitude values. Since our gradients are much more reasonable than previously, where they were nearly all zero, we are more confident that learning will occur at a reasonable rate, even for a large deep neural network (20 layers). We’ve successfully used batch normalization to fix one of the most common issues in training deep neural networks!</p>
<h4 id="intuition-for-why-batch-normalization-helps-with-better-gradient-signals">Intuition for why Batch Normalization helps with better gradient signals</h4>
<p>When gradient descent updates a certain layer in our network with the gradient <script type="math/tex">\frac{dL}{dW_i}</script>, it is ignorant of the changes in statistics in other layers - for example, it implicitly assumes that the distribution of the activations of the previous layer (and hence the input into this layer) stay the same as it updates the current layer it is on. Without batch normalization, this assumption isn’t true: gradient descent also eventually updates the weights in the previous layer, therefore changing the statistics of the output activations for that layer. Therefore, there could be a case where we update layer <script type="math/tex">i</script> , but the distribution of the inputs into that layer change such that the update actually does <em>worse</em> on these new inputs. Batch normalization fixes this, by guaranteeing that the statistics of the input into each layer stay the same throughout the learning process. See <a href="https://www.youtube.com/watch?v=Xogn6veSyxA&feature=youtu.be&t=325">this explanation</a> by Goodfellow for more on this.</p>
<p>P.S. - all the code used to generate the plots used in this answer are available <a href="https://github.com/rohan-varma/nn-init-demo/">here</a>.</p>
<h4 id="references">References</h4>
<ol>
<li><a href="https://arxiv.org/abs/1502.03167">Batch Normalization Paper</a></li>
<li><a href="cs231n.stanford.edu">CS 231n Lecture on Batch Normlization</a></li>
</ol>
<h4 id="notes">Notes</h4>
<p>[2/19/18] - I originally wrote this as an <a href="https://www.quora.com/How-does-batch-normalization-help/answer/Rohan-Varma-8">answer on Quora</a></p>
<p>[2/21/18] - The code used in the forward and backward pass isn’t completely accurate with respect to scaling the outputs by parameters <script type="math/tex">\gamma</script> and <script type="math/tex">\beta</script>. In actuality, there is supposed to be a <script type="math/tex">\gamma_i</script> and a <script type="math/tex">\beta_i</script> for <em>each</em> activation in <em>each</em> hidden layer - for example, if we have a batch of <script type="math/tex">n</script> activations and each activation has shape <script type="math/tex">1000</script>, there should be <script type="math/tex">1000</script> <script type="math/tex">\gamma_i</script>s and <script type="math/tex">1000</script> <script type="math/tex">\beta_i</script>s in each layer. I didn’t bother to actually implement it this way as it doesn’t affect the normalization process for the one step I illustrated.</p>
<p>[2/22/18] - I applied batch normalization <em>after</em> the ReLU nonlinearity, whereas the original paper states that it is applied after the affine layer and <em>before</em> the nonlinearity. Apparently, their actual code applies it after the ReLU as well, and it was mis-stated in their paper. See <a href="https://www.reddit.com/r/MachineLearning/comments/67gonq/d_batch_normalization_before_or_after_relu/">this Reddit thread</a> for more discussion.</p>Picking Loss Functions - A comparison between MSE, Cross Entropy, and Hinge Loss2018-01-09T00:00:00+00:002018-01-09T00:00:00+00:00https://rohanvarma.me/Loss-Functions<p><img src="https://raw.githubusercontent.com/rohan-varma/rohan-blog/gh-pages/images/loss3.jpg" alt="loss" /></p>
<p>Loss functions are a key part of any machine learning model: they define an objective against which the performance of your model is measured, and the setting of weight parameters learned by the model is determined by minimizing a chosen loss function. There are several different common loss functions to choose from: the cross-entropy loss, the mean-squared error, the huber loss, and the hinge loss - just to name a few. Given a particular model, each loss function has particular properties that make it interesting - for example, the (L2-regularized) hinge loss comes with the maximum-margin property, and the mean-squared error when used in conjunction with linear regression comes with convexity guarantees.</p>
<p>In this post, I’ll discuss three common loss functions: the mean-squared (MSE) loss, cross-entropy loss, and the hinge loss. These are the most commonly used functions I’ve seen used in traditional machine learning and deep learning models, so I thought it would be a good idea to figure out the underlying theory behind each one, and when to prefer one over the others.</p>
<h4 id="the-mean-squared-loss-probabalistic-interpretation">The Mean-Squared Loss: Probabalistic Interpretation</h4>
<p>For a model prediction such as <script type="math/tex">h_\theta(x_i) = \theta_0 + \theta_1x</script> (a simple linear regression in 2 dimensions) where the inputs are a feature vector <script type="math/tex">x_i</script>, the mean-squared error is given by summing across all <script type="math/tex">N</script> training examples, and for each example, calculating the squared difference from the true label <script type="math/tex">y_i</script> and the prediction <script type="math/tex">h_\theta(x_i)</script>:</p>
<script type="math/tex; mode=display">J = \frac{1}{N} \sum_{i=1}^{N} (y_i - h_\theta(x_i))^2</script>
<p>It turns out we can derive the mean-squared loss by considering a typical linear regression problem.</p>
<p>With linear regression, we seek to model our real-valued labels <script type="math/tex">Y</script> as being a linear function of our inputs <script type="math/tex">X</script>, corrupted by some noise. Let’s write out this assumption:</p>
<script type="math/tex; mode=display">Y = \theta_0 + \theta_1x + \eta</script>
<p>And to solidify our assumption, we’ll say that <script type="math/tex">\eta</script> is Gaussian noise with 0 mean and unit variance, that is <script type="math/tex">\eta \sim N(0, 1)</script>. This means that <script type="math/tex">E[Y] = E[\theta_0 + \theta_1x + \eta] = \theta_0 + \theta_1x</script> and <script type="math/tex">Var[Y] = Var[\theta_0 + \theta_1x + \eta] =</script>,1 so <script type="math/tex">Y</script> is also Gaussian with mean <script type="math/tex">\theta_0 + \theta_1x</script> and variance 1.</p>
<p>We can write out the probability of observing a single <script type="math/tex">(x_i, y_i)</script> sample:</p>
<script type="math/tex; mode=display">p(y_i \vert x_i) = e^{-\frac{(y_{i} - (\theta_{0} + \theta_{1}x_{i}))^2}{2}}</script>
<p>Summing across <script type="math/tex">N</script> of these samples in our dataset, we can write down the likelihood - essentially the probability of observing all <script type="math/tex">N</script> of our samples. Note that we also make the assumption that our data are independent of each other, so we can write out the likelihood as a simple product over each individual probability:</p>
<script type="math/tex; mode=display">L(x, y) = \prod_{i=1}^{N}e^{-\frac{(y_i - (\theta_0 + \theta_1x_i))^2}{2}}</script>
<p>Next, we can take the log of our likelihood function to obtain the log-likelihood, a function that is easier to differentiate and overall nicer to work with:</p>
<script type="math/tex; mode=display">l(x, y) = -\frac{1}{2}\sum_{i=1}^{N}(y_i - (\theta_0 + \theta_1x_i))^2</script>
<p>This gives us the MSE:</p>
<script type="math/tex; mode=display">J = \frac{1}{2}\sum_{i=1}^{N}(y_i - \theta^Tx_i)^2</script>
<p>Essentially, this means that using the MSE loss makes sense if the assumption that your outputs are a real-valued function of your inputs, with a certain amount of irreducible Gaussian noise, with constant mean and variance. If these assumptions don’t hold true (such as in the context of classification), the MSE loss may not be the best bet.</p>
<h4 id="the-cross-entropy-loss-probabalistic-interpretation">The Cross-Entropy Loss: Probabalistic Interpretation</h4>
<p>In the context of classification, our model’s prediction <script type="math/tex">h_\theta(x_i)</script> will be given by <script type="math/tex">\sigma(Wx_i + b)</script> which produces a value between <script type="math/tex">0</script> and <script type="math/tex">1</script> that can be interpreted as a probability of example <script type="math/tex">x_i</script> belonging to the positive class. If this probability were less than <script type="math/tex">0.5</script> we’d classify it as a negative example, otherwise we’d classify it as a positive example. This means that we can write down the probabilily of observing a negative or positive instance:</p>
<p><script type="math/tex">p(y_i = 1 \vert x_i) = h_\theta(x_i)</script> and <script type="math/tex">p(y_i = 0 \vert x_i) = 1 - h_\theta(x_i)</script></p>
<p>We can combine these two cases into one expression:</p>
<script type="math/tex; mode=display">p(y_i | x_i) = [h_\theta(x_i)]^{(y_i)} [1 - h_\theta(x_i)]^{(1 - y_i)}</script>
<p>Invoking our assumption that the data are independent and identically distributed, we can write down the likelihood by simply taking the product across the data:</p>
<script type="math/tex; mode=display">L(x, y) = \prod_{i = 1}^{N}[h_\theta(x_i)]^{(y_i)} [1 - h_\theta(x_i)]^{(1 - y_i)}</script>
<p>Similar to above, we can take the log of the above expression and use properties of logs to simplify, and finally invert our entire expression to obtain the cross entropy loss:</p>
<script type="math/tex; mode=display">J = -\sum_{i=1}^{N} y_i\log (h_\theta(x_i)) + (1 - y_i)\log(1 - h_\theta(x_i))</script>
<h4 id="the-cross-entropy-loss-in-the-case-of-multi-class-classification">The Cross-Entropy Loss in the case of multi-class classification</h4>
<p>Let’s supposed that we’re now interested in applying the cross-entropy loss to multiple (> 2) classes. The idea behind the loss function doesn’t change, but now since our labels <script type="math/tex">y_i</script> are one-hot encoded, we write down the loss (slightly) differently:</p>
<script type="math/tex; mode=display">-\sum_{i=1}^{N} \sum_{j=1}^{K} y_{ij} \log(h_{\theta}(x_{i}){_j})</script>
<p>This is pretty similar to the binary cross entropy loss we defined above, but since we have multiple classes we need to sum over all of them. The loss <script type="math/tex">L_i</script> for a particular training example is given by</p>
<p><script type="math/tex">L_{i} = - \log p(Y = y_{i} \vert X = x_{i})</script>.</p>
<p>In particular, in the inner sum, only one term will be non-zero, and that term will be the <script type="math/tex">\log</script> of the (normalized) probability assigned to the correct class. Intuitively, this makes sense because <script type="math/tex">\log(x)</script> is increasing on the interval <script type="math/tex">(0, 1)</script> so <script type="math/tex">-\log(x)</script> is decreasing on that interval. For example, if we have a score of 0.8 for the correct label, our loss will be 0.09, if we have a score of .08 our loss would be 1.09.</p>
<p>Another variant on the cross entropy loss for multi-class classification also adds the other predicted class scores to the loss:</p>
<script type="math/tex; mode=display">- \sum_{i=1}^{N} \sum_{j=1}^{K} y_{ij} \log(h_{\theta}(x_{i})_{j}) + (1-y_{ij})log(1 - h_{\theta}(x_{i})_{j})</script>
<p>The second term in the inner sum essentially inverts our labels and score assignments: it gives the other predicted classes a probability of <script type="math/tex">1 - s_j</script>, and penalizes them by the <script type="math/tex">\log</script> of that amount (here, <script type="math/tex">s_j</script> denotes the <script type="math/tex">j</script>th score, which is the <script type="math/tex">j</script>th element of <script type="math/tex">h_\theta(x_i)</script>).</p>
<p>This again makes sense - penalizing the incorrect classes in this way will encourage the values <script type="math/tex">1 - s_j</script> (where each <script type="math/tex">s_j</script> is a probability assigned to an incorrect class) to be large, which will in turn encourage <script type="math/tex">s_j</script> to be low. This alternative version seems to tie in more closely to the binary cross entropy that we obtained from the maximum likelihood estimate, but the first version appears to be more commonly used both in practice and in teaching.</p>
<p>It turns out that it doesn’t really matter which variant of cross-entropy you use for multiple-class classification, as they both decrease at similar rates and are just offset, with the second variant discussed having a higher loss for a particular setting of scores. To show this, I <a href="https://github.com/rohan-varma/machine-learning-notes/blob/master/loss-experiment/loss.py">wrote some code</a> to plot these 2 loss functions against each other, for probabilities for the correct class ranging from 0.01 to 0.98, and obtained the following plot:</p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/machine-learning-notes/master/loss-experiment/loss.png" alt="loss" /></p>
<h4 id="cross-entropy-loss-an-information-theory-perspective">Cross Entropy Loss: An information theory perspective</h4>
<p>As mentioned in the <a href="https://cs231n.github.io/linear-classify/">CS 231n lectures</a>, the cross-entropy loss can be interpreted via information theory. In information theory, the Kullback-Leibler (KL) divergence measures how “different” two probability distributions are. We can think of our classification problem as having 2 different probability distributions: first, the distribution for our actual labels, where all the probability mass is concentrated on the correct label, and there is no probability mass on the rest, and second, the distribution which we are learning, where the concentrations of probability mass are given by the outputs of the running our raw scores through a softmax function.</p>
<p>In an ideal world, our learned distribution would match the actual distribution, with 100% probability being assigned to the correct label. This can’t really happen since that would mean our raw scores would have to be <script type="math/tex">\infty</script> and <script type="math/tex">-\infty</script> for our correct and incorrect classes respectively, and, more practically, constraints we impose on our model (i.e. using logistic regression instead of a deep neural net) will limit our ability to correctly classify every example with high probability on the correct label.</p>
<p>Interpreting the cross-entropy loss as minimizing the KL divergence between 2 distributions is interesting if we consider how we can extend cross-entropy to different scenarios. For example, a lot of datasets are only partially labelled or have noisy (i.e. occasionally incorrect) labels. If we could probabilistically assign labels to the unlabelled portion of a dataset, or interpret the incorrect labels as being sampled from a probabalistic noise distribution, we can still apply the idea of minimizing the KL-divergence, although our ground-truth distribution will no longer concentrate all the probability mass over a single label.</p>
<h4 id="differences-in-learning-speed-for-classification">Differences in learning speed for classification</h4>
<p>It turns out that if we’re given a typical classification problem and a model <script type="math/tex">h_\theta(x) = \sigma(Wx_i + b)</script>, we can show that (at least theoretically) the cross-entropy loss leads to quicker learning through gradient descent than the MSE loss. This is primarily due to the use of the sigmoid function. First, let’s recall the gradient descent update rule:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>For i = 1 ... N:
Compute dJ/dw_i for i = 1 ... M parameters
Let w_i = w_i - learning_rate * dJ/dw_i
</code></pre></div></div>
<p>(Note that the gradient terms <script type="math/tex">\frac{dJ}{dw_i}</script> should all be computed before applying the updates). Essentially, the gradient descent algorithm computes partial derivatives for all the parameters in our network, and updates the parameters by decrementing the parameters by their respective partial derivatives, times a constant known as the learning rate, taking a step towards a local minimum.</p>
<p>This means that the “speed” of learning is dictated by two things: the learning rate and the size of the partial derivative. The learning rate is a hyperparameter that we must tune, so we’ll focus on the size of the partial derivatives for now. Consider the following binary classification scenario: we have an input feature vector <script type="math/tex">x_i</script>, a label <script type="math/tex">y_i</script>, and a prediction <script type="math/tex">\hat{y_i} = h_\theta(x_i)</script>.</p>
<p>We’ll show that given our model <script type="math/tex">h_\theta(x) = \sigma(Wx_i + b)</script>, learning can occur much faster during the beginning phases of training if we used the cross-entropy loss instead of the MSE loss. And we want this to happen, since at the beginning of training, our model is performing poorly due to the weights being randomly initialized.</p>
<p>First, given our prediction <script type="math/tex">\hat{y_i} = \sigma(Wx_i + b)</script> and our loss <script type="math/tex">J = \frac{1}{2}(y_i - \hat{y_i})^2</script> , we first obtain the partial derivative <script type="math/tex">\frac{dJ}{dW}</script>, applying the chain rule twice:</p>
<script type="math/tex; mode=display">\frac{dJ}{dW} = (y_i - \hat{y_i})\sigma'(Wx_i + b)x_i</script>
<p>This derivative has the term <script type="math/tex">\sigma'(Wx_i + b)</script> in it. This can be expressed as <script type="math/tex">\sigma(Wx_i + b)(1 - \sigma(Wx_i + b))</script> (see here for a proof). Since we initialized our weights randomly with values close to 0, this expression will be very close to 0, which will make the partial derivative nearly vanish during the early stages of training. A plot of the sigmoid curve’s derivative is shown below [6], indicating that the gradients are small whenever the outputs are close to <script type="math/tex">0</script> or <script type="math/tex">1</script>:</p>
<p><img src="https://raw.githubusercontent.com/rohan-varma/rohan-blog/gh-pages/images/sigmoid_derivative.jpg" alt="sigmoid" /></p>
<p>This can lead to slower learning at the beginning stages of gradient descent, since the smaller derivatives change each weight by only a small amount, and gradient descent takes a while to get out of this loop and make larger updates towards a minima.</p>
<p>On the other hand, given the cross entropy loss:</p>
<script type="math/tex; mode=display">J = -\sum_{i=1}^{N} y_i\log(\sigma (Wx_i + b)) + (1-y_i)\log(1 - \sigma(Wx_i + b))</script>
<p>We can obtain the partial derivative <script type="math/tex">\frac{dJ}{dW}</script> as follows (with the substitution <script type="math/tex">\sigma(z) = \sigma(Wx_i + b)</script>:</p>
<script type="math/tex; mode=display">\frac{dJ}{dW} = -\sum_{i=1}^{N} \frac{y_i x_i\sigma'(z)}{\sigma(z)} - \frac{(1-y_i)x_i \sigma'(z)}{1 - \sigma(z)}</script>
<p>Simplifying, we obtain a nice expression for the gradient of the loss function with respect to the weights:</p>
<script type="math/tex; mode=display">\sum_{i=1}^{N} x_i(\sigma(z) - y_i)</script>
<p>This derivative does not have a <script type="math/tex">\sigma'</script> term in it, and we can see that the magnitude of the derivative is entirely dependent on the magnitude of our error <script type="math/tex">\sigma(z) - y_i</script> - how far off our prediction was from the ground truth. This is great, since that means early on in learning, the derivatives will be large, and later on in learning, the derivatives will get smaller and smaller, corresponding to smaller adjustments to the weight variables, which makes intuitive sense since if our error is small, then we’d want to avoid large adjustments that could cause us to jump out of the minima. Michael Nielsen in his <a href="https://neuralnetworksanddeeplearning">book</a> has an in-depth discussion and illustration of this that is really helpful.</p>
<h4 id="hinge-loss-vs-cross-entropy-loss">Hinge Loss vs Cross-Entropy Loss</h4>
<p>There’s actually another commonly used type of loss function in classification related tasks: the hinge loss. The (L2-regularized) hinge loss leads to the canonical support vector machine model with the max-margin property: the margin is the smallest distance from the line (or more generally, hyperplane) that separates our points into classes and defines our classification:</p>
<p><img src="https://docs.opencv.org/2.4.13.4/_images/optimal-hyperplane.png" alt="svm" /></p>
<p>The hinge loss penalizes predictions not only when they are incorrect, but even when they are correct but not confident. It penalizes gravely wrong predictions significantly, correct but not confident predictions a little less, and only confident, correct predictions are not penalized at all. Let’s formalize this by writing out the hinge loss in the case of binary classification:</p>
<script type="math/tex; mode=display">\sum_{i} max(0, 1 - y_{i} * h_{\theta}(x_{i}))</script>
<p>Our labels <script type="math/tex">y_{i}</script> are either -1 or 1, so the loss is only zero when the signs match and <script type="math/tex">\vert (h_{\theta}(x_{i}))\vert \geq 1</script>. For example, if our score for a particular training example was <script type="math/tex">0.2</script> but the label was <script type="math/tex">-1</script>, we’d incur a penalty of <script type="math/tex">1.2</script>, if our score was <script type="math/tex">-0.7</script> (meaning that this instance was predicted to have label <script type="math/tex">-1</script>) we’d still incur a penalty of <script type="math/tex">0.3</script>, but if we predicted <script type="math/tex">-1.1</script> then we would incur no penalty. A visualization of the hinge loss (in green) compared to other cost functions is given below:</p>
<p><img src="https://i.stack.imgur.com/4DFDU.png" alt="hinge loss" /></p>
<p>The main difference between the hinge loss and the cross entropy loss is that the former arises from trying to maximize the margin between our decision boundary and data points - thus attempting to ensure that each point is correctly and confidently classified*, while the latter comes from a maximum likelihood estimate of our model’s parameters. The softmax function, whose scores are used by the cross entropy loss, allows us to interpret our model’s scores as relative probabilities against each other. For example, the cross-entropy loss would invoke a much higher loss than the hinge loss if our (un-normalized) scores were <script type="math/tex">[10, 8, 8]</script> versus <script type="math/tex">[10, -10, -10]</script>, where the first class is correct. In fact, the (multi-class) hinge loss would recognize that the correct class score already exceeds the other scores by more than the margin, so it will invoke zero loss on both scores. Once the margins are satisfied, the SVM will no longer optimize the weights in an attempt to “do better” than it is already.</p>
<h4 id="wrap-up">Wrap-Up</h4>
<p>In this post, we’ve show that the MSE loss comes from a probabalistic interpretation of the regression problem, and the cross-entropy loss comes from a probabalistic interpretaion of binary classification. The MSE loss is therefore better suited to regression problems, and the cross-entropy loss provides us with faster learning when our predictions differ significantly from our labels, as is generally the case during the first several iterations of model training. We’ve also compared and contrasted the cross-entropy loss and hinge loss, and discussed how using one over the other leads to our models learning in different ways. Thanks for reading, and hope you enjoyed the post!</p>
<h4 id="sources">Sources</h4>
<ol>
<li>
<p><a href="https://neuralnetworksanddeeplearning.com/chap3.html">Michael Nielsen’s Neural Networks and Deep Learning, Chapter 3</a></p>
</li>
<li>
<p><a href="https://cs231n.github.io/linear-classify/">Stanford CS 231n notes on cross entropy and hinge loss</a></p>
</li>
<li>
<p><a href="https://docs.opencv.org/2.4.13.4/doc/tutorials/ml/introduction_to_svm/introduction_to_svm.html">OpenCV introduction to SVMs</a></p>
</li>
<li>
<p><a href="https://math.stackexchange.com/questions/782586/how-do-you-minimize-hinge-loss">StackExchange answer on hinge loss minimization</a></p>
</li>
<li>
<p><a href="https://www.cs.princeton.edu/courses/archive/fall16/cos402/lectures/402-lec5.pdf">Machine Learning, Princeton University</a></p>
</li>
<li>
<p><a href="https://ronny.rest/blog/post_2017_08_10_sigmoid/">Ronny Restrepo, sigmoid functions</a></p>
</li>
</ol>
<h4 id="notes">Notes</h4>
<p>[4/16/19] - Fixed broken links and clarified the particular model for which the learning speed of MSE loss is slower than cross-entropy</p>Paper Analysis - Sequence to Sequence Learning2018-01-02T00:00:00+00:002018-01-02T00:00:00+00:00https://rohanvarma.me/Seq-2-Seq<p><img src="https://raw.githubusercontent.com/rohan-varma/rohan-blog/gh-pages/images/seq2seq.png" alt="seq2seq" /></p>
<p><a href="https://arxiv.org/pdf/1409.3215.pdf">Link to paper</a>
<a href="https://github.com/rohan-varma/paper-analysis/blob/master/seq2seq-paper/Pytorch%20Seq%202%20Seq%20Model.ipynb">Link to example implementation</a></p>
<p><strong>Abstract</strong></p>
<ul>
<li>
<p>Traditional DNNs have achieved good performance whenever large labelled training datasets are available, but cannot map sequences to sequences</p>
</li>
<li>
<p>Main approach of the paper is to use a multilayer LSTM to map input sequence to a fixed-length vector, and then another deep LSTM to decode the fixed length vector into a sequence</p>
</li>
<li>
<p>The LSTM model also learned useful phrase & sentence representations that are sensitive to word order and invariant to passive/active voice</p>
</li>
<li>
<p>indicates that the actual structure of the language was mostly captured, the representation of “He ate the cookie” and “the cookie was eaten by him” aren’t too different</p>
</li>
<li>
<p>Reversing word order in source sentence helped because it introduced more short-term dependencies</p>
</li>
</ul>
<p><strong>Introduction</strong></p>
<ul>
<li>DNNs are powerful, example: 2 layer neural network of quadratic size can learn to sort n n-bit numbers</li>
<li>DNNs are only useful for problems who’s inputs and labels can be expressed as fixed-length vectors in some way</li>
<li>This is limiting, DNNs can’t do many tasks who’s inputs are best represented as sequences such as machine translation, POS tagging, speech recognition</li>
<li>Main idea: use one LSTM to read the input sequence (of variable length) one time step at a time, and map this to a fixed-length vector. Then a second LSTM takes this fixed-length vector as an input and produces an output sequence</li>
<li>The second LSTM is basically an RNN language model that is conditioned on the encoded representation</li>
<li>The researchers trained an ensemble of 5 deep LSTMs (each with 384 million parameters) and used a beam search decoder to achieve state of the art performance on the WMT english to french translation task</li>
<li>Reversing the words in the source sentence helped train the LSTMs a lot, because this introduced many more short term dependencies to make it easier to train the LSTM with SGD</li>
</ul>
<p><strong>Model</strong></p>
<ul>
<li>
<p>RNN inputs: sequence <script type="math/tex">x_1 … x_T</script> as inputs, computes a sequence of outputs <script type="math/tex">y_1 … y_T</script></p>
</li>
<li>
<p>At each timestep, RNN computes a hidden state <script type="math/tex">h_t</script> and an output <script type="math/tex">y_t</script>. We can think of the hidden state as encapsulating the information encountered at previous timesteps</p>
</li>
<li>
<script type="math/tex; mode=display">h_t = tanh(W_{hx}x_t + W_{hh}h_{t-1})</script>
</li>
<li>
<script type="math/tex; mode=display">y_t = W_{hy}h_t</script>
</li>
<li>
<p>This is for a “single-layer” RNN that does not have layers of hidden states. If there are multiple layers of hidden states, then instead of <script type="math/tex">x_t</script> as the input into a later hidden layer, the input is the <script type="math/tex">h_t</script> at the previous layer, same timestep</p>
</li>
<li>
<p>For general/variable-length sequence to sequence learning, the general idea is to map the input sequence to a fixed-length vector using one RNN and then map the fixed-length vector to the target sequence with another RNN</p>
</li>
<li>
<p>However, in practice RNNs aren’t very good with learning longer-term dependencies. LSTMs have been shown to do much better at learning longer-term dependencies because they don’t fall victim to the vanishing gradient problem like traditional RNN cells do</p>
</li>
<li>
<p>Goal of the LSTM is to estimate the following conditional probability:</p>
<ul>
<li><script type="math/tex">p(y_{1}, … y_{T'} \vert x_{1} … x_{T})</script> , where the length of the 2 sequences differ from each other</li>
<li>The LSTM does this by computing a fixed-dimensional representation <script type="math/tex">v</script> after observing the input sequence <script type="math/tex">x_1 … x_T</script></li>
<li>Then, conditioned on this <script type="math/tex">v</script>, we can produce the output sequence via the following formulation:
<ul>
<li>
<script type="math/tex; mode=display">p(y_1 | x_1 … x_T) = p(y_1 | v)</script>
</li>
<li>
<script type="math/tex; mode=display">P(y_1, y_2 | x_1 … x_T) = p(y_2 | v, y_1) p(y_1 | v)</script>
</li>
<li>i.e., at each time step the the <script type="math/tex">i</script>th output is conditioned on the fixed length vector <script type="math/tex">v</script> and the previous outputs, if any</li>
<li>In general, <script type="math/tex">P(y_1, y_2 … y_{T'} \vert x_1, … x_T) = \prod_{t = 1}^{T'} p(y_t \vert v, y_1… y_{T'-1})</script></li>
<li>Each of these distributions are represented with a softmax over the vocabulary</li>
</ul>
</li>
<li>2 different LSTMs are used since this doesn’t increase the number of model parameters by much and makes it easier to train the LSTM on multiple language pairs</li>
<li>A deep LSTM was used with four layers; this was found to significantly outperform shallow LSTMs</li>
<li>Reversed order of words in input sequences was found to help a lot</li>
<li>The dataset had 12m sentences with 348m french words and 304m english words</li>
</ul>
<p><strong>Training</strong></p>
<ul>
<li>Model was trained to maximize probability of producing a correct translation given a source sentence:</li>
<li>
<script type="math/tex; mode=display">\frac{1}{N} \sum_{i=1}^{N} \log p(T_i | S_i)</script>
</li>
<li>Translations are produced by finding the most likely translation given by the LSTM: <script type="math/tex">T* = argmax_T p(T \vert S)</script></li>
<li>Beam search used to find the most likely translations. At each time, we maintain a list of partial hypotheses and then extend each partial hypothesis with every word in the vocabulary. Then we discard all but <script type="math/tex">B</script> of the most likely hypotheses (where the likelihood is given by the model’s log probability).</li>
<li>A hypothesis is finished once the end-of-sentence tag <EOS> is emitted</EOS></li>
<li>As discussed, reversing the words in the source sentences helps the translation task a lot</li>
<li>An intuitive explanation for this is by reversing the source sentence, the average distance between corresponding words decreases so there is less of an overall time lag</li>
<li>Therefore, backpropagation has an easier time communicating between the source sentence and the target sentence.</li>
<li>Ex: “I like to eat the apples” and “Me gusta comer las manzanitas” vs “Apples the eat to like I” and “Me gusta comer las manzanitas”, the second pair has more words closer to the corresponding word in the translated sentence</li>
<li>A deep LSTM with 4 layers and 1000 cells at each layer was used. 100 dimensional word embeddings</li>
<li>Initialization was uniform random between -0.08 and 0.08</li>
<li>SGD without momentum with lr = 0.7, and then the learning rate was halved from epochs 5 to 7.5</li>
<li>batch size = 128</li>
<li>To avoid exploding gradients, the researchers enforced a hard cap on the norm of the gradients and the gradiewnts weres scaled down when the norm exceeded a threshold</li>
<li>Each layer was trained on a different GPU and communicated its activations when it was one. Spent about 10 days training</li>
</ul>
<p><strong>Results</strong></p>
<ul>
<li>
<p>The model achieved state of the art accuracy on english to french translation tasks</p>
</li>
<li>
<p>The fixed length vectors learned were pretty meaningful, in that they were sensitive to order of the words (i.e. John admires Mary was further away from Mary admires John, but Mary admires John and Mary respects John were relatively close together)</p>
</li>
<li>
<p>Other approaches included using convolutional networks to map sentences to fixed length vectors, using attention mechanisms to overcome issues with long sentence translation, or taking phrase-based approaches to achieve smoother translations</p>
</li>
</ul>
</li>
</ul>Interpreting Regularization as a Bayesian Prior2017-08-24T00:00:00+00:002017-08-24T00:00:00+00:00https://rohanvarma.me/Regularization<p><img src="https://raw.githubusercontent.com/rohan-varma/rohan-blog/gh-pages/images/reg.png" alt="img" /></p>
<h3 id="introductionbackground">Introduction/Background</h3>
<p>In machine learning, we often start off by writing down a probabalistic model that defines our data. We then go on to write down a likelihood or some type of loss function, which we then optimize over to get the optimal settings for the parameters that we seek to estimate. Along the way, techniques such as regularization, hyperparameter tuning, and cross-validation can be used to ensure that we don’t overfit on our training dataset and our model generalizes well to unseen data.</p>
<p>Specifically, we have a few key functions and variables: the underlying probability distribution <script type="math/tex">p(x, y)</script> which generate our training examples (pairs of features and labels), a training set <script type="math/tex">(x, y)_{i = 1}^{D}</script> of <script type="math/tex">D</script> examples which we observe, and a model <script type="math/tex">h(x) : x \rightarrow{} y</script> which we wish to learn in order to produce a mapping from <script type="math/tex">x</script> to <script type="math/tex">y</script>. This function <script type="math/tex">h</script> is selected from a larger function space <script type="math/tex">H</script>.</p>
<p>For example, if we are in the context of linear regression models, then all functions in the function space of <script type="math/tex">H</script> will take on the form <script type="math/tex">y_i = x_{i}^T \beta</script> where a particular setting of our parameters <script type="math/tex">\beta</script> will result in a particular <script type="math/tex">h(x)</script>. We also have some function <script type="math/tex">L(h(x), y)</script> that takes in our predictions and labels, and quantifies how accurate our model is across some data.</p>
<p>Ideally, we’d like to minimize the risk function</p>
<script type="math/tex; mode=display">R[h(x)] = \sum_{(x, y)} L( h(x), y) p(x, y)</script>
<p>across all possible <script type="math/tex">(x, y)</script> pairs. However, this is impossible since we don’t know the underlying probability distribution that describes our dataset, so instead we seek to approximate the risk function by minimizing a loss function across the data that we have observed:</p>
<script type="math/tex; mode=display">\frac{1}{N} \sum_{i = 1}^{N} L(h(x), y)</script>
<h3 id="linear-models">Linear Models</h3>
<p>If we assume that our data are roughly linear, then we can write a relationship between our features and real-valued outputs: <script type="math/tex">y_i = x_i^T \beta + \epsilon</script> where <script type="math/tex">\epsilon \tilde{} N(0, \sigma^2)</script>. This essentially means that our data has a linear relationship that is corrupted by random Gaussian noise that has zero mean and constant variance.</p>
<p>This has the implication that <script type="math/tex">y_i</script> is a Gaussian random variable, and we can compute its expectation and variance:</p>
<script type="math/tex; mode=display">E[y_i] = E[x_i^T \beta + \epsilon] = x_i^T \beta</script>
<script type="math/tex; mode=display">Var[y_i] = Var[x_i^T \beta + \epsilon] = \sigma^2</script>
<p>We can now write down the probability of observing a value <script type="math/tex">y_i</script> given a certain set of features <script type="math/tex">x</script>:</p>
<script type="math/tex; mode=display">p(y_i | x_i) = N(y_i | x_i^T \beta, \sigma^2)</script>
<p>Next, we can write down the probability of observing the entire dataset of <script type="math/tex">(x, y)</script> pairs. This is known as the likelihood, and it’s simply the product of observing each of the individual feature, label pairs:</p>
<script type="math/tex; mode=display">L(x,y) = \prod_{i = 1}^{n} N(y_i | x_i \beta, \sigma^2)</script>
<p>As a note, writing down the likelihood this way does assume that our training data are independent and identically distributed, meaning that we are assuming that each of the training samples have the same probability distribution, and are mutually independent.</p>
<p>If we want to find the <script type="math/tex">\hat{\beta}</script> that maximizes the chance of us observing the training examples that we observed, then it makes sense to maximize the above likelihood. This is known as <strong>maximum likelihood estimation</strong>, and is a common approach to many machine learning problems such as linear and logistic regression.</p>
<p>In other words, we want to find</p>
<script type="math/tex; mode=display">\hat{\beta} = argmax_{\beta} \prod_{i = 1}^{n} N(y_i | x_i \beta, \sigma^2)</script>
<p>To simplify this a little bit, we can write out the normal distribution, and also take the log of the function, since the <script type="math/tex">\hat{\beta}</script> that maximizes <script type="math/tex">L</script> will also maximize <script type="math/tex">log(L)</script>. We end up with</p>
<script type="math/tex; mode=display">\hat{\beta} = argmax_{\beta} log \prod_{i = 1}^{n} \frac{1}{\sqrt(2 \pi \sigma^2}e^-\frac{(y_i - x_i \beta)^2}{2 \sigma^2}</script>
<p>Distributing the log and dropping constants (since they don’t affect the value of our parameter which maximizes the expression), we obtain</p>
<script type="math/tex; mode=display">\hat{\beta} = argmax_{\beta} \sum_{i = 1}^{N} -(y_i - x_i \beta)^2</script>
<p>Since minimizing the opposite of a function is the same as maximizing it, we can turn the above into a minimization problem:</p>
<script type="math/tex; mode=display">\hat{\beta} = argmin_{\beta} \sum_{i = 1}^{N} (y_i - x_i \beta)^2</script>
<p>This is the familiar least squares estimator, which says that the optimal parameter is the one that minimizes the <script type="math/tex">L2</script> squared norm between the predictions and actual values. We can use gradient descent with some initial setting of <script type="math/tex">\beta</script> and be guaranteed to get to a global minimum (since the function is convex) or we can explicitly solve for <script type="math/tex">\beta</script> and obtain the same answer.</p>
<p>Right now is a good time to think about the assumptions of this linear regression model. Like many models, it assumes that the data are drawn independently from the same data generating distribution. Furthermore, it assumes that this distribution is normal with a linear mean and constant variance. It also has a more implicit assumption: that the parameter <script type="math/tex">\beta</script> which we wish to estimate is not a random variable itself, and we will show how relaxing this assumption leads to a regularized linear model.</p>
<h3 id="regularization">Regularization</h3>
<p>Regularization is a popular approach to reducing a model’s predisposition to overfit on the training data and thus hopefully increasing the generalization ability of the model. Previously, we sought to learn the optimial <script type="math/tex">h(x)</script> from the space of functions <script type="math/tex">H</script>. However, if the whole function space can be explored, and our samples were observed with some amount of noise, then the model will likely select a function that overfits on the observed data. One way we can combat this is by limiting our search to a subspace within <script type="math/tex">H</script>, and this is exactly what regularization does.</p>
<p>To regularize a model, we take our loss function and add a regularizer to it. Regularizers take the form <script type="math/tex">\lambda R(\beta)</script> where <script type="math/tex">R(\beta)</script> is some function of our parameters, and <script type="math/tex">\lambda</script> is a hyperparameter describing our regularization constant. Using this rule, we can write out a regularized version of our loss function above, giving us a model known as ridge regression:</p>
<script type="math/tex; mode=display">\hat{\beta} = argmin_{\beta} \sum_{i = 1}^{N} (y_i - x_i \beta)^2 + \lambda \sum_{i = 1}^{j} \beta_j^2</script>
<p>What’s interesting about regularization is that it can be more deeply understood if we reconsider our original probabalistic model. In our original model, we conditioned our outputs on a linear function of the parameter which we wish to learn <script type="math/tex">\beta</script>. Instead of considering <script type="math/tex">\beta</script> as a fixed quantity that we want a point estimate of, we can consider <script type="math/tex">\beta</script> itself as a random variable drawn from a certain probability distribution. This is known as the <strong>prior</strong> probability distribution, because we assign <script type="math/tex">\beta</script> some probability without having observed the associated <script type="math/tex">(x, y)</script> pairs. Imposing a prior would be especially useful if we had some information about the parameter before observing any of the training data (possibly from domain knowledge), but it turns out that imposing a Gaussian prior even in the absence of actual prior knowledge leads to interesting properties (see the <a href="http://www.deeplearningbook.org/">Deep Learning Book chapter on regularization</a> for more details about this). In particular, we can condition <script type="math/tex">\beta</script> as on a Gaussian with 0 mean and constant variance [1]:</p>
<script type="math/tex; mode=display">\beta \tilde{} N(0, \lambda^{-1})</script>
<p>As a consequence, we must adjust our probability of observing a particular <script type="math/tex">(x, y)</script> pair to accommodate the probability of observing the parameter that generated this pair. We obtain a new expression for our likelihood:</p>
<script type="math/tex; mode=display">L(x,y) = \prod_{i = 1}^{n} N(y_i | x_i \beta, \sigma^2) N(\beta | 0, \lambda^{-1})</script>
<p>Similar to the previously discussed method of maximum likelihood estimation, we can estimate the parameter <script type="math/tex">\beta</script> to be the <script type="math/tex">\hat{\beta}</script> that maximizes the above function:</p>
<script type="math/tex; mode=display">\hat{\beta} = argmax_{\beta} \sum_{i = 1}^{N} log N(y_i | x_i \beta, \sigma^2) + log N(\beta | 0, \lambda^{-1})</script>
<p>This is the maximum a posteriori estimate of <script type="math/tex">\beta</script>, and it only differs from the maximum likelihood estimate in that the former takes into account previous information, or a prior distribution, on the parameter <script type="math/tex">\beta</script>. In fact, the maximum likelihood estimate of the parameter can be seen as a special case of the maximum a posteriori estimate, where we take the prior probability distribution on the parameter to just be a constant.</p>
<p>Since (dropping unneeded constants) <script type="math/tex">N(\beta, 0, \lambda^{-1}) = exp(\frac{- \beta^{2}}{2 \lambda^{-1}})</script>, after taking the log, and minimizing the negative of the above function we obtain the familiar regularizer <script type="math/tex">\frac{1}{2} \lambda \beta^2</script> and our squared loss function <script type="math/tex">\sum_{i = 1}^{N} (y_i - x_i \beta)^2</script> is the same as the loss function we obtained without regularization. In this way, <script type="math/tex">L2</script> regularization on a linear model can be thought of as imposing a Bayesian prior on the underlying parameters which we wish to estimate.</p>
<h3 id="aside-interpreting-regularization-in-the-context-of-bias-and-variance">Aside: interpreting regularization in the context of bias and variance</h3>
<p>The error of a statistical model can be decomposed into three distinct sources of error: error due to bias, error due to variance, and irreducible error. They are related as follows:</p>
<script type="math/tex; mode=display">Err(x) = bias(X)^2 + var(x) + \epsilon</script>
<p>Given a constant error, this means that there will always be a tradeoff between bias and variance. Having too much bias or too much variance isn’t good for a model, but for different reasons. A high bias, low variance model will likely end up being inaccurate across both the training and testing datasets, and its predictions will likely not deviate too much based on the data sample it is trained on. On the other hand, a low-bias, high-variance model will likely give good results on a training dataset, but fail to generalize as well on a testing dataset.</p>
<p>The Gauss-Markov theorem states that in a linear regression problem, the least squares estimator has the lowest variance out of all other unbiased estimators. However, if we consider biased estimators such as the estimator given by ridge regression, we can arrive at a lower variance, higher-bias solution. In particular, the expectation of the ridge estimator (derived <a href="http://math.bu.edu/people/cgineste/classes/ma575/p/w14_1.pdf">here</a>) is given by:</p>
<script type="math/tex; mode=display">\beta - \lambda (X^TX + \lambda I)^{-1} \beta</script>
<p>The bias of an estimator is defined as the difference between the parameter’s expected value and the true parameter <script type="math/tex">\beta</script>: <script type="math/tex">bias(\hat{\beta}) = E[\hat{\beta}] - \beta</script></p>
<p>As you can see, the bias is proportional to <script type="math/tex">\lambda</script> and <script type="math/tex">\lambda = 0</script> gives us the unbiased least squares estimator since <script type="math/tex">E[\hat{\beta}] = \beta</script>. Therefore, assuming a constant total error for the least squares estimator and the ridge estimator, the variance for the ridge estimator is lower. A more complete discussion, including formal calculations for the bias and variance of the ridge estimator compared to the least squares estimator, is given <a href="https://math.bu.edu/people/cgineste/classes/ma575/p/w14_1.pdf">here</a>.</p>
<h3 id="a-linear-algebra-perspective">A linear algebra perspective</h3>
<p>To see why regularization makes sense from a linear algebra perspective, we can write down our least squares estimate in vectorized form:</p>
<script type="math/tex; mode=display">argmin_{\beta} { (y - X\beta)^T (y - X \beta) }</script>
<p>Next, we can expand this and simplify a little bit:</p>
<script type="math/tex; mode=display">argmin_{\beta} (y^T - \beta^TX^T)(y - X\beta)</script>
<script type="math/tex; mode=display">= argmin_{\beta} -2y^TX\beta + \beta^TX^TX\beta</script>
<p>where we have dropped the terms that are not a factor of <script type="math/tex">\beta</script> since they will zero out when we differentiate.</p>
<p>To minimize, we differentiate with respect to <script type="math/tex">\beta</script>:</p>
<script type="math/tex; mode=display">\frac{\delta L}{\delta \beta} = -2 y^TX + 2X^TX\beta</script>
<p>Setting the derivative equal to zero gives us the closed form solution of <script type="math/tex">\beta</script> which is the least-squares estimate [2]:</p>
<script type="math/tex; mode=display">\hat{\beta} = (X^TX)^{-1} y^TX</script>
<p>As we can see, in order to actually compute this quantity the matrix <script type="math/tex">X^T X</script> must be invertible. The matrix <script type="math/tex">X^T X</script> being invertible corresponds exactly to showing that the matrix is positive definite, which means that the scalar quantity <script type="math/tex">z^T X^T X z > 0</script> for any real, non-zero vectors <script type="math/tex">z</script>. However, the best we can do is show that <script type="math/tex">X^T X</script> is positive semidefinite.</p>
<p>To show that <script type="math/tex">X^TX</script> is positive semidefinite, we must show that the quantity <script type="math/tex">z^T X^T X z \geq 0</script> for any real, non-zero vectors <script type="math/tex">z</script>.</p>
<p>If we expand out the quantity <script type="math/tex">X^T X</script>, we obtain <script type="math/tex">\sum_{i = 1}^{N} x_i x_i^T</script> and it follows that the quantity <script type="math/tex">z^T (\sum_{i = 1}^{N} x_i x_i^T) z = \sum_{i = 1}^{N} (x_i^Tz)^2 \geq 0</script>. This means that in sitautions where this quantity is exactly <script type="math/tex">0</script>, the matrix <script type="math/tex">X^T X</script> cannot be inverted and a closed-form least squares solution cannot be computed.</p>
<p>On the other hand, expanding out our ridge estimate which has an extra regulariztion term <script type="math/tex">\lambda \sum_{i} \beta_i^2</script>, we obtain the derivative</p>
<script type="math/tex; mode=display">\frac{\delta L}{\delta \beta} = -2 y^TX + 2X^TX\beta + 2 \lambda \beta</script>
<p>Setting this quantity equal to zero, and rewriting <script type="math/tex">\lambda \beta</script> as <script type="math/tex">\lambda I \beta</script> (using the property of multiplication with the identity matrix), we now obtain</p>
<script type="math/tex; mode=display">\beta (X^TX + \lambda I) = y^T X</script>
<p>giving us the ridge estimate</p>
<script type="math/tex; mode=display">\hat{\beta_{ridge}} = (X^TX + \lambda I)^{-1} y^TX</script>
<p>The only difference in this closed-form solution is the addition of the <script type="math/tex">\lambda I</script> term to the quantity that gets inverted, so we are now sure that this quantity is positive definite if <script type="math/tex">\lambda > 0</script>. In other words, even when the matrix <script type="math/tex">X^T X</script> is not invertible, we can still compute a ridge estimate from our data [3].</p>
<h3 id="regularizers-in-neural-networks">Regularizers in neural networks</h3>
<p>While techniques such as L2 regularization can be used while training a neural network, employing techniques such as dropout, which randomly discards some proportion of the activations at a per-layer level during training, have been shown to be much more successful. There is also a different type of regularizer that takes into account the idea that a neural network should have sparse activations for any particular input. There are several theoretical reeasons for why sparsity is important, a topic covered very well by Glorot et al. in a <a href="https://proceedings.mlr.press/v15/glorot11a/glorot11a.pdf">2011 paper</a>.</p>
<p>Since sparsity is important in neural networks, we can introduce a constraint that can gaurantee us some degree of sparsity. Specifically, we can constrain the average activation of a particular neuron in a particular hidden layer.</p>
<p>In particular, the average activation of a neuron in a particular layer, weighted by the input into the neuron, can be given by summing over all of the activation - input pairs: <script type="math/tex">\hat{\rho} = \frac{1}{m} \sum_{i = 1}^{N} x_i a_i^2</script>. Next, we can choose a hyperparameter <script type="math/tex">\rho</script> for this particular neuron, which represents the average activation we want it to have - for example, if we wanted this neuron to activate sparsely, we might set <script type="math/tex">\rho = 0.05</script>. In order to ensure that our model learns neurons which sparsely activate, we must incorporate some function of <script type="math/tex">\hat{\rho}</script> and <script type="math/tex">\rho</script> into our cost function.</p>
<p>One way to do this is with the <a href="https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence">KL divergence</a>, which computes how much one probability distribution (in this case, our current average activation <script type="math/tex">\hat\rho</script>) and another expected probability distribution (<script type="math/tex">\rho</script>) diverge from each other. If we minimize the KL divergence for each of our neuron’s activations then our model will learn sparse activations. The cost function may be:</p>
<script type="math/tex; mode=display">J_{sparse} (W, b) = J(W, b) + \lambda \sum_{i = 1}^{M} KL(\rho_i || \hat{\rho_i})</script>
<p>where <script type="math/tex">J(W, b)</script> is a regular cost function used in neural networks, such as the cross-entropy loss. The hyperparameter <script type="math/tex">\lambda</script> indicates how important sparsity is to us - as <script type="math/tex">\lambda \rightarrow{} \infty</script>, we disregard the actual loss function and only aim to learn a sparse representation, and as <script type="math/tex">\lambda \rightarrow{} 0</script> we disregard the importance of sparse activations and only minimize the original loss function. Additional details on this type of regularization with application to sparse autoencoders are given <a href="https://ufldl.stanford.edu/wiki/index.php/Autoencoders_and_Sparsity">here</a>.</p>
<h3 id="recap">Recap</h3>
<p>As we have seen, regularization can be interpreted in several different ways, each of which gives us additional insight into what exactly regularization accomplishes. A few of the different interpretations are:</p>
<p>1) As a Bayesian prior on the paramaters which we are trying to learn.</p>
<p>2) As a term added to the loss function of our model which penalizes some function of our parameters, thereby introducing a tradeoff between minimizing the original loss function and ensuring our weights do not deviate too much from what we want them to be.</p>
<p>3) As a constraint on the model which we are trying to learn. This means we can take the original optimization problem and frame it in a constrained fashion, thereby ensuring that the magnitude of our weights never exceed a certain threshold (in the case of <script type="math/tex">L2</script> regularization).</p>
<p>4) As a method of reducing the function search space <script type="math/tex">H</script> to a new function search space <script type="math/tex">H'</script> that is smaller than <script type="math/tex">H</script>. Without regularization, we may search for our optimal function <script type="math/tex">h</script> in a much larger space, and constraining this to a smaller subspace can lead us to select models with better generalization ability.</p>
<p>Overall, regularization is a useful technique that is often employed to reduce the overall variance of a model, thereby improving its generalization capability. Of course, there’s tradeoffs in using regularization, most notably having to tune the hyperparameter <script type="math/tex">\lambda</script> which can be costly in terms of computational time. Thanks for reading!</p>
<h3 id="sources">Sources</h3>
<ol>
<li>
<p><a href="https://math.bu.edu/people/cgineste/classes/ma575/p/w14_1.pdf">Boston University Linear Models Course by Cedric Ginestet</a></p>
</li>
<li>
<p><a href="https://ufldl.stanford.edu/wiki/index.php/Autoencoders_and_Sparsity">Autoencoders and Sparsity, Stanford UFDL</a></p>
</li>
<li>
<p><a href="https://math.stackexchange.com/questions/1582348/simple-example-of-maximum-a-posteriori/1582407">Explanation of MAP Estimation</a></p>
</li>
</ol>
<h4 id="notes">Notes</h4>
<p>[1] Imposing different prior distributions on the parameter leads to different types of regularization. A normal distribution with zero mean and constant variance leads to <script type="math/tex">L2</script> regularization, while a Laplacean prior would lead to <script type="math/tex">L1</script> regularization.</p>
<p>[2] Technically, we’ve only shown that the <script type="math/tex">\hat{\beta}</script> we’ve found is a local optimum. We actually want to verify that this is indeed a global minimum, which can be done by showing that the function we are minimizing is convex.</p>
<p>[3] For completeness, it is worth mentioning that there are other solutions if the inverse of the matrix <script type="math/tex">X^T X</script> does not exist. One common workaround is to use the <a href="https://en.wikipedia.org/wiki/Moore%E2%80%93Penrose_pseudoinverse">Moore-Penrose Psuedoinverse</a> which can be computed using the singular value decompisition of the matrix being psuedo-inverted. This is commonly used in implementations of PCA algorithms.</p>
<p>[7/15/19] - Added a note abou the underlying parameters of a model coming from a prior probability distribution.</p>