[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"work:node-bottleneck:en":3},[4,29,32,35,37,39,41,43],{"type":5,"items":6},"toc",[7,11,14,17,20,23,26],{"id":8,"level":9,"title":10},"events-scare-me",2,"Events Scare Me",{"id":12,"level":9,"title":13},"first-hypothesis-the-next-js-cache","First Hypothesis: The Next.js Cache",{"id":15,"level":9,"title":16},"second-hypothesis-something-is-blocking-the-event-loop","Second Hypothesis: Something Is Blocking the Event Loop!",{"id":18,"level":9,"title":19},"remove-work-that-node-does-not-need-to-do","Remove Work That Node Does Not Need to Do",{"id":21,"level":9,"title":22},"the-event-returns","The Event Returns",{"id":24,"level":9,"title":25},"then-was-gc-the-cause","Then Was GC the Cause?",{"id":27,"level":9,"title":28},"closing-thoughts","Closing Thoughts",{"type":30,"html":31},"html","\u003Ch2 id=\"events-scare-me\">Events Scare Me\u003C\u002Fh2>\n\u003Cp>\u003Cimg src=\"__WORK_ASSET__event-load-overview.webp\" alt=\"\" loading=\"lazy\">\u003C\u002Fp>\n\u003Cp>During an event, user traffic surged and the site’s response time began to slow down.\u003C\u002Fp>\n\u003Cp>The trace logs showed that the APIs were fast, but all the load was occurring on the web servers.\u003C\u002Fp>\n",{"type":33,"name":34},"directive","node-load-test-chart",{"type":30,"html":36},"\u003Cp>The APIs were fast, but page responses were far too slow. Most of the load was occurring in Next.js.\u003C\u002Fp>\n\u003Cp>Profiling with our monitoring tool revealed two symptoms.\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Delayed Next.js fetch-cache I\u002FO\u003C\u002Fli>\n\u003Cli>Delays inside the runtime that were not classified under a clear application span\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>There were only three days left before the next event, so I had to find a way to fix it.\u003C\u002Fp>\n\u003Ch2 id=\"first-hypothesis-the-next-js-cache\">First Hypothesis: The Next.js Cache\u003C\u002Fh2>\n\u003Cp>Under load, reading the fetch cache into memory and writing it to the system were delayed. At the time, the environment used a PM2-based cluster with multiple Node instances running on a single server.\u003C\u002Fp>\n\u003Cp>All of these instances read and wrote cache data under the same directory. I thought that moving them to a shared cache store would reduce file-system I\u002FO load and might improve performance.\u003C\u002Fp>\n\u003Cp>I immediately added a custom Next.js cache handler that used a shared Redis cache.\u003C\u002Fp>\n\u003Cpre class=\"shiki github-light\" style=\"background-color:#fff;color:#24292e\" tabindex=\"0\">\u003Ccode>\u003Cspan class=\"line\">\u003Cspan style=\"color:#D73A49\">import\u003C\u002Fspan>\u003Cspan style=\"color:#24292E\"> { createClient } \u003C\u002Fspan>\u003Cspan style=\"color:#D73A49\">from\u003C\u002Fspan>\u003Cspan style=\"color:#032F62\"> 'redis'\u003C\u002Fspan>\u003Cspan style=\"color:#24292E\">;\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#D73A49\">const\u003C\u002Fspan>\u003Cspan style=\"color:#005CC5\"> redisClient\u003C\u002Fspan>\u003Cspan style=\"color:#D73A49\"> =\u003C\u002Fspan>\u003Cspan style=\"color:#6F42C1\"> createClient\u003C\u002Fspan>\u003Cspan style=\"color:#24292E\">({\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#24292E\">\turl: process.env.\u003C\u002Fspan>\u003Cspan style=\"color:#005CC5\">REDIS_URL\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#D73A49\">\t...\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#24292E\">)}\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#D73A49\">...\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fpre>\u003Cp>\u003Cimg src=\"__WORK_ASSET__redis-cache-handler-results.webp\" alt=\"\" loading=\"lazy\">\u003C\u002Fp>\n\u003Cp>The structural change greatly improved heap memory efficiency and resilience against OOM errors, but it did not improve performance very much under load.\u003C\u002Fp>\n",{"type":33,"name":38},"node-redis-cache-chart",{"type":30,"html":40},"\u003Cp>(Locust, 30 users, 1,000 RPS, same scenario)\u003C\u002Fp>\n\u003Cp>It seemed that I had taken the wrong direction, so I tried the next approach.\u003C\u002Fp>\n\u003Ch2 id=\"second-hypothesis-something-is-blocking-the-event-loop\">Second Hypothesis: Something Is Blocking the Event Loop!\u003C\u002Fh2>\n\u003Cp>Node processes work within a single event loop. The API returned a response in only 50 ms, but another 7,000 ms passed afterward. This suggested that runtime logic, rather than network communication, might be blocking the event loop and delaying the response.\u003C\u002Fp>\n\u003Cp>I began to suspect the cause only after calling the most frequently requested API. It returned a huge 500 KB JSON response.\u003C\u002Fp>\n\u003Cpre class=\"shiki github-light\" style=\"background-color:#fff;color:#24292e\" tabindex=\"0\">\u003Ccode>\u003Cspan class=\"line\">\u003Cspan style=\"color:#24292E\">{\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#005CC5\">\t\"itemCode\"\u003C\u002Fspan>\u003Cspan style=\"color:#24292E\">: \u003C\u002Fspan>\u003Cspan style=\"color:#032F62\">\"abc123\"\u003C\u002Fspan>\u003Cspan style=\"color:#24292E\">,\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#B31D28;font-style:italic\">\t...\u003C\u002Fspan>\u003Cspan style=\"color:#B31D28;font-style:italic\"> [500kb]\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#24292E\">}\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fpre>\u003Cp>With traffic remaining high, \u003Ca href=\"https:\u002F\u002Fnodejs.org\u002Flearn\u002Fasynchronous-work\u002Fdont-block-the-event-loop\" target=\"_blank\" rel=\"noreferrer\">serializing and deserializing JSON of this size can put considerable load on Node\u003C\u002Fa>.\u003C\u002Fp>\n\u003Cpre class=\"shiki github-light\" style=\"background-color:#fff;color:#24292e\" tabindex=\"0\">\u003Ccode>\u003Cspan class=\"line\">\u003Cspan style=\"color:#D73A49\">const\u003C\u002Fspan>\u003Cspan style=\"color:#005CC5\"> response\u003C\u002Fspan>\u003Cspan style=\"color:#D73A49\"> =\u003C\u002Fspan>\u003Cspan style=\"color:#D73A49\"> await\u003C\u002Fspan>\u003Cspan style=\"color:#6F42C1\"> fetch\u003C\u002Fspan>\u003Cspan style=\"color:#24292E\">(apiUrl)\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003Cspan style=\"color:#D73A49\">return\u003C\u002Fspan>\u003Cspan style=\"color:#D73A49\"> await\u003C\u002Fspan>\u003Cspan style=\"color:#24292E\"> response.\u003C\u002Fspan>\u003Cspan style=\"color:#6F42C1\">json\u003C\u002Fspan>\u003Cspan style=\"color:#24292E\">()\u003C\u002Fspan>\u003C\u002Fspan>\n\u003Cspan class=\"line\">\u003C\u002Fspan>\u003C\u002Fcode>\u003C\u002Fpre>\u003Cp>As a first step, I removed unnecessary fields from the response and reduced it to about 150 KB. That was still not small enough, so it needed further optimization.\u003C\u002Fp>\n\u003Cp>gzip could reduce the amount of data transferred, but it could not remove the cost of parsing and serializing JSON inside Node. If Node also performed the compression, it might consume even more of the already limited CPU. Splitting the JSON into smaller pieces would have required redesigning the business logic, and there was not enough time for that.\u003C\u002Fp>\n\u003Cp>After thinking it over, another idea came to mind.\u003C\u002Fp>\n\u003Ch2 id=\"remove-work-that-node-does-not-need-to-do\">Remove Work That Node Does Not Need to Do\u003C\u002Fh2>\n\u003Cp>I wondered whether this was a structural problem. Did Next.js need to process this API response in the first place?\u003C\u002Fp>\n\u003Cp>If the Next.js proxy was only acting as a pass-through without applying any logic, the structure itself could be the bottleneck. The public API was exactly such a case because it only exchanged JSON and required no separate authentication.\u003C\u002Fp>\n\u003Cp>\u003Cimg src=\"__WORK_ASSET__direct-routing-architecture.svg\" alt=\"\" loading=\"lazy\">\u003C\u002Fp>\n\u003Cp>I changed the routing so that APIs that only exchanged JSON did not pass through the proxy. The performance changed immediately. Under the same test environment as before, the Node load finally dropped considerably.\u003C\u002Fp>\n",{"type":33,"name":42},"node-direct-routing-chart",{"type":30,"html":44},"\u003Cp>(Locust, 30 users, 1,000 RPS, same scenario)\u003C\u002Fp>\n\u003Ch2 id=\"the-event-returns\">The Event Returns\u003C\u002Fh2>\n\u003Cp>The scheduled event began three days later.\u003C\u002Fp>\n\u003Cp>The conditions were not exactly the same. But when I compared the first 30 minutes of events with similar TPS levels, the Node application’s average response time had dropped by about 70%.\u003C\u002Fp>\n\u003Cp>Node responses also no longer slowed sharply right after the event began, and the event ended without any notable failures.\u003C\u002Fp>\n\u003Ch2 id=\"then-was-gc-the-cause\">Then Was GC the Cause?\u003C\u002Fh2>\n\u003Cp>Performance improved greatly after removing the Node proxy layer, but this result alone did not fully explain why. Was it simply because there were fewer requests to process, or had something else improved as well?\u003C\u002Fp>\n\u003Cp>After the event, I looked again at the heap usage graphs. Comparing periods with the same TPS before and after the change showed a clear difference in GC frequency.\u003C\u002Fp>\n\u003Cp>\u003Cimg src=\"__WORK_ASSET__gc_frequency_comparison.svg\" alt=\"\" loading=\"lazy\">\u003C\u002Fp>\n\u003Cp>Parsing and processing a 500 KB JSON response on every request would have created many temporary objects at once. I believe these objects filled the Young Generation quickly, causing frequent minor GCs that also paused the event loop.\u003C\u002Fp>\n\u003Cp>\u003Cimg src=\"__WORK_ASSET__js-gc-example.webp\" alt=\"\" loading=\"lazy\">\u003C\u002Fp>\n\u003Cp>I cannot be completely certain, but the problem may have been the “post-processing after fast communication.” Removing the Node proxy layer eliminated the processing of large JSON responses, and the number of GCs fell along with it.\u003C\u002Fp>\n\u003Ch2 id=\"closing-thoughts\">Closing Thoughts\u003C\u002Fh2>\n\u003Cp>Trying something under uncertain conditions is difficult. But that seems to make solving it feel even more rewarding.\u003C\u002Fp>\n\u003Cp>This time, I am still uneasy because the problem was resolved without identifying the exact cause. Next time, I hope to be a better engineer who can find the cause even under uncertain conditions.\u003C\u002Fp>\n"]