feed.xml

<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="4.4.1">Jekyll</generator><link href="https://ibug.io/feed.xml" rel="self" type="application/atom+xml" /><link href="https://ibug.io/" rel="alternate" type="text/html" /><updated>2026-03-11T07:47:26+00:00</updated><id>https://ibug.io/feed.xml</id><title type="html">iBug</title><subtitle>The little personal site for iBug</subtitle><author><name>iBug</name></author><entry><title type="html">BanD Width! It’s MyBOND!!</title><link href="https://ibug.io/blog/2026/02/bonding-iperf3/" rel="alternate" type="text/html" title="BanD Width! It’s MyBOND!!" /><published>2026-02-16T00:00:00+00:00</published><updated>2026-02-24T11:27:10+00:00</updated><id>https://ibug.io/blog/2026/02/bonding-iperf3</id><content type="html" xml:base="https://ibug.io/blog/2026/02/bonding-iperf3/"><![CDATA[<p>My workstation in my univerity lab has consistently generated some 600&nbsp;TiB of annual upload across two PT sites. After an upgrade to the room a few months back, it received the most wanted upgrade for years: A second 1 Gbps line to the campus network. I immediately made the two NICs into one bonded 2 Gbps interface, and it has shined for many times with multiple popular torrents.</p>
			<p>Inspired by my friend <a href="https://blog.yfycloud.site/about/">@TheRainstorm</a> who managed to double his upload by load-balancing WireGuard over two lines from the same ISP (China Mobile), I figured I’d leverage this chance and make a better understanding though some more detailed experiments.</p>
			<h2 id="setup">Setup</h2>
			<h3 id="sender">Sender</h3>
			<p>My workstation:</p>
			<ul>
				<li>Ubuntu 24.04 (Kernel 6.8)</li>
				<li>Two Intel I210 NICs, connected to the same switch of campus network</li>
				<li>iperf 3.16</li>
			</ul>
			<p>Controlled variables:</p>
			<ul>
				<li>Bond mode: Round-robin (<code class="language-plaintext highlighter-rouge">balance-rr</code>, 0) vs Transmit Load Balancing (<code class="language-plaintext highlighter-rouge">balance-tlb</code>, 5)
					<ul>
						<li>For <code class="language-plaintext highlighter-rouge">balance-tlb</code> mode, <code class="language-plaintext highlighter-rouge">xmit_hash_policy</code> is set to <code class="language-plaintext highlighter-rouge">layer3+4</code>.</li>
					</ul>
				</li>
				<li>TCP congestion control (CC) algorithm: CUBIC vs BBR</li>
				<li>Parallelism (the value to <code class="language-plaintext highlighter-rouge">-P</code> option of iperf3): 1 vs 4
					<ul>
						<li>Note that due to the way <code class="language-plaintext highlighter-rouge">balance-tlb</code> works, with only one connection, bonding is not going to be different than using only one NIC.
							&lt;!–</li>
					</ul>
				</li>
				<li>Whether another application is constantly generating background upload activity.
					<ul>
						<li>This is implemented by letting qBittorrent run in the background with upload rate limited to 1&nbsp;MB/s, and is labeled <code class="language-plaintext highlighter-rouge">qB</code> during the experiment.
							–&gt;</li>
					</ul>
				</li>
			</ul>
			<p>These variables combine into 8 different scenarios, which is enough dimensions to look at in isolation.</p>
			<h3 id="receivers">Receivers</h3>
			<p>I sourced three destination (receiver) hosts with &gt; 2 Gbps download bandwidth, labeled as following:</p>
			<ul>
				<li><strong>A</strong>. One of our usually idle lab servers.</li>
				<li><strong>B</strong>. <a href="https://mirrors.ustc.edu.cn/">USTC Mirrors</a> server.</li>
				<li><strong>C</strong>. Friend-sponsored home broadband in Shanghai.</li>
			</ul>
			<p>Typical traits of these destinations are:</p>
			<table>
				<thead>
					<tr>
						<th style="text-align: center">Destination</th>
						<th style="text-align: center">Download BW</th>
						<th style="text-align: center">Latency</th>
						<th style="text-align: center">BDP*</th>
						<th style="text-align: left">Other notes</th>
					</tr>
				</thead>
				<tbody>
					<tr>
						<td style="text-align: center">A</td>
						<td style="text-align: center">10 Gbps</td>
						<td style="text-align: center">250 ± 30 us</td>
						<td style="text-align: center">500&nbsp;KB</td>
						<td style="text-align: left">Mostly idle</td>
					</tr>
					<tr>
						<td style="text-align: center">B</td>
						<td style="text-align: center">10 Gbps</td>
						<td style="text-align: center">300 ± 200 us</td>
						<td style="text-align: center">600&nbsp;KB</td>
						<td style="text-align: left">Under constant load</td>
					</tr>
					<tr>
						<td style="text-align: center">C</td>
						<td style="text-align: center">~2.2 Gbps</td>
						<td style="text-align: center">28 ± 0.2 ms</td>
						<td style="text-align: center">56&nbsp;MB</td>
						<td style="text-align: left">Mostly idle</td>
					</tr>
				</tbody>
			</table>
			<p>* Because my workstation can only generate upload at a theoretical maximum speed of 2 Gbps, BDP is calculated at this speed.</p>
			<h2 id="analyses">Analyses</h2>
			<p>iperf3 provides three indicators: Transmission bitrate, number of retransmissions and the congestion window size.</p>
			<p>We cannot attach too much importance to the bond mode when testing bonding performance, so let’s get straight to it.</p>
			<h3 id="single-stream">Single stream</h3>
			<table>
				<thead>
					<tr>
						<th style="text-align: center">Dest</th>
						<th style="text-align: center">P</th>
						<th style="text-align: center">CC</th>
						<th style="text-align: right">Bitrate (RR)</th>
						<th style="text-align: right">Bitrate (TLB)</th>
						<th style="text-align: right">Retr (RR)</th>
						<th style="text-align: right">Retr (TLB)</th>
						<th style="text-align: right">Cwnd (RR)</th>
						<th style="text-align: right">Cwnd (TLB)</th>
					</tr>
				</thead>
				<tbody>
					<tr>
						<td style="text-align: center">A</td>
						<td style="text-align: center">1</td>
						<td style="text-align: center">BBR</td>
						<td style="text-align: right">1.78 Gbps</td>
						<td style="text-align: right">940 Mbps</td>
						<td style="text-align: right">20079</td>
						<td style="text-align: right">20</td>
						<td style="text-align: right">331&nbsp;KB</td>
						<td style="text-align: right">233&nbsp;KB</td>
					</tr>
					<tr>
						<td style="text-align: center">A</td>
						<td style="text-align: center">1</td>
						<td style="text-align: center">CUBIC</td>
						<td style="text-align: right">1.19 Gbps</td>
						<td style="text-align: right">936 Mbps</td>
						<td style="text-align: right">7258</td>
						<td style="text-align: right">110</td>
						<td style="text-align: right">103&nbsp;KB</td>
						<td style="text-align: right">385&nbsp;KB</td>
					</tr>
					<tr>
						<td style="text-align: center">B</td>
						<td style="text-align: center">1</td>
						<td style="text-align: center">BBR</td>
						<td style="text-align: right">1.62 Gbps</td>
						<td style="text-align: right">944 Mbps</td>
						<td style="text-align: right">37018</td>
						<td style="text-align: right">51</td>
						<td style="text-align: right">343&nbsp;KB</td>
						<td style="text-align: right">241&nbsp;KB</td>
					</tr>
					<tr>
						<td style="text-align: center">B</td>
						<td style="text-align: center">1</td>
						<td style="text-align: center">CUBIC</td>
						<td style="text-align: right">1.19 Gbps</td>
						<td style="text-align: right">941 Mbps</td>
						<td style="text-align: right">6914</td>
						<td style="text-align: right">72</td>
						<td style="text-align: right">98&nbsp;KB</td>
						<td style="text-align: right">338&nbsp;KB</td>
					</tr>
					<tr>
						<td style="text-align: center">C</td>
						<td style="text-align: center">1</td>
						<td style="text-align: center">BBR</td>
						<td style="text-align: right">1.11 Gbps</td>
						<td style="text-align: right">935 Mbps</td>
						<td style="text-align: right">0</td>
						<td style="text-align: right">0</td>
						<td style="text-align: right">8.87&nbsp;MB</td>
						<td style="text-align: right">7.67&nbsp;MB</td>
					</tr>
					<tr>
						<td style="text-align: center">C</td>
						<td style="text-align: center">1</td>
						<td style="text-align: center">CUBIC</td>
						<td style="text-align: right">1.16 Gbps</td>
						<td style="text-align: right">931 Mbps</td>
						<td style="text-align: right">0</td>
						<td style="text-align: right">0</td>
						<td style="text-align: right">6.44&nbsp;MB</td>
						<td style="text-align: right">4.20&nbsp;MB</td>
					</tr>
				</tbody>
			</table>
			<p>We first look at TLB bond mode.
				As expected, with one single stream, it runs on only one slave interface (confirmed by watching <code class="language-plaintext highlighter-rouge">bmon -p eth0,eth1,bond0</code> during execution).
				And with the entire route being able to carry nearly 1.8 Gbps, there’s no surprise that different congestion algorithms don’t affect single-stream performance in all scenarios.</p>
			<p>We then note the huge difference between the BBR and CUBIC algorithms in RR bond mode.
				Because destinations A and B both have a very low BDP, any fluctuation in latency hits hard on CUBIC in forms of out-of-order packet deliveries, which reflects clearly in the difference of retransmitted packets and the Cwnd size.
				In TLB mode, with only one active NIC, retransmission is kept low, but in RR mode it skyrocketed.</p>
			<p>For destination C however, that’s an entirely different case (zero retransmission) with its own reason:
				There’s enough BDP for the sender to raise Cwnd up to 8&nbsp;MiB, but the receiver reports a window of only 4&nbsp;MiB, so the entire transmission is limited by the receiver, not congestion or link load.</p>
			<h3 id="parallel-streams">Parallel streams</h3>
			<p>During the experiment, I found it very often that all 4 streams gets load-balanced onto one NIC in <code class="language-plaintext highlighter-rouge">balance-tlb</code> mode, so I had to re-run the tests multiple times to obtain the result where 4 streams are equally balanced onto both NICs.
				Results from imbalanced stream distributions are discarded during retry.</p>
			<p>For 4 parallel streams, the result looks much better:</p>
			<table>
				<thead>
					<tr>
						<th style="text-align: center">Dest</th>
						<th style="text-align: center">P</th>
						<th style="text-align: center">CC</th>
						<th style="text-align: right">Bitrate (RR)</th>
						<th style="text-align: right">Bitrate (TLB)</th>
						<th style="text-align: right">Retr (RR)</th>
						<th style="text-align: right">Retr (TLB)</th>
						<th style="text-align: right">Cwnd (RR)</th>
						<th style="text-align: right">Cwnd (TLB)</th>
					</tr>
				</thead>
				<tbody>
					<tr>
						<td style="text-align: center">A</td>
						<td style="text-align: center">4</td>
						<td style="text-align: center">BBR</td>
						<td style="text-align: right">1.79 Gbps</td>
						<td style="text-align: right">1.77 Gbps</td>
						<td style="text-align: right">41357</td>
						<td style="text-align: right">18</td>
						<td style="text-align: right">222&nbsp;KB</td>
						<td style="text-align: right">178&nbsp;KB</td>
					</tr>
					<tr>
						<td style="text-align: center">A</td>
						<td style="text-align: center">4</td>
						<td style="text-align: center">CUBIC</td>
						<td style="text-align: right">1.80 Gbps</td>
						<td style="text-align: right">1.77 Gbps</td>
						<td style="text-align: right">13254</td>
						<td style="text-align: right">82</td>
						<td style="text-align: right">48&nbsp;KB</td>
						<td style="text-align: right">306&nbsp;KB</td>
					</tr>
					<tr>
						<td style="text-align: center">B</td>
						<td style="text-align: center">4</td>
						<td style="text-align: center">BBR</td>
						<td style="text-align: right">1.79 Gbps</td>
						<td style="text-align: right">1.72 Gbps</td>
						<td style="text-align: right">61810</td>
						<td style="text-align: right">9368</td>
						<td style="text-align: right">237&nbsp;KB</td>
						<td style="text-align: right">236&nbsp;KB</td>
					</tr>
					<tr>
						<td style="text-align: center">B</td>
						<td style="text-align: center">4</td>
						<td style="text-align: center">CUBIC</td>
						<td style="text-align: right">1.77 Gbps</td>
						<td style="text-align: right">1.74 Gbps</td>
						<td style="text-align: right">15549</td>
						<td style="text-align: right">4243</td>
						<td style="text-align: right">31&nbsp;KB</td>
						<td style="text-align: right">174&nbsp;KB</td>
					</tr>
					<tr>
						<td style="text-align: center">C</td>
						<td style="text-align: center">4</td>
						<td style="text-align: center">BBR</td>
						<td style="text-align: right">1.70 Gbps</td>
						<td style="text-align: right">1.82 Gbps</td>
						<td style="text-align: right">185</td>
						<td style="text-align: right">0</td>
						<td style="text-align: right">4.04&nbsp;MB</td>
						<td style="text-align: right">3.84&nbsp;MB</td>
					</tr>
					<tr>
						<td style="text-align: center">C</td>
						<td style="text-align: center">4</td>
						<td style="text-align: center">CUBIC</td>
						<td style="text-align: right">1.74 Gbps</td>
						<td style="text-align: right">1.82 Gbps</td>
						<td style="text-align: right">20</td>
						<td style="text-align: right">17</td>
						<td style="text-align: right">2.91&nbsp;MB</td>
						<td style="text-align: right">2.89&nbsp;MB</td>
					</tr>
				</tbody>
			</table>
			<p>The gap between RR and TLB no longer exists, and differences in retransmissions and Cwnd size can be attributed entirely to the CC algorithms themselves.
				In particular, due to Cwnd falling under 4&nbsp;MiB, or possibly just random network fluctuation, retransmission to destination C is observed for the first time.
				It’s also interesting to note that the difference in Cwnd size between BBR and CUBIC increased to 5x, compared to some 3 ~ 3.5x in single-stream case.</p>
			<p>However, the numbers of retransmissions are lower by magnitudes in TLB bond mode than in RR mode.
				I believe this is because the latencies to destinations A and B have very high variation compared to their means (averages), which will result in high number of reordered packets when packets from the same stream are distributed through two underlying interfaces.</p>
			<h2 id="bottom-line">Bottom line</h2>
			<p>I did a set of rudimentary experiments in this article, and TCP upload is only a very generic use case.
				For example, @TheRainstorm is running Sunshine streaming inside load-balanced WireGuard tunnel, and turning up forward error correction (FEC) level is one way to offset fluctuation and packet loss.
				I’ve been running qBittorrent uploads under <code class="language-plaintext highlighter-rouge">balance-tlb</code> mode for months, and it’s been very stable because qBittorrent uploads torrent content over many connections.</p>
			<p>If anyone wants a more reliable multi-NIC bonding setup, I’d definitely recommend getting a capable switch and doing bonding in <code class="language-plaintext highlighter-rouge">802.3ad</code> (LACP) mode.
				But for a small homelab, <code class="language-plaintext highlighter-rouge">balance-rr</code> + BBR would probably suffice for optimizing single-stream transmission speed, at the cost of some overhead on bandwidth.</p>
			]]></content><author><name>iBug</name></author><category term="linux" /><category term="networking" /><summary type="html"><![CDATA[My workstation in my univerity lab has consistently generated some 600&nbsp;TiB of annual upload across two PT sites. After an upgrade to the room a few months back, it received the most wanted upgrade for years: A second 1 Gbps line to the campus network. I immediately made the two NICs into one bonded 2 Gbps interface, and it has shined for many times with multiple popular torrents.]]></summary></entry><entry><title type="html">Manually resize Guild Chest in Palworld 0.6</title><link href="https://ibug.io/blog/2025/07/palworld-0.6-resize-guild-chest/" rel="alternate" type="text/html" title="Manually resize Guild Chest in Palworld 0.6" /><published>2025-07-28T00:00:00+00:00</published><updated>2025-07-28T14:02:46+00:00</updated><id>https://ibug.io/blog/2025/07/palworld-0.6-resize-guild-chest</id><content type="html" xml:base="https://ibug.io/blog/2025/07/palworld-0.6-resize-guild-chest/"><![CDATA[<p>The Guild Chest, first introduced in Feybreak Update, easily became the best way to move items between bases.
			However, it only has 54 slots and is nowhere near enough for all the items you want to store.
			For those who want to increase the size of the Guild Chest, especially in single player worlds, there’s a tutorial on how to do it manually:
			<a href="https://www.nexusmods.com/palworld/mods/2419">Guild chest resizer after solo play started (Manual)</a>.</p>
		<p>A few months back, I followed this tutorial and successfully resized my Guild Chest to 270 slots.
			But this time on my friend’s save, the script no longer worked.</p>
		<div class="language-console highlighter-rouge">
			<div class="highlight">
				<pre class="highlight"><code><span class="gp">PS D:\Downloads\GuildChestResizer&gt;</span><span class="w"> </span>python sav-to-gvas.py D:<span class="se">\D</span>ownloads<span class="se">\G</span>uildChestResizer<span class="se">\L</span>evel.sav
<span class="go">Traceback (most recent call last):
</span><span class="gp">  File "D:\Downloads\GuildChestResizer\sav-to-gvas.py", line 30, in &lt;module&gt;</span><span class="w">
</span><span class="go">    main()
  File "D:\Downloads\GuildChestResizer\sav-to-gvas.py", line 18, in main
    uncompressed_data = zlib.decompress(data[12:])
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
zlib.error: Error -3 while decompressing data: incorrect header check
</span><span class="gp">PS D:\Downloads\GuildChestResizer&gt;</span><span class="w">
</span></code></pre>
			</div>
		</div>
		<p>Fortunately, I had my own save file where I performed the steps, so I could investigate the issue.</p>
		<h2 id="investigation">Investigation</h2>
		<p>Notice how the script tries to decompress the data using zlib, with the first 12&nbsp;bytes skipped.
			So the first thing to check is to compare the header of the save files.</p>
		<p>My save file (that worked):</p>
		<div class="language-text highlighter-rouge">
			<div class="highlight">
				<pre class="highlight"><code>00000000  7d d3 67 01 c8 51 23 00  50 6c 5a 32 78 9c 64 7a  |}.g..Q#.PlZ2x.dz|
</code></pre>
			</div>
		</div>
		<p>My friend’s that no longer works:</p>
		<div class="language-text highlighter-rouge">
			<div class="highlight">
				<pre class="highlight"><code>00000000  09 c2 8b 01 a7 f1 20 00  50 6c 4d 31 8c 0a 00 36  |...... .PlM1...6|
</code></pre>
			</div>
		</div>
		<p>According to <a href="https://stackoverflow.com/q/9050260/5958455">Stack Overflow</a>, <code class="language-plaintext highlighter-rouge">78 9c</code> indicates default compression for zlib, while <code class="language-plaintext highlighter-rouge">8c 0a</code> is not a valid zlib header.
			After playing around with the save file, I concluded that the save file was not compressed with zlib at all, so time to look for other clues.</p>
		<p>By examining the Python script, particularly the “convert back” one (<code class="language-plaintext highlighter-rouge">gvas-to-sav.py</code>), I figured out the header for the <code class="language-plaintext highlighter-rouge">.sav</code> file:</p>
		<ul>
			<li>Bytes 0-4: Size of uncompressed data (4&nbsp;bytes, little-endian)</li>
			<li>Bytes 4-8: Size of compressed data (4&nbsp;bytes, little-endian)</li>
			<li>Bytes 8-11: Magic number (3&nbsp;bytes, <code class="language-plaintext highlighter-rouge">PlZ</code>)</li>
			<li>Bytes 12: Possibly type information, as the script checks for <code class="language-plaintext highlighter-rouge">0x32</code></li>
		</ul>
		<p>It’s easy to notice the discrepancies between the failing save file and the above structure:</p>
		<ul>
			<li>The magic number is <code class="language-plaintext highlighter-rouge">PlM</code> instead of <code class="language-plaintext highlighter-rouge">PlZ</code>.</li>
			<li>The potential type information is <code class="language-plaintext highlighter-rouge">0x31</code> instead of <code class="language-plaintext highlighter-rouge">0x32</code>.</li>
		</ul>
		<p>The uncompressed size and compressed size fields are correct for that file itself, so most likely the new save file employs a different compression algorithm.
			Time to identify it.</p>
		<h2 id="finding-the-compression-algorithm">Finding the compression algorithm</h2>
		<p>So here comes the trick: I know about this other tool called <a href="https://www.nexusmods.com/palworld/mods/104">PalEdit</a>, which is also written in Python and is <a href="https://github.com/EternalWraith/PalEdit">open-source on GitHub</a>, so it’s easy to read the code for clues and insights.</p>
		<p>There’s a <code class="language-plaintext highlighter-rouge">SaveConverter.py</code> file that catches my attention, which refers:</p>
		<div class="language-python highlighter-rouge">
			<div class="highlight">
				<pre class="highlight"><code><span class="kn">from</span> <span class="n">palworld_save_tools.palsav</span> <span class="kn">import</span> <span class="n">compress_gvas_to_sav</span><span class="p">,</span> <span class="n">decompress_sav_to_gvas</span>
</code></pre>
			</div>
		</div>
		<p>… except there’s no <code class="language-plaintext highlighter-rouge">palworld_save_tools</code> directory anywhere in the repository.
			Though interestingly, there <em>is</em> a <a href="https://github.com/EternalWraith/PalEdit/blob/main/palworld_save_tools.zip"><code class="language-plaintext highlighter-rouge">palworld_save_tools.zip</code> file</a> in the root of the repository.</p>
		<p>Recalling how Python can <a href="https://docs.python.org/3/using/cmdline.html#interface-options">execute</a> a directory or a zip file if it contains a <code class="language-plaintext highlighter-rouge">__main__.py</code> script, it sure can <em>import</em> it as well if it contains a <code class="language-plaintext highlighter-rouge">__init__.py</code>, so time to unpack that ZIP archive and inspect the contents.
			Within a few steps you’ll land at <code class="language-plaintext highlighter-rouge">compressor/__init__.py</code> which looks like this at its beginning:</p>
		<div class="language-python highlighter-rouge">
			<div class="highlight">
				<pre class="highlight"><code><span class="k">class</span> <span class="nc">SaveType</span><span class="p">:</span>
    <span class="n">CNK</span> <span class="o">=</span> <span class="mh">0x30</span>  <span class="c1"># Zlib compressed on xbox
</span>    <span class="n">PLM</span> <span class="o">=</span> <span class="mh">0x31</span>  <span class="c1"># Oodle compressed
</span>    <span class="n">PLZ</span> <span class="o">=</span> <span class="mh">0x32</span>  <span class="c1"># Zlib compressed
</span>
<span class="c1"># [code omitted]...
</span>
<span class="k">class</span> <span class="nc">MagicBytes</span><span class="p">:</span>
    <span class="n">CNK</span> <span class="o">=</span> <span class="sa">b</span><span class="sh">"</span><span class="s">CNK</span><span class="sh">"</span>  <span class="c1"># Zlib magic on xbox
</span>    <span class="n">PLZ</span> <span class="o">=</span> <span class="sa">b</span><span class="sh">"</span><span class="s">PlZ</span><span class="sh">"</span>  <span class="c1"># Zlib magic
</span>    <span class="n">PLM</span> <span class="o">=</span> <span class="sa">b</span><span class="sh">"</span><span class="s">PlM</span><span class="sh">"</span>  <span class="c1"># Oodle magic
</span></code></pre>
			</div>
		</div>
		<p>Now the details become clear:
			The new save file contains <code class="language-plaintext highlighter-rouge">PlM</code> as its magic bytes and a matching <code class="language-plaintext highlighter-rouge">0x31</code> type information, so mystery solved and the next step is to figure out how to decompress it.</p>
		<h2 id="patching-the-scripts">Patching the scripts</h2>
		<p>Searching for <code class="language-plaintext highlighter-rouge">Oodle compression</code> on Google produces multiple results from Epic Games and Unreal Engine, guess that’s their latest feature maybe?
			So I turned my focus back to PalEdit, where there’s the <code class="language-plaintext highlighter-rouge">compressor/oozlib.py</code> file with this:</p>
		<div class="language-python highlighter-rouge">
			<div class="highlight">
				<pre class="highlight"><code><span class="k">try</span><span class="p">:</span>
    <span class="kn">import</span> <span class="n">ooz</span>
<span class="k">except</span> <span class="nb">ImportError</span><span class="p">:</span>
    <span class="k">raise</span> <span class="nc">ImportError</span><span class="p">(</span>
        <span class="sa">f</span><span class="sh">"</span><span class="s">Failed to import </span><span class="sh">'</span><span class="s">ooz</span><span class="sh">'</span><span class="s"> module. Make sure the Ooz library exists in </span><span class="si">{</span><span class="n">local_ooz_path</span><span class="si">}</span><span class="s"> or latest pyooz is installed in your Python environment. Install using </span><span class="sh">'</span><span class="s">pip install git+https://github.com/MRHRTZ/pyooz.git</span><span class="sh">'"</span>
    <span class="p">)</span>
</code></pre>
			</div>
		</div>
		<p>Problem solved, but I’m on Windows without a C toolchain (I no longer do Windows development), so I moved all relevant files onto my Linux server.
			Then it’s just a matter of <code class="language-plaintext highlighter-rouge">pip install</code> command to get the requirements ready.</p>
		<p>I also patched the decompressor script:</p>
		<div class="language-python highlighter-rouge">
			<div class="highlight">
				<pre class="highlight"><code><span class="c1">#!/usr/bin/env python3
</span>
<span class="kn">import</span> <span class="n">subprocess</span>
<span class="kn">import</span> <span class="n">sys</span>
<span class="kn">import</span> <span class="n">glob</span>
<span class="kn">import</span> <span class="n">zlib</span>
<span class="kn">import</span> <span class="n">ooz</span>

<span class="k">def</span> <span class="nf">main</span><span class="p">():</span>
    <span class="k">if</span> <span class="nf">len</span><span class="p">(</span><span class="n">sys</span><span class="p">.</span><span class="n">argv</span><span class="p">)</span> <span class="o">&lt;</span> <span class="mi">2</span><span class="p">:</span>
        <span class="nf">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
    <span class="n">sav_file</span> <span class="o">=</span> <span class="n">sys</span><span class="p">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
    <span class="k">with</span> <span class="nf">open</span><span class="p">(</span><span class="n">sav_file</span><span class="p">,</span> <span class="sh">'</span><span class="s">rb</span><span class="sh">'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
        <span class="n">data</span> <span class="o">=</span> <span class="n">f</span><span class="p">.</span><span class="nf">read</span><span class="p">()</span>
        <span class="n">uncompressed_len</span> <span class="o">=</span> <span class="nb">int</span><span class="p">.</span><span class="nf">from_bytes</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="mi">4</span><span class="p">],</span> <span class="n">byteorder</span><span class="o">=</span><span class="sh">'</span><span class="s">little</span><span class="sh">'</span><span class="p">)</span>
        <span class="n">compressed_len</span> <span class="o">=</span> <span class="nb">int</span><span class="p">.</span><span class="nf">from_bytes</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="mi">4</span><span class="p">:</span><span class="mi">8</span><span class="p">],</span> <span class="n">byteorder</span><span class="o">=</span><span class="sh">'</span><span class="s">little</span><span class="sh">'</span><span class="p">)</span>
        <span class="n">magic_bytes</span> <span class="o">=</span> <span class="n">data</span><span class="p">[</span><span class="mi">8</span><span class="p">:</span><span class="mi">11</span><span class="p">]</span>
        <span class="k">assert</span> <span class="n">magic_bytes</span> <span class="o">==</span> <span class="sa">b</span><span class="sh">'</span><span class="s">PlM</span><span class="sh">'</span>
        <span class="n">save_type</span> <span class="o">=</span> <span class="n">data</span><span class="p">[</span><span class="mi">11</span><span class="p">]</span>
        <span class="k">assert</span> <span class="n">save_type</span> <span class="o">==</span> <span class="mh">0x31</span>
        <span class="n">uncompressed_data</span> <span class="o">=</span> <span class="n">ooz</span><span class="p">.</span><span class="nf">decompress</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="mi">12</span><span class="p">:],</span> <span class="n">uncompressed_len</span><span class="p">)</span>
        <span class="k">if</span> <span class="n">uncompressed_len</span> <span class="o">!=</span> <span class="nf">len</span><span class="p">(</span><span class="n">uncompressed_data</span><span class="p">):</span>
            <span class="k">return</span>
        <span class="n">gvas_file</span> <span class="o">=</span> <span class="n">sav_file</span><span class="p">.</span><span class="nf">replace</span><span class="p">(</span><span class="sh">'</span><span class="s">.sav</span><span class="sh">'</span><span class="p">,</span> <span class="sh">'</span><span class="s">.sav.gvas</span><span class="sh">'</span><span class="p">)</span>
        <span class="k">with</span> <span class="nf">open</span><span class="p">(</span><span class="n">gvas_file</span><span class="p">,</span> <span class="sh">'</span><span class="s">wb</span><span class="sh">'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
            <span class="n">f</span><span class="p">.</span><span class="nf">write</span><span class="p">(</span><span class="nf">bytes</span><span class="p">(</span><span class="n">uncompressed_data</span><span class="p">))</span>

<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="sh">"</span><span class="s">__main__</span><span class="sh">"</span><span class="p">:</span>
    <span class="nf">main</span><span class="p">()</span>
</code></pre>
			</div>
		</div>
		<p>Now it correctly decompresses my friend’s <code class="language-plaintext highlighter-rouge">Level.sav</code> into <code class="language-plaintext highlighter-rouge">Level.sav.gvas</code> where I can follow the rest of <a href="https://www.nexusmods.com/palworld/mods/2419">the instructions</a> manually.
			For compressing the edited file back, I just used the provided <code class="language-plaintext highlighter-rouge">gvas-to-sav.py</code> file in the belief that the previous zlib-based compression method is still supported by the game.</p>
		<p>After sending the edited <code class="language-plaintext highlighter-rouge">Level.sav</code> file back to the game, we’re glad to see our Guild Chest containing 270 slots, which we probably would never use up.</p>
		<h2 id="postface">Postface</h2>
		<p>If you’re on Windows or don’t have a C/C++ compiler for installing <code class="language-plaintext highlighter-rouge">pyooz</code> yourself, the same <code class="language-plaintext highlighter-rouge">palworld_save_tools.zip</code> provides these files that you may find useful:</p>
		<div class="language-text highlighter-rouge">
			<div class="highlight">
				<pre class="highlight"><code>lib/linux_arm64/ooz.so
lib/linux_x86_64/ooz.so
lib/windows/ooz.pyd
</code></pre>
			</div>
		</div>
		<p>Of course, how to utilize these files is left as an exercise for the readers.</p>
		]]></content><author><name>iBug</name></author><category term="games" /><category term="palworld" /><summary type="html"><![CDATA[The Guild Chest, first introduced in Feybreak Update, easily became the best way to move items between bases. However, it only has 54 slots and is nowhere near enough for all the items you want to store. For those who want to increase the size of the Guild Chest, especially in single player worlds, there’s a tutorial on how to do it manually: Guild chest resizer after solo play started (Manual).]]></summary></entry><entry><title type="html">A trip to Japan</title><link href="https://ibug.io/blog/2025/07/japan-june-2025/" rel="alternate" type="text/html" title="A trip to Japan" /><published>2025-07-23T00:00:00+00:00</published><updated>2025-07-23T00:59:25+00:00</updated><id>https://ibug.io/blog/2025/07/japan-june-2025</id><content type="html" xml:base="https://ibug.io/blog/2025/07/japan-june-2025/"><![CDATA[<p>As a more meaningful way to celebrate our graduation as Master students and spend this last truly free time in a long while, I joined my friends on a trip to Japan.</p>
	<h2 id="preparations">Preparations</h2>
	<p>Kudos to my friends for planning the itinerary and discussing the details.
		We planned to visit Tokyo, Kyoto, and Osaka, with a day trip to Nara.</p>
	<p>Other than planning, these are many important things to prepare before the trip:</p>
	<ul>
		<li>
			<p><strong>Visa</strong>: Japan has an embassy and many consulates in China, and each is responsible for a specific region.
				In particular, the Japan Embassy requires individuals to apply for a visa through a designated travel agency.
				This is not particularly complex, as you can just pick an agency in your region and follow their instructions, or head over to TaoBao and purchase a visa application service.
				Prepare the required documents, deliver them to the agency, and wait for your passport to be returned with the visa.</p>
			<p>One thing to note is that Japan no longer issues physical single-entry visas that are stamped in your passport, but instead issues a digital visa that can be opened and displayed on your phone.
				This will be useful later.</p>
		</li>
		<li>
			<p><strong>Flights</strong>: My friends handled the flight bookings. For departure, we flew from Shanghai (PVG) to Tokyo (NRT), and returned from Osaka (KIX) to Shanghai (PVG).
				The flights were direct and took about 3 hours each way.
				To save some money, we booked the flights in advance and chose a budget airline (Spring Airlines), which later caused some issues.
				I personally wanted to return home directly after the trip, so I booked a separate flight from Osaka to Shenzhen (SZX) on the same day, skipping Shanghai.</p>
		</li>
		<li>
			<p><strong>Accommodation</strong>: We booked hotels in Kyoto for our stay there, but opted for Airbnb in Tokyo and Osaka, as hotel prices in these cities were orbitantly high (partially accounting for Expo 2025).
				For homestays, we chose places close to the city center and had good reviews, to ensure convenience and comfort.</p>
		</li>
		<li>
			<p><strong>Transportation</strong>: Japan’s metro system is very convenient and efficient, so why not?
				We inspected the IC card system and decided that a Suica card would be sufficient for most of our needs.
				Since we were going to land at Narita International Airport, we purchased a Skyliner + 3-day Tokyo Subway pass combo online, which allowed us to take the Skyliner train from Narita Airport to downtown Tokyo and convenient subway rides during our stay there.</p>
			<p>For iPhone users, the Suica card can be added to Apple Wallet and recharged through Apple Pay, which is even more convenient than physical cards.
				I unfortunately am an Android user, so I had to buy a physical card at NRT, priced at JP¥ 500 for the card itself.</p>
		</li>
		<li>
			<p><strong>Connectivity</strong>: From our friends’ recommendations, we purchased <a href="https://global.cmlink.com/en/plan-detail?PlanID=D2206291737376685419">data SIM cards from CMLink</a> for our phones.
				There’s one advantage of CMLink over local Japanese SIM cards: Data traffic exits through Hong Kong, which means good connectivity to both mainland China and the rest of the world.</p>
		</li>
		<li>
			<p><strong>Money &amp; Payments</strong>: Japan is a very well-developed country, so cashless payments are prevalent.
				However, some cash is still required occasionally, so we exchanged some cash at our local bank before the trip.
				Three of us brought JP¥ 30,000 (≈ US$ 208) each, and I brought a bit less at JP¥ 20,000 (≈ US$ 140).
				We also brought our bank cards for convenience, as VISA and Mastercard are widely accepted in Japan.</p>
			<p>My primary credit card is a UnionPay and VISA dual-branded card, which due to regulatory reasons, is a magnetic stripe card with no IC chip. So I additionally brought my <a href="https://www.bochk.com/dam/more/bocdebitcard/card/en.html">Bank of China (Hong Kong) Mastercard debit card</a>, which is an IC-and-NFC card and should cover the cases where the mag-stripe card is unusable.</p>
		</li>
		<li>
			<p><strong>Other items</strong>: Mobile phone, power banks, chargers, and a suitable backpack because you will need to carry your stuff around including your passport.</p>
			<p>Unlike many other countries, Japan uses a different standard for power delivery, at 100V and 50/60 Hz, so you will need to double check your power adapter to ensure it’s compatible.</p>
			<p>Also, one of us brought a roll of garbage bags, as Japan has a very strict garbage sorting system and doesn’t install garbage bins in public places.
				This means we had to carry our garbage around until we found a suitable bin, so these bags would prevent our backpacks from getting dirty.</p>
		</li>
	</ul>
	<h3 id="before-the-trip">Before the trip</h3>
	<p>There’s a <a href="https://www.vjw.digital.go.jp/">Visit Japan Web</a> system that allows you to register your trip beforehand.
		It combines visa verification and customs declaration into one, and streamlines the disembarkation process.</p>
	<h3 id="day-0">Day 0: Shanghai</h3>
	<p>Because our flight was in the morning, we decided to stay overnight at a hotel near PVG.
		This also brings us a chance to meet up with our friends who were working or studying in Shanghai, and we had a nice dinner together.
		To save some stamina, we headed to the hotel right after dinner and rested for the night, so that we could get up early the next day without feeling too tired.</p>
	<p>I was surprised to find that Shanghai metro uses 3 languages for the on-train radio announcements: Mandarin Chinese, English, and <strong>Shanghai dialect</strong>, at least on Line 16.</p>
	<h2 id="day-1">Day 1: Arrival in Tokyo</h2>
	<p>Waking up at 6 AM, we rode the hotel shuttle bus to PVG, and it was an ordinary flight to NRT.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250606130555.webp" class="image-popup" title="Shortly before landing in Narita International Airport
"><img src="https://image.ibugone.com/jp-2025/IMG20250606130555.webp" alt="Shortly before landing in Narita International Airport" loading="lazy" /></a>
		<figcaption>
			Shortly before landing in Narita International Airport
		</figcaption>
	</figure>
	<p>We landed at Terminal 3 and went through the immigration and customs checks smoothly.
		Because we prepared our Visit Japan Web registration, all we had to do was to show the QR code to the self-service kiosk and then present our passports to the immigration officer.</p>
	<p>The first thing we interacted with after disembarkation and getting out of T3 was a vending machine.
		Interestingly, it supported both cash and card payments, and we used our NFC-enabled bank cards to buy a bottle of water.
		The prices were substantially higher than in Shanghai, at JP¥ 160 for a 500ml bottle, 2x as much as that in Shanghai, but the difference was certainly negligible for such a one-off case.</p>
	<p>We then took another shuttle bus to Terminal 2, where we redeemed our Skyliner and the Tokyo Subway pass tickets and purchased our Suica cards.
		Worth commending is that the labels and signs throughout Japan are highly accessible, with everything written in 4 languages: Japanese, English, Chinese (Simplified), and Korean.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250606141629.webp" class="image-popup" title="First sight in Narita International Airport Terminal 2
"><img src="https://image.ibugone.com/jp-2025/IMG20250606141629.webp" alt="First sight in Narita International Airport Terminal 2" loading="lazy" /></a>
		<figcaption>
			First sight in Narita International Airport Terminal 2
		</figcaption>
	</figure>
	<p>Skyliner is a all-reserved train service that connects Narita International Airport to downtown Tokyo, and we took it to the other end of the line, Keisei-Ueno Station.
		Its high frequency means that we didn’t have to wait long and the seats were rarely full.
		Needless to say, the train was very clean and comfortable, with spacious seats and large windows compared to High-Speed Rail trains in China.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250606150720.webp" class="image-popup" title="Scenic view from the Skyliner train on the way
"><img src="https://image.ibugone.com/jp-2025/IMG20250606150720.webp" alt="Scenic view from the Skyliner train on the way" loading="lazy" /></a>
		<figcaption>
			Scenic view from the Skyliner train on the way
		</figcaption>
	</figure>
	<p>At Keisei-Ueno Station, we transferred to the Tokyo Metro and took the Hibiya Line to our Airbnb in Minowa.
		How nice it is that every station is numbered, so that those who are not familiar with the station names can just call out the station number.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250606155502.webp" class="image-popup" title="A subway station guideboard for Hibiya Line at Ueno Station
"><img src="https://image.ibugone.com/jp-2025/IMG20250606155502.webp" alt="A subway station guideboard for Hibiya Line at Ueno Station" loading="lazy" /></a>
		<figcaption>
			A subway station guideboard for Hibiya Line at Ueno Station
		</figcaption>
	</figure>
	<p>In this way, we can name this ride “from H-18 to H-20”.</p>
	<p>The next sensation was the narrowness of exit stairways in the subway stations in Tokyo.
		Subway stations in China are generally spacious, with wide stairways and escalators, and very well-lit.
		Taking this station as example, the exit stairway is very narrow with no escalator installed, though there are elevators for those in need.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250606175247.webp" class="image-popup" title="A narrow exit stairway in a Tokyo subway station
"><img src="https://image.ibugone.com/jp-2025/IMG20250606175247.webp" alt="A narrow exit stairway in a Tokyo subway station" loading="lazy" /></a>
		<figcaption>
			A narrow exit stairway in a Tokyo subway station
		</figcaption>
	</figure>
	<p>My first impression of Japanese streets are cleanliness and orderliness.
		As anticipated, despite having no trash cans on the streets, it’s still very clean and tidy.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250606161011.webp" class="image-popup" title="A street view in Minowa, Tokyo
"><img src="https://image.ibugone.com/jp-2025/IMG20250606161011.webp" alt="A street view in Minowa, Tokyo" loading="lazy" /></a>
		<figcaption>
			A street view in Minowa, Tokyo
		</figcaption>
	</figure>
	<figure class="third ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250606161137.webp" title="Shortly before landing in Narita International Airport">
			<img src="https://image.ibugone.com/jp-2025/IMG20250606161137.webp" alt="Shortly before landing in Narita International Airport" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250606161422.webp" title="An alleyway view">
			<img src="https://image.ibugone.com/jp-2025/IMG20250606161422.webp" alt="An alley in Minowa, Tokyo" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250606162634.webp" title="View from the roof of our stay">
			<img src="https://image.ibugone.com/jp-2025/IMG20250606162634.webp" alt="Minowa street view" loading="lazy" />
		</a>
	</figure>
	<p>After a short break, we went out to explore the neighborhood and had dinner.
		It didn’t take long to notice the abundance of vending machines in Japan, which all sell an assortment of drinks, with around half of them being coffee.
		How the Japanese love coffee!
		Also the prices are very inconsistent, and it’s often the case that the same bottle of drink becomes as much as JP¥ 30 cheaper or pricier in a few meters.
		These vending machines also appear to be operated by different companies, as the design and payment options vary a lot.
		Some machines are cash-only, while others accept cash, card, and even QR code payments through WeChat Pay and Alipay.
		These vending machines made up our water supply throughout our entire journey in Japan.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250606175114.webp" class="image-popup" title="A Kirin vending machine in the back of some sort of garage
"><img src="https://image.ibugone.com/jp-2025/IMG20250606175114.webp" alt="A Kirin vending machine" loading="lazy" /></a>
		<figcaption>
			A Kirin vending machine in the back of some sort of garage
		</figcaption>
	</figure>
	<p>We headed back to the metro station and rode to Asakusa, making a transfer at Ueno Station.
		Here came the next surprise: the Tokyo Metro doesn’t have cross-platform transfers, so we had to exit the turnstiles for Hibiya Line and enter those for Ginza Line.
		We entered Asakusa ROX and found our way through the crowded shopping mall, only to find that the renowned Kura Sushi was fully booked for the night.
		After some wandering, we decided to try a random restaurant, and noticed the nice-looking tonkatsu outside Tokiwa Restaurant.
		To be honest, it <em>was</em> delicious, and its quality was well reflected in the price, at JP¥ 1,980, rather typical of Asakusa (indeed).</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250606185709.webp" class="image-popup" title="My JP¥ 1,980 dinner at Asakusa
"><img src="https://image.ibugone.com/jp-2025/IMG20250606185709.webp" alt="A tonkatsu set meal at Asakusa" loading="lazy" /></a>
		<figcaption>
			My JP¥ 1,980 dinner at Asakusa
		</figcaption>
	</figure>
	<p>What’s even more typical is the pathetic supply of rice, which isn’t even half of a bowl.
		A refill is sold separately at JP¥ 100.
		Well, what can I say, this is Tokyo after all!</p>
	<p>On our free roam at the perimeter, we noticed the in-car subway map looking like spaghetti.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250606193028.webp" class="image-popup" title="Spaghetti subway map in a car of Asakusa Line
"><img src="https://image.ibugone.com/jp-2025/IMG20250606193028.webp" alt="A subway map in a Tokyo subway car" loading="lazy" /></a>
		<figcaption>
			Spaghetti subway map in a car of Asakusa Line
		</figcaption>
	</figure>
	<p>We made a brief visit to the Tokyo Skytree and the Tokyo Tower, before returning to our stay.
		On our way back shopping for supplies at a 7-Eleven nearby, I noticed a peculiar pricing of bottled water:</p>
	<ul>
		<li>JP¥ 118 for a 500 mL package</li>
		<li>JP¥ 131 for a 2 L package</li>
	</ul>
	<p>I guess the prices all went to the bottle instead of the water.
		No wonder why the Japanese are so keen on recycling plastic bottles, that’s some good revenue!</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250606202642.webp" class="image-popup" title="Water prices in 7-Eleven in Tokyo
"><img src="https://image.ibugone.com/jp-2025/IMG20250606202642.webp" alt="Bottled water on a store shelf" loading="lazy" /></a>
		<figcaption>
			Water prices in 7-Eleven in Tokyo
		</figcaption>
	</figure>
	<h2 id="day-2">Day 2: Kyu-Furukawa Gardens &amp; Sunshine Aquarium</h2>
	<p>Day 2 was arranged for anime pilgrimage by my friends (for <em>BanG Dream! It’s MyGO!!!!!</em> in particular), so they sure provided a joyful itinerary for the day despite my innocence in anime.</p>
	<p>First stop was Kyu-Furukawa Gardens, a tranquil garden in the Kita ward of Tokyo.
		Educated by the experience of the previous day, I was for sure surprised by the humble price of the entrance ticket, at JP¥ 150.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250607105909.webp" class="image-popup" title="Kyu-Furukawa Gardens ticket
"><img src="https://image.ibugone.com/jp-2025/IMG20250607105909.webp" alt="Kyu-Furukawa Gardens ticket" loading="lazy" /></a>
		<figcaption>
			Kyu-Furukawa Gardens ticket
		</figcaption>
	</figure>
	<p>The garden is very well maintained, along with a beautiful Western-style mansion.
		Problem was, it was already summer, well past the blooming season of most flowers, so the garden was not as colorful as it could be.</p>
	<figure class="third ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250607110213.webp" title="Main garden of Kyu-Furukawa Gardens">
			<img src="https://image.ibugone.com/jp-2025/IMG20250607110213.webp" alt="Main garden of Kyu-Furukawa Gardens" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250607110744.webp" title="The western-style mansion in Kyu-Furukawa Gardens">
			<img src="https://image.ibugone.com/jp-2025/IMG20250607110744.webp" alt="The western-style mansion in Kyu-Furukawa Gardens" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250607110644.webp" title="Path towards the tranquil center of Kyu-Furukawa Gardens">
			<img src="https://image.ibugone.com/jp-2025/IMG20250607110644.webp" alt="Path towards the tranquil center of Kyu-Furukawa Gardens" loading="lazy" />
		</a>
	</figure>
	<p>Somehow, the roads in Kita really reminded me of <em>Doraemon</em>.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250607112325.webp" class="image-popup" title="An ordinary road in Kita, Tokyo
"><img src="https://image.ibugone.com/jp-2025/IMG20250607112325.webp" alt="An ordinary road in Kita, Tokyo" loading="lazy" /></a>
		<figcaption>
			An ordinary road in Kita, Tokyo
		</figcaption>
	</figure>
	<p>We then rode the city bus to Asukayama Park nearby for more pilgrimage.
		The bus felt good, but again except for the fare, which was JP¥ 180 for a 10-minute ride.
		In contrast, bus rides in Shanghai only cost a few Chinese yuans (≤ JP¥ 100).</p>
	<p>The sun made the visit to the park very enjoyable, with a steam locomotive on display and a nice playground, and the kids having fun brought an energetic vibe to the atmosphere.</p>
	<figure class="third ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250607113736.webp" title="The D51 853 steam locomotive on display at Asukayama Park">
			<img src="https://image.ibugone.com/jp-2025/IMG20250607113736.webp" alt="The D51 853 steam locomotive on display at Asukayama Park" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250607114356.webp" title="Kids in a water play area at Asukayama Park">
			<img src="https://image.ibugone.com/jp-2025/IMG20250607114356.webp" alt="Kids in a water play area at Asukayama Park" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250607114749.webp" title="A lovely bird in Asukayama Park, not afraid of humans">
			<img src="https://image.ibugone.com/jp-2025/IMG20250607114749.webp" alt="A lovely bird in Asukayama Park" loading="lazy" />
		</a>
	</figure>
	<p>Following another ride on the Tokyo Sakura Tram, we arrived at Chitose Bridge, where we had lunch at a nearby restaurant after takings a few photos over the roads.
		For the first time in Japan, I made a payment using my magnetic stripe credit card.
		I was delighted to know that the card remained functional after years of cold storage in my drawer.
		It was a nice beef don, except that the ragú was extremely salty.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250607132038.webp" class="image-popup" title="My lunch for today
"><img src="https://image.ibugone.com/jp-2025/IMG20250607132038.webp" alt="A beef don set meal" loading="lazy" /></a>
		<figcaption>
			My lunch for today
		</figcaption>
	</figure>
	<p>The assortment of seasonings on the table was definitely remarkable.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250607132627.webp" class="image-popup" title="Seasonings on the table
"><img src="https://image.ibugone.com/jp-2025/IMG20250607132627.webp" alt="Seasonings on the table" loading="lazy" /></a>
		<figcaption>
			Seasonings on the table
		</figcaption>
	</figure>
	<p>For the afternoon, we visited the Sunshine City in Ikebukuro.
		It’s a large shopping mall, though most of the stores were not of our interest.
		We briefly browsed through an anime merchandise store, and then headed towards the Sunshine Aquarium on the top floor.</p>
	<p>As famous as it is, the first thing to notice is the <em>Detective Conan</em> (a.k.a. <em>Case Closed</em>) wall at the entrance of the aquarium.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250607143116.webp" class="image-popup" title="Detective Conan decorations at the entrance of Sunshine Aquarium
"><img src="https://image.ibugone.com/jp-2025/IMG20250607143116.webp" alt="Detective Conan decorations at the entrance of Sunshine Aquarium" loading="lazy" /></a>
		<figcaption>
			Detective Conan decorations at the entrance of Sunshine Aquarium
		</figcaption>
	</figure>
	<p>And a horror-looking vending machine for edible insects.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250607143239.webp" class="image-popup" title="A vending machine for edible insects
"><img src="https://image.ibugone.com/jp-2025/IMG20250607143239.webp" alt="A vending machine for edible insects" loading="lazy" /></a>
		<figcaption>
			A vending machine for edible insects
		</figcaption>
	</figure>
	<p>Inside the aquarium are hundreds of tanks at display, with a wide variety of marine life.
		I’m no expert in marine biology, so here are just some photos of the exhibits.</p>
	<figure class="third ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250607150201.webp" title="Jellyfish lit up in blue">
			<img src="https://image.ibugone.com/jp-2025/IMG20250607150201.webp" alt="Jellyfish lit up in blue" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250607143920.webp" title="Coral reefs and tropical fish">
			<img src="https://image.ibugone.com/jp-2025/IMG20250607143920.webp" alt="Coral reefs and tropical fish" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250607145754.webp" title="A swarm of fish above coral reefs">
			<img src="https://image.ibugone.com/jp-2025/IMG20250607145754.webp" alt="A swarm of fish above coral reefs" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250607150921.webp" title="An overhead jellyfish tank">
			<img src="https://image.ibugone.com/jp-2025/IMG20250607150921.webp" alt="An overhead jellyfish tank" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250607154041.webp" title="Translucent fish around fallen logs">
			<img src="https://image.ibugone.com/jp-2025/IMG20250607154041.webp" alt="Translucent fish around fallen logs" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250607154621.webp" title="Axolotl">
			<img src="https://image.ibugone.com/jp-2025/IMG20250607154621.webp" alt="Axolotl" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250607155207.webp" title="Blue tang, like *Dory* from *Finding Nemo*">
			<img src="https://image.ibugone.com/jp-2025/IMG20250607155207.webp" alt="Blue tang, like *Dory* from *Finding Nemo*" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250607155249.webp" title="Clownfish, like *Nemo* from *Finding Nemo*">
			<img src="https://image.ibugone.com/jp-2025/IMG20250607155249.webp" alt="Clownfish, like *Nemo* from *Finding Nemo*" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250607162258.webp" title="Penguins in Tokyo">
			<img src="https://image.ibugone.com/jp-2025/IMG20250607162258.webp" alt="Penguins in Tokyo" loading="lazy" />
		</a>
	</figure>
	<p>Coming out of the aquarium, we met with friends studying and working in Tokyo and headed to an arcade around Ueno, where we tried out some Japan-native arcade games like <em>maimai DX</em>.
		With 8 people together, we had our yakiniku dinner at a restaurant in Ueno, which was a great way to end the day.</p>
	<h2 id="day-3">Day 3: Shibuya &amp; Shimokitazawa</h2>
	<p>Day 3 continues as anime pilgrimage in another direction.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250608141602.webp" class="image-popup" title="A busy crossroad in Shibuya
"><img src="https://image.ibugone.com/jp-2025/IMG20250608141602.webp" alt="A busy crossroad in Shibuya" loading="lazy" /></a>
		<figcaption>
			A busy crossroad in Shibuya
		</figcaption>
	</figure>
	<p>During our exploration of Shibuya, we encountered the Pride March of Tokyo, where multiple groups of people marched through the streets, decorated with rainbow flags and banners.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250608143118.webp" class="image-popup" title="A truck leading a group of marchers
"><img src="https://image.ibugone.com/jp-2025/IMG20250608143118.webp" alt="A truck leading a group of marchers" loading="lazy" /></a>
		<figcaption>
			A truck leading a group of marchers
		</figcaption>
	</figure>
	<p>The march added to the crowd in Shibuya, which was already bustling with people.
		Strangely, streets in Shibuya were barely equipped with vending machines, and the crowd added to the scarcity of water supply.
		We squeezed our way through the crowd and through a park, only to find a group of vending machines that already had queues built up long before them.
		So out of desperation, we quickly found our way to the next available subway station and continued to Shimokitazawa.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250608161232.webp" class="image-popup" title="Honda Theater at Shimokitazawa
"><img src="https://image.ibugone.com/jp-2025/IMG20250608161232.webp" alt="Honda Theater at Shimokitazawa" loading="lazy" /></a>
		<figcaption>
			Honda Theater at Shimokitazawa
		</figcaption>
	</figure>
	<p>My friends purchased some merchandise at Honda Theater, and then we managed to locate three prominent scenes from <em>Bocchi the Rock!</em> in the neighborhood.</p>
	<figure class="half ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250608163653.webp" title="A group of three vending machines under a stairway">
			<img src="https://image.ibugone.com/jp-2025/IMG20250608163653.webp" alt="A group of three vending machines under a stairway" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250608164455.webp" title="The side of a house painted with a colorful tree">
			<img src="https://image.ibugone.com/jp-2025/IMG20250608164455.webp" alt="The side of a house painted with a colorful tree" loading="lazy" />
		</a>
		<figcaption>Famous scenes from <em>Bocchi the Rock!</em>
		</figcaption>
	</figure>
	<p>For dinner, we went to the Uobei Sushi in Shibuya, a conveyor belt sushi restaurant.
		I was certainly shocked by the prices, as an average salmon sushi plate is priced at JP¥ 160, even less than half of that in Shanghai.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250608174755.webp" class="image-popup" title="Dishes at Uobei Sushi
"><img src="https://image.ibugone.com/jp-2025/IMG20250608174755.webp" alt="Dishes at Uobei Sushi" loading="lazy" /></a>
		<figcaption>
			Dishes at Uobei Sushi
		</figcaption>
	</figure>
	<p>Adding to the superior quality and tasty flavor, this is definitely the <em>best</em> sushi I have ever had.</p>
	<p>Following dinner, we went to Shibuya PARCO, a large shopping mall nearby, and enjoyed ourselves with various brand stores and a Pokémon Center.</p>
	<figure class="third ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250608184514.webp" title="Equipments at display in Nintendo Tokyo store">
			<img src="https://image.ibugone.com/jp-2025/IMG20250608184514.webp" alt="Equipments at display in Nintendo Tokyo store" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250608185421.webp" title="A trial play of *Mario Kart World* in Nintendo Tokyo store">
			<img src="https://image.ibugone.com/jp-2025/IMG20250608185421.webp" alt="A trial play of *Mario Kart World* in Nintendo Tokyo store" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250608190958.webp" title="Stuffed toys for sale in Pokémon Center">
			<img src="https://image.ibugone.com/jp-2025/IMG20250608190958.webp" alt="Stuffed toys for sale in Pokémon Center" loading="lazy" />
		</a>
		<figcaption>Shibuya PARCO
		</figcaption>
	</figure>
	<h2 id="day-4">Day 4: Tokyo Skytree &amp; Akihabara</h2>
	<p>Our last day in Tokyo encompassed a visit to the Tokyo Skytree, the tallest structure in Japan.</p>
	<p>We arrived by metro at Oshiage Station, and was then met with a complex of escalators leading up into the shopping mall beneath the Skytree.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250609111635.webp" class="image-popup" title="The escalator complex leading to the Tokyo Skytree
"><img src="https://image.ibugone.com/jp-2025/IMG20250609111635.webp" alt="The escalator complex leading to the Tokyo Skytree" loading="lazy" /></a>
		<figcaption>
			The escalator complex leading to the Tokyo Skytree
		</figcaption>
	</figure>
	<p>After traversing through the shopping mall and queuing up for admission, the first sight to behold was the well-decorated elevator leading up to the observation deck.
		The decoration is said to be seasonal, with four variants throughout the year.
		Presumably, this is the summer variant, judging by our time of visit.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250609113326.webp" class="image-popup" title="Decorations in the elevator going up Tokyo Skytree
"><img src="https://image.ibugone.com/jp-2025/IMG20250609113326.webp" alt="Decorations in the elevator going up Tokyo Skytree" loading="lazy" /></a>
		<figcaption>
			Decorations in the elevator going up Tokyo Skytree
		</figcaption>
	</figure>
	<p>From the main observation deck at 350 m above ground, we had a panoramic view of Tokyo.
		Very typical of Japan, the streets are very well organized, with a grid-like layout but narrow roads.
		Also the buildings are generally not very tall unlike in Shanghai, presumably due to the earthquake risk as Japan sits on the Pacific Ring of Fire.</p>
	<figure class="third ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250609113522.webp" title="A view towards the city from the observation deck of Tokyo Skytree">
			<img src="https://image.ibugone.com/jp-2025/IMG20250609113522.webp" alt="A view towards the city from the observation deck of Tokyo Skytree" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250609114109.webp" title="The wish stand for tying wish strips at Tokyo Skytree">
			<img src="https://image.ibugone.com/jp-2025/IMG20250609114109.webp" alt="The wish stand for tying wish strips at Tokyo Skytree" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250609115324.webp" title="A *Detective Conan* display stand at the observation deck of Tokyo Skytree">
			<img src="https://image.ibugone.com/jp-2025/IMG20250609115324.webp" alt="A *Detective Conan* display stand at the observation deck of Tokyo Skytree" loading="lazy" />
		</a>
		<figcaption>Views from the main observation deck
		</figcaption>
	</figure>
	<p>At the 340 m level, there is a glass floor that allows you to see straight down to the ground, similar to the one in the Oriental Pearl Tower in Shanghai.
		However, the glass floor here is too small to provide any excitement, and the view is nowhere near as impressive as that in the Oriental Pearl Tower.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250609122312.webp" class="image-popup" title="The glass floor at 340m featuring a straight view down
"><img src="https://image.ibugone.com/jp-2025/IMG20250609122312.webp" alt="The glass floor at 340m featuring a straight view down" loading="lazy" /></a>
		<figcaption>
			The glass floor at 340m featuring a straight view down
		</figcaption>
	</figure>
	<p>Coming out of Tokyo Skytree, we returned to the metro station and headed towards Ningyocho for another renowned ramen restaurant.
		In the metro station, there was a vending machine for ice creams that caught our attention.
		We bought an ice cream each, and I was surprised at its decent taste under a reasonable price.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250609132842.webp" class="image-popup" title="An ice cream vending machine in a Tokyo subway station
"><img src="https://image.ibugone.com/jp-2025/IMG20250609132842.webp" alt="An ice cream vending machine in a Tokyo subway station" loading="lazy" /></a>
		<figcaption>
			An ice cream vending machine in a Tokyo subway station
		</figcaption>
	</figure>
	<p>Arriving at the destined restaurant, we were delighted to find readily available seats (as it was 2 PM already).
		Unlike other ramen noodles, this one is served with the soup and toppings in a separate bowl, which is quite unique.
		You are supposed to add the noodle to the soup bowl in batches before enjoying it.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250609141018.webp" class="image-popup" title="A bowl of ramen with soup and toppings in a separate bowl
"><img src="https://image.ibugone.com/jp-2025/IMG20250609141018.webp" alt="A bowl of ramen with soup and toppings in a separate bowl" loading="lazy" /></a>
		<figcaption>
			A bowl of ramen with soup and toppings in a separate bowl
		</figcaption>
	</figure>
	<p>We then spent the rest of the afternoon in Akihabara, the anime and electronics district of Tokyo.</p>
	<figure class="third ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250609144851.webp" title="A street view of Akihabara">
			<img src="https://image.ibugone.com/jp-2025/IMG20250609144851.webp" alt="A street view of Akihabara" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250609145641.webp" title="Figures on display in a store">
			<img src="https://image.ibugone.com/jp-2025/IMG20250609145641.webp" alt="Figures on display in a store" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250609152930.webp" title="Books in an underground bookstore">
			<img src="https://image.ibugone.com/jp-2025/IMG20250609152930.webp" alt="Books in an underground bookstore" loading="lazy" />
		</a>
		<figcaption>Akihabara sightseeing
		</figcaption>
	</figure>
	<p>The first few stores were not of my interest, as they were mostly anime merchandise stores.
		But when it comes to the Sofmap, the wide variety of electronics and gadgets definitely warrants a visit.</p>
	<p>The first floor (ground floor in British English) is for second-handed Apple mobile devices, mostly iPhones and iPads.
		The upper floors are for various computer components, peripherals, and gaming devices.</p>
	<figure class="third ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250609155824.webp" title="2F, AIO coolers and motherboards">
			<img src="https://image.ibugone.com/jp-2025/IMG20250609155824.webp" alt="2F, AIO coolers and motherboards" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250609160051.webp" title="2F, high-end gaming routers">
			<img src="https://image.ibugone.com/jp-2025/IMG20250609160051.webp" alt="2F, high-end gaming routers" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250609160242.webp" title="3F, second-handed laptops">
			<img src="https://image.ibugone.com/jp-2025/IMG20250609160242.webp" alt="3F, second-handed laptops" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250609160735.webp" title="5F, a beautifully built gaming PC">
			<img src="https://image.ibugone.com/jp-2025/IMG20250609160735.webp" alt="5F, a beautifully built gaming PC" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250609162545.webp" title="7F, various shortcut devices and peripherals">
			<img src="https://image.ibugone.com/jp-2025/IMG20250609162545.webp" alt="7F, various shortcut devices and peripherals" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250609163306.webp" title="7F, mouse pads and gaming mice">
			<img src="https://image.ibugone.com/jp-2025/IMG20250609163306.webp" alt="7F, mouse pads and gaming mice" loading="lazy" />
		</a>
		<figcaption>Akihabara Sofmap
		</figcaption>
	</figure>
	<p>Following Sofmap, we visited a few more stores in Akihabara, including Surugaya, Mandarake and AmiAmi, before giving the longest stop of the day at K-Books, a large store for anime merchandise.</p>
	<p>The merchandise was well beyond my understanding, so here are just some photos of the offerings.</p>
	<figure class="third ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250609172336.webp" title="Badges">
			<img src="https://image.ibugone.com/jp-2025/IMG20250609172336.webp" alt="Badges" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250609172941.webp" title="Acrylic stands and posters">
			<img src="https://image.ibugone.com/jp-2025/IMG20250609172941.webp" alt="Acrylic stands and posters" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250609173548.webp" title="Holographic cards">
			<img src="https://image.ibugone.com/jp-2025/IMG20250609173548.webp" alt="Holographic cards" loading="lazy" />
		</a>
		<figcaption>Akihabara K-Books
		</figcaption>
	</figure>
	<p>For the last dinner in Tokyo, we booked a table at <em>Niku no Kirikata</em> (肉の切り方) in Nihombashi, a restaurant specializing in grilled meat.
		Despite the pricing at JP¥ 7,200 per person, the quality and the experience were unparalleled.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250609195115.webp" class="image-popup" title="An assortment of meat, with 5 copies of each type
"><img src="https://image.ibugone.com/jp-2025/IMG20250609195115.webp" alt="An assortment of meat, with 5 copies of each type" loading="lazy" /></a>
		<figcaption>
			An assortment of meat, with 5 copies of each type
		</figcaption>
	</figure>
	<p>And this concludes our stay in Tokyo.</p>
	<h2 id="day-5">Day 5: Shinkansen to Kyoto</h2>
	<p>After getting up and packing up our luggages, we took the metro to Tokyo Station and proceeded with our plan of Kyoto.
		Since Shinkansen from Tokyo to western Japan has a frequent headway of every 5 minutes, we didn’t have to worry about ticket and seat availability, so tickets were purchased on the spot.</p>
	<p>Unseen elsewhere, purchasing the Shinkansen tickets required us to produce our metro tickets or Suica cards, presumably because the ticket office is located inside before the metro turnstiles.
		We purchased tickets for Nozomi 35, one the fastest Shinkansen lines with the least stops, and the tickets were priced at JP¥ 13,970 per person.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250610120236.webp" class="image-popup" title="A set of Shinkansen tickets
"><img src="https://image.ibugone.com/jp-2025/IMG20250610120236.webp" alt="A set of Shinkansen tickets" loading="lazy" /></a>
		<figcaption>
			A set of Shinkansen tickets
		</figcaption>
	</figure>
	<p>Given its high headway frequency, it’s not surprising that the platform looks similar to a metro platform.</p>
	<figure class="half ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250610123518.webp" title="Shinkansen platform at Tokyo Station">
			<img src="https://image.ibugone.com/jp-2025/IMG20250610123518.webp" alt="Shinkansen platform at Tokyo Station" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250610123524.webp" title="Shinkansen platform at Tokyo Station">
			<img src="https://image.ibugone.com/jp-2025/IMG20250610123524.webp" alt="Shinkansen platform at Tokyo Station" loading="lazy" />
		</a>
		<figcaption>Shinkansen platform at Tokyo Station
		</figcaption>
	</figure>
	<p>The other thing that caught our attention was a Shinkansen Coffee machine.
		It offered a variety of coffee drinks, and what’s more intriguing is that the beverage is brewed on the spot, with a small cup of coffee beans ground and brewed into a cup of coffee.
		There’s even a screen showing the internals of the machine, which is quite fascinating.</p>
	<figure class="half ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250610124224.webp" title="Various offerings from the Shinkansen Coffee machine">
			<img src="https://image.ibugone.com/jp-2025/IMG20250610124224.webp" alt="Various offerings from the Shinkansen Coffee machine" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250610124035.webp" title="Screen showing the brewing coffee">
			<img src="https://image.ibugone.com/jp-2025/IMG20250610124035.webp" alt="Screen showing the brewing coffee" loading="lazy" />
		</a>
		<figcaption>Shinkansen Coffee machine
		</figcaption>
	</figure>
	<p>Our journey on the Shinkansen was very comfortable and we enjoyed the rural scenery along the way.</p>
	<figure class="third ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250610132229.webp" title="A street view in Shinagawa">
			<img src="https://image.ibugone.com/jp-2025/IMG20250610132229.webp" alt="A street view in Shinagawa" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250610134031.webp" title="Irrigation fields, with thick clouds in the distance">
			<img src="https://image.ibugone.com/jp-2025/IMG20250610134031.webp" alt="Irrigation fields, with thick clouds in the distance" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250610142512.webp" title="A solar farm">
			<img src="https://image.ibugone.com/jp-2025/IMG20250610142512.webp" alt="A solar farm" loading="lazy" />
		</a>
		<figcaption>Scenery from the Shinkansen
		</figcaption>
	</figure>
	<p>After a 2-hour ride, we arrived at Kyoto Station, a modern yet tremendous and complex station.
		We checked in at our hotel before heading out to explore the city.
		We took our lunch-and-dinner for the day at yet another random ramen restaurant, which featured a unique taste.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250610171021.webp" class="image-popup" title="Our first meal in Kyoto. There can never be too much ramen.
"><img src="https://image.ibugone.com/jp-2025/IMG20250610171021.webp" alt="A bowl of ramen" loading="lazy" /></a>
		<figcaption>
			Our first meal in Kyoto. There can never be too much ramen.
		</figcaption>
	</figure>
	<p>My first impression of Kyoto is its simplistic and even primitive metro system, with nonexistent platform doors, making it look very dated.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250610165712.webp" class="image-popup" title="A Kyoto metro station platform
"><img src="https://image.ibugone.com/jp-2025/IMG20250610165712.webp" alt="A Kyoto metro station platform" loading="lazy" /></a>
		<figcaption>
			A Kyoto metro station platform
		</figcaption>
	</figure>
	<p>We then took a stroll around the Shijo Bridge, a bridge over the Kamo River, and enjoyed the view of the river and the city.
		It was drizzling at the time, but previous showers had boosted the current of the Kamo River, making it look more lively.</p>
	<figure class="third ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250610173149.webp" title="The stone plaque for Shijo Bridge">
			<img src="https://image.ibugone.com/jp-2025/IMG20250610173149.webp" alt="The stone plaque for Shijo Bridge" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250610173558.webp" title="View of the Kamo River from the west bank">
			<img src="https://image.ibugone.com/jp-2025/IMG20250610173558.webp" alt="View of the Kamo River from the west bank" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250610174112.webp" title="Dense vegetation alongside the Kamo River over a plain area">
			<img src="https://image.ibugone.com/jp-2025/IMG20250610174112.webp" alt="Dense vegetation alongside the Kamo River over a plain area" loading="lazy" />
		</a>
		<figcaption>Kamo River views
		</figcaption>
	</figure>
	<p>We returned to our hotel for a break and fortunately stayed over another downpour.
		Late in the evening, we went hungry and headed for a nearby kebab restaurant.
		What a nice way to call it a day!</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250610221153.webp" class="image-popup" title="Our nighttime snack in Kyoto
"><img src="https://image.ibugone.com/jp-2025/IMG20250610221153.webp" alt="Kebab at a restaurant" loading="lazy" /></a>
		<figcaption>
			Our nighttime snack in Kyoto
		</figcaption>
	</figure>
	<h2 id="day-6">Day 6: Nara and Fushimi Inari Taisha</h2>
	<p>The next day in Kyoto was arranged for a short trip to Nara.
		Slightly to my surprise, several common tourist target cities are closely located, including Kyoto, Nara, and Osaka.
		The transportation to Nara is only a 40-minute ride on the Kintetsu Nara Line, costing JP¥ 1,440.
		Also, I was happy that Kintetsu stations support tap-to-ride with NFC bank cards, so I could save some Suica balance and consequently cash<sup id="fnref:suica-recharge"><a href="#fn:suica-recharge" class="footnote" rel="footnote" role="doc-noteref">1</a></sup>.</p>
	<p>Except that Kintetsu station platforms are the narrowest I have ever seen…</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250611111246.webp" class="image-popup" title="Extremely narrow Kintetsu station platform
"><img src="https://image.ibugone.com/jp-2025/IMG20250611111246.webp" alt="Extremely narrow Kintetsu station platform" loading="lazy" /></a>
		<figcaption>
			Extremely narrow Kintetsu station platform
		</figcaption>
	</figure>
	<p>It was another drizzling day in Nara, though fortunately the rain was not prohibitive to our visit.
		Shortly after coming out of Kintetsu Nara Station, we noticed the first deer and became excited to take photos of it.
		Not many steps did we take before we crossed paths with a herd of deer, which were roaming freely in the streets and parks.</p>
	<figure class="half ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250611112508.webp" title="A horned deer in Nara">
			<img src="https://image.ibugone.com/jp-2025/IMG20250611112508.webp" alt="A horned deer in Nara" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250611112649.webp" title="A fawn in Nara">
			<img src="https://image.ibugone.com/jp-2025/IMG20250611112649.webp" alt="A fawn in Nara" loading="lazy" />
		</a>
		<figcaption>Nara deers
		</figcaption>
	</figure>
	<p>However, as we approached the Nara Park, the situation quickly turned chaotic.
		The rain had made the ground muddy, and the deers were roaming everywhere, bringing a lot of mud onto the walkways.
		Tourists were feeding the deers with special deer crackers, which drove the deers even more aggressive.</p>
	<figure class="third ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250611114010.webp" title="Deers and tourists crowded in Nara Park">
			<img src="https://image.ibugone.com/jp-2025/IMG20250611114010.webp" alt="Deers and tourists crowded in Nara Park" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250611114126.webp" title="Deers hunting for deer crackers">
			<img src="https://image.ibugone.com/jp-2025/IMG20250611114126.webp" alt="Deers hunting for deer crackers" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250611115930.webp" title="A deer resting on the ground">
			<img src="https://image.ibugone.com/jp-2025/IMG20250611115930.webp" alt="A deer resting on the ground" loading="lazy" />
		</a>
		<figcaption>More deers in Nara Park
		</figcaption>
	</figure>
	<p>As interesting as it was, deers are not fended off from potentially dangerous places like vehicle roads, so here it comes:</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250611115318.webp" class="image-popup" title="A road sign reading 鹿の飛び出し (Caution: Deer jumping out)
"><img src="https://image.ibugone.com/jp-2025/IMG20250611115318.webp" alt="A road sign reading 鹿の飛び出し (Caution: Deer jumping out)" loading="lazy" /></a>
		<figcaption>
			A road sign reading 鹿の飛び出し (Caution: Deer jumping out)
		</figcaption>
	</figure>
	<p>Vending machines in this area are also very localized, with deer crackers and other deer-related products for sale.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250611123941.webp" class="image-popup" title="A vending machine selling deer crackers
"><img src="https://image.ibugone.com/jp-2025/IMG20250611123941.webp" alt="A vending machine selling deer crackers" loading="lazy" /></a>
		<figcaption>
			A vending machine selling deer crackers
		</figcaption>
	</figure>
	<p>Through a long walk through the woods, we arrived at Kasuga-taisha Shrine.
		With uncooperative weather and the rain-soaked ground, considering our plan for the afternoon to a similarly-styled shrine, we decided to cut the visit short here and start searching for our pre-determined tonkatsu restaurant.</p>
	<p>This is the best tonkatsu I have ever had at a very reasonable price of JP¥ 1,848.
		Even more to my surprise, the meal included free refills of everything but the tonkatsu itself.
		Notably, this is the first time in Japan that I have seen free refills of rice, whose price is not known for being generous in Japan.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250611142734.webp" class="image-popup" title="A tonkatsu set meal
"><img src="https://image.ibugone.com/jp-2025/IMG20250611142734.webp" alt="A tonkatsu set meal" loading="lazy" /></a>
		<figcaption>
			A tonkatsu set meal
		</figcaption>
	</figure>
	<p>With the delicious tonkatsu, I ended up refilling everything twice and coming out extremely satisfied.</p>
	<p>We headed back to Kyoto in the afternoon and dropped off halfway for Fushimi Inari Taisha, a famous shrine known for its thousands of vermilion torii gates.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250611160157.webp" class="image-popup" title="Spacious interior of Keihan Line trains
"><img src="https://image.ibugone.com/jp-2025/IMG20250611160157.webp" alt="Spacious interior of Keihan Line trains" loading="lazy" /></a>
		<figcaption>
			Spacious interior of Keihan Line trains
		</figcaption>
	</figure>
	<p>Similar to anime merchandise stores, the cultural significance of the shrine is beyond my understanding, so here come the photos.</p>
	<figure class="third ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250611162519.webp" title="First building">
			<img src="https://image.ibugone.com/jp-2025/IMG20250611162519.webp" alt="First building" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250611163116.webp" title="Small torii gate charms hanging on a wishing site">
			<img src="https://image.ibugone.com/jp-2025/IMG20250611163116.webp" alt="Small torii gate charms hanging on a wishing site" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250611163558.webp" title="The first set of a few gates">
			<img src="https://image.ibugone.com/jp-2025/IMG20250611163558.webp" alt="The first set of a few gates" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250611163833.webp" title="As we went deepter, torii gates became denser">
			<img src="https://image.ibugone.com/jp-2025/IMG20250611163833.webp" alt="As we went deepter, torii gates became denser" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250611165558.webp" title="A pond in the middle of the Mount Inari">
			<img src="https://image.ibugone.com/jp-2025/IMG20250611165558.webp" alt="A pond in the middle of the Mount Inari" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250611171850.webp" title="After the rain in the morning, the sky looks clear and nice">
			<img src="https://image.ibugone.com/jp-2025/IMG20250611171850.webp" alt="After the rain in the morning, the sky looks clear and nice" loading="lazy" />
		</a>
		<figcaption>Fushimi Inari Taisha
		</figcaption>
	</figure>
	<p>Before the part of densest vegetation, there’s a toilet and a vending machine, noting that up the mountain is considered a sacred place and these are the last facilities you will find.</p>
	<p>On our way back to the station, we bought some snacks from the streetside stores.
		The first store amazed us with overpriced cucumbers, which we laughed at and moved on.
		We then bought some ice cream from another store at JP¥ 400, thinking that it would be the most reasonable price we could find near such an attraction (which it later turned out to be even more true).
		The ice cream tasted good, except I’m not a fan of <strong>功</strong>克力.</p>
	<figure class="third ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250611173905.webp" title="Iced cucumbers... but JP¥ 600 each???">
			<img src="https://image.ibugone.com/jp-2025/IMG20250611173905.webp" alt="Iced cucumbers" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250611174113.webp" title="Assorted ice cream options">
			<img src="https://image.ibugone.com/jp-2025/IMG20250611174113.webp" alt="Assorted ice cream options" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250611175657.webp" title="A miniature on the side of the road">
			<img src="https://image.ibugone.com/jp-2025/IMG20250611175657.webp" alt="A miniature on the side of the road" loading="lazy" />
		</a>
		<figcaption>On our way back to Keihan line station
		</figcaption>
	</figure>
	<p>Walking further away, we were slightly surprised to find that the JP¥ 400 ice cream was actually the cheapest we could find, as every other store sold ice creams at at least JP¥ 500,and Taiwan Tapiocas even went up to JP¥ 800.</p>
	<p>Coming back to Kyoto, we went to (again!) a ramen restaurant for dinner, and took a walk around the city for the evening.
		For once, I was surprised to find that the driers in a public laundry were out of service.
		Guess Japanese products aren’t always <em>that</em> reliable after all.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250611221827.webp" class="image-popup" title="3 out of 4 driers are out of order
"><img src="https://image.ibugone.com/jp-2025/IMG20250611221827.webp" alt="3 out of 4 driers are out of order" loading="lazy" /></a>
		<figcaption>
			3 out of 4 driers are out of order
		</figcaption>
	</figure>
	<h2 id="day-7">Day 7: Kyoto Railway Museum &amp; Kiyomizu-dera</h2>
	<p>The last day in Kyoto began at the Kyoto Railway Museum.
		Into the museum are three large locomotives and a photo service.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250612111908.webp" class="image-popup" title="Three large locomotives
"><img src="https://image.ibugone.com/jp-2025/IMG20250612111908.webp" alt="Three large locomotives" loading="lazy" /></a>
		<figcaption>
			Three large locomotives
		</figcaption>
	</figure>
	<p>Further into the museum are a few more carriages of various types, including a Shinkansen carriage, a dining carriage, and some passenger carriages redecorated to display parts and artifacts from the past.</p>
	<figure class="third ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250612113732.webp" title="The dining car as a restaurant">
			<img src="https://image.ibugone.com/jp-2025/IMG20250612113732.webp" alt="The dining car as a restaurant" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250612113902.webp" title="Internal of a metro car">
			<img src="https://image.ibugone.com/jp-2025/IMG20250612113902.webp" alt="Internal of a metro car" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250612113955.webp" title="A rail switching apparatus">
			<img src="https://image.ibugone.com/jp-2025/IMG20250612113955.webp" alt="A rail switching apparatus" loading="lazy" />
		</a>
	</figure>
	<p>Notably, the dining car is used as a restaurant and sells many train-shaped take-out meals, which are very popular among visitors.
		By the time we arrived at the selling stand, half of the meals were already sold out, and we had to settle for less popular ones.</p>
	<p>After lunch, we went for a live steam locomotive ride, which only launched once every hour, priced separately at JP¥ 300.</p>
	<figure class="third ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250612115648.webp" title="The steam locomotive ride">
			<img src="https://image.ibugone.com/jp-2025/IMG20250612115648.webp" alt="The steam locomotive ride" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250612120719.webp" title="Nice scenery along the short ride">
			<img src="https://image.ibugone.com/jp-2025/IMG20250612120719.webp" alt="Nice scenery along the short ride" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250612121057.webp" title="The steam locomotive, loaded with coal">
			<img src="https://image.ibugone.com/jp-2025/IMG20250612121057.webp" alt="The steam locomotive, loaded with coal" loading="lazy" />
		</a>
	</figure>
	<p>Further inside the museum is a large exhibition hall, which features a wide range of exhibits, including more cars, station equipments and artifacts, and many railway models.</p>
	<figure class="third ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250612121201.webp" title="Locomotive depot">
			<img src="https://image.ibugone.com/jp-2025/IMG20250612121201.webp" alt="Locomotive depot" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250612121247.webp" title="A Thomas &amp; Friends locomotive">
			<img src="https://image.ibugone.com/jp-2025/IMG20250612121247.webp" alt="A Thomas &amp; Friends locomotive" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250612124913.webp" title="Steam engine apparatus">
			<img src="https://image.ibugone.com/jp-2025/IMG20250612124913.webp" alt="Steam engine apparatus" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250612131431.webp" title="Various track lights">
			<img src="https://image.ibugone.com/jp-2025/IMG20250612131431.webp" alt="Various track lights" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250612132650.webp" title="A huge sandbox model of Kyoto railway system">
			<img src="https://image.ibugone.com/jp-2025/IMG20250612132650.webp" alt="A huge sandbox model of Kyoto railway system" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250612131938.webp" title="An interactive playground for the interested">
			<img src="https://image.ibugone.com/jp-2025/IMG20250612131938.webp" alt="An interactive playground for the interested" loading="lazy" />
		</a>
	</figure>
	<p>The best thing about the museum is its many interactive exhibits, including a train control simulator with RC trains inside a large podium, a real ticket machine and gate, and a signal control simulator.
		These exhibits are very well designed and provide a great experience for visitors of all ages.</p>
	<p>The afternoon was arranged for a visit to Kiyomizu-dera, a famous Buddhist temple in Kyoto.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250612150414.webp" class="image-popup" title="Front of a small shrine
"><img src="https://image.ibugone.com/jp-2025/IMG20250612150414.webp" alt="Front of a small shrine" loading="lazy" /></a>
		<figcaption>
			Front of a small shrine
		</figcaption>
	</figure>
	<p>Unlike previous days, the weather on this day was extra sunny, so natural scenery around the temple was alive and vibrant.</p>
	<figure class="third ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250612151341.webp" title="Halfway up the temple, a view towards the distance">
			<img src="https://image.ibugone.com/jp-2025/IMG20250612151341.webp" alt="Halfway up the temple, a view towards the distance" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250612151557.webp" title="Three surrounding buildings">
			<img src="https://image.ibugone.com/jp-2025/IMG20250612151557.webp" alt="Three surrounding buildings" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250612152314.webp" title="View from the main temple towards the exit path">
			<img src="https://image.ibugone.com/jp-2025/IMG20250612152314.webp" alt="View from the main temple towards the exit path" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250612155835.webp" title="Halfway on the exit path">
			<img src="https://image.ibugone.com/jp-2025/IMG20250612155835.webp" alt="Halfway on the exit path" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250612160338.webp" title="Pine trees on the mountain forest">
			<img src="https://image.ibugone.com/jp-2025/IMG20250612160338.webp" alt="Pine trees on the mountain forest" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250612161716.webp" title="View from the exit path">
			<img src="https://image.ibugone.com/jp-2025/IMG20250612161716.webp" alt="View from the exit path" loading="lazy" />
		</a>
	</figure>
	<p>Surrounded by the forest and on a mountain, the temple features stunning nature views.</p>
	<p>On our way back to the city, we came by Shijo Bridge again, this time to find it more gentle and peaceful, as it did not rain today.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250612174026.webp" class="image-popup" title="Shijo Bridge again
"><img src="https://image.ibugone.com/jp-2025/IMG20250612174026.webp" alt="Shijo Bridge over the Kamo River" loading="lazy" /></a>
		<figcaption>
			Shijo Bridge again
		</figcaption>
	</figure>
	<p>For dinner, we planned a prestigious kaiseki meal near Kyoto Station, which is a traditional multi-course Japanese dinner.
		It’s a very unique experience, with each course being carefully prepared and presented, and the ingredients being seasonal and local.</p>
	<figure class="third ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250612182342.webp" title="First dish of the Kaiseki meal, with a variety of appetizers">
			<img src="https://image.ibugone.com/jp-2025/IMG20250612182342.webp" alt="First dish of the Kaiseki meal, with a variety of appetizers" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250612183042.webp" title="Second dish of the Kaiseki meal, a bowl of miso soup">
			<img src="https://image.ibugone.com/jp-2025/IMG20250612183042.webp" alt="Second dish of the Kaiseki meal, a bowl of miso soup" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250612183558.webp" title="Third dish of the Kaiseki meal, a small sashimi set">
			<img src="https://image.ibugone.com/jp-2025/IMG20250612183558.webp" alt="Third dish of the Kaiseki meal, a small sashimi set" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250612184909.webp" title="Fourth dish of the Kaiseki meal, grilled meat">
			<img src="https://image.ibugone.com/jp-2025/IMG20250612184909.webp" alt="Fourth dish of the Kaiseki meal, grilled meat" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250612185414.webp" title="Fifth dish of the Kaiseki meal, a shrimp ball and some vegetables">
			<img src="https://image.ibugone.com/jp-2025/IMG20250612185414.webp" alt="Fifth dish of the Kaiseki meal, a shrimp ball and some vegetables" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250612190836.webp" title="The main dish of the Kaiseki meal, a bowl of unadon with rice">
			<img src="https://image.ibugone.com/jp-2025/IMG20250612190836.webp" alt="The main dish of the Kaiseki meal, a bowl of unadon with rice" loading="lazy" />
		</a>
		<figcaption>Kaiseki dinner servings
		</figcaption>
	</figure>
	<p>After the main dish, we were served a dessert of matcha cake and ice cream, which was a perfect way to end the meal.</p>
	<p>In the evening, we took (yet another) walk back to the Shijo Bridge, where we enjoyed the night view of the Kamo River and the city, unencumbered by rain.</p>
	<figure class="third ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250612211507.webp">
			<img src="https://image.ibugone.com/jp-2025/IMG20250612211507.webp" alt="" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250612212837.webp">
			<img src="https://image.ibugone.com/jp-2025/IMG20250612212837.webp" alt="" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250612213625.webp">
			<img src="https://image.ibugone.com/jp-2025/IMG20250612213625.webp" alt="" loading="lazy" />
		</a>
		<figcaption>Kamo River at night
		</figcaption>
	</figure>
	<p>On our way back to the hotel, we got lost in the complex of Kyoto Station while trying to refund an unused metro pass.</p>
	<p>Because this entire day was planned for Kyoto, we bought a single-day metro pass, thinking it would be useful.
		However, two of us ended up using it only once and gettings it rejected by the ticket machine at almost everywhere else.
		It was throughout the day that we realized that the metro pass is only valid for Kyoto Municipal Subway (with only two lines!), while there are at least four other metro operators in Kyoto that covers the rest of the city.
		Obviously, these alternative operators do not accept the metro pass, so we decided to refund the last unused one.
		What a mess!</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250612202835.webp" class="image-popup" title="The Kyoto Station complex from its north side
"><img src="https://image.ibugone.com/jp-2025/IMG20250612202835.webp" alt="The Kyoto Station complex from its north side" loading="lazy" /></a>
		<figcaption>
			The Kyoto Station complex from its north side
		</figcaption>
	</figure>
	<h2 id="day-8">Day 8: Osaka Castle</h2>
	<p>The next day, we took the Hankyu Line to Osaka, which is only a 40-minute ride.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250613105951.webp" class="image-popup" title="Rural scenery on the way to Osaka
"><img src="https://image.ibugone.com/jp-2025/IMG20250613105951.webp" alt="Rural scenery on the way to Osaka" loading="lazy" /></a>
		<figcaption>
			Rural scenery on the way to Osaka
		</figcaption>
	</figure>
	<p>After arriving at Osaka, we located our Airbnb, unloaded our luggages, and headed out for lunch.
		Unsurprisingly, there can never be too much ramen again.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250613130406.webp" class="image-popup" title="Needy Ramen Overdose in Osaka
"><img src="https://image.ibugone.com/jp-2025/IMG20250613130406.webp" alt="*Needy Ramen Overdose* in Osaka" loading="lazy" /></a>
		<figcaption>
			<em>Needy Ramen Overdose</em> in Osaka
		</figcaption>
	</figure>
	<p>The first interesting thing we noticed in Osaka was that (almost) all bottle trash cans in the metro stations were covered up, labeling</p>
	<blockquote>
		<p>Due to special vigilance, trash box can NOT be used.</p>
	</blockquote>
	<p>Well, the Japanese sometimes do weird things.</p>
	<p>Anyways, we went on to Osaka Castle for a nice trip.</p>
	<p>Coming out of Tanimachi 4-chome Station, the first thing to see are two landmark buildings.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250613140045.webp" class="image-popup" title="NHK Osaka Hall and the Osaka Museum of History
"><img src="https://image.ibugone.com/jp-2025/IMG20250613140045.webp" alt="NHK Osaka Hall and the Osaka Museum of History" loading="lazy" /></a>
		<figcaption>
			NHK Osaka Hall and the Osaka Museum of History
		</figcaption>
	</figure>
	<p>It was a super sunny day and the grass was very green there, so for sure the scenery was very pleasant.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250613141839.webp" class="image-popup" title="View of a surrounding building across the moat
"><img src="https://image.ibugone.com/jp-2025/IMG20250613141839.webp" alt="View of a surrounding building across the moat" loading="lazy" /></a>
		<figcaption>
			View of a surrounding building across the moat
		</figcaption>
	</figure>
	<p>We rode a cart from the outside to the entrance to the castle at JP¥ 300 per person, skipping a long walk under the sun.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250613142225.webp" class="image-popup" title="Entrance to Osaka Castle on the inner wall
"><img src="https://image.ibugone.com/jp-2025/IMG20250613142225.webp" alt="Entrance to Osaka Castle on the inner wall" loading="lazy" /></a>
		<figcaption>
			Entrance to Osaka Castle on the inner wall
		</figcaption>
	</figure>
	<p>To my surprise, the inner moat has already dried up and is now a grassy field.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250613142526.webp" class="image-popup" title="The inner moat of Osaka Castle
"><img src="https://image.ibugone.com/jp-2025/IMG20250613142526.webp" alt="The inner moat of Osaka Castle" loading="lazy" /></a>
		<figcaption>
			The inner moat of Osaka Castle
		</figcaption>
	</figure>
	<p>We redeemed our tickets and entered the castle, now a museum for the history of Osaka and much of Japan.
		The ground floor elevator took us to the 5th floor, where the museum starts.</p>
	<p>5F is equipped with many video screens showing Japanese history, primarily focusing on the Sengoku period, with image and text descriptions about the battles and notable figures of the time.
		The same went for 6F.</p>
	<figure class="third ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250613144258.webp" title="Various battles and sieges in the history of Japan">
			<img src="https://image.ibugone.com/jp-2025/IMG20250613144258.webp" alt="Various battles and sieges in the history of Japan" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250613144447.webp" title="Notable figures of the Tokugawa clan">
			<img src="https://image.ibugone.com/jp-2025/IMG20250613144447.webp" alt="Notable figures of the Tokugawa clan" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250613145931.webp" title="Video snippets, with parallax effect">
			<img src="https://image.ibugone.com/jp-2025/IMG20250613145931.webp" alt="Video snippets, with parallax effect" loading="lazy" />
		</a>
	</figure>
	<p>Interestingly, the video screens employed a smart trick with mirrors to create a parallax effect, granting the scene a depth-of-field effect.</p>
	<p>The castle somehow did not have a 7th floor (or was skipped entirely), and the 8th floor is the observation deck.
		From this vantage point, we had a panoramic view of Osaka, with the Kyocera Dome and the Umeda Sky Building in sight.</p>
	<figure class="third ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250613151034.webp" title="View towards the south">
			<img src="https://image.ibugone.com/jp-2025/IMG20250613151034.webp" alt="View towards the south" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250613151450.webp" title="View towards the north">
			<img src="https://image.ibugone.com/jp-2025/IMG20250613151450.webp" alt="View towards the north" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250613151821.webp" title="View towards the west">
			<img src="https://image.ibugone.com/jp-2025/IMG20250613151821.webp" alt="View towards the west" loading="lazy" />
		</a>
	</figure>
	<p>Also, while there are alternate stairways connecting up to 5F, the central spiral stairway is the only way to go up to 8F.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250613152457.webp" class="image-popup" title="The spiral stairway inside Osaka Castle
"><img src="https://image.ibugone.com/jp-2025/IMG20250613152457.webp" alt="The spiral stairway inside Osaka Castle" loading="lazy" /></a>
		<figcaption>
			The spiral stairway inside Osaka Castle
		</figcaption>
	</figure>
	<p>After coming down from the observation deck, we visited 4F and 3F, which are dedicated to the display of artifacts.
		The artifacts are mostly from the Sengoku period, including weapons, armor, and everyday items.
		They also rotate every few months, so the exhibits are always fresh and interesting.
		Unfortunately, these floors were no-photo zones, so there are no photos of the exhibits.</p>
	<p>Coming out of the castle, we turned back to appreciate the castle from the outside and its surrounding square.
		On our way out, we chose to walk back so we could take in the scenery at a more leisurely pace.</p>
	<figure class="third ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250613155012.webp" title="A tree in the Osaka Castle Park">
			<img src="https://image.ibugone.com/jp-2025/IMG20250613155012.webp" alt="A tree in the Osaka Castle Park" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250613160630.webp" title="The outer moat of Osaka Castle">
			<img src="https://image.ibugone.com/jp-2025/IMG20250613160630.webp" alt="The outer moat of Osaka Castle" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250613160932.webp" title="The outer moat of Osaka Castle in another direction">
			<img src="https://image.ibugone.com/jp-2025/IMG20250613160932.webp" alt="The outer moat of Osaka Castle in another direction" loading="lazy" />
		</a>
	</figure>
	<p>For dinner, we went to a pre-ordered yakiniku tabehodai (all-you-can-eat grilled meat) restaurant.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250613193605.webp" class="image-popup" title="A table of grilled meat
"><img src="https://image.ibugone.com/jp-2025/IMG20250613193605.webp" alt="A table of grilled meat" loading="lazy" /></a>
		<figcaption>
			A table of grilled meat
		</figcaption>
	</figure>
	<p>When we were shopping for supplies at a nearby convenience store, I noticed a peculiar banner over the refrigerator section:</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250613210057.webp" class="image-popup" title="Are you OK?
"><img src="https://image.ibugone.com/jp-2025/IMG20250613210057.webp" alt="A peculiar banner over the refrigerator section" loading="lazy" /></a>
		<figcaption>
			Are you OK?
		</figcaption>
	</figure>
	<p>Time to rest and recharge for the next day!</p>
	<h2 id="day-9">Day 9: Expo 2025</h2>
	<p>First thing to note, while previous trips were all filled with joy and excitement, the Expo 2025 was the first (and so far the only) to bring disappointment and frustration, a stark contrast to the previous experiences.</p>
	<p>The Expo 2025 introduced a booking and lottery system for granting access to some popular pavilions, which could have been a great idea to avoid overcrowding and long queues.
		However, the website was like running on potatoes, loading extremely slowly.
		To add salt to the wound, it even came up with an online queuing system, where you had to keep the web page open for <em>hours</em>, watching the queue number going down from some 70,000, only to start looking for pavilions to book.
		You can book any pavilion until 7 days before the ticket date, where bookings are randomly approved in a lottery system.
		Granted, the 4 of us were lucky enough to get precisely <strong>no</strong> bookings from the lottery, and we were for sure delighted to find it very fair for the 4 of us.
		Anyways, as the ticket was non-refundable, we proceeded to the Expo 2025 site and sought for available pavilions.</p>
	<p>As many other sites in Osaka, the venue was built entirely on reclaimed land.
		Transportation to the Expo 2025 site was as accessible as it should be, with an extension of the Chuo Line that rides to Yumeshima Station.
		It was yet another drizzling day, but the rain did not stop tourists from crowding to the venue.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250614102817.webp" class="image-popup" title="Heavily crowded Yumeshima Station
"><img src="https://image.ibugone.com/jp-2025/IMG20250614102817.webp" alt="Heavily crowded Yumeshima Station" loading="lazy" /></a>
		<figcaption>
			Heavily crowded Yumeshima Station
		</figcaption>
	</figure>
	<p>The site also split incoming visitors by their booking time, and we were glad to come slightly later than our booked time, when the designated queue was already short.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250614103620.webp" class="image-popup" title="The queue for 11 AM getting crowded
"><img src="https://image.ibugone.com/jp-2025/IMG20250614103620.webp" alt="The queue for 11 AM getting crowded" loading="lazy" /></a>
		<figcaption>
			The queue for 11 AM getting crowded
		</figcaption>
	</figure>
	<p>As one of us got stuck in the queue for a while, we took the time to browse through a nearby merchandise store.</p>
	<p>Not gonna lie, the design of the Expo 2025 mascot is quite absurd.
		Totally beyond my aesthetics.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250614110825.webp" class="image-popup" title="The Expo 2025 mascot for sale
"><img src="https://image.ibugone.com/jp-2025/IMG20250614110825.webp" alt="The Expo 2025 mascot for sale" loading="lazy" /></a>
		<figcaption>
			The Expo 2025 mascot for sale
		</figcaption>
	</figure>
	<p>With the rain clearing up, we were able to start appreciating the site unencumbered.
		Notably, the largest wooden structure called <em>The Ring</em> caught our attention, with escalators leading up to the top.
		We saved it for later, and went on finding available pavilions that did not require bookings or lotteries.</p>
	<figure class="half ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250614115435.webp" title="The Ring at Expo 2025">
			<img src="https://image.ibugone.com/jp-2025/IMG20250614115435.webp" alt="The Ring at Expo 2025" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250614115846.webp" title="View towards the pavilions from the Ring">
			<img src="https://image.ibugone.com/jp-2025/IMG20250614115846.webp" alt="View towards the pavilions from the Ring" loading="lazy" />
		</a>
	</figure>
	<p>The first pavilion we managed to enter was the Malaysia Pavilion, which was a large building surrounded by bamboo.</p>
	<figure class="third ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250614120020.webp" title="The Malaysia Pavilion">
			<img src="https://image.ibugone.com/jp-2025/IMG20250614120020.webp" alt="The Malaysia Pavilion" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250614121959.webp" title="A miniature of the streets">
			<img src="https://image.ibugone.com/jp-2025/IMG20250614121959.webp" alt="A miniature of the streets" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250614122128.webp" title="A realistic food stall">
			<img src="https://image.ibugone.com/jp-2025/IMG20250614122128.webp" alt="A realistic food stall" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250614122332.webp" title="Another food stall">
			<img src="https://image.ibugone.com/jp-2025/IMG20250614122332.webp" alt="Another food stall" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250614122414.webp" title="An eco-friendly artificial tree">
			<img src="https://image.ibugone.com/jp-2025/IMG20250614122414.webp" alt="An eco-friendly artificial tree" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250614123323.webp" title="A sandbox with projectors, themed &quot;A City For All&quot;">
			<img src="https://image.ibugone.com/jp-2025/IMG20250614123323.webp" alt="A sandbox with projectors, themed &quot;A City For All&quot;" loading="lazy" />
		</a>
		<figcaption>Malaysia Pavilion
		</figcaption>
	</figure>
	<p>The next pavilion was the <em>Earth at Night</em> pavilion with a relatively short queue, backed by a similarly short exhibition.</p>
	<figure class="third ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250614130502.webp" title="The large globe">
			<img src="https://image.ibugone.com/jp-2025/IMG20250614130502.webp" alt="The large globe" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250614130736.webp" title="A &quot;Tokyo at Night&quot; display">
			<img src="https://image.ibugone.com/jp-2025/IMG20250614130736.webp" alt="A &quot;Tokyo at Night&quot; display" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250614130835.webp" title="Traditional Crafts of Japan">
			<img src="https://image.ibugone.com/jp-2025/IMG20250614130835.webp" alt="Traditional Crafts of Japan" loading="lazy" />
		</a>
		<figcaption>Earth at Night Pavilion
		</figcaption>
	</figure>
	<p>We also paid a brief visit to the Turkey Pavilion, but didn’t enter deeply as it approached lunch time.
		There’s a Turkish food stall right outside the pavilion, and it was still raining lightly, so we decided to take the convenience of the stall.
		To our dismay, the stall only offered a limited selection of food, and they were rather ordinary.
		Despite the mediocre food, the Turkey Pavilion also held an ice cream stall, which was very stylish and affordable (at JP¥ 800).</p>
	<p>After some random roaming, we discovered a large round ground surrounded by wooden pillars, with mist rising from the ground.
		It feels very magical indeed.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250614143034.webp" class="image-popup" title="A “Summoning Altar” maybe?
"><img src="https://image.ibugone.com/jp-2025/IMG20250614143034.webp" alt="A large round ground surrounded by wooden pillars" loading="lazy" /></a>
		<figcaption>
			A “Summoning Altar” maybe?
		</figcaption>
	</figure>
	<p>For the afternoon, we queued up for the China Pavilion which fortunately did not require a booking.
		The queue was long, and we were held in rain for an hour before reaching sheltered areas.
		In spite of that, we enjoyed the fantastic display the China Pavilion offered.</p>
	<figure class="third ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250614144146.webp" title="Huge bamboo slips outside the China Pavilion">
			<img src="https://image.ibugone.com/jp-2025/IMG20250614144146.webp" alt="Huge bamboo slips outside the China Pavilion" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250614150404.webp" title="The pavilion sign as we were still queuing up">
			<img src="https://image.ibugone.com/jp-2025/IMG20250614150404.webp" alt="The pavilion sign as we were still queuing up" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250614153036.webp" title="Interactive display of ancient artifacts">
			<img src="https://image.ibugone.com/jp-2025/IMG20250614153036.webp" alt="Interactive display of ancient artifacts" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250614154156.webp" title="Wall sculptures showing a China-Japan meeting">
			<img src="https://image.ibugone.com/jp-2025/IMG20250614154156.webp" alt="Wall sculptures showing a China-Japan meeting" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250614154515.webp" title="An 8-minute video room about daily life in China">
			<img src="https://image.ibugone.com/jp-2025/IMG20250614154515.webp" alt="An 8-minute video room about daily life in China" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250614155420.webp" title="A sandbox city showing China&#39;s ambitions for the future">
			<img src="https://image.ibugone.com/jp-2025/IMG20250614155420.webp" alt="A sandbox city showing China&#39;s ambitions for the future" loading="lazy" />
		</a>
		<figcaption>China Pavilion
		</figcaption>
	</figure>
	<p>Coming out of the China Pavilion, we spent the rest of our visit to Expo on the outside.</p>
	<p>For exmaple, I almost mistake this statue of <em>Lapras</em> for <em>Chillet</em> in <em>Palworld</em>.
		Guess Nintendo didn’t sue them out of nothing.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250614161004.webp" class="image-popup" title="A statue of Lapras
"><img src="https://image.ibugone.com/jp-2025/IMG20250614161004.webp" alt="A statue of Lapras" loading="lazy" /></a>
		<figcaption>
			A statue of Lapras
		</figcaption>
	</figure>
	<p>As the rain cleared for a while, we finally took the chance to visit The Ring and enjoy the view of Osaka from the top.</p>
	<figure class="third ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250614160719.webp" title="The Ring over the sea">
			<img src="https://image.ibugone.com/jp-2025/IMG20250614160719.webp" alt="The Ring over the sea" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250614163022.webp" title="Another view towards the pavilions from the Ring">
			<img src="https://image.ibugone.com/jp-2025/IMG20250614163022.webp" alt="Another view towards the pavilions from the Ring" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250614163127.webp" title="A watch tower in the distance">
			<img src="https://image.ibugone.com/jp-2025/IMG20250614163127.webp" alt="A watch tower in the distance" loading="lazy" />
		</a>
	</figure>
	<hr />
	<p>Coming out of Expo, we went to the dinner booked with a friend, who is already the head of a multi-national enterprise with global influence.</p>
	<figure class="third ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250614193633.webp" title="Fried shrimp">
			<img src="https://image.ibugone.com/jp-2025/IMG20250614193633.webp" alt="Fried shrimp" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250614193816.webp" title="A sashimi set">
			<img src="https://image.ibugone.com/jp-2025/IMG20250614193816.webp" alt="A sashimi set" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250614202953.webp" title="Salted lobsters">
			<img src="https://image.ibugone.com/jp-2025/IMG20250614202953.webp" alt="Salted lobsters" loading="lazy" />
		</a>
	</figure>
	<p>We had a great time catching up with each other and exchanging ideas.
		Following the dinner, we went to a Don Quijote from our friend’s suggestions, to shop for souvenirs and snacks before our return.</p>
	<figure class="third ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250614220039.webp" title="Starbucks-branded matcha latte">
			<img src="https://image.ibugone.com/jp-2025/IMG20250614220039.webp" alt="Starbucks-branded matcha latte" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250614220202.webp" title="Food and seasoning section for sashimi and sushi">
			<img src="https://image.ibugone.com/jp-2025/IMG20250614220202.webp" alt="Food and seasoning section for sashimi and sushi" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250614222502.webp" title="More seasoning, and Haidilao self-heating hot pot">
			<img src="https://image.ibugone.com/jp-2025/IMG20250614222502.webp" alt="More seasoning, and Haidilao self-heating hot pot" loading="lazy" />
		</a>
	</figure>
	<p>I bought some snacks and cosmetics for my family, including two boxes of <em>Pocky</em> and an Osaka-exclusive cup cake.
		It’s a tax-free store for anyone presenting a passport and a gross purchase of at least JP¥ 5,000, so I happily saved the consumption tax.</p>
	<h2 id="day-10">Day 10: Return to China</h2>
	<p>As my friends were returning to Shanghai while I would be returning to Shenzhen on my own, this last day left me on my own with an extra hour.
		After packing up my luggages and enjoying one last ramen in Osaka, I took a brief walk around the neighborhood.</p>
	<figure class=""><a href="https://image.ibugone.com/jp-2025/IMG20250615124826.webp" class="image-popup" title="Iconic Dotonbori view
"><img src="https://image.ibugone.com/jp-2025/IMG20250615124826.webp" alt="Iconic Dotonbori view" loading="lazy" /></a>
		<figcaption>
			Iconic Dotonbori view
		</figcaption>
	</figure>
	<p>As time went by, I took the Nankai Line from Namba Station to Kansai International Airport, a 50-minute ride, and proceeded with my return flight.</p>
	<figure class="third ">
		<a href="https://image.ibugone.com/jp-2025/IMG20250615135858.webp" title="Inside Kansai International Airport">
			<img src="https://image.ibugone.com/jp-2025/IMG20250615135858.webp" alt="Inside Kansai International Airport" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250615165253.webp" title="Flight meal">
			<img src="https://image.ibugone.com/jp-2025/IMG20250615165253.webp" alt="Flight meal" loading="lazy" />
		</a>
		<a href="https://image.ibugone.com/jp-2025/IMG20250615185114.webp" title="Finally landed at Shenzhen Bao&#39;an International Airport">
			<img src="https://image.ibugone.com/jp-2025/IMG20250615185114.webp" alt="Finally landed at Shenzhen Bao&#39;an International Airport" loading="lazy" />
		</a>
	</figure>
	<p>Anecdotally, I landed in Shenzhen safely, returning home on time, while my friends’ flight diverted to Nanjing due to heavy downpour in Shanghai, and only returned to Shanghai near midnight.</p>
	<h2 id="thoughts">Thoughts</h2>
	<p>This trip to Japan surely brought an unforgettable experience and broadened my horizons.
		The culture and the people were all amazing, and food and dining were all delicious.
		Given Japan’s proximity to China, it definitely stands out as a great destination for short trips, and time permitting, I will for sure visit Japan again in the future.</p>
	<s>Also I previously arranged a Chinese version of this article, but it has since been scrapped as <i>why wouldn't the English version suffice alone?</i></s>
	<div class="footnotes" role="doc-endnotes">
		<ol>
			<li id="fn:suica-recharge">
				<p>Physical Suica cards can only be recharged with cash and no bank cards, not even debit cards. <a href="#fnref:suica-recharge" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
			</li>
		</ol>
	</div>
	]]></content><author><name>iBug</name></author><category term="life" /><summary type="html"><![CDATA[As a more meaningful way to celebrate our graduation as Master students and spend this last truly free time in a long while, I joined my friends on a trip to Japan.]]></summary></entry><entry><title type="html">Fixing OneDrive not expanding in Explorer sidebar on Windows 10 LTSC</title><link href="https://ibug.io/blog/2025/04/windows-10-ltsc-onedrive-explorer-sidebar-fix/" rel="alternate" type="text/html" title="Fixing OneDrive not expanding in Explorer sidebar on Windows 10 LTSC" /><published>2025-04-22T00:00:00+00:00</published><updated>2025-04-23T00:39:07+00:00</updated><id>https://ibug.io/blog/2025/04/windows-10-ltsc-onedrive-explorer-sidebar-fix</id><content type="html" xml:base="https://ibug.io/blog/2025/04/windows-10-ltsc-onedrive-explorer-sidebar-fix/"><![CDATA[<p>I’ve recently re-installed Windows 10 LTSC for both of my computers, and after setting up OneDrive (from Office 365 suite), I noticed that the OneDrive folder in the Explorer sidebar was not expanding when clicked, showing a static gray arrow.
		This <em>could</em> be a known issue with Windows 10 LTSC, as it doesn’t include OneDrive integration by default, unlike consumer versions.</p>
	<p>For readers who want to skip the details, just head to the <a href="#solution"><strong>Solution</strong></a> section.</p>
	<h2 id="gathering-information">Gathering Information</h2>
	<p>Looking for “Windows 10 OneDrive doesn’t expand” on Google, <a href="https://answers.microsoft.com/en-us/msoffice/forum/all/one-drive-folder-tab-not-expanding/04eac1d9-2665-4c5e-b02b-9976fbcf2c49">One drive folder tab not expanding</a> shows up first.
		It contains no useful information other than a link to another: <a href="https://answers.microsoft.com/en-us/windows/forum/all/cannot-access-onedrive-personal-folder-through/fb21a27c-bf5a-4371-926e-9ff869070f03">Cannot access Onedrive - Personal folder through Quick Access of File Explorer</a>.</p>
	<p>The second link is much better populated with information, notably there’s a valid workaround:</p>
	<blockquote>
		<div class="language-ini highlighter-rouge">
			<div class="highlight">
				<pre class="highlight"><code><span class="na">Windows</span><span class="w"> </span><span class="na">Registry</span><span class="w"> </span><span class="na">Editor</span><span class="w"> </span><span class="na">Version</span><span class="w"> </span><span class="na">5.00</span><span class="w">

</span><span class="nn">[HKEY_CLASSES_ROOT\CLSID\{018D5C66-4533-4307-9B53-224DE2ED1FE6}\Instance\InitPropertyBag]</span><span class="w">
</span><span class="na">"TargetKnownFolder"=""</span><span class="w">

</span><span class="nn">[HKEY_CLASSES_ROOT\WOW6432Node\CLSID\{018D5C66-4533-4307-9B53-224DE2ED1FE6}\Instance\InitPropertyBag]</span><span class="w">
</span><span class="na">"TargetKnownFolder"=""</span><span class="w">
</span></code></pre>
			</div>
  </div>
		<p>Save them as a .reg file, run it each time you log on (and after One Drive running)</p>
	</blockquote>
	<p>Note how this needs to be applied every time OneDrive starts, which is something to further investigate.</p>
	<p>With a bit more trying and Googling, it turns out OneDrive sets these keys to <code class="language-plaintext highlighter-rouge">{a52bba46-e9e1-435f-b3d9-28daa648c0f6}</code>, which is the KnownFolderID for the OneDrive folder, and is missing from the registry.</p>
	<p>From the two <code class="language-plaintext highlighter-rouge">bbs.pcbeta.com</code> links (Chinese: 远景论坛, <a href="https://bbs.pcbeta.com/viewthread-1933367-2-2.html">1</a> #32, <a href="https://bbs.pcbeta.com/viewthread-1936542-1-2.html">2</a> #17) mentioned in the thread, adding back the missing registry keys for <code class="language-plaintext highlighter-rouge">{a52bba46-e9e1-435f-b3d9-28daa648c0f6}</code> should fix the issue.</p>
	<h2 id="solution">Solution</h2>
	<p>Just import the missing registry keys for OneDrive.
		These registry keys are exported from a working Windows 10 Enterprise installation.</p>
	<div class="language-ini highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code><span class="na">Windows</span><span class="w"> </span><span class="na">Registry</span><span class="w"> </span><span class="na">Editor</span><span class="w"> </span><span class="na">Version</span><span class="w"> </span><span class="na">5.00</span><span class="w">

</span><span class="nn">[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\FolderDescriptions\{A52BBA46-E9E1-435f-B3D9-28DAA648C0F6}]</span><span class="w">
</span><span class="na">"Attributes"=dword:00000001</span><span class="w">
</span><span class="na">"Category"=dword:00000004</span><span class="w">
</span><span class="na">"DefinitionFlags"=dword:00000040</span><span class="w">
</span><span class="na">"Icon"=hex(2):25,00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,74,</span><span class="se">\
</span><span class="w">  </span><span class="na">00,25,00,5c,00,73,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,00,5c,00,69,00,</span><span class="se">\
</span><span class="w">  </span><span class="na">6d,00,61,00,67,00,65,00,72,00,65,00,73,00,2e,00,64,00,6c,00,6c,00,2c,00,2d,</span><span class="se">\
</span><span class="w">  </span><span class="na">00,31,00,30,00,34,00,30,00,00,00</span><span class="w">
</span><span class="na">"LocalizedName"=hex(2):40,00,25,00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,</span><span class="se">\
</span><span class="w">  </span><span class="na">6f,00,6f,00,74,00,25,00,5c,00,53,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,</span><span class="se">\
</span><span class="w">  </span><span class="na">00,5c,00,53,00,65,00,74,00,74,00,69,00,6e,00,67,00,53,00,79,00,6e,00,63,00,</span><span class="se">\
</span><span class="w">  </span><span class="na">43,00,6f,00,72,00,65,00,2e,00,64,00,6c,00,6c,00,2c,00,2d,00,31,00,30,00,32,</span><span class="se">\
</span><span class="w">  </span><span class="na">00,34,00,00,00</span><span class="w">
</span><span class="na">"LocalRedirectOnly"=dword:00000001</span><span class="w">
</span><span class="na">"Name"="OneDrive"</span><span class="w">
</span><span class="na">"ParentFolder"="{5E6C858F-0E22-4760-9AFE-EA3317B67173}"</span><span class="w">
</span><span class="na">"ParsingName"="shell:::{018D5C66-4533-4307-9B53-224DE2ED1FE6}"</span><span class="w">
</span><span class="na">"RelativePath"="OneDrive"</span><span class="w">

</span><span class="nn">[HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\Windows\CurrentVersion\Explorer\FolderDescriptions\{A52BBA46-E9E1-435f-B3D9-28DAA648C0F6}]</span><span class="w">
</span><span class="na">"Attributes"=dword:00000001</span><span class="w">
</span><span class="na">"Category"=dword:00000004</span><span class="w">
</span><span class="na">"DefinitionFlags"=dword:00000040</span><span class="w">
</span><span class="na">"Icon"=hex(2):25,00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,74,</span><span class="se">\
</span><span class="w">  </span><span class="na">00,25,00,5c,00,73,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,00,5c,00,69,00,</span><span class="se">\
</span><span class="w">  </span><span class="na">00,31,00,30,00,34,00,30,00,00,00</span><span class="w">
</span><span class="na">"LocalizedName"=hex(2):40,00,25,00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,</span><span class="se">\
</span><span class="w">  </span><span class="na">6f,00,6f,00,74,00,25,00,5c,00,53,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,</span><span class="se">\
</span><span class="w">  </span><span class="na">00,5c,00,53,00,65,00,74,00,74,00,69,00,6e,00,67,00,53,00,79,00,6e,00,63,00,</span><span class="se">\
</span><span class="w">  </span><span class="na">43,00,6f,00,72,00,65,00,2e,00,64,00,6c,00,6c,00,2c,00,2d,00,31,00,30,00,32,</span><span class="se">\
</span><span class="w">  </span><span class="na">00,34,00,00,00</span><span class="w">
</span><span class="na">"LocalRedirectOnly"=dword:00000001</span><span class="w">
</span><span class="na">"Name"="OneDrive"</span><span class="w">
</span><span class="na">"ParentFolder"="{5E6C858F-0E22-4760-9AFE-EA3317B67173}"</span><span class="w">
</span><span class="na">"ParsingName"="shell:::{018D5C66-4533-4307-9B53-224DE2ED1FE6}"</span><span class="w">
</span><span class="na">"RelativePath"="OneDrive"</span><span class="w">
</span></code></pre>
		</div>
	</div>
	<p>Note that this article happens to reproduce <a href="https://lolicp.com/windows/202309626.html">this blog</a> from lolicp.com, which may be of Chinese readers’ interest.</p>
	]]></content><author><name>iBug</name></author><category term="windows" /><summary type="html"><![CDATA[I’ve recently re-installed Windows 10 LTSC for both of my computers, and after setting up OneDrive (from Office 365 suite), I noticed that the OneDrive folder in the Explorer sidebar was not expanding when clicked, showing a static gray arrow. This could be a known issue with Windows 10 LTSC, as it doesn’t include OneDrive integration by default, unlike consumer versions.]]></summary></entry><entry><title type="html">Beating $3k SSD with $2k HDD?</title><link href="https://ibug.io/blog/2024/10/ustc-mirrors-zfs-rebuild/" rel="alternate" type="text/html" title="Beating $3k SSD with $2k HDD?" /><published>2024-10-27T00:00:00+00:00</published><updated>2025-07-11T11:18:31+00:00</updated><id>https://ibug.io/blog/2024/10/ustc-mirrors-zfs-rebuild</id><content type="html" xml:base="https://ibug.io/blog/2024/10/ustc-mirrors-zfs-rebuild/"><![CDATA[<p>A.K.A. Practical ZFS application on USTC Mirrors. A writeup of the talk I gave at Nanjing University this August.</p>
	<p>Also, a <a href="https://lug.ustc.edu.cn/planet/2024/12/ustc-mirrors-zfs-rebuild/">Chinese version</a> is available if you want.</p>
	<h2 id="background">Background</h2>
	<p><a href="https://mirrors.ustc.edu.cn/">USTC Open-Source Software Mirrors</a> is one of the largest public mirror sites in China. In the two months of May and June 2024, we served an average daily egress traffic of some 36&nbsp;TiB, which breaks down as follows:</p>
	<ul>
		<li>19&nbsp;TiB from HTTP/HTTPS, among 17M requests</li>
		<li>10.3&nbsp;TiB from rsync, among 21.8k requests (if we count one absurd client in, the number of requests goes to 147.8k)</li>
	</ul>
	<p>Over the years, as mirror repositories have grown and new repositories have been added, we have been running tight on disk space. For our two servers responsible for the mirror service, we have reached unhealthy levels of disk usage:</p>
	<ul>
		<li>HTTP server (XFS): 63.3&nbsp;TiB used out of 66.0&nbsp;TiB (96%, achieved on December 18, 2023)</li>
		<li>Rsync server (ZFS): 42.4&nbsp;TiB used out of 43.2&nbsp;TiB (98%, achieved on November 21, 2023)</li>
	</ul>
	<p>The servers have the following configurations:</p>
	<dl>
		<dt>HTTP server</dt>
		<dd>
			<ul>
				<li>Set up in Fall 2020</li>
				<li>Intel Cascade Lake CPU, 256&nbsp;GB DDR4 RAM</li>
				<li>Twelve 10&nbsp;TB HDDs + One 2&nbsp;TB SSD</li>
				<li>XFS on LVM on hardware RAID</li>
				<li>Reserved free PEs on LVM VG level as XFS cannot be shrunk</li>
			</ul>
		</dd>
		<dt>Rsync server</dt>
		<dd>
			<ul>
				<li>Set up in Winter 2016</li>
				<li>Intel Broadwell CPU, 256&nbsp;GB DDR4 RAM</li>
				<li>Twelve 6&nbsp;TB HDDs + some smaller SSDs for OS and cache</li>
				<li>RAID-Z3 on ZFS, 8 data disks + 3 parity disks + 1 hot spare</li>
				<li>All default parameters (except <code class="language-plaintext highlighter-rouge">zfs_arc_max</code>)</li>
			</ul>
		</dd>
	</dl>
	<p>These servers are constantly running at an I/O utilization of over 90%, which results in less than 50&nbsp;MB/s download speed even from within USTC campus. Clearly this is not the ideal performance for this kind of dedicated storage servers.</p>
	<figure class=""><a href="https://image.ibugone.com/grafana/mirrors-io-utilization-may-2024.png" class="image-popup" title="I/O load of two servers from USTC Mirrors in May 2024
"><img src="https://image.ibugone.com/grafana/mirrors-io-utilization-may-2024.png" alt="I/O load of two servers from USTC Mirrors in May 2024" loading="lazy" /></a>
		<figcaption>
			I/O load of two servers from USTC Mirrors in May 2024
		</figcaption>
	</figure>
	<h2 id="zfs">ZFS</h2>
	<p>ZFS is usually known for being the ultimate single-node storage solution. It combines RAID, volume management, and filesystem in one, and provides advanced features like snapshots, clones and send/receive. Everything in ZFS is checksummed, ensuring data integrity. For servers dedicated to storage, ZFS appears to be a “fire and forget” solution, which is easily challenged by its tremendous amount of tunables and parameters.</p>
	<p>As preliminary learning and experiments, I sourced some drives for my own workstation and set up two ZFS pools on them. Then I signed up for some private tracker (PT) sites for I/O load to tune for. The results were quite satisfying: In two years and a half, my single-node PT station has generated 1.20&nbsp;PiB of uploads.</p>
	<p>Over the years, I have gathered some of my most important sources for learning ZFS:</p>
	<ul>
		<li>Chris’s Wiki: <a href="https://utcc.utoronto.ca/~cks/space/blog/">https://utcc.utoronto.ca/~cks/space/blog/</a></li>
		<li>OpenZFS Documentation: <a href="https://openzfs.github.io/openzfs-docs/">https://openzfs.github.io/openzfs-docs/</a></li>
		<li>My own blog: <a href="/p/62">Understanding ZFS block sizes</a>
			<ul>
				<li>Plus all references in the article</li>
			</ul>
		</li>
	</ul>
	<figure class=""><a href="https://image.ibugone.com/grafana/qb/2024-06-05.png" class="image-popup" title="A byproduct of my ZFS learning: A Grafana dashboard for qBittorrent (lol…)
"><img src="https://image.ibugone.com/grafana/qb/2024-06-05.png" alt="A Grafana dashboard for qBittorrent" loading="lazy" /></a>
		<figcaption>
			A byproduct of my ZFS learning: A Grafana dashboard for qBittorrent (lol…)
		</figcaption>
	</figure>
	<p>After these years of learning ZFS, I realized that there’s a substantial room for improvement in our mirror servers, by embracing ZFS and tuning it properly.</p>
	<h2 id="mirrors">Mirrors</h2>
	<p>Before we move on to rebuilding the ZFS pool, we need to understand our I/O workload. In essence, a mirror site:</p>
	<ul>
		<li>Provides file downloads</li>
		<li>Also (begrudgingly) serves as speed tests</li>
		<li>Mostly reads, and almost all reads are whole-file sequential reads</li>
		<li>Can withstand minimal data loss as mirror contents can be easily re-synced</li>
	</ul>
	<figure class=""><a href="https://image.ibugone.com/server/mirrors-file-size-distribution-2024-08.png" class="image-popup" title="File size distribution of USTC Mirrors in August 2024
"><img src="https://image.ibugone.com/server/mirrors-file-size-distribution-2024-08.png" alt="File size distribution of USTC Mirrors in August 2024" loading="lazy" /></a>
		<figcaption>
			File size distribution of USTC Mirrors in August 2024
		</figcaption>
	</figure>
	<p>With those in mind, we analyzed our mirror content. As can be seen from the graph above, half of the 40M files are less than 10&nbsp;KiB in size, and 90% of the files are less than 1&nbsp;MiB. Still, the files are averaged at 1.6&nbsp;MiB.</p>
	<h2 id="mirrors2">Rebuilding the Rsync server</h2>
	<p>In June, we set out to rebuild the Rsync server as it had a lower service traffic and importance, yet a disproportionately higher disk usage. We laid out the following plan:</p>
	<ul>
		<li>First, the RAID overhead of RAID-Z3 was too high (reiterating: half of the files are less than 10&nbsp;KiB, and the disks have 4&nbsp;KiB sectors), so we decided to switch to RAID-Z2 as well as split the RAID group into two. Two RAIDZ vdevs also implies double the IOPS, as each “block” (in ZFS parlance) is stored on only one vdev.</li>
		<li>We then carefully select dataset properties to optimize for our workload:
			<ul>
				<li><code class="language-plaintext highlighter-rouge">recordsize=1M</code> to maximize sequential throughput and minimize fragmentation</li>
				<li><code class="language-plaintext highlighter-rouge">compression=zstd</code> to (try to) save some disk space
					<ul>
						<li>
							<p>Since OpenZFS 2.2, a mechanism called “early-abort” has been extended to Zstd compression (level 3+), which saves CPU cycles by testing data compressibility with LZ4 then Zstd 1, before actually trying to compress with Zstd.</p>
							<p>We know that most of our mirror content is already compressed (like software packages and ISOs), so early-abort is urging us to use Zstd.</p>
						</li>
					</ul>
				</li>
				<li><code class="language-plaintext highlighter-rouge">xattr=off</code> as we don’t need extended attributes for mirror content.</li>
				<li><code class="language-plaintext highlighter-rouge">atime=off</code> as we don’t need access time. Also cuts off a lot of writes.</li>
				<li><code class="language-plaintext highlighter-rouge">setuid=off</code>, <code class="language-plaintext highlighter-rouge">exec=off</code>, <code class="language-plaintext highlighter-rouge">devices=off</code> to disable what we don’t need.</li>
				<li><code class="language-plaintext highlighter-rouge">secondarycache=metadata</code> to cache metadata only, as this Rsync server has a much more uniform access pattern than the HTTP server. We would like to save our SSDs from unnecessary writes.</li>
			</ul>
		</li>
		<li>Some slightly dangerous properties:
			<ul>
				<li><code class="language-plaintext highlighter-rouge">sync=disabled</code> to disable synchronous writes. This allows ZFS to buffer writes up to <code class="language-plaintext highlighter-rouge">zfs_txg_timeout</code> seconds and make better allocation decisions.</li>
				<li><code class="language-plaintext highlighter-rouge">redundant_metadata=some</code> to trade some metadata redundancy for better write performance.</li>
			</ul>
			<p>We believe these changes are in alignment with our evaluation of data safety and loss tolerance.</p>
		</li>
		<li>
			<p>For ZFS module parameters, the sheer number of 290+ tunables is overwhelming. Thanks to @happyaron, the current ZFS maintainer in Debian and administrator of BFSU Mirror, we selected a handful of them:</p>
			<div class="language-shell highlighter-rouge">
				<div class="highlight">
					<pre class="highlight"><code><span class="c"># Set ARC size to 160-200&nbsp;GiB, keep 16&nbsp;GiB free for OS</span>
options zfs <span class="nv">zfs_arc_max</span><span class="o">=</span>214748364800
options zfs <span class="nv">zfs_arc_min</span><span class="o">=</span>171798691840
options zfs <span class="nv">zfs_arc_sys_free</span><span class="o">=</span>17179869184

<span class="c"># Favor metadata to data by 20x (OpenZFS 2.2+)</span>
options zfs <span class="nv">zfs_arc_meta_balance</span><span class="o">=</span>2000

<span class="c"># Allow up to 80% of ARC to be used for dnodes</span>
options zfs <span class="nv">zfs_arc_dnode_limit_percent</span><span class="o">=</span>80

<span class="c"># See man page section "ZFS I/O Scheduler"</span>
options zfs <span class="nv">zfs_vdev_async_read_max_active</span><span class="o">=</span>8
options zfs <span class="nv">zfs_vdev_async_read_min_active</span><span class="o">=</span>2
options zfs <span class="nv">zfs_vdev_scrub_max_active</span><span class="o">=</span>5
options zfs <span class="nv">zfs_vdev_max_active</span><span class="o">=</span>20000

<span class="c"># Never throttle the ARC</span>
options zfs <span class="nv">zfs_arc_lotsfree_percent</span><span class="o">=</span>0

<span class="c"># Tune L2ARC</span>
options zfs <span class="nv">l2arc_headroom</span><span class="o">=</span>8
options zfs <span class="nv">l2arc_write_max</span><span class="o">=</span>67108864
options zfs <span class="nv">l2arc_noprefetch</span><span class="o">=</span>0
</code></pre>
				</div>
    </div>
			<p>And also <code class="language-plaintext highlighter-rouge">zfs_dmu_offset_next_sync</code>, which is enabled by default since OpenZFS 2.1.5, so it’s omitted from our list.</p>
		</li>
	</ul>
	<p>After relocating Rsync service to our primary server (HTTP server), we broke up the existing ZFS pool and rebuilt it anew, before syncing previous repositories back from external sources. To our surprise, the restoration took only 3 days, much faster than we had anticipated. Other numbers also looked promising:</p>
	<ul>
		<li>
			<p>Compression ratio: 39.5T / 37.1T (1.07x)</p>
			<p>We’d like to point out that ZFS only provides two digits after the decimal point for compression ratio, so if you want a higher precision, you need take the raw numbers and calculate it yourself:</p>
			<div class="language-shell highlighter-rouge">
				<div class="highlight">
					<pre class="highlight"><code>zfs list <span class="nt">-po</span> name,logicalused,used
</code></pre>
				</div>
    </div>
			<p>Our actual number was 1 + 6.57%, at 2.67&nbsp;TB (2.43&nbsp;TiB) saved, which means equivalently 9 copies of WeChat data <a href="https://image.ibugone.com/teaser/lenovo-legion-wechat-data.jpg">as advertised by Lenovo Legion</a>.</p>
		</li>
		<li>
			<p>And most importantly, a much saner I/O load:</p>
			<figure class=""><a href="https://image.ibugone.com/grafana/mirrors2-io-utilization-and-free-space-june-july-2024.png" class="image-popup" title="I/O load of server “mirrors2” before and after the rebuild
"><img src="https://image.ibugone.com/grafana/mirrors2-io-utilization-and-free-space-june-july-2024.png" alt="I/O load of server mirrors2 before and after the rebuild" loading="lazy" /></a>
				<figcaption>
					I/O load of server “mirrors2” before and after the rebuild
				</figcaption>
			</figure>
		</li>
	</ul>
	<p>We can see that, after a few days of warm-up, the I/O load has maintained at around 20%, whereas it was constantly at 90% before the rebuild.</p>
	<h2 id="mirrors4">Rebuilding the HTTP server</h2>
	<p>Our HTTP server was set up in late 2020 and under a different background.
		When we were first deciding the technology stack, we were not confident in ZFS and were discouraged by the abysmal performance of our Rsync server.
		So we opted for an entirely different stack for this server: hardware RAID, LVM (because the RAID controller didn’t allow RAID groups across two controllers), and XFS.
		For memory caching, we relied on kernel’s page cache, and for SSD caching, we tried LVMcache, which was quite new at the moment and rather immature.</p>
	<p>These unpracticed technologies have, without a doubt, ended up a pain.</p>
	<ul>
		<li>XFS cannot be shrunk, so we had to reserve free PEs at LVM VG level. We also cannot fill the FS, so there are two levels of free space reservation. Double the waste.</li>
		<li>We initially allocated 1.5&nbsp;TB of SSD cache, but given LVMcache’s recommendation of no more than 1 million chunks, we opted for just 1&nbsp;TiB (1&nbsp;MiB chunk size × 1 Mi chunks).</li>
		<li>There were no options for cache eviction policy, so later we dug into the kernel source code and found that it was a 64-level LRU.</li>
		<li>The first thing to die was GRUB2. Due to GRUB’s parsing of LVM metadata, it was unable to boot from a VG where a cached volume was present. We had to <a href="https://github.com/taoky/grub/commit/85b260baec91aa4f7db85d7592f6be92d549a0ae">patch</a> GRUB for it to handle this case.</li>
		<li>With an incorrect understanding of chunk size and number of chunks, our SSD ran severely over its write endurance in under 2 years, and we had to replace it with a new one.</li>
	</ul>
	<p>Even after understanding the algorithm and still going for 128&nbsp;KiB chunk size and over 8 Mi chunks, LVMcache still didn’t offer a competitive hit rate:</p>
	<figure class=""><a href="https://image.ibugone.com/grafana/mirrors4-dmcache-may-june-2024.png" class="image-popup" title="LVMcache hit rate over May to June 2024
"><img src="https://image.ibugone.com/grafana/mirrors4-dmcache-may-june-2024.png" alt="LVMcache hit rate over May to June 2024" loading="lazy" /></a>
		<figcaption>
			LVMcache hit rate over May to June 2024
		</figcaption>
	</figure>
	<p>We had already been fed up with those troubles through the years, and the success with our Rsync server rebuild gave us great confidence with ZFS.
		So in less than a month, we laid out a similar plan for our HTTP server, but trying something new:</p>
	<ul>
		<li>We updated the kernel to <code class="language-plaintext highlighter-rouge">6.8.8-3-pve</code>, which bundles the latest <code class="language-plaintext highlighter-rouge">zfs.ko</code> for us. This means we don’t have to waste time on DKMS.</li>
		<li>Since the number of disks is the same (12 disks), we also went for two RAID-Z2 vdevs with 6 disks each.
			<ul>
				<li>As this server provides HTTP service to end users, the access pattern will have a greater hot/cold distinction than the Rsync server. So we keep <code class="language-plaintext highlighter-rouge">secondarycache=all</code> for this server (leave the default value unchanged).</li>
				<li>This newer server has a better CPU, so we increased compression level to <code class="language-plaintext highlighter-rouge">zstd-8</code> in hope for a better compression ratio.</li>
			</ul>
		</li>
		<li>Since we already have the Rsync server running ZFS with desired parameters, we have <code class="language-plaintext highlighter-rouge">zfs send -Lcp</code> available when syncing the data back. This allows us to restore 50+ TiB of data in just 36 hours.</li>
		<li>Due to having a slightly different set of repositories, the compression ratio is slightly lower at 1 + 3.93% (2.42&nbsp;TiB / 2.20&nbsp;TiB saved).</li>
	</ul>
	<p>We put the I/O loads of both servers together for comparison:</p>
	<figure class=""><a href="https://image.ibugone.com/grafana/mirrors2-4-io-utilization-june-july-2024.png" class="image-popup" title="I/O load of two servers from USTC Mirrors before and after rebuild
"><img src="https://image.ibugone.com/grafana/mirrors2-4-io-utilization-june-july-2024.png" alt="I/O load of two servers from USTC Mirrors before and after rebuild" loading="lazy" /></a>
		<figcaption>
			I/O load of two servers from USTC Mirrors before and after rebuild
		</figcaption>
	</figure>
	<p>This graph starts with the initial state. The first server was rebuilt at 1/3, and the second server was rebuilt at 2/3.</p>
	<p>The hit rate of ZFS ARC is also quite satisfying:</p>
	<figure class=""><a href="https://image.ibugone.com/grafana/mirrors2-4-zfs-arc-hit-rate.png" class="image-popup" title="ZFS ARC hit rate of two servers
"><img src="https://image.ibugone.com/grafana/mirrors2-4-zfs-arc-hit-rate.png" alt="ZFS ARC hit rate of two servers" loading="lazy" /></a>
		<figcaption>
			ZFS ARC hit rate of two servers
		</figcaption>
	</figure>
	<p>The stablized I/O load is even lower after both servers were rebuilt.</p>
	<figure class=""><a href="https://image.ibugone.com/grafana/mirrors2-4-disk-io-after-rebuild.png" class="image-popup" title="Sustained disk I/O of two servers after rebuild
"><img src="https://image.ibugone.com/grafana/mirrors2-4-disk-io-after-rebuild.png" alt="Sustained disk I/O of two servers after rebuild" loading="lazy" /></a>
		<figcaption>
			Sustained disk I/O of two servers after rebuild
		</figcaption>
	</figure>
	<h2 id="misc">Misc</h2>
	<h3 id="zfs-compression">ZFS compression</h3>
	<p>We are slightly surprised to see that so many repositories are well-compressible:</p>
	<table>
		<thead>
			<tr>
				<th style="text-align: left">NAME</th>
				<th style="text-align: right">LUSED</th>
				<th style="text-align: right">USED</th>
				<th style="text-align: right">RATIO</th>
			</tr>
		</thead>
		<tbody>
			<tr>
				<td style="text-align: left">pool0/repo/crates.io-index</td>
				<td style="text-align: right">2.19G</td>
				<td style="text-align: right">1.65G</td>
				<td style="text-align: right">3.01x</td>
			</tr>
			<tr>
				<td style="text-align: left">pool0/repo/elpa</td>
				<td style="text-align: right">3.35G</td>
				<td style="text-align: right">2.32G</td>
				<td style="text-align: right">1.67x</td>
			</tr>
			<tr>
				<td style="text-align: left">pool0/repo/rfc</td>
				<td style="text-align: right">4.37G</td>
				<td style="text-align: right">3.01G</td>
				<td style="text-align: right">1.56x</td>
			</tr>
			<tr>
				<td style="text-align: left">pool0/repo/debian-cdimage</td>
				<td style="text-align: right">1.58T</td>
				<td style="text-align: right">1.04T</td>
				<td style="text-align: right">1.54x</td>
			</tr>
			<tr>
				<td style="text-align: left">pool0/repo/tldp</td>
				<td style="text-align: right">4.89G</td>
				<td style="text-align: right">3.78G</td>
				<td style="text-align: right">1.48x</td>
			</tr>
			<tr>
				<td style="text-align: left">pool0/repo/loongnix</td>
				<td style="text-align: right">438G</td>
				<td style="text-align: right">332G</td>
				<td style="text-align: right">1.34x</td>
			</tr>
			<tr>
				<td style="text-align: left">pool0/repo/rosdistro</td>
				<td style="text-align: right">32.2M</td>
				<td style="text-align: right">26.6M</td>
				<td style="text-align: right">1.31x</td>
			</tr>
		</tbody>
	</table>
	<p>A few numbers (notably the first one) don’t make sense, which we attribute to <a href="https://github.com/openzfs/zfs/issues/7639"><i class="fab fa-github"></i> openzfs/zfs#7639</a>.</p>
	<p>If we sort the table by difference, it would be:</p>
	<table>
		<thead>
			<tr>
				<th style="text-align: left">NAME</th>
				<th style="text-align: right">LUSED</th>
				<th style="text-align: right">USED</th>
				<th style="text-align: right">DIFF</th>
			</tr>
		</thead>
		<tbody>
			<tr>
				<td style="text-align: left">pool0/repo</td>
				<td style="text-align: right">58.3T</td>
				<td style="text-align: right">56.1T</td>
				<td style="text-align: right">2.2T</td>
			</tr>
			<tr>
				<td style="text-align: left">pool0/repo/debian-cdimage</td>
				<td style="text-align: right">1.6T</td>
				<td style="text-align: right">1.0T</td>
				<td style="text-align: right">549.6G</td>
			</tr>
			<tr>
				<td style="text-align: left">pool0/repo/opensuse</td>
				<td style="text-align: right">2.5T</td>
				<td style="text-align: right">2.3T</td>
				<td style="text-align: right">279.7G</td>
			</tr>
			<tr>
				<td style="text-align: left">pool0/repo/turnkeylinux</td>
				<td style="text-align: right">1.2T</td>
				<td style="text-align: right">1.0T</td>
				<td style="text-align: right">155.2G</td>
			</tr>
			<tr>
				<td style="text-align: left">pool0/repo/loongnix</td>
				<td style="text-align: right">438.2G</td>
				<td style="text-align: right">331.9G</td>
				<td style="text-align: right">106.3G</td>
			</tr>
			<tr>
				<td style="text-align: left">pool0/repo/alpine</td>
				<td style="text-align: right">3.0T</td>
				<td style="text-align: right">2.9T</td>
				<td style="text-align: right">103.9G</td>
			</tr>
			<tr>
				<td style="text-align: left">pool0/repo/openwrt</td>
				<td style="text-align: right">1.8T</td>
				<td style="text-align: right">1.7T</td>
				<td style="text-align: right">70.0G</td>
			</tr>
		</tbody>
	</table>
	<p><code class="language-plaintext highlighter-rouge">debian-cdimage</code> alone contributes to a quarter of the saved space.</p>
	<h3 id="grafana-for-zfs-io">Grafana for ZFS I/O</h3>
	<p>We also fixed a Grafana panel for ZFS I/O so it’s displaying the correct numbers.
		Because ZFS I/O statistics are exported through <code class="language-plaintext highlighter-rouge">/proc/spl/kstat/zfs/$POOL/objset-$OBJSETID_HEX</code> and is cumulative per “object set” (i.e. dataset), we need to calculate the derivative of the numbers and <em>then</em> sum by pool.
		This means the use of subqueries is inevitable.</p>
	<div class="language-sql highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code><span class="k">SELECT</span>
  <span class="n">non_negative_derivative</span><span class="p">(</span><span class="k">sum</span><span class="p">(</span><span class="nv">"reads"</span><span class="p">),</span> <span class="mi">1</span><span class="n">s</span><span class="p">)</span> <span class="k">AS</span> <span class="nv">"read"</span><span class="p">,</span>
  <span class="n">non_negative_derivative</span><span class="p">(</span><span class="k">sum</span><span class="p">(</span><span class="nv">"writes"</span><span class="p">),</span> <span class="mi">1</span><span class="n">s</span><span class="p">)</span> <span class="k">AS</span> <span class="nv">"write"</span>
<span class="k">FROM</span> <span class="p">(</span>
  <span class="k">SELECT</span>
    <span class="k">first</span><span class="p">(</span><span class="nv">"reads"</span><span class="p">)</span> <span class="k">AS</span> <span class="nv">"reads"</span><span class="p">,</span>
    <span class="k">first</span><span class="p">(</span><span class="nv">"writes"</span><span class="p">)</span> <span class="k">AS</span> <span class="nv">"writes"</span>
  <span class="k">FROM</span> <span class="nv">"zfs_pool"</span>
  <span class="k">WHERE</span> <span class="p">(</span><span class="nv">"host"</span> <span class="o">=</span> <span class="s1">'taokystrong'</span> <span class="k">AND</span> <span class="nv">"pool"</span> <span class="o">=</span> <span class="s1">'pool0'</span><span class="p">)</span> <span class="k">AND</span> <span class="err">$</span><span class="n">timeFilter</span>
  <span class="k">GROUP</span> <span class="k">BY</span> <span class="nb">time</span><span class="p">(</span><span class="err">$</span><span class="n">interval</span><span class="p">),</span> <span class="nv">"host"</span><span class="p">::</span><span class="n">tag</span><span class="p">,</span> <span class="nv">"pool"</span><span class="p">::</span><span class="n">tag</span><span class="p">,</span> <span class="nv">"dataset"</span><span class="p">::</span><span class="n">tag</span> <span class="n">fill</span><span class="p">(</span><span class="k">null</span><span class="p">)</span>
<span class="p">)</span>
<span class="k">WHERE</span> <span class="err">$</span><span class="n">timeFilter</span>
<span class="k">GROUP</span> <span class="k">BY</span> <span class="nb">time</span><span class="p">(</span><span class="err">$</span><span class="n">interval</span><span class="p">),</span> <span class="nv">"pool"</span><span class="p">::</span><span class="n">tag</span> <span class="n">fill</span><span class="p">(</span><span class="n">linear</span><span class="p">)</span>
</code></pre>
		</div>
	</div>
	<p>This query is a bit slow (due to the subquery) and unfortunately there’s not much we can do about it.</p>
	<p>To display I/O bandwidth, simply replace <code class="language-plaintext highlighter-rouge">reads</code> and <code class="language-plaintext highlighter-rouge">writes</code> with <code class="language-plaintext highlighter-rouge">nread</code> and <code class="language-plaintext highlighter-rouge">nwritten</code> in the inner query.</p>
	<figure class=""><a href="https://image.ibugone.com/grafana/mirrors2-4-zfs-io-count.png" class="image-popup" title="ZFS I/O count and bandwidth
"><img src="https://image.ibugone.com/grafana/mirrors2-4-zfs-io-count.png" alt="ZFS I/O count and bandwidth" loading="lazy" /></a>
		<figcaption>
			ZFS I/O count and bandwidth
		</figcaption>
	</figure>
	<p>We are astonished to see an HDD array can sustain 15k IOPS and peaking at 50k IOPS.
		This becomes all explained when we discovered that these numbers took ARC hits into account, and a minimal proportion were actually hitting the disks.</p>
	<h3 id="apparmor">AppArmor</h3>
	<p>It didn’t take long before we noticed all our sync tasks were failing.
		We found <code class="language-plaintext highlighter-rouge">rsync</code> failing with <code class="language-plaintext highlighter-rouge">EPERM</code> for <code class="language-plaintext highlighter-rouge">socketpair(2)</code> calls, which never manifested before.
		Interestingly, these were denied by AppArmor.
		We traced down the cause to be Ubuntu’s addition to the kernel, <code class="language-plaintext highlighter-rouge">security/apparmor/af_unix.c</code>.
		As Proxmox VE forks its kernel from Ubuntu, this change also made its way into our server.</p>
	<p>We also found PVE packaging their own copy of AppArmor <code class="language-plaintext highlighter-rouge">features</code>, so we decided to adopt the same approach:</p>
	<div class="language-shell highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code>dpkg-divert <span class="nt">--package</span> lxc-pve <span class="nt">--rename</span> <span class="nt">--divert</span> /usr/share/apparmor-features/features.stock <span class="nt">--add</span> /usr/share/apparmor-features/features
wget <span class="nt">-O</span> /usr/share/apparmor-features/features https://github.com/proxmox/lxc/raw/master/debian/features
</code></pre>
		</div>
	</div>
	<h3 id="file-deduplication">File deduplication</h3>
	<p>For a small set of repositories, possibly due to limitations of syncing methods, we noticed a lot of identically-looking directories.</p>
	<figure class=""><a href="https://image.ibugone.com/server/ls-zerotier-redhat-el.png" class="image-popup" title="Some folders from ZeroTier repository
"><img src="https://image.ibugone.com/server/ls-zerotier-redhat-el.png" alt="Some folders from ZeroTier repository" loading="lazy" /></a>
		<figcaption>
			Some folders from ZeroTier repository
		</figcaption>
	</figure>
	<p>ZFS deduplication immediately came to our mind, so we made a preliminary test on ZT:</p>
	<div class="language-shell highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code>zfs create <span class="nt">-o</span> <span class="nv">dedup</span><span class="o">=</span>on pool0/repo/zerotier
<span class="c"># dump content into it</span>
</code></pre>
		</div>
	</div>
	<div class="language-console highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code><span class="gp">#</span><span class="w"> </span>zdb <span class="nt">-DDD</span> pool0
<span class="go">dedup = 4.93, compress = 1.23, copies = 1.00, dedup * compress / copies = 6.04
</span></code></pre>
		</div>
	</div>
	<p>The results look promising, but we are still hesitant to enable deduplication due to the potential performance impact even on these selected datasets.</p>
	<p>Guess what we ended up with?</p>
	<div class="language-shell highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code><span class="c"># post-sync.sh</span>
<span class="c"># Do file-level deduplication for select repos</span>
<span class="k">case</span> <span class="s2">"</span><span class="nv">$NAME</span><span class="s2">"</span> <span class="k">in
  </span>docker-ce|influxdata|nginx|openresty|proxmox|salt|tailscale|zerotier<span class="p">)</span>
    jdupes <span class="nt">-L</span> <span class="nt">-Q</span> <span class="nt">-r</span> <span class="nt">-q</span> <span class="s2">"</span><span class="nv">$DIR</span><span class="s2">"</span> <span class="p">;;</span>
<span class="k">esac</span>
</code></pre>
		</div>
	</div>
	<p>As attractive as it looks, this userspace file deduplication tool is as good as ZFS can do, but without the performance loss.</p>
	<table>
		<thead>
			<tr>
				<th>Name</th>
				<th>Orig</th>
				<th>Dedup</th>
				<th>Diff</th>
				<th>Ratio</th>
			</tr>
		</thead>
		<tbody>
			<tr>
				<td>proxmox</td>
				<td>395.4G</td>
				<td>162.6G</td>
				<td>232.9G</td>
				<td>2.43x</td>
			</tr>
			<tr>
				<td>docker-ce</td>
				<td>539.6G</td>
				<td>318.2G</td>
				<td>221.4G</td>
				<td>1.70x</td>
			</tr>
			<tr>
				<td>influxdata</td>
				<td>248.4G</td>
				<td>54.8G</td>
				<td>193.6G</td>
				<td>4.54x</td>
			</tr>
			<tr>
				<td>salt</td>
				<td>139.0G</td>
				<td>87.2G</td>
				<td>51.9G</td>
				<td>1.59x</td>
			</tr>
			<tr>
				<td>nginx</td>
				<td>94.9G</td>
				<td>59.7G</td>
				<td>35.2G</td>
				<td>1.59x</td>
			</tr>
			<tr>
				<td>zerotier</td>
				<td>29.8G</td>
				<td>6.1G</td>
				<td>23.7G</td>
				<td>4.88x</td>
			</tr>
			<tr>
				<td>mysql-repo</td>
				<td>647.8G</td>
				<td>632.5G</td>
				<td>15.2G</td>
				<td>1.02x</td>
			</tr>
			<tr>
				<td>openresty</td>
				<td>65.1G</td>
				<td>53.4G</td>
				<td>11.7G</td>
				<td>1.22x</td>
			</tr>
			<tr>
				<td>tailscale</td>
				<td>17.9G</td>
				<td>9.0G</td>
				<td>9.0G</td>
				<td>2.00x</td>
			</tr>
		</tbody>
	</table>
	<p>We decided to exclude <code class="language-plaintext highlighter-rouge">mysql-repo</code> as the deduplication ratio is too low to justify the I/O load after each sync.</p>
	<h2 id="conclusion">Conclusion</h2>
	<p>ZFS solved a number of problems we had with our mirror servers, and with the current setup, we are delighted to announce that ZFS is <em>the</em> best solution for mirrors.</p>
	<p>With ZFS:</p>
	<ul>
		<li>We no longer need to worry about partitioning, as ZFS can grow and shrink as needed.</li>
		<li>Our HDD array is now running faster than SSDs. Amazing!
			<ul>
				<li>Be the first one to no longer <strong>envy</strong> TUNA’s SSD server!</li>
			</ul>
		</li>
		<li>Extra capacity at no cost, thanks to ZFS compression.
			<ul>
				<li>Even more so with deduplication.</li>
			</ul>
		</li>
	</ul>
	<h3 id="considerations">Considerations</h3>
	<p>While our ZFS looks very promising, we’re aware that ZFS is not known for its long-term performance stability due to fragmentation.
		We’ll continue to monitor our servers and see if this performance is sustainable.</p>
	]]></content><author><name>iBug</name></author><category term="linux" /><category term="server" /><category term="zfs" /><summary type="html"><![CDATA[A.K.A. Practical ZFS application on USTC Mirrors. A writeup of the talk I gave at Nanjing University this August.]]></summary></entry><entry><title type="html">Make Python 3.12 install user packages without complaints</title><link href="https://ibug.io/blog/2024/09/python3.12-user-packages/" rel="alternate" type="text/html" title="Make Python 3.12 install user packages without complaints" /><published>2024-09-02T00:00:00+00:00</published><updated>2024-09-03T11:59:51+00:00</updated><id>https://ibug.io/blog/2024/09/python3.12-user-packages</id><content type="html" xml:base="https://ibug.io/blog/2024/09/python3.12-user-packages/"><![CDATA[<p>I have a habit of <code class="language-plaintext highlighter-rouge">pip3 install --user</code> and then expecting these packages under <code class="language-plaintext highlighter-rouge">~/.local/lib/</code> to be available for my Python scripts whenever I need them. However, with PEP 668 landing in Python 3.12, I now have to add <code class="language-plaintext highlighter-rouge">--break-system-packages</code> even for <em>user</em> packages. This is super annoying considered that I have multiple projects sharing the same set of common packages (e.g. <a href="https://squidfunk.github.io/mkdocs-material/"><code class="language-plaintext highlighter-rouge">mkdocs-material</code></a>, a nice MkDocs theme). So time to tell <code class="language-plaintext highlighter-rouge">pip</code> to jerk off on that complaint.</p>
	<p>Obviously, aliasing <code class="language-plaintext highlighter-rouge">pip3</code> (as per my personal habit, I always prefer <code class="language-plaintext highlighter-rouge">python3</code> and <code class="language-plaintext highlighter-rouge">pip3</code> over <code class="language-plaintext highlighter-rouge">python</code> and <code class="language-plaintext highlighter-rouge">pip</code>) to <code class="language-plaintext highlighter-rouge">pip3 --break-system-packages</code> could work, with all the limitations that any other shell alias bear.</p>
	<p>The key here is, by examining how virtual environments work, we can trick Python into thinking that <code class="language-plaintext highlighter-rouge">~/.local</code> is one of them. This is already documented in the <a href="https://docs.python.org/3/library/site.html"><code class="language-plaintext highlighter-rouge">site</code> package</a>:</p>
	<blockquote>
		<p>If a file named <code class="language-plaintext highlighter-rouge">pyvenv.cfg</code> exists one directory above <code class="language-plaintext highlighter-rouge">sys.executable</code> …</p>
	</blockquote>
	<p>So here’s the solution, assuming <code class="language-plaintext highlighter-rouge">~/.local/bin</code> is already in your <code class="language-plaintext highlighter-rouge">$PATH</code>:</p>
	<ol>
		<li>Symlink <code class="language-plaintext highlighter-rouge">/usr/bin/python3</code> to <code class="language-plaintext highlighter-rouge">~/.local/bin/python3</code></li>
		<li>Copy <code class="language-plaintext highlighter-rouge">/usr/bin/pip3</code> to <code class="language-plaintext highlighter-rouge">~/.local/bin/pip3</code>, and change the shebang line to <code class="language-plaintext highlighter-rouge">#!/home/example/.local/bin/python3</code> (you’ll have to use the absolute path here, though).</li>
		<li>
			<p>Create <code class="language-plaintext highlighter-rouge">~/.local/pyvenv.cfg</code> with just one line of content:</p>
			<div class="language-ini highlighter-rouge">
				<div class="highlight">
					<pre class="highlight"><code><span class="w"> </span><span class="py">include-system-site-packages</span><span class="w"> </span><span class="p">=</span><span class="w"> </span><span class="s">true</span>
</code></pre>
				</div>
    </div>
			<p>You can, of course, add other settings for <code class="language-plaintext highlighter-rouge">venv</code>, which is completely optional and up to you.</p>
		</li>
	</ol>
	<p>Now whenever you install something with <code class="language-plaintext highlighter-rouge">pip3</code>, it’ll happily install it under <code class="language-plaintext highlighter-rouge">~/.local/lib/python3.12/site-packages</code> even without the need for <code class="language-plaintext highlighter-rouge">--user</code>.</p>
	<p>If you prefer <code class="language-plaintext highlighter-rouge">python</code> or <code class="language-plaintext highlighter-rouge">pip</code> commands, you can just change the file names in the above steps accordingly.</p>
	<p>Noteworthy points are:</p>
	<ol>
		<li>This method certainly can work for the system Python installation (at <code class="language-plaintext highlighter-rouge">/usr</code>), which I wouldn’t recommend for obvious reasons. If you insist, you should at least do this under <code class="language-plaintext highlighter-rouge">/usr/local</code> instead.</li>
		<li>
			<s>This method (when applied to `~/.local`) is ineffective against scripts already shebanged with `#!/usr/bin/python3`. Consider developing the habit of running `python3 script.py` instead of `./script.py` like I do.</s>
		</li>
	</ol>
	<h2 id="corrigenda">Corrigenda</h2>
	<ul>
		<li>Scripts shebanged with <code class="language-plaintext highlighter-rouge">#!/usr/bin/python3</code> <em>will</em> pick up packages under <code class="language-plaintext highlighter-rouge">~/.local/lib/</code> as this is the default user site directory, which comes in <code class="language-plaintext highlighter-rouge">sys.path</code> even before the system site directory.</li>
	</ul>
	]]></content><author><name>iBug</name></author><category term="linux" /><category term="python" /><summary type="html"><![CDATA[I have a habit of pip3 install --user and then expecting these packages under ~/.local/lib/ to be available for my Python scripts whenever I need them. However, with PEP 668 landing in Python 3.12, I now have to add --break-system-packages even for user packages. This is super annoying considered that I have multiple projects sharing the same set of common packages (e.g. mkdocs-material, a nice MkDocs theme). So time to tell pip to jerk off on that complaint.]]></summary></entry><entry><title type="html">镜像站 ZFS 调优实践</title><link href="https://ibug.io/blog/2024/08/nju-talk/" rel="alternate" type="text/html" title="镜像站 ZFS 调优实践" /><published>2024-08-17T00:00:00+00:00</published><updated>2025-07-28T14:51:43+00:00</updated><id>https://ibug.io/blog/2024/08/nju-talk</id><content type="html" xml:base="https://ibug.io/blog/2024/08/nju-talk/"><![CDATA[<section id="title">
		<h1 class="title">2000 元的机械硬盘 &gt; 3000 元的固态硬盘？</h1>
		<h2 style="font-weight: normal;">A.K.A. 镜像站 ZFS 调优实践</h2>
		<hr />
		<p class="date">iBug @ USTC</p>
		<p class="date">2024 年 8 月 17 日<br />
			南京大学 开源软件论坛</p>
	</section>
	<section>
		<section id="background">
			<h2>USTC Mirrors</h2>
			<ul>
				<li>日均服务量：（2024-05 ~ 2024-06）
					<ul>
						<li>出流量 ~36&nbsp;TiB</li>
						<li>HTTP 请求数 17M，响应流量 19&nbsp;TiB</li>
						<li>Rsync 请求数 147.8K（21.8K），输出流量 10.3&nbsp;TiB</li>
					</ul>
				</li>
				<li>极限情况的仓库容量：
					<ul>
						<li>HTTP 服务器（XFS）：63.3&nbsp;TiB / 66.0&nbsp;TiB (96%, 2023-12-18)</li>
						<li>Rsync 服务器（ZFS）：42.4&nbsp;TiB / 43.2&nbsp;TiB (98%, 2023-11-21)</li>
					</ul>
				</li>
			</ul>
		</section>
		<section id="background-2">
			<h2>背景</h2>
			<ul>
				<li>HTTP 服务器：
					<ul>
						<li>2020 年下半年搭建</li>
						<li>10&nbsp;TB <i class="fas fa-compact-disc fa-spin"></i> &times; 12</li>
						<li>2&nbsp;TB <i class="fas fa-floppy-disk"></i> &times; 1</li>
						<li>XFS on LVM on HW RAID</li>
						<li>考虑到 XFS 不能缩，VG 留了 free PE</li>
					</ul>
				</li>
				<li>Rsync 服务器：
					<ul>
						<li>2016 年下半年搭建</li>
						<li>6&nbsp;TB <i class="fas fa-compact-disc fa-spin"></i> &times; 12</li>
						<li>240&nbsp;GB <i class="fas fa-floppy-disk"></i> &times; 2 + 480&nbsp;GB <i class="fas fa-floppy-disk"></i> &times; 1 (Optane 900p)</li>
						<li>RAID-Z3（8 data + 3 parity + 1 hot spare）</li>
						<li>全默认参数（除了 <code>zfs_arc_max</code>）</li>
					</ul>
				</li>
			</ul>
			<p>硬盘 I/O 日常 &gt; 90%，校内下载 iso 不足 50&nbsp;MB/s</p>
		</section>
		<section id="background-2-image">
			<div class="img-container">
				<img src="https://image.ibugone.com/grafana/mirrors-io-utilization-may-2024.png" />
				<p>USTC 镜像站两台服务器在 2024 年 5 月期间的磁盘负载</p>
			</div>
		</section>
	</section>
	<section>
		<section id="zfs">
			<h2>ZFS</h2>
			<ul>
				<li>单机存储的终极解决方案</li>
				<li>集 RAID、LVM、FS 于一体</li>
				<li>所有数据都有 checksum</li>
				<li><s>Fire and forget</s></li>
				<li>好多参数啊</li>
			</ul>
		</section>
		<section id="zfs-learning">
			<h3>前期学习与实验</h3>
			<ul>
				<li><s>从（另一个）老师那嫖了点盘</s>装上了 ZFS，用于 研&ensp;究&ensp;学&ensp;习</li>
				<li>I/O 负载来源？<s>上 PT 站</s></li>
				<li>练习时长两年半的成果：<i class="fas fa-arrow-up"></i> 1.20&nbsp;PiB, <i class="fas fa-arrow-down"></i> 1.83&nbsp;TiB</li>
			</ul>
			<hr />
			<p>重要学习资料：</p>
			<ul>
				<li><a href="https://utcc.utoronto.ca/~cks/space/blog/">Chris Siebenmann's blog</a></li>
				<li><a href="https://openzfs.github.io/openzfs-docs/">OpenZFS Documentation</a></li>
				<li>iBug's blog: <a href="/p/62">Understanding ZFS block sizes</a> (<a href="/p/62">ibug.io/p/62</a>)
					<ul>
						<li>以及这篇 blog 底下的众多参考文献</li>
					</ul>
				</li>
			</ul>
		</section>
		<section id="zfs-image">
			<div class="img-container">
				<img src="https://image.ibugone.com/grafana/qb/2024-06-05.png" />
				<p>好像给什么奇怪的东西加入了 Grafana</p>
			</div>
		</section>
	</section>
	<section>
		<section id="mirrors">
			<h2>镜像站</h2>
			<ul>
				<li>提供文件下载服务</li>
				<li><s>也提供「家庭宽带上下行流量比例平衡」服务</s></li>
				<li>读多写少，并且几乎所有操作都是整个文件顺序读写</li>
				<li>少量的数据损坏没啥不良后果</li>
			</ul>
		</section>
		<section id="mirrors-file-distrib">
			<div class="img-container">
				<img src="https://image.ibugone.com/server/mirrors-file-size-distribution-2024-08.png" />
				<p>2024 年 8 月 USTC 镜像仓库内的文件大小分布
					<br>
					其中中位数为 9.83&nbsp;KiB，平均大小为 1.60&nbsp;MiB</p>
			</div>
		</section>
		<section id="mirrors2">
			<h3>重建 Rsync 服务器</h3>
			<ul>
				<li>RAID-Z3 的 overhead 较高，而且拆成两组 RAID-Z2 = 两倍的 IOPS</li>
				<li>镜像站调优计划：
					<ul>
						<li><code>recordsize=1M</code>：反正都是全文件顺序读</li>
						<li><code>compression=zstd</code>：至少可以压掉 &gt; 1M 文件的 padding
							<ul>
								<li>OpenZFS 2.2 将 early abort 机制推广到了 Zstd 3+，不必担心性能问题</li>
							</ul>
						</li>
						<li><code>xattr=off</code>：谁家镜像需要 xattr？</li>
						<li><code>atime=off</code>, <code>setuid=off</code>, <code>exec=off</code>, <code>devices=off</code>：开着干啥？</li>
						<li><code>secondarycache=metadata</code>：Rsync 就别来消耗固态寿命了</li>
					</ul>
				</li>
				<li>Danger Zone：
					<ul>
						<li><code>sync=disabled</code>：囤到 <code>zfs_txg_timeout</code> 再写盘</li>
						<li><code>redundant_metadata=some</code>：偶尔坏个文件也没事</li>
					</ul>
				</li>
				<li>Full version: <a href="https://docs.ustclug.org/services/mirrors/zfs/#setup">LUG @ USTC Documentation</a></li>
			</ul>
		</section>
		<section id="zfs-parameters">
			<h3>ZFS 参数</h3>
			<ul>
				<li>290+ 参数不能个个都学习嘛（感谢 Aron Xu @ BFSU）</li>
				<li>ARC 容量：
					<pre><code class="language-sh" data-trim>
# Set ARC size to 160-200&nbsp;GiB, keep 16&nbsp;GiB free for OS
options zfs zfs_arc_max=214748364800
options zfs zfs_arc_min=171798691840
options zfs zfs_arc_sys_free=17179869184
        </code></pre>
				</li>
				<li>ARC 内容：
					<pre><code class="language-sh" data-trim>
# Favor metadata to data by 20x (OpenZFS 2.2+)
options zfs zfs_arc_meta_balance=2000

# Allow up to 80% of ARC to be used for dnodes
options zfs zfs_arc_dnode_limit_percent=80
        </code></pre>
				</li>
				<li>I/O 队列深度：
					<pre><code class="language-sh" data-trim>
# See man page section "ZFS I/O Scheduler"
options zfs zfs_vdev_async_read_max_active=8
options zfs zfs_vdev_async_read_min_active=2
options zfs zfs_vdev_scrub_max_active=5
options zfs zfs_vdev_max_active=20000
        </code></pre>
				</li>
				<li>Full version: <a href="https://docs.ustclug.org/services/mirrors/zfs/#zfs-kernel-module">LUG @ USTC Documentation</a></li>
			</ul>
		</section>
		<section id="mirrors2-rebuild-results">
			<h3>重建成果</h3>
			<ul>
				<li>略感惊喜的压缩率：39.5T / 37.1T = 1.07x
					<ul>
						<li>正确用法：<code>zfs list -po name,logicalused,used</code></li>
						<li>实际压缩率：1 + 6.57%（-2.67&nbsp;TB / -2.43&nbsp;TiB）</li>
						<li><s>等于删了 <a href="https://image.ibugone.com/teaser/lenovo-legion-wechat-data.jpg">9 份微信数据</a></s></li>
					</ul>
				</li>
				<li>合理的磁盘 I/O</li>
			</ul>
		</section>
		<section id="mirrors2-io-image">
			<div class="img-container">
				<img src="https://image.ibugone.com/grafana/mirrors2-io-utilization-and-free-space-june-july-2024.png" />
				<p>重建前后 Rsync 服务器的磁盘负载与空闲空间比较</p>
			</div>
		</section>
	</section>
	<section>
		<section id="mirrors4">
			<h2>HTTP 服务器</h2>
			<ul>
				<li>硬件 RAID + LVM + XFS + Kernel page cache（开箱即用？）</li>
				<li>SSD？LVMCache！
					<ul>
						<li>1M extents? Block size? Algorithm?</li>
						<li><i class="fas fa-skull"></i> GRUB2</li>
						<li><i class="fas fa-skull"></i> "oldssd"</li>
					</ul>
				</li>
				<li>XFS 不能缩，所以 VG 和 FS 两层都要留空间备用</li>
			</ul>
		</section>
		<section id="mirrors4-dmcache-image">
			<div class="img-container">
				<img src="https://image.ibugone.com/grafana/mirrors4-dmcache-may-june-2024.png" />
				<p>重建前 HTTP 服务器采用的 LVMcache 方案的命中率</p>
			</div>
		</section>
		<section id="mirrors4-rebuild">
			<h2>如法炮制</h2>
			<ul>
				<li>体验一下更加先进的 kernel：<code>6.8.8-3-pve</code>（无需 DKMS 哦）</li>
				<li>重建为两组 RAID-Z2，开压缩
					<ul>
						<li>面向 HTTP 用户的服务器，所以 <code>secondarycache=all</code>（放着不动）</li>
						<li>更好的 CPU，所以 <code>compression=zstd-8</code></li>
					</ul>
				</li>
				<li>更快的 <code>zfs send -Lcp</code>：36 小时倒完 50+ TiB 仓库</li>
				<li>压缩率：1 + 3.93%（-2.42&nbsp;TB / -2.20&nbsp;TiB）</li>
			</ul>
		</section>
		<section id="mirrors2-4-io-image">
			<div class="img-container">
				<img src="https://image.ibugone.com/grafana/mirrors2-4-io-utilization-june-july-2024.png" />
				<p>重建前后两台服务器的磁盘负载比较
					<br>
					左边为重建前，中间为仅 Rsync 服务器重建后，右边为两台服务器均重建后的负载</p>
			</div>
		</section>
		<section id="mirrors2-4-zfs-arc-image">
			<div class="img-container">
				<img src="https://image.ibugone.com/grafana/mirrors2-4-zfs-arc-hit-rate.png" />
				<p>两台服务器的 ZFS ARC 命中率</p>
			</div>
		</section>
		<section id="mirrors2-4-recent-io-image">
			<div class="img-container">
				<img src="https://image.ibugone.com/grafana/mirrors2-4-disk-io-after-rebuild.png" />
				<p>两台服务器重建后稳定的磁盘利用率</p>
			</div>
		</section>
	</section>
	<section>
		<section id="misc">
			<h2>杂项</h2>
		</section>
		<section id="zfs-compressratio">
			<h3>ZFS 压缩率</h3>
			<table>
				<thead>
					<tr>
						<th>NAME</th>
						<th>LUSED</th>
						<th>USED</th>
						<th>RATIO</th>
					</tr>
				</thead>
				<tbody>
					<tr>
						<td>pool0/repo/crates.io-index</td>
						<td>2.19G</td>
						<td>1.65G</td>
						<td>3.01x</td>
					</tr>
					<tr>
						<td>pool0/repo/elpa</td>
						<td>3.35G</td>
						<td>2.32G</td>
						<td>1.67x</td>
					</tr>
					<tr>
						<td>pool0/repo/rfc</td>
						<td>4.37G</td>
						<td>3.01G</td>
						<td>1.56x</td>
					</tr>
					<tr>
						<td>pool0/repo/debian-cdimage</td>
						<td>1.58T</td>
						<td>1.04T</td>
						<td>1.54x</td>
					</tr>
					<tr>
						<td>pool0/repo/tldp</td>
						<td>4.89G</td>
						<td>3.78G</td>
						<td>1.48x</td>
					</tr>
					<td>pool0/repo/loongnix</td>
					<td>438G</td>
					<td>332G</td>
					<td>1.34x</td>
				</tr>
				<tr>
					<td>pool0/repo/rosdistro</td>
					<td>32.2M</td>
					<td>26.6M</td>
					<td>1.31x</td>
				</tr>
				<tr>
				</tbody>
			</table>
			<p>我数学不好：<a href="https://github.com/openzfs/zfs/issues/7639"><i class="fab fa-github"></i> openzfs/zfs#7639</a></p>
		</section>
		<section id="zfs-compressratio-diff">
			<h3>ZFS 压缩量</h3>
			<table>
				<thead>
					<tr>
						<th>NAME</th>
						<th>LUSED</th>
						<th>USED</th>
						<th>DIFF</th>
					</tr>
				</thead>
				<tbody>
					<tr>
						<td>pool0/repo</td>
						<td>58.3T</td>
						<td>56.1T</td>
						<td>2.2T</td>
					</tr>
					<tr>
						<td>pool0/repo/debian-cdimage</td>
						<td>1.6T</td>
						<td>1.0T</td>
						<td>549.6G</td>
					</tr>
					<tr>
						<td>pool0/repo/opensuse</td>
						<td>2.5T</td>
						<td>2.3T</td>
						<td>279.7G</td>
					</tr>
					<tr>
						<td>pool0/repo/turnkeylinux</td>
						<td>1.2T</td>
						<td>1.0T</td>
						<td>155.2G</td>
					</tr>
					<tr>
						<td>pool0/repo/loongnix</td>
						<td>438.2G</td>
						<td>331.9G</td>
						<td>106.3G</td>
					</tr>
					<tr>
						<td>pool0/repo/alpine</td>
						<td>3.0T</td>
						<td>2.9T</td>
						<td>103.9G</td>
					</tr>
					<tr>
						<td>pool0/repo/openwrt</td>
						<td>1.8T</td>
						<td>1.7T</td>
						<td>70.0G</td>
					</tr>
				</tbody>
			</table>
		</section>
		<section id="grafana-zfs-io">
			<h3>Grafana I/O 统计</h3>
			<pre><code data-trim>
SELECT
  non_negative_derivative(sum("reads"), 1s) AS "read",
  non_negative_derivative(sum("writes"), 1s) AS "write"
FROM (
  SELECT
    first("reads") AS "reads",
    first("writes") AS "writes"
  FROM "zfs_pool"
  WHERE ("host" = 'taokystrong' AND "pool" = 'pool0') AND $timeFilter
  GROUP BY time($interval), "host"::tag, "pool"::tag, "dataset"::tag fill(null)
)
WHERE $timeFilter
GROUP BY time($interval), "pool"::tag fill(linear)
</code></pre>
			<p>跑得有点慢（毕竟要先 <code>GROUP BY</code> 每个 ZFS dataset 再一起 <code>sum</code>）</p>
			<p>I/O 带宽：把里层的 <code>reads</code> 和 <code>writes</code> 换成 <code>nread</code> 和 <code>nwritten</code> 即可</p>
		</section>
		<section id="grafana-zfs-io-image">
			<div class="img-container">
				<img src="https://image.ibugone.com/grafana/mirrors2-4-zfs-io-count.png" />
			</div>
			<p></p>
			<ul>
				<li>如何用机械盘跑出平均 15K、最高 50K 的 IOPS？</li>
				<li><s>把 ARC hit 算进去</s></li>
			</ul>
		</section>
	</section>
	<section>
		<section id="hearse">
			<h2>灵车时间</h2>
		</section>
		<section id="pve-kernel">
			<h2>Proxmox Kernel</h2>
			<ul>
				<li>≈ Ubuntu Kernel</li>
				<li><i class="fas fa-skull"></i> Rsync 容器</li>
				<li><code>security/apparmor/af_unix.c</code>???</li>
				<li><a href="https://docs.ustclug.org/faq/apparmor/">LUG Documentation: AppArmor</a></li>
			</ul>
			<pre><code class="language-sh" data-trim>
dpkg-divert --package lxc-pve --rename --divert /usr/share/apparmor-features/features.stock --add /usr/share/apparmor-features/features
wget -O /usr/share/apparmor-features/features https://github.com/proxmox/lxc/raw/master/debian/features
    </code></pre>
		</section>
		<section id="zerotier-data">
			<div class="img-container">
				<img src="https://image.ibugone.com/server/ls-zerotier-redhat-el.png" />
				<p>ZeroTier 仓库中一眼重复的内容</p>
			</div>
		</section>
		<section id="dedup">
			<h3>Dedup!</h3>
			<pre><code class="language-sh">zfs create -o dedup=on pool0/repo/zerotier</code></pre>
			<pre><code class="language-sh" data-trim>
# zdb -DDD pool0
dedup = 4.93, compress = 1.23, copies = 1.00, dedup * compress / copies = 6.04
    </code></pre>
			<p>效果倒是不错，但是不想像 ZFS dedup 这么灵怎么办？</p>
		</section>
		<section id="jdupes">
			<h3>jdupes</h3>
			<pre><code class="language-sh" data-trim>
# post-sync.sh
# Do file-level deduplication for select repos
case "$NAME" in
  docker-ce|influxdata|nginx|openresty|proxmox|salt|tailscale|zerotier)
    jdupes -L -Q -r -q "$DIR" ;;
esac
    </code></pre>
		</section>
		<section id="jdupes-table">
			<h3>jdupes 效果</h3>
			<table>
				<thead>
					<tr>
						<th>Name</th>
						<th>Orig</th>
						<th>Dedup</th>
						<th>Diff</th>
						<th>Ratio</th>
					</tr>
				</thead>
				<tbody>
					<tr>
						<td>proxmox</td>
						<td>395.4G</td>
						<td>162.6G</td>
						<td>232.9G</td>
						<td>2.43x</td>
					</tr>
					<tr>
						<td>docker-ce</td>
						<td>539.6G</td>
						<td>318.2G</td>
						<td>221.4G</td>
						<td>1.70x</td>
					</tr>
					<tr>
						<td>influxdata</td>
						<td>248.4G</td>
						<td>54.8G</td>
						<td>193.6G</td>
						<td>4.54x</td>
					</tr>
					<tr>
						<td>salt</td>
						<td>139.0G</td>
						<td>87.2G</td>
						<td>51.9G</td>
						<td>1.59x</td>
					</tr>
					<tr>
						<td>nginx</td>
						<td>94.9G</td>
						<td>59.7G</td>
						<td>35.2G</td>
						<td>1.59x</td>
					</tr>
					<tr>
						<td>zerotier</td>
						<td>29.8G</td>
						<td>6.1G</td>
						<td>23.7G</td>
						<td>4.88x</td>
					</tr>
					<tr>
						<td>mysql-repo</td>
						<td>647.8G</td>
						<td>632.5G</td>
						<td>15.2G</td>
						<td>1.02x</td>
					</tr>
					<tr>
						<td>openresty</td>
						<td>65.1G</td>
						<td>53.4G</td>
						<td>11.7G</td>
						<td>1.22x</td>
					</tr>
					<tr>
						<td>tailscale</td>
						<td>17.9G</td>
						<td>9.0G</td>
						<td>9.0G</td>
						<td>2.00x</td>
					</tr>
				</tbody>
			</table>
		</section>
	</section>
	<section id="conclusion">
		<h2>只要 ZFS 用得好</h2>
		<ul>
			<li>妈妈再也不用担心我的硬盘分区</li>
			<li>机械硬盘 <s>比西方的固态硬盘跑得还快</s></li>
			<li>成为第一个不再<b>羡慕</b> TUNA 全闪的镜像站</li>
			<li>免费的额外容量
				<ul>
					<li>Dedup 会员红包</li>
				</ul>
			</li>
			<li>碎片率？</li>
		</ul>
	</section>
	<section id="outro">
		<h1>谢谢！</h1>
		<small>
			<p>本页面的链接：<a href="/p/72"><i class="fas fa-fw fa-link"></i> ibug.io/p/72</a></p>
			<p>
				<a href="https://lug.ustc.edu.cn/planet/2024/12/ustc-mirrors-zfs-rebuild/"><i class="fas fa-fw fa-link"></i> 中文文章</a>
				|
				<a href="/p/74"><i class="fas fa-fw fa-link"></i> English article</a>
			</p>
			<p>友情链接：2023 年南京大学报告：<a href="/p/59"><i class="fas fa-fw fa-link"></i> ibug.io/p/59</a></p>
		</small>
	</section>
	]]></content><author><name>iBug</name></author></entry><entry><title type="html">Why my IPv4 gets stuck? - Debugging network issues with bpftrace</title><link href="https://ibug.io/blog/2024/08/first-touch-bpftrace/" rel="alternate" type="text/html" title="Why my IPv4 gets stuck? - Debugging network issues with bpftrace" /><published>2024-08-03T00:00:00+00:00</published><updated>2024-08-03T03:22:32+00:00</updated><id>https://ibug.io/blog/2024/08/first-touch-bpftrace</id><content type="html" xml:base="https://ibug.io/blog/2024/08/first-touch-bpftrace/"><![CDATA[<p>I run a Debian-based software router on my home network. It’s connected to multiple ISPs, so I have some policy routing rules to balance the traffic between them. Some time ago, I noticed that the IPv4 connectivity got stuck intermittently when it didn’t use to, while IPv6 was working fine. It’s also interesting that the issue only happened with one specific ISP, in the egress direction, and only a few specific devices were affected.</p>
	<p>At first I suspected the ISP’s equipment, but a clue quickly dismissed that suspicion: Connection to the same ISP worked fine when initiated from the router itself, as well as many other unaffected devices. So the issue must be within the router.</p>
	<p>As usual, every network debugging begins with a packet capture. I start <code class="language-plaintext highlighter-rouge">tcpdump</code> on both the LAN interface and the problematic WAN interface, then try <code class="language-plaintext highlighter-rouge">curl</code>-ing something from an affected device. Packet capture shows a few back-and-forth packets, then the device keeps sending packets but the router doesn’t forward them to the WAN interface anymore. Time for a closer look.</p>
	<h2 id="identifying-the-issue">Identifying the issue</h2>
	<p>On an affected device, <code class="language-plaintext highlighter-rouge">curl</code> gets stuck somewhere in the middle:</p>
	<div class="language-console highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code><span class="gp">$</span><span class="w"> </span>curl <span class="nt">-vso</span> /dev/null https://www.cloudflare.com/
<span class="go">*   Trying 104.16.124.96:443...
</span><span class="gp">* Connected to www.cloudflare.com (104.16.124.96) port 443 (#</span>0<span class="o">)</span>
<span class="go">* ALPN: offers h2,http/1.1
} [5&nbsp;bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512&nbsp;bytes data]
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
{ [5&nbsp;bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122&nbsp;bytes data]
^C
</span></code></pre>
		</div>
	</div>
	<p><code class="language-plaintext highlighter-rouge">tcpdump</code> shows nothing special:</p>
	<div class="language-console highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code><span class="gp">#</span><span class="w"> </span>tcpdump <span class="nt">-ni</span> any <span class="s1">'host 104.16.124.96 and tcp port 443'</span>
<span class="gp">02:03:47.403905 lan0 In  IP 172.17.0.2.49194 &gt;</span><span class="w"> </span>104.16.124.96.443: Flags <span class="o">[</span>S], <span class="nb">seq </span>1854398703, win 65535, options <span class="o">[</span>mss 1460,sackOK,TS val 1651776756 ecr 0,nop,wscale 10], length 0
<span class="gp">02:03:47.403956 ppp0 Out IP 10.250.193.4.49194 &gt;</span><span class="w"> </span>104.16.124.96.443: Flags <span class="o">[</span>S], <span class="nb">seq </span>1854398703, win 65535, options <span class="o">[</span>mss 1432,sackOK,TS val 1651776756 ecr 0,nop,wscale 10], length 0
<span class="gp">02:03:47.447663 ppp0 In  IP 104.16.124.96.443 &gt;</span><span class="w"> </span>10.250.193.4.49194: Flags <span class="o">[</span>S.], <span class="nb">seq </span>1391350792, ack 1854398704, win 65535, options <span class="o">[</span>mss 1460,sackOK,TS val 141787839 ecr 1651776756,nop,wscale 13], length 0
<span class="gp">02:03:47.447696 lan0 Out IP 104.16.124.96.443 &gt;</span><span class="w"> </span>172.17.0.2.49194: Flags <span class="o">[</span>S.], <span class="nb">seq </span>1391350792, ack 1854398704, win 65535, options <span class="o">[</span>mss 1460,sackOK,TS val 141787839 ecr 1651776756,nop,wscale 13], length 0
<span class="gp">02:03:47.447720 lan0 In  IP 172.17.0.2.49194 &gt;</span><span class="w"> </span>104.16.124.96.443: Flags <span class="o">[</span>.], ack 1, win 64, options <span class="o">[</span>nop,nop,TS val 1651776800 ecr 141787839], length 0
<span class="gp">02:03:47.452705 lan0 In  IP 172.17.0.2.49194 &gt;</span><span class="w"> </span>104.16.124.96.443: Flags <span class="o">[</span>P.], <span class="nb">seq </span>1:518, ack 1, win 64, options <span class="o">[</span>nop,nop,TS val 1651776804 ecr 141787839], length 517
<span class="gp">02:03:47.452751 ppp0 Out IP 10.250.193.4.49194 &gt;</span><span class="w"> </span>104.16.124.96.443: Flags <span class="o">[</span>P.], <span class="nb">seq </span>1:518, ack 1, win 64, options <span class="o">[</span>nop,nop,TS val 1651776804 ecr 141787839], length 517
<span class="gp">02:03:47.496507 ppp0 In  IP 104.16.124.96.443 &gt;</span><span class="w"> </span>10.250.193.4.49194: Flags <span class="o">[</span>.], ack 518, win 9, options <span class="o">[</span>nop,nop,TS val 141787888 ecr 1651776804], length 0
<span class="gp">02:03:47.496527 lan0 Out IP 104.16.124.96.443 &gt;</span><span class="w"> </span>172.17.0.2.49194: Flags <span class="o">[</span>.], ack 518, win 9, options <span class="o">[</span>nop,nop,TS val 141787888 ecr 1651776804], length 0
<span class="gp">02:03:47.498147 ppp0 In  IP 104.16.124.96.443 &gt;</span><span class="w"> </span>10.250.193.4.49194: Flags <span class="o">[</span>P.], <span class="nb">seq </span>1:2737, ack 518, win 9, options <span class="o">[</span>nop,nop,TS val 141787890 ecr 1651776804], length 2736
<span class="gp">02:03:47.498165 lan0 Out IP 104.16.124.96.443 &gt;</span><span class="w"> </span>172.17.0.2.49194: Flags <span class="o">[</span>P.], <span class="nb">seq </span>1:2737, ack 518, win 9, options <span class="o">[</span>nop,nop,TS val 141787890 ecr 1651776804], length 2736
<span class="gp">02:03:47.498175 lan0 In  IP 172.17.0.2.49194 &gt;</span><span class="w"> </span>104.16.124.96.443: Flags <span class="o">[</span>.], ack 2737, win 70, options <span class="o">[</span>nop,nop,TS val 1651776850 ecr 141787890], length 0
<span class="gp">02:03:47.498195 ppp0 In  IP 104.16.124.96.443 &gt;</span><span class="w"> </span>10.250.193.4.49194: Flags <span class="o">[</span>P.], <span class="nb">seq </span>2737:3758, ack 518, win 9, options <span class="o">[</span>nop,nop,TS val 141787890 ecr 1651776804], length 1021
<span class="gp">02:03:47.498228 ppp0 Out IP 10.250.193.4.49194 &gt;</span><span class="w"> </span>104.16.124.96.443: Flags <span class="o">[</span>R], <span class="nb">seq </span>1854399221, win 0, length 0
<span class="go">^C
711 packets captured
720 packets received by filter
0 packets dropped by kernel
</span></code></pre>
		</div>
	</div>
	<p>Considering the complexity of the policy routing, I tried inspecting conntrack status in parallel. Nothing unusual there either, until I tried matching conntrack events with <code class="language-plaintext highlighter-rouge">tcpdump</code>:</p>
	<div class="language-console highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code><span class="gp">#</span><span class="w"> </span>conntrack <span class="nt">-E</span> <span class="nt">-s</span> 172.17.0.2 <span class="nt">-p</span> tcp <span class="nt">--dport</span> 443 2&gt;/dev/null | ts %.T
<span class="go">02:03:47.404103     [NEW] tcp      6 120 SYN_SENT src=172.17.0.2 dst=104.16.124.96 sport=49194 dport=443 [UNREPLIED] src=104.16.124.96 dst=10.250.193.4 sport=443 dport=49194
02:03:47.447748  [UPDATE] tcp      6 60 SYN_RECV src=172.17.0.2 dst=104.16.124.96 sport=49194 dport=443 src=104.16.124.96 dst=10.250.193.4 sport=443 dport=49194 mark=48
02:03:47.447843 [DESTROY] tcp      6 432000 ESTABLISHED src=172.17.0.2 dst=104.16.124.96 sport=49194 dport=443 src=104.16.124.96 dst=10.250.193.4 sport=443 dport=49194 [ASSURED] mark=48
02:03:47.452798     [NEW] tcp      6 300 ESTABLISHED src=172.17.0.2 dst=104.16.124.96 sport=49194 dport=443 [UNREPLIED] src=104.16.124.96 dst=10.250.193.4 sport=443 dport=49194
02:03:47.496572  [UPDATE] tcp      6 300 src=172.17.0.2 dst=104.16.124.96 sport=49194 dport=443 src=104.16.124.96 dst=10.250.193.4 sport=443 dport=49194 mark=48
02:03:47.498195  [UPDATE] tcp      6 300 src=172.17.0.2 dst=104.16.124.96 sport=49194 dport=443 src=104.16.124.96 dst=10.250.193.4 sport=443 dport=49194 [ASSURED] mark=48
02:03:47.498243 [DESTROY] tcp      6 432000 ESTABLISHED src=172.17.0.2 dst=104.16.124.96 sport=49194 dport=443 src=104.16.124.96 dst=10.250.193.4 sport=443 dport=49194 [ASSURED] mark=48
^C
</span></code></pre>
		</div>
	</div>
	<p>With <code class="language-plaintext highlighter-rouge">ts</code> (from <a href="https://packages.debian.org/stable/moreutils"><code class="language-plaintext highlighter-rouge">moreutils</code></a>) adding timestamps to conntrack events, I can see that the conntrack entry is destroyed right after (+123μs) the second packet comes in from the device. Subsequent packets causes (+93μs) the same conntrack entry to be recreated, so <code class="language-plaintext highlighter-rouge">curl</code> could somehow complete the SSL handshake to a point where it only sends one packet and nothing afterwards for the connection to be recreated for a third time.</p>
	<p>Clearly the second packet should be considered <code class="language-plaintext highlighter-rouge">ESTABLISHED</code> by conntrack and makes no sense to trigger a <code class="language-plaintext highlighter-rouge">DESTROY</code> event. I’m at a loss here and start trying random things hoping to find a clue. I tried downgrading the kernel to 5.10 (from Bullseye) and upgrading to 6.9 (from Bookworm backports), but nothing changed, eliminating the possibility of a kernel bug.</p>
	<p>After scrutinizing my firewall rules, I noticed a small difference between IPv4 and IPv6 rules:</p>
	<div class="language-shell highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code><span class="c"># rules.v4</span>
<span class="k">*</span>nat
<span class="c"># ...</span>
<span class="nt">-A</span> POSTROUTING <span class="nt">-o</span> ppp+ <span class="nt">-j</span> MASQUERADE
COMMIT

<span class="k">*</span>mangle
:PREROUTING ACCEPT <span class="o">[</span>0:0]
<span class="c"># ...</span>
<span class="nt">-A</span> PREROUTING <span class="nt">-j</span> CONNMARK <span class="nt">--restore-mark</span>
<span class="nt">-A</span> PREROUTING <span class="nt">-m</span> mark <span class="o">!</span> <span class="nt">--mark</span> 0 <span class="nt">-j</span> ACCEPT
<span class="c">#A PREROUTING -m conntrack --ctstate NEW,RELATED -j MARK --set-xmark 0x100/0x100</span>
<span class="nt">-A</span> PREROUTING <span class="nt">-m</span> mark <span class="nt">--mark</span> 0/0xff <span class="nt">-j</span> ExtraConn
<span class="nt">-A</span> PREROUTING <span class="nt">-m</span> mark <span class="nt">--mark</span> 0/0xff <span class="nt">-j</span> IntraConn
<span class="nt">-A</span> PREROUTING <span class="nt">-m</span> mark <span class="nt">--mark</span> 0/0xff <span class="nt">-j</span> MARK <span class="nt">--set-xmark</span> 0x30/0xff
<span class="nt">-A</span> PREROUTING <span class="nt">-j</span> CONNMARK <span class="nt">--save-mark</span>
<span class="c"># ...</span>
<span class="nt">-A</span> ExtraConn <span class="nt">-i</span> ppp0 <span class="nt">-j</span> MARK <span class="nt">--set-xmark</span> 0x30/0xff
<span class="c"># ...</span>
<span class="nt">-A</span> IntraConn <span class="nt">-s</span> 172.17.0.2/32 <span class="nt">-j</span> iBugOptimized
<span class="c"># ...</span>
<span class="nt">-A</span> iBugOptimized <span class="nt">-j</span> MARK <span class="nt">--set-xmark</span> 0x36/0xff
<span class="nt">-A</span> iBugOptimized <span class="nt">-j</span> ACCEPT
COMMIT
</code></pre>
		</div>
	</div>
	<p>However, <code class="language-plaintext highlighter-rouge">rules.v6</code> is missing the last rule in <code class="language-plaintext highlighter-rouge">iBugOptimized</code>, and IPv6 is somehow exempt from the conntrack issue. Removing this extra <code class="language-plaintext highlighter-rouge">ACCEPT</code> rule from <code class="language-plaintext highlighter-rouge">rules.v4</code> fully restores the connectivity. So this is certainly the cause, but how is it related to the actual issue?</p>
	<h2 id="investigation">Investigation</h2>
	<p><em>I know there are some decent tools on GitHub that aids in debugging iptables, which is notorious for its complexity. But since I wrote the entire firewall rule set and am still maintaining it by hand, I’m going for the hard route of watching and understanding every single rule.</em></p>
	<p>The difference for that single <code class="language-plaintext highlighter-rouge">ACCEPT</code> rule is, it skips the <code class="language-plaintext highlighter-rouge">--save-mark</code> step, so the assigned firewall mark is not saved to its corresponding conntrack entry. When a reply packet comes in, conntrack has nothing for the <code class="language-plaintext highlighter-rouge">--restore-mark</code> step, so the packet gets assigned the “default” mark of <code class="language-plaintext highlighter-rouge">0x30</code> and <em>then</em> this value gets saved. I should have noticed the wrong conntrack mark earlier, as <code class="language-plaintext highlighter-rouge">conntrack -L</code> clearly showed a mark of 48 instead of the intended 54 (<code class="language-plaintext highlighter-rouge">0x36</code> from <code class="language-plaintext highlighter-rouge">iBugOptimized</code>). This narrows the cause down to a discrepancy between the packet mark and the conntrack mark.</p>
	<p>Firewall marks are a more flexible way to implement slightly complicated policy-based routing, as it defers the routing decision to the <code class="language-plaintext highlighter-rouge">mangle/PREROUTING</code> chain instead of the single-chain global routing rules. In my case, every ISP gets assigned a fwmark routing rule like this:</p>
	<div class="language-text highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code>9:      from all fwmark 0x30/0xff lookup eth0 proto static
9:      from all fwmark 0x31/0xff lookup eth1 proto static
9:      from all fwmark 0x36/0xff lookup ppp0 proto static
</code></pre>
		</div>
	</div>
	<p>Presumably, subsequent packets from the same connection should be routed to <code class="language-plaintext highlighter-rouge">eth0</code> because it has the mark <code class="language-plaintext highlighter-rouge">0x30</code> restored from conntrack entry. This is not the case, however, as <code class="language-plaintext highlighter-rouge">tcpdump</code> shows nothing on <code class="language-plaintext highlighter-rouge">eth0</code> and everything on <code class="language-plaintext highlighter-rouge">ppp0</code>.</p>
	<p>Unless there’s some magic in the kernel for it to decide to destroy a connection simply for a packet mark mismatch, this is not close enough to the root cause. Verifying the magic is relatively easy:</p>
	<div class="language-shell highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code>iptables <span class="nt">-I</span> PREROUTING <span class="nt">-s</span> 172.17.0.2/32 <span class="nt">-j</span> Test
iptables <span class="nt">-A</span> Test <span class="nt">-m</span> conntrack <span class="nt">--ctstate</span> NEW <span class="nt">-j</span> MARK <span class="nt">--set-xmark</span> 0x36/0xff
iptables <span class="nt">-A</span> Test <span class="nt">-m</span> conntrack <span class="nt">--ctstate</span> ESTABLISHED <span class="nt">-j</span> MARK <span class="nt">--set-xmark</span> 0x30/0xff
</code></pre>
		</div>
	</div>
	<p>This time, even if <code class="language-plaintext highlighter-rouge">conntrack</code> shows no mark (i.e. zero) on the connection, the packets are still routed correctly to <code class="language-plaintext highlighter-rouge">ppp0</code>, and curl gets stuck as the same place as before. So the kernel doesn’t care about the conntrack mark at all.</p>
	<p>Unfortunately, this is about as far as userspace inspection can go. I need to find out why exactly the kernel decides to destroy the conntrack entry.</p>
	<h2 id="bpftrace-comes-in"><code class="language-plaintext highlighter-rouge">bpftrace</code> comes in</h2>
	<p>I’ve seen professional kernel network developers extensively running <code class="language-plaintext highlighter-rouge">bpftrace</code> to debug network issues (THANK YOU to the guy behind the Telegram channel <em>Welcome to the Black Parade</em>), so I’m giving it a try.</p>
	<p>First thing is to figure out what to hook. Searching through Google did not reveal a trace point for conntrack events, but I get to know about the conntrack path. With help from ChatGPT, I begin with <code class="language-plaintext highlighter-rouge">kprobe:nf_ct_delete</code> and putting together all struct definitions starting from <code class="language-plaintext highlighter-rouge">struct nf_conn</code>:</p>
	<div class="language-c highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code><span class="cp">#include</span> <span class="cpf">&lt;linux/socket.h&gt;</span><span class="cp">
#include</span> <span class="cpf">&lt;net/netfilter/nf_conntrack.h&gt;</span><span class="cp">
</span>
<span class="n">kprobe</span><span class="o">:</span><span class="n">nf_ct_delete</span>
<span class="p">{</span>
    <span class="c1">// The first argument is the struct nf_conn</span>
    <span class="err">$</span><span class="n">ct</span> <span class="o">=</span> <span class="p">(</span><span class="k">struct</span> <span class="n">nf_conn</span> <span class="o">*</span><span class="p">)</span><span class="n">arg0</span><span class="p">;</span>

    <span class="c1">// Check if the connection is for IPv4</span>
    <span class="k">if</span> <span class="p">(</span><span class="err">$</span><span class="n">ct</span><span class="o">-&gt;</span><span class="n">tuplehash</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="n">tuple</span><span class="p">.</span><span class="n">src</span><span class="p">.</span><span class="n">l3num</span> <span class="o">==</span> <span class="n">AF_INET</span><span class="p">)</span> <span class="p">{</span>
        <span class="err">$</span><span class="n">src_ip</span> <span class="o">=</span> <span class="err">$</span><span class="n">ct</span><span class="o">-&gt;</span><span class="n">tuplehash</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="n">tuple</span><span class="p">.</span><span class="n">src</span><span class="p">.</span><span class="n">u3</span><span class="p">.</span><span class="n">ip</span><span class="p">;</span>
        <span class="err">$</span><span class="n">dst_ip</span> <span class="o">=</span> <span class="err">$</span><span class="n">ct</span><span class="o">-&gt;</span><span class="n">tuplehash</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="n">tuple</span><span class="p">.</span><span class="n">dst</span><span class="p">.</span><span class="n">u3</span><span class="p">.</span><span class="n">ip</span><span class="p">;</span>
        <span class="n">printf</span><span class="p">(</span><span class="s">"Conntrack destroyed (IPv4): src=%s dst=%s</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span>
                <span class="n">ntop</span><span class="p">(</span><span class="err">$</span><span class="n">src_ip</span><span class="p">),</span> <span class="n">ntop</span><span class="p">(</span><span class="err">$</span><span class="n">dst_ip</span><span class="p">));</span>
    <span class="p">}</span>
<span class="p">}</span>
</code></pre>
		</div>
	</div>
	<p>Seems all good, except it won’t compile:</p>
	<div class="language-text highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code>ERROR: Can not access field 'u3' on expression of type 'none'
        $dst_ip = $ct-&gt;tuplehash[0].tuple.dst.u3.ip;
                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
</code></pre>
		</div>
	</div>
	<p>After another half-hour of struggling and bothering with ChatGPT, I gave up trying to access the destination tuple, and thought I’d be fine with inspecting the stack trace:</p>
	<div class="language-c highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code><span class="cp">#include</span> <span class="cpf">&lt;linux/socket.h&gt;</span><span class="cp">
#include</span> <span class="cpf">&lt;net/netfilter/nf_conntrack.h&gt;</span><span class="cp">
</span>
<span class="n">kprobe</span><span class="o">:</span><span class="n">nf_ct_delete</span>
<span class="p">{</span>
    <span class="c1">// The first argument is the struct nf_conn</span>
    <span class="err">$</span><span class="n">ct</span> <span class="o">=</span> <span class="p">(</span><span class="k">struct</span> <span class="n">nf_conn</span> <span class="o">*</span><span class="p">)</span><span class="n">arg0</span><span class="p">;</span>

    <span class="c1">// Check if the connection is for IPv4</span>
    <span class="k">if</span> <span class="p">(</span><span class="err">$</span><span class="n">ct</span><span class="o">-&gt;</span><span class="n">tuplehash</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="n">tuple</span><span class="p">.</span><span class="n">src</span><span class="p">.</span><span class="n">l3num</span> <span class="o">==</span> <span class="n">AF_INET</span><span class="p">)</span> <span class="p">{</span>
        <span class="err">$</span><span class="n">tuple_orig</span> <span class="o">=</span> <span class="err">$</span><span class="n">ct</span><span class="o">-&gt;</span><span class="n">tuplehash</span><span class="p">[</span><span class="mi">0</span><span class="p">].</span><span class="n">tuple</span><span class="p">;</span>
        <span class="err">$</span><span class="n">src_ip</span> <span class="o">=</span> <span class="err">$</span><span class="n">tuple_orig</span><span class="p">.</span><span class="n">src</span><span class="p">.</span><span class="n">u3</span><span class="p">.</span><span class="n">ip</span><span class="p">;</span>
        <span class="err">$</span><span class="n">src_port_n</span> <span class="o">=</span> <span class="err">$</span><span class="n">tuple_orig</span><span class="p">.</span><span class="n">src</span><span class="p">.</span><span class="n">u</span><span class="p">.</span><span class="n">all</span><span class="p">;</span>
        <span class="err">$</span><span class="n">src_port</span> <span class="o">=</span> <span class="p">(</span><span class="err">$</span><span class="n">src_port_n</span> <span class="o">&gt;&gt;</span> <span class="mi">8</span><span class="p">)</span> <span class="o">|</span> <span class="p">((</span><span class="err">$</span><span class="n">src_port_n</span> <span class="o">&lt;&lt;</span> <span class="mi">8</span><span class="p">)</span> <span class="o">&amp;</span> <span class="mh">0x00FF00</span><span class="p">);</span>
        <span class="k">if</span> <span class="p">(</span><span class="err">$</span><span class="n">src_ip</span> <span class="o">!=</span> <span class="mh">0x020011ac</span><span class="p">)</span> <span class="p">{</span>
            <span class="k">return</span><span class="p">;</span>
        <span class="p">}</span>
        <span class="err">$</span><span class="n">mark</span> <span class="o">=</span> <span class="err">$</span><span class="n">ct</span><span class="o">-&gt;</span><span class="n">mark</span><span class="p">;</span>
        <span class="n">printf</span><span class="p">(</span><span class="s">"Conntrack destroyed (IPv4): src=%s sport=%d mark=%d</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span>
                <span class="n">ntop</span><span class="p">(</span><span class="err">$</span><span class="n">src_ip</span><span class="p">),</span> <span class="err">$</span><span class="n">src_port</span><span class="p">,</span> <span class="err">$</span><span class="n">mark</span><span class="p">);</span>

        <span class="n">printf</span><span class="p">(</span><span class="s">"%s</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">kstack</span><span class="p">());</span>
    <span class="p">}</span>
<span class="p">}</span>
</code></pre>
		</div>
	</div>
	<p>Noteworthy is that I have to filter the connections in the program, otherwise my screen gets flooded with unrelated events.</p>
	<p>The output comes promising:</p>
	<div class="language-text highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code>Attaching 1 probe...
Conntrack destroyed (IPv4): src=172.17.0.2 sport=39456 mark=0 proto=6

    nf_ct_delete+1
    nf_nat_inet_fn+188
    nf_nat_ipv4_out+80
    nf_hook_slow+70
    ip_output+220
    ip_forward_finish+132
    ip_forward+1296
    ip_rcv+404
    __netif_receive_skb_one_core+145
    __netif_receive_skb+21
    netif_receive_skb+300
    ...
</code></pre>
		</div>
	</div>
	<p>Reading the source code from the top few functions of the call stack:</p>
	<div class="language-c highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code><span class="c1">// net/netfilter/nf_nat_proto.c</span>

<span class="k">static</span> <span class="kt">unsigned</span> <span class="kt">int</span>
<span class="nf">nf_nat_ipv4_out</span><span class="p">(</span><span class="kt">void</span> <span class="o">*</span><span class="n">priv</span><span class="p">,</span> <span class="k">struct</span> <span class="n">sk_buff</span> <span class="o">*</span><span class="n">skb</span><span class="p">,</span>
                <span class="k">const</span> <span class="k">struct</span> <span class="n">nf_hook_state</span> <span class="o">*</span><span class="n">state</span><span class="p">)</span>
<span class="p">{</span>
<span class="cp">#ifdef CONFIG_XFRM
</span>    <span class="k">const</span> <span class="k">struct</span> <span class="n">nf_conn</span> <span class="o">*</span><span class="n">ct</span><span class="p">;</span>
    <span class="k">enum</span> <span class="n">ip_conntrack_info</span> <span class="n">ctinfo</span><span class="p">;</span>
    <span class="kt">int</span> <span class="n">err</span><span class="p">;</span>
<span class="cp">#endif
</span>    <span class="kt">unsigned</span> <span class="kt">int</span> <span class="n">ret</span><span class="p">;</span>

    <span class="n">ret</span> <span class="o">=</span> <span class="n">nf_nat_ipv4_fn</span><span class="p">(</span><span class="n">priv</span><span class="p">,</span> <span class="n">skb</span><span class="p">,</span> <span class="n">state</span><span class="p">);</span> <span class="c1">// &lt;-- call to nf_nat_ipv4_fn</span>
<span class="cp">#ifdef CONFIG_XFRM
</span>    <span class="k">if</span> <span class="p">(</span><span class="n">ret</span> <span class="o">!=</span> <span class="n">NF_ACCEPT</span><span class="p">)</span>
        <span class="k">return</span> <span class="n">ret</span><span class="p">;</span>
</code></pre>
		</div>
	</div>
	<div class="language-c highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code><span class="c1">// net/netfilter/nf_nat_proto.c</span>

<span class="k">static</span> <span class="kt">unsigned</span> <span class="kt">int</span>
<span class="nf">nf_nat_ipv4_fn</span><span class="p">(</span><span class="kt">void</span> <span class="o">*</span><span class="n">priv</span><span class="p">,</span> <span class="k">struct</span> <span class="n">sk_buff</span> <span class="o">*</span><span class="n">skb</span><span class="p">,</span>
               <span class="k">const</span> <span class="k">struct</span> <span class="n">nf_hook_state</span> <span class="o">*</span><span class="n">state</span><span class="p">)</span>
<span class="p">{</span>
    <span class="c1">// ...</span>

    <span class="k">return</span> <span class="n">nf_nat_inet_fn</span><span class="p">(</span><span class="n">priv</span><span class="p">,</span> <span class="n">skb</span><span class="p">,</span> <span class="n">state</span><span class="p">);</span>
<span class="p">}</span>
</code></pre>
		</div>
	</div>
	<div class="language-c highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code><span class="c1">// net/netfilter/nf_nat_core.c</span>

<span class="kt">unsigned</span> <span class="kt">int</span>
<span class="nf">nf_nat_inet_fn</span><span class="p">(</span><span class="kt">void</span> <span class="o">*</span><span class="n">priv</span><span class="p">,</span> <span class="k">struct</span> <span class="n">sk_buff</span> <span class="o">*</span><span class="n">skb</span><span class="p">,</span>
               <span class="k">const</span> <span class="k">struct</span> <span class="n">nf_hook_state</span> <span class="o">*</span><span class="n">state</span><span class="p">)</span>
<span class="p">{</span>
    <span class="c1">// ...</span>
        <span class="k">if</span> <span class="p">(</span><span class="n">nf_nat_oif_changed</span><span class="p">(</span><span class="n">state</span><span class="o">-&gt;</span><span class="n">hook</span><span class="p">,</span> <span class="n">ctinfo</span><span class="p">,</span> <span class="n">nat</span><span class="p">,</span> <span class="n">state</span><span class="o">-&gt;</span><span class="n">out</span><span class="p">))</span>
            <span class="k">goto</span> <span class="n">oif_changed</span><span class="p">;</span>
    <span class="c1">// ...</span>

<span class="nl">oif_changed:</span>
    <span class="n">nf_ct_kill_acct</span><span class="p">(</span><span class="n">ct</span><span class="p">,</span> <span class="n">ctinfo</span><span class="p">,</span> <span class="n">skb</span><span class="p">);</span>
    <span class="k">return</span> <span class="n">NF_DROP</span><span class="p">;</span>
<span class="p">}</span>
</code></pre>
		</div>
	</div>
	<p>As far as function inlining goes, there’s only one way <code class="language-plaintext highlighter-rouge">nf_nat_inet_fn</code> calls into <code class="language-plaintext highlighter-rouge">nf_ct_delete</code>, which is through <code class="language-plaintext highlighter-rouge">nf_ct_kill_acct</code>. And the only reason for that is <code class="language-plaintext highlighter-rouge">nf_nat_oif_changed</code>.</p>
	<h2 id="conclusion">Conclusion</h2>
	<p>Now everything makes sense. With a badly placed <code class="language-plaintext highlighter-rouge">ACCEPT</code> rule, the conntrack connection gets saved a different mark than desired, and then destroyed because subsequent packets are routed differently for having the wrong mark. The timestamp difference of related events also roughly matches up the distance of the code path. It also must be a NAT’ed connection, as this way of <code class="language-plaintext highlighter-rouge">nf_ct_delete</code> is only reachable when the packet is about to be sent to the egress interface.</p>
	]]></content><author><name>iBug</name></author><category term="linux" /><category term="networking" /><summary type="html"><![CDATA[I run a Debian-based software router on my home network. It’s connected to multiple ISPs, so I have some policy routing rules to balance the traffic between them. Some time ago, I noticed that the IPv4 connectivity got stuck intermittently when it didn’t use to, while IPv6 was working fine. It’s also interesting that the issue only happened with one specific ISP, in the egress direction, and only a few specific devices were affected.]]></summary></entry><entry><title type="html">Driving pppd with systemd</title><link href="https://ibug.io/blog/2024/07/pppd-with-systemd/" rel="alternate" type="text/html" title="Driving pppd with systemd" /><published>2024-07-07T00:00:00+00:00</published><updated>2024-07-16T01:25:55+00:00</updated><id>https://ibug.io/blog/2024/07/pppd-with-systemd</id><content type="html" xml:base="https://ibug.io/blog/2024/07/pppd-with-systemd/"><![CDATA[<p>I moved my soft router (Intel N5105, Debian) from school to home, and at home it’s behind an ONU on bridge mode, so it’ll have to do PPPoE itself.</p>
	<p>Getting started with PPPoE on Debian is exactly the same as on Ubuntu: Install <code class="language-plaintext highlighter-rouge">pppoeconf</code> and run <code class="language-plaintext highlighter-rouge">pppoeconf</code>, then fill in the DSL username and password. Then I can see <code class="language-plaintext highlighter-rouge">ppp0</code> interface up and working.</p>
	<p>However, as I use <code class="language-plaintext highlighter-rouge">systemd-networkd</code> on my router while <code class="language-plaintext highlighter-rouge">pppd</code> appears to bundle ifupdown, I’ll have to fix everything needed for <code class="language-plaintext highlighter-rouge">pppd</code> to work with systemd-networkd.</p>
	<h2 id="systemd-service">Systemd service</h2>
	<p>The first thing is to get it to start at boot. Looking through Google, a <a href="https://gist.github.com/rany2/330c8fe202b318cacdcb54830c20f98c">Gist</a> provides the exact systemd service file I need. After copying it to <code class="language-plaintext highlighter-rouge">/etc/systemd/system/ppp@.service</code>, I tried to start it with <code class="language-plaintext highlighter-rouge">systemctl start pppd@dsl-provider</code>. It seems like there’s a misconfiguration:</p>
	<div class="language-text highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code>/usr/sbin/pppd: Can't open options file /etc/ppp/peers/dsl/provider: No such file or directory
</code></pre>
		</div>
	</div>
	<p>The instance name is surely <code class="language-plaintext highlighter-rouge">dsl-provider</code> and not <code class="language-plaintext highlighter-rouge">dsl/provider</code>, so I look more closely at the service file.</p>
	<div class="language-ini highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code><span class="nn">[...]</span><span class="w">
</span><span class="py">Description</span><span class="p">=</span><span class="s">PPP connection for %I</span>
<span class="nn">[...]</span><span class="w">
</span><span class="py">ExecStart</span><span class="p">=</span><span class="s">/usr/sbin/pppd up_sdnotify nolog call %I</span>
</code></pre>
		</div>
	</div>
	<p>The systemd man page <a href="https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html"><code class="language-plaintext highlighter-rouge">systemd.unit(5)</code></a> says:</p>
	<blockquote>
		<table>
			<thead>
				<tr>
					<th>Specifier</th>
					<th>Meaning</th>
					<th>Details</th>
				</tr>
			</thead>
			<tbody>
				<tr>
					<td>“%i”</td>
					<td>Instance name</td>
					<td>For instantiated units this is the string between the first “@” character and the type suffix. Empty for non-instantiated units.</td>
				</tr>
				<tr>
					<td>“%I”</td>
					<td>Unescaped instance name</td>
					<td>Same as “%i”, but with escaping undone.</td>
				</tr>
			</tbody>
		</table>
	</blockquote>
	<p>Fair enough, let’s change <code class="language-plaintext highlighter-rouge">%I</code> to <code class="language-plaintext highlighter-rouge">%i</code> and try starting <code class="language-plaintext highlighter-rouge">pppd@dsl-provider</code> again.</p>
	<h2 id="systemd-networkd">systemd-networkd</h2>
	<p>Now that <code class="language-plaintext highlighter-rouge">ppp0</code> is up, time to configure routes and routing rules with <code class="language-plaintext highlighter-rouge">systemd-networkd</code>. I created a file <code class="language-plaintext highlighter-rouge">/etc/systemd/network/10-ppp0.network</code>.</p>
	<div class="language-ini highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code><span class="nn">[Match]</span><span class="w">
</span><span class="py">Name</span><span class="p">=</span><span class="s">ppp0</span>
<span class="w">
</span><span class="nn">[Network]</span><span class="w">
</span><span class="py">DHCP</span><span class="p">=</span><span class="s">yes</span>
<span class="c"># ...
</span></code></pre>
		</div>
	</div>
	<p>After restarting systemd-networkd, I was disappointed to see the PPP-negotiated IP address removed, only leaving an SLAAC IPv6 address behind. With some searching through <code class="language-plaintext highlighter-rouge">systemd.network(5)</code>, I found <code class="language-plaintext highlighter-rouge">KeepConfiguration=yes</code> was what I was looking for.</p>
	<h2 id="start-order">Start order</h2>
	<p>One problem still remains: At the time systemd-networkd starts, <code class="language-plaintext highlighter-rouge">ppp0</code> is not yet up, and systemd-networkd simply skips its configuration. A solution seems trivial:</p>
	<div class="language-ini highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code><span class="c"># systemctl edit pppd@dsl-provider
</span><span class="nn">[Unit]</span><span class="w">
</span><span class="py">Before</span><span class="p">=</span><span class="s">systemd-networkd.service</span>
</code></pre>
		</div>
	</div>
	<p>… except it doesn’t seem to have any effect.</p>
	<p>I wouldn’t bother digging into pppd, so I look around for something analogous to ifupdown’s <code class="language-plaintext highlighter-rouge">up</code> script, which is <code class="language-plaintext highlighter-rouge">/etc/ppp/ip-up.d/</code>. So I could just drop another script to notify systemd-networkd.</p>
	<div class="language-shell highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code><span class="c"># /etc/ppp/ip-up.d/1systemd-networkd</span>
<span class="c">#!/bin/sh</span>

networkctl reconfigure <span class="s2">"</span><span class="nv">$PPP_IFACE</span><span class="s2">"</span>
</code></pre>
		</div>
	</div>
	<p>I also noticed that when bringing in ifupdown, the <code class="language-plaintext highlighter-rouge">pppoeconf</code>-created config looks like this:</p>
	<div class="language-shell highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code>auto dsl-provider
iface dsl-provider inet ppp
    pre-up /bin/ip <span class="nb">link set </span>enp3s0 up <span class="c"># line maintained by pppoeconf</span>
    provider dsl-provider
</code></pre>
		</div>
	</div>
	<p>So to maintain behavioral compatibility, I configured the systemd service like this:</p>
	<div class="language-ini highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code><span class="c"># systemctl edit pppd@dsl-provider
</span><span class="nn">[Unit]</span><span class="w">
</span><span class="py">BindsTo</span><span class="p">=</span><span class="s">sys-subsystem-net-devices-enp3s0.device</span>
<span class="py">After</span><span class="p">=</span><span class="s">sys-subsystem-net-devices-enp3s0.device</span>
</code></pre>
		</div>
	</div>
	<p>After multiple reboots and manual restarts of <code class="language-plaintext highlighter-rouge">pppd@dsl-provider.service</code>, I’m convinced that this is a reliable solution.</p>
	<h2 id="extra">Extra: IPv6 PD</h2>
	<p>As the home ISP provides IPv6 Prefix Delegation (but my school didn’t), it would be nice to take it and distribute it to the LAN. Online tutorials are abundant, e.g. <a href="https://major.io/p/dhcpv6-prefix-delegation-with-systemd-networkd/" rel="nofollow noopener">this one</a>. With everything set supposedly up, I was again disappointed to see only a single SLAAC IPv6 address on <code class="language-plaintext highlighter-rouge">ppp0</code> itself, and <code class="language-plaintext highlighter-rouge">journalctl -eu systemd-networkd</code> shows no sign of receiving a PD allocation.</p>
	<p>After poking around with <code class="language-plaintext highlighter-rouge">IPv6AcceptRA=</code> and <code class="language-plaintext highlighter-rouge">[DHCPv6] PrefixDelegationHint=</code> settings for a while, I decided to capture some packets for investigation. I started <code class="language-plaintext highlighter-rouge">tcpdump -i ppp0 -w /tmp/ppp0.pcap icmp6 or udp port 546</code> and restarted <code class="language-plaintext highlighter-rouge">systemd-networkd</code>. After a few seconds, the pcap file contains exactly 4 packets that I need (some items omitted for brevity):</p>
	<div class="language-markdown highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code><span class="p">-</span> ICMPv6: Router Solicitation from 00:00:00:00:00:00
<span class="p">-</span> ICMPv6: Router Advertisement from 00:00:5e:00:01:99
<span class="p">  -</span> Flags: 0x40 (only O)
<span class="p">  -</span> ICMPv6 Option: Prefix information (2001:db8::/64)
<span class="p">    -</span> Flags: L + A
<span class="p">-</span> DHCPv6: Information-request XID: 0x8bf4f0 CID: 00020000ab11503f79e54f10745d
<span class="p">  -</span> Option Request
<span class="p">    -</span> Option: Option Request (6)
<span class="p">    -</span> Length: 10
<span class="p">    -</span> Requested Option code: DNS recursive name server (23)
<span class="p">    -</span> Requested Option code: Simple Network Time Protocol Server (31)
<span class="p">    -</span> Requested Option code: Lifetime (32)
<span class="p">    -</span> Requested Option code: NTP Server (56)
<span class="p">    -</span> Requested Option code: INF_MAX_RT (83)
<span class="p">-</span> DHCPv6: Reply XID: 0x8bf4f0 CID: 00020000ab11503f79e54f10745d
</code></pre>
		</div>
	</div>
	<p>Clearly the client isn’t even requesting a PD allocation with <code class="language-plaintext highlighter-rouge">PrefixDelegationHint=</code> set. With some more Google-ing, I added <code class="language-plaintext highlighter-rouge">[DHCPv6] WithoutRA=solicit</code> to <code class="language-plaintext highlighter-rouge">10-ppp0.network</code> and restarted <code class="language-plaintext highlighter-rouge">systemd-networkd</code>. There are 6 packets, but the order appears a little bit off:</p>
	<div class="language-markdown highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code><span class="p">-</span> Solicit XID: 0x2bc2aa CID: 00020000ab11503f79e54f10745d
<span class="p">-</span> Advertise XID: 0x2bc2aa CID: 00020000ab11503f79e54f10745d
<span class="p">-</span> Request XID: 0xf8c1dd CID: 00020000ab11503f79e54f10745d
<span class="p">  -</span> Identity Association for Prefix Delegation
<span class="p">-</span> Reply XID: 0xf8c1dd CID: 00020000ab11503f79e54f10745d
<span class="p">-</span> Router Solicitation from 00:00:00:00:00:00
<span class="p">-</span> Router Advertisement from 00:00:5e:00:01:99
</code></pre>
		</div>
	</div>
	<p>This time DHCP request comes <em>before</em> the RS/RA pair, which is not what I expected. But at least it’s now requesting a PD prefix.</p>
	<p>Then I found <a href="https://unix.stackexchange.com/a/715025/211239">this answer</a> straight to the point, summarized as:</p>
	<ul>
		<li>The “managed” (M) flag indicates the client should acquire an address via DHCPv6, and triggers DHCPv6 Solicit and Request messages.</li>
		<li>The “other” (O) flag indicates the client should do SLAAC while acquiring other configuration information via DHCPv6, and triggers DHCPv6 Information-request messages.</li>
		<li>When both flags are present, the O flag is superseded by the M flag and has no effect.</li>
	</ul>
	<p>So systemd-networkd is implementing everything correctly, and I should configure systemd-networkd to always send Solicit messages regardless of the RA flags received. This is done by setting <code class="language-plaintext highlighter-rouge">[IPv6AcceptRA] DHCPv6Client=always</code></p>
	<p>Now with every detail understood, after a restart of <code class="language-plaintext highlighter-rouge">systemd-networkd</code>, I finally see the PD prefix allocated:</p>
	<div class="language-text highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code>systemd-networkd[528]: ppp0: DHCP: received delegated prefix 2001:db8:0:a00::/60
systemd-networkd[528]: enp1s0: DHCP-PD address 2001:db8:0:a00:2a0:c9ff:feee:c4b/64 (valid for 2d 23h 59min 59s, preferred for 1d 23h 59min 59s)
systemd-networkd[528]: enp2s0: DHCP-PD address 2001:db8:0:a01:2a0:c9ff:feee:c4c/64 (valid for 2d 23h 59min 59s, preferred for 1d 23h 59min 59s)
</code></pre>
		</div>
	</div>
	<h2 id="update-1">Update: Stuck booting</h2>
	<p>A few days after this blog post, my local ISP ran into an outage that rendered the PPPoE connection unoperational.
		When I couldn’t identify the issue initially, I tried rebooting the router and it never came back up again.
		I plugged in a monitor and a keyboard, only to find systemd repeatedly trying to bring up <code class="language-plaintext highlighter-rouge">pppd@dsl-provider.service</code> when it would not succeed.
		The failure to start <code class="language-plaintext highlighter-rouge">pppd</code> resulted in complete unavailability of the network stack.</p>
	<p>I recalled that with OpenWRT this wasn’t the case, as the PPPoE interface being down would not impact any other interfaces.
		So I ended up removing all dependencies on <code class="language-plaintext highlighter-rouge">pppd@.service</code>, making it an ordinary system service that’s only <code class="language-plaintext highlighter-rouge">WantedBy=multi-user.target</code>.
		Considering that pppd will call <code class="language-plaintext highlighter-rouge">networkctl reconfigure</code> when it establishes the <code class="language-plaintext highlighter-rouge">ppp0</code> interface, the removal of systemd dependences shouldn’t have any consequences.</p>
	<h2 id="sum-up">Sum up</h2>
	<ul>
		<li>Use systemd to start <code class="language-plaintext highlighter-rouge">pppd</code> as a system service.
			<ul>
				<li>No need to bother with ordering.</li>
			</ul>
		</li>
		<li>Add <code class="language-plaintext highlighter-rouge">KeepConfiguration=yes</code> to <code class="language-plaintext highlighter-rouge">ppp0.network</code>.</li>
		<li>Use a custom script in <code class="language-plaintext highlighter-rouge">ip-up.d</code> to invoake systemd-networkd to reconfigure after it’s up.</li>
		<li>
			<p>For IPv6 PD, use both:</p>
			<div class="language-ini highlighter-rouge">
				<div class="highlight">
					<pre class="highlight"><code><span class="nn">[DHCPv6]</span><span class="w">
</span><span class="py">PrefixDelegationHint</span><span class="p">=</span><span class="s">::/60</span>
<span class="w">
</span><span class="nn">[IPv6AcceptRA]</span><span class="w">
</span><span class="py">DHCPv6Client</span><span class="p">=</span><span class="s">always</span>
</code></pre>
				</div>
    </div>
		</li>
	</ul>
	]]></content><author><name>iBug</name></author><category term="linux" /><category term="networking" /><summary type="html"><![CDATA[I moved my soft router (Intel N5105, Debian) from school to home, and at home it’s behind an ONU on bridge mode, so it’ll have to do PPPoE itself.]]></summary></entry><entry><title type="html">Migrating Ubuntu onto ZFS</title><link href="https://ibug.io/blog/2024/05/migrate-rootfs-to-zfs/" rel="alternate" type="text/html" title="Migrating Ubuntu onto ZFS" /><published>2024-05-14T00:00:00+00:00</published><updated>2024-05-14T23:37:49+00:00</updated><id>https://ibug.io/blog/2024/05/migrate-rootfs-to-zfs</id><content type="html" xml:base="https://ibug.io/blog/2024/05/migrate-rootfs-to-zfs/"><![CDATA[<p>As part of a planned disk migration, I decided to move my Ubuntu installation from a traditional ext4 setup to ZFS.
		I did a lot of preparation and research, but things went much smoother than I had previously anticipated.
		I did not even have to consult IPMI for any recovery.</p>
	<p>Existing partition layout:</p>
	<div class="language-console highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code><span class="gp">#</span><span class="w"> </span>fdisk <span class="nt">-l</span> /dev/nvme1n1
<span class="go">[...]
Device             Start        End    Sectors  Size Type
/dev/nvme1n1p1      2048    1050623    1048576  512M EFI System
/dev/nvme1n1p2   1050624  269486079  268435456  128G Linux filesystem
/dev/nvme1n1p3 269486080 3907029134 3637543055  1.7T Solaris /usr &amp; Apple ZFS
</span></code></pre>
		</div>
	</div>
	<p>Since I already have <code class="language-plaintext highlighter-rouge">/home</code> running on ZFS <code class="language-plaintext highlighter-rouge">pool0</code>, there’s not much to prepare.
		All I need to move is the rootfs itself, which has around 20&nbsp;GB of data.</p>
	<p>Start by installing anything necessary:</p>
	<div class="language-shell highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code>apt <span class="nb">install </span>zfs-initramfs arch-install-scripts
</code></pre>
		</div>
	</div>
	<p>Then create the dataset layout:</p>
	<div class="language-shell highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code><span class="c"># pool0 already has xattr=sa</span>
zfs create <span class="se">\</span>
  <span class="nt">-o</span> <span class="nv">canmount</span><span class="o">=</span>off <span class="se">\</span>
  <span class="nt">-o</span> <span class="nv">mountpoint</span><span class="o">=</span>none <span class="se">\</span>
  <span class="nt">-o</span> <span class="nv">acltype</span><span class="o">=</span>posix <span class="se">\</span>
  pool0/ROOT
zfs create <span class="nt">-o</span> <span class="nv">mountpoint</span><span class="o">=</span>/mnt/new pool0/ROOT/ubuntu

rsync <span class="nt">-avSHAXx</span> <span class="nt">--delete</span> / /mnt/new/
</code></pre>
		</div>
	</div>
	<p>Now there’s a little deviation from common setup.
		I don’t trust GRUB’s ZFS support, so I’m going to merge <code class="language-plaintext highlighter-rouge">/boot</code> into the EFI partition (which has a decent 512&nbsp;MB of capacity).
		This is a decision made after surveying my friends’ setup.</p>
	<div class="language-shell highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code><span class="c"># Merge data</span>
rsync <span class="nt">-ax</span> /boot/ /boot/efi/ <span class="c"># Ignore any errors</span>
umount /boot/efi
vim /etc/fstab
<span class="c"># Change /boot/efi to /boot</span>
<span class="c"># Also remove the current rootfs entry</span>
systemctl daemon-reload
mount /boot
</code></pre>
		</div>
	</div>
	<p>Now prepare GRUB:</p>
	<div class="language-shell highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code>zpool <span class="nb">set </span><span class="nv">bootfs</span><span class="o">=</span>pool0/ROOT/ubuntu pool0
mount <span class="nt">-o</span> <span class="nb">bind</span> /boot /mnt/new/boot
arch-chroot /mnt/new
</code></pre>
		</div>
	</div>
	<div class="language-console highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code><span class="gp">#</span><span class="w"> </span>grub-install
<span class="go">Installing for x86_64-efi platform.
grub-install: error: cannot find EFI directory.
</span></code></pre>
		</div>
	</div>
	<p>Well, if only <code class="language-plaintext highlighter-rouge">grub-install</code> didn’t hard-code <code class="language-plaintext highlighter-rouge">/boot/efi</code> (which is against the FHS standard anyways).
		Fortunately, I recall a small detail that could make this work in another convenient way:</p>
	<div class="language-shell highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code>dpkg-reconfigure grub-efi-amd64
</code></pre>
		</div>
	</div>
	<p>Also regenerate GRUB configuration:</p>
	<div class="language-shell highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code>zfs <span class="nb">set </span><span class="nv">mountpoint</span><span class="o">=</span>/ pool0/ROOT/ubuntu
update-grub
</code></pre>
		</div>
	</div>
	<p>Now double-check the GRUB configuration at <code class="language-plaintext highlighter-rouge">/boot/grub/grub.cfg</code> and make sure there are lines like this:</p>
	<div class="language-text highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code>linux /vmlinuz [...] root=ZFS=pool0/ROOT/ubuntu [...]
</code></pre>
		</div>
	</div>
	<p>After verifying paths to the kernel and the initrd image are correct, reboot:</p>
	<div class="language-shell highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code>reboot
</code></pre>
		</div>
	</div>
	<p>In just a minute, I noticed my server came back up.
		Time to confirm everything is working as expected:</p>
	<div class="language-console highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code><span class="gp">#</span><span class="w"> </span>mount
<span class="go">pool0/ROOT/ubuntu on / type zfs (rw,relatime,xattr,posixacl,casesensitive)

</span><span class="gp">#</span><span class="w"> </span><span class="nb">df</span> <span class="nt">-h</span> /
<span class="go">Filesystem         Size  Used Avail Use% Mounted on
pool0/ROOT/ubuntu  1.2T   11G  1.1T   1% /

</span><span class="gp">#</span><span class="w"> </span>zfs get compressratio pool0/ROOT
<span class="go">NAME        PROPERTY       VALUE  SOURCE
pool0/ROOT  compressratio  2.02x  -
</span></code></pre>
		</div>
	</div>
	<p>The last thing is to rewrite my rootfs backup script to take snapshots directly, instead of rsync-ing to another ZFS pool before taking a snapshot there.
		After taking a snapshot, I can also send it away as a “backup against disk failure”.</p>
	<p>A slightly revised version of my snapshotting script, sans the sending part:</p>
	<div class="language-shell highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code><span class="c">#!/bin/sh</span>

<span class="nb">set</span> <span class="nt">-e</span>

<span class="nv">DATASET</span><span class="o">=</span>pool0/ROOT/ubuntu
<span class="nv">DATE</span><span class="o">=</span><span class="si">$(</span><span class="nb">date</span> +%Y%m%d<span class="si">)</span>
<span class="nv">SNAPSHOT</span><span class="o">=</span><span class="s2">"</span><span class="nv">$DATASET</span><span class="s2">@</span><span class="nv">$DATE</span><span class="s2">"</span>
<span class="nv">RETENTION_DAYS</span><span class="o">=</span><span class="s2">"</span><span class="k">${</span><span class="nv">1</span><span class="k">:-</span><span class="nv">7</span><span class="k">}</span><span class="s2">"</span>
<span class="nv">RETENTION</span><span class="o">=</span><span class="s2">"</span><span class="k">$((</span>RETENTION_DAYS <span class="o">*</span> <span class="m">86400</span><span class="k">))</span><span class="s2">"</span>

<span class="nv">NOW</span><span class="o">=</span><span class="s2">"</span><span class="k">$((</span><span class="si">$(</span><span class="nb">date</span> +%s<span class="si">)</span> <span class="o">-</span> <span class="m">3600</span><span class="k">))</span><span class="s2">"</span>
<span class="k">if</span> <span class="o">[</span> <span class="s2">"</span><span class="si">$(</span>zfs list <span class="nt">-Hpo</span> name <span class="s2">"</span><span class="nv">$SNAPSHOT</span><span class="s2">"</span><span class="si">)</span><span class="s2">"</span> <span class="o">=</span> <span class="s2">"</span><span class="nv">$SNAPSHOT</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
  </span><span class="nb">echo</span> <span class="s2">"Snapshot exists: </span><span class="nv">$SNAPSHOT</span><span class="s2">"</span>
<span class="k">else
  </span>zfs snapshot <span class="nt">-ro</span> ibug:retention<span class="o">=</span><span class="s2">"</span><span class="nv">$RETENTION</span><span class="s2">"</span> <span class="s2">"</span><span class="nv">$SNAPSHOT</span><span class="s2">"</span>
<span class="k">fi

</span>zfs list <span class="nt">-Hpt</span> snapshot <span class="nt">-o</span> name,creation,ibug:retention <span class="s2">"</span><span class="nv">$DATASET</span><span class="s2">"</span> |
  <span class="k">while </span><span class="nb">read</span> <span class="nt">-r</span> zNAME zCREATION zRETENTION<span class="p">;</span> <span class="k">do
  if</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$zRETENTION</span><span class="s2">"</span> <span class="o">=</span> <span class="s2">"-"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span>
    <span class="c"># assume default value</span>
    <span class="nv">zRETENTION</span><span class="o">=</span><span class="s2">"</span><span class="k">$((</span><span class="m">7</span> <span class="o">*</span> <span class="m">86400</span><span class="k">))</span><span class="s2">"</span>
  <span class="k">fi
  </span><span class="nv">UNTIL</span><span class="o">=</span><span class="s2">"</span><span class="k">$((</span>zCREATION <span class="o">+</span> zRETENTION<span class="k">))</span><span class="s2">"</span>
  <span class="nv">UNTIL_DATE</span><span class="o">=</span><span class="s2">"</span><span class="si">$(</span><span class="nb">date</span> <span class="nt">-d</span> <span class="s2">"@</span><span class="nv">$UNTIL</span><span class="s2">"</span> <span class="s2">"+%Y-%m-%d %H:%M:%S"</span><span class="si">)</span><span class="s2">"</span>
  <span class="nb">echo</span> <span class="s2">"</span><span class="nv">$zNAME</span><span class="s2">: </span><span class="nv">$UNTIL_DATE</span><span class="s2">"</span>
  <span class="k">if</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$NOW</span><span class="s2">"</span> <span class="nt">-ge</span> <span class="s2">"</span><span class="nv">$UNTIL</span><span class="s2">"</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
    </span>zfs destroy <span class="nt">-rv</span> <span class="s2">"</span><span class="nv">$zNAME</span><span class="s2">"</span>
  <span class="k">fi
done</span>
</code></pre>
		</div>
	</div>
	<div class="language-shell highlighter-rouge">
		<div class="highlight">
			<pre class="highlight"><code><span class="c"># crontab</span>
15 4 <span class="k">*</span> <span class="k">*</span> 1,5     /root/backup.sh 30
15 4 <span class="k">*</span> <span class="k">*</span> 0,2-4,6 /root/backup.sh  7
</code></pre>
		</div>
	</div>
	]]></content><author><name>iBug</name></author><category term="linux" /><category term="server" /><category term="zfs" /><summary type="html"><![CDATA[As part of a planned disk migration, I decided to move my Ubuntu installation from a traditional ext4 setup to ZFS. I did a lot of preparation and research, but things went much smoother than I had previously anticipated. I did not even have to consult IPMI for any recovery.]]></summary></entry></feed>