<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>Columnar Storage on codedump notes</title>
    <link>https://www.codedump.info/en/tags/columnar-storage/</link>
    <description>Recent content in Columnar Storage on codedump notes</description>
    <generator>Hugo</generator>
    <language>en</language>
    <lastBuildDate>Wed, 18 Nov 2026 10:17:06 +0800</lastBuildDate>
    <atom:link href="https://www.codedump.info/en/tags/columnar-storage/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>Understanding Lance (1): File Organization</title>
      <link>https://www.codedump.info/en/post/20261118-lance-1/</link>
      <pubDate>Wed, 18 Nov 2026 10:17:06 +0800</pubDate>
      <guid>https://www.codedump.info/en/post/20261118-lance-1/</guid>
      <description>&lt;h1 class=&#34;heading&#34; id=&#34;characteristics-of-object-storage&#34;&gt;&#xA;  Characteristics of Object Storage&#xA;  &lt;a class=&#34;anchor&#34; href=&#34;#characteristics-of-object-storage&#34;&gt;#&lt;/a&gt;&#xA;&lt;/h1&gt;&#xA;&lt;p&gt;Lance is a columnar storage format that primarily targets S3 object storage. Compared with local disk storage, object storage has the following characteristics:&lt;/p&gt;&#xA;&lt;table&gt;&#xA;  &lt;thead&gt;&#xA;      &lt;tr&gt;&#xA;          &lt;th&gt;Constraint&lt;/th&gt;&#xA;          &lt;th&gt;Consequence&lt;/th&gt;&#xA;      &lt;/tr&gt;&#xA;  &lt;/thead&gt;&#xA;  &lt;tbody&gt;&#xA;      &lt;tr&gt;&#xA;          &lt;td&gt;There are no real directories&lt;/td&gt;&#xA;          &lt;td&gt;Directories are just key prefixes; &lt;code&gt;tree/&lt;/code&gt; and &lt;code&gt;_indices/{uuid}/&lt;/code&gt; are naming conventions&lt;/td&gt;&#xA;      &lt;/tr&gt;&#xA;      &lt;tr&gt;&#xA;          &lt;td&gt;No transactions across objects&lt;/td&gt;&#xA;          &lt;td&gt;You cannot &amp;ldquo;write multiple files at once and have them take effect atomically&amp;rdquo;&lt;/td&gt;&#xA;      &lt;/tr&gt;&#xA;      &lt;tr&gt;&#xA;          &lt;td&gt;Only single-object operations are atomic&lt;/td&gt;&#xA;          &lt;td&gt;&lt;code&gt;PUT&lt;/code&gt; / &lt;code&gt;PUT-if-not-exists&lt;/code&gt; / &lt;code&gt;DELETE&lt;/code&gt;&lt;/td&gt;&#xA;      &lt;/tr&gt;&#xA;      &lt;tr&gt;&#xA;          &lt;td&gt;rename is not always atomic&lt;/td&gt;&#xA;          &lt;td&gt;On S3 it is copy+delete, so rename cannot be used for commit&lt;/td&gt;&#xA;      &lt;/tr&gt;&#xA;      &lt;tr&gt;&#xA;          &lt;td&gt;list can be expensive&lt;/td&gt;&#xA;          &lt;td&gt;Every directory listing is a request; on S3 Express it is even O(n)&lt;/td&gt;&#xA;      &lt;/tr&gt;&#xA;      &lt;tr&gt;&#xA;          &lt;td&gt;Requests are rate-limited&lt;/td&gt;&#xA;          &lt;td&gt;File names need to be spread across prefixes&lt;/td&gt;&#xA;      &lt;/tr&gt;&#xA;      &lt;tr&gt;&#xA;          &lt;td&gt;Consistency models differ&lt;/td&gt;&#xA;          &lt;td&gt;You cannot assume &amp;ldquo;it shows up in list immediately after writing&amp;rdquo;&lt;/td&gt;&#xA;      &lt;/tr&gt;&#xA;  &lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;Therefore, any design that &lt;strong&gt;&amp;ldquo;modifies multiple files together&amp;rdquo;&lt;/strong&gt; is infeasible.&lt;/p&gt;</description>
    </item>
  </channel>
</rss>
