This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-staging
in repository https://gitbox.apache.org/repos/asf/datafusion-python.git
The following commit(s) were added to refs/heads/asf-staging by this push:
new 8754e22b Publish built docs triggered by
4b215724565cec4257ed9dfa25271c5481c9f7b4
8754e22b is described below
commit 8754e22b7091c3d249160bde44d3d78ed2946c71
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Fri Mar 27 15:48:09 2026 +0000
Publish built docs triggered by 4b215724565cec4257ed9dfa25271c5481c9f7b4
---
_sources/autoapi/datafusion/context/index.rst.txt | 11 ++++++
_sources/autoapi/datafusion/index.rst.txt | 11 ++++++
_sources/user-guide/data-sources.rst.txt | 26 ++++++++----
autoapi/datafusion/context/index.html | 26 ++++++++++++
autoapi/datafusion/index.html | 26 ++++++++++++
genindex.html | 10 ++++-
objects.inv | Bin 9511 -> 9520 bytes
searchindex.js | 2 +-
user-guide/common-operations/aggregations.html | 38 +++++++++---------
user-guide/common-operations/basic-info.html | 46 +++++++++++-----------
user-guide/common-operations/functions.html | 20 +++++-----
user-guide/data-sources.html | 29 ++++++++++----
12 files changed, 175 insertions(+), 70 deletions(-)
diff --git a/_sources/autoapi/datafusion/context/index.rst.txt
b/_sources/autoapi/datafusion/context/index.rst.txt
index 1e6952f8..bdf0107f 100644
--- a/_sources/autoapi/datafusion/context/index.rst.txt
+++ b/_sources/autoapi/datafusion/context/index.rst.txt
@@ -282,6 +282,17 @@ Module Contents
+ .. py:method:: with_extension(extension: Any) -> SessionConfig
+
+ Create a new configuration using an extension.
+
+ :param extension: A custom configuration extension object. These are
+ :param shared from another DataFusion extension library.:
+
+ :returns: A new :py:class:`SessionConfig` object with the updated
setting.
+
+
+
.. py:method:: with_information_schema(enabled: bool = True) ->
SessionConfig
Enable or disable the inclusion of ``information_schema`` virtual tables.
diff --git a/_sources/autoapi/datafusion/index.rst.txt
b/_sources/autoapi/datafusion/index.rst.txt
index 5fd82597..457b539d 100644
--- a/_sources/autoapi/datafusion/index.rst.txt
+++ b/_sources/autoapi/datafusion/index.rst.txt
@@ -2274,6 +2274,17 @@ Package Contents
+ .. py:method:: with_extension(extension: Any) -> SessionConfig
+
+ Create a new configuration using an extension.
+
+ :param extension: A custom configuration extension object. These are
+ :param shared from another DataFusion extension library.:
+
+ :returns: A new :py:class:`SessionConfig` object with the updated
setting.
+
+
+
.. py:method:: with_information_schema(enabled: bool = True) ->
SessionConfig
Enable or disable the inclusion of ``information_schema`` virtual tables.
diff --git a/_sources/user-guide/data-sources.rst.txt
b/_sources/user-guide/data-sources.rst.txt
index 26f1303c..48ff4c01 100644
--- a/_sources/user-guide/data-sources.rst.txt
+++ b/_sources/user-guide/data-sources.rst.txt
@@ -224,25 +224,37 @@ A common technique for organizing tables is using a three
level hierarchical app
supports this form of organizing using the
:py:class:`~datafusion.catalog.Catalog`,
:py:class:`~datafusion.catalog.Schema`, and
:py:class:`~datafusion.catalog.Table`. By default,
a :py:class:`~datafusion.context.SessionContext` comes with a single Catalog
and a single Schema
-with the names ``datafusion`` and ``default``, respectively.
+with the names ``datafusion`` and ``public``, respectively.
The default implementation uses an in-memory approach to the catalog and
schema. We have support
-for adding additional in-memory catalogs and schemas. This can be done like in
the following
+for adding additional in-memory catalogs and schemas. You can access tables
registered in a schema
+either through the Dataframe API or via sql commands. This can be done like in
the following
example:
.. code-block:: python
+ import pyarrow as pa
from datafusion.catalog import Catalog, Schema
+ from datafusion import SessionContext
+
+ ctx = SessionContext()
my_catalog = Catalog.memory_catalog()
- my_schema = Schema.memory_schema()
+ my_schema = Schema.memory_schema()
+ my_catalog.register_schema('my_schema_name', my_schema)
+ ctx.register_catalog_provider('my_catalog_name', my_catalog)
- my_catalog.register_schema("my_schema_name", my_schema)
+ # Create an in-memory table
+ table = pa.table({
+ 'name': ['Bulbasaur', 'Charmander', 'Squirtle'],
+ 'type': ['Grass', 'Fire', 'Water'],
+ 'hp': [45, 39, 44],
+ })
+ df = ctx.create_dataframe([table.to_batches()], name='pokemon')
- ctx.register_catalog("my_catalog_name", my_catalog)
+ my_schema.register_table('pokemon', df)
-You could then register tables in ``my_schema`` and access them either through
the DataFrame
-API or via sql commands such as ``"SELECT * from
my_catalog_name.my_schema_name.my_table"``.
+ ctx.sql('SELECT * FROM my_catalog_name.my_schema_name.pokemon').show()
User Defined Catalog and Schema
-------------------------------
diff --git a/autoapi/datafusion/context/index.html
b/autoapi/datafusion/context/index.html
index 010a73d6..1c7c50f9 100644
--- a/autoapi/datafusion/context/index.html
+++ b/autoapi/datafusion/context/index.html
@@ -651,6 +651,15 @@
</code>
</a>
</li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link"
href="#datafusion.context.SessionConfig.with_extension">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ SessionConfig.with_extension()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h4 nav-item toc-entry">
<a class="reference internal nav-link"
href="#datafusion.context.SessionConfig.with_information_schema">
<code class="docutils literal notranslate">
@@ -1592,6 +1601,23 @@ automatically created.</p>
</dl>
</dd></dl>
+<dl class="py method">
+<dt class="sig sig-object py"
id="datafusion.context.SessionConfig.with_extension">
+<span class="sig-name descname"><span
class="pre">with_extension</span></span><span class="sig-paren">(</span><em
class="sig-param"><span class="n"><span
class="pre">extension</span></span><span class="p"><span
class="pre">:</span></span><span class="w"> </span><span class="n"><span
class="pre">Any</span></span></em><span class="sig-paren">)</span> <span
class="sig-return"><span class="sig-return-icon">→</span> <span
class="sig-return-typehint"><a class="reference internal" href=" [...]
+<dd><p>Create a new configuration using an extension.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters<span class="colon">:</span></dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>extension</strong> – A custom configuration extension object.
These are</p></li>
+<li><p><strong>library.</strong> (<em>shared from another DataFusion
extension</em>)</p></li>
+</ul>
+</dd>
+<dt class="field-even">Returns<span class="colon">:</span></dt>
+<dd class="field-even"><p>A new <a class="reference internal"
href="#datafusion.context.SessionConfig"
title="datafusion.context.SessionConfig"><code class="xref py py-class docutils
literal notranslate"><span class="pre">SessionConfig</span></code></a> object
with the updated setting.</p>
+</dd>
+</dl>
+</dd></dl>
+
<dl class="py method">
<dt class="sig sig-object py"
id="datafusion.context.SessionConfig.with_information_schema">
<span class="sig-name descname"><span
class="pre">with_information_schema</span></span><span
class="sig-paren">(</span><em class="sig-param"><span class="n"><span
class="pre">enabled</span></span><span class="p"><span
class="pre">:</span></span><span class="w"> </span><span class="n"><span
class="pre">bool</span></span><span class="w"> </span><span class="o"><span
class="pre">=</span></span><span class="w"> </span><span
class="default_value"><span class="pre">True</span></span></em><span [...]
diff --git a/autoapi/datafusion/index.html b/autoapi/datafusion/index.html
index 7c909610..e31529d0 100644
--- a/autoapi/datafusion/index.html
+++ b/autoapi/datafusion/index.html
@@ -2886,6 +2886,15 @@
</code>
</a>
</li>
+ <li class="toc-h4 nav-item toc-entry">
+ <a class="reference internal nav-link"
href="#datafusion.SessionConfig.with_extension">
+ <code class="docutils literal notranslate">
+ <span class="pre">
+ SessionConfig.with_extension()
+ </span>
+ </code>
+ </a>
+ </li>
<li class="toc-h4 nav-item toc-entry">
<a class="reference internal nav-link"
href="#datafusion.SessionConfig.with_information_schema">
<code class="docutils literal notranslate">
@@ -5804,6 +5813,23 @@ automatically created.</p>
</dl>
</dd></dl>
+<dl class="py method">
+<dt class="sig sig-object py" id="datafusion.SessionConfig.with_extension">
+<span class="sig-name descname"><span
class="pre">with_extension</span></span><span class="sig-paren">(</span><em
class="sig-param"><span class="n"><span
class="pre">extension</span></span><span class="p"><span
class="pre">:</span></span><span class="w"> </span><span class="n"><span
class="pre">Any</span></span></em><span class="sig-paren">)</span> <span
class="sig-return"><span class="sig-return-icon">→</span> <span
class="sig-return-typehint"><a class="reference internal" href=" [...]
+<dd><p>Create a new configuration using an extension.</p>
+<dl class="field-list simple">
+<dt class="field-odd">Parameters<span class="colon">:</span></dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>extension</strong> – A custom configuration extension object.
These are</p></li>
+<li><p><strong>library.</strong> (<em>shared from another DataFusion
extension</em>)</p></li>
+</ul>
+</dd>
+<dt class="field-even">Returns<span class="colon">:</span></dt>
+<dd class="field-even"><p>A new <a class="reference internal"
href="#datafusion.SessionConfig" title="datafusion.SessionConfig"><code
class="xref py py-class docutils literal notranslate"><span
class="pre">SessionConfig</span></code></a> object with the updated setting.</p>
+</dd>
+</dl>
+</dd></dl>
+
<dl class="py method">
<dt class="sig sig-object py"
id="datafusion.SessionConfig.with_information_schema">
<span class="sig-name descname"><span
class="pre">with_information_schema</span></span><span
class="sig-paren">(</span><em class="sig-param"><span class="n"><span
class="pre">enabled</span></span><span class="p"><span
class="pre">:</span></span><span class="w"> </span><span class="n"><span
class="pre">bool</span></span><span class="w"> </span><span class="o"><span
class="pre">=</span></span><span class="w"> </span><span
class="default_value"><span class="pre">True</span></span></em><span [...]
diff --git a/genindex.html b/genindex.html
index 91f551b7..d3a52142 100644
--- a/genindex.html
+++ b/genindex.html
@@ -3782,6 +3782,12 @@
<ul>
<li><a
href="autoapi/datafusion/options/index.html#datafusion.options.CsvReadOptions.with_escape">(datafusion.options.CsvReadOptions
method)</a>
+</li>
+ </ul></li>
+ <li><a
href="autoapi/datafusion/context/index.html#datafusion.context.SessionConfig.with_extension">with_extension()
(datafusion.context.SessionConfig method)</a>
+
+ <ul>
+ <li><a
href="autoapi/datafusion/index.html#datafusion.SessionConfig.with_extension">(datafusion.SessionConfig
method)</a>
</li>
</ul></li>
<li><a
href="autoapi/datafusion/context/index.html#datafusion.context.RuntimeEnvBuilder.with_fair_spill_pool">with_fair_spill_pool()
(datafusion.context.RuntimeEnvBuilder method)</a>
@@ -3808,14 +3814,14 @@
<li><a
href="autoapi/datafusion/options/index.html#datafusion.options.CsvReadOptions.with_file_sort_order">(datafusion.options.CsvReadOptions
method)</a>
</li>
</ul></li>
+ </ul></td>
+ <td style="width: 33%; vertical-align: top;"><ul>
<li><a
href="autoapi/datafusion/context/index.html#datafusion.context.RuntimeEnvBuilder.with_greedy_memory_pool">with_greedy_memory_pool()
(datafusion.context.RuntimeEnvBuilder method)</a>
<ul>
<li><a
href="autoapi/datafusion/index.html#datafusion.RuntimeEnvBuilder.with_greedy_memory_pool">(datafusion.RuntimeEnvBuilder
method)</a>
</li>
</ul></li>
- </ul></td>
- <td style="width: 33%; vertical-align: top;"><ul>
<li><a
href="autoapi/datafusion/index.html#datafusion.CsvReadOptions.with_has_header">with_has_header()
(datafusion.CsvReadOptions method)</a>
<ul>
diff --git a/objects.inv b/objects.inv
index 1afe9bd4..c3c25003 100644
Binary files a/objects.inv and b/objects.inv differ
diff --git a/searchindex.js b/searchindex.js
index c80e54cd..663a9273 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"API Reference": [[19, null]], "Additional
Resources": [[37, "additional-resources"]], "Aggregate Functions": [[24,
"aggregate-functions"], [31, "aggregate-functions"], [33,
"aggregate-functions"]], "Aggregation": [[24, null]], "Alternative Approach":
[[20, "alternative-approach"]], "Apache Iceberg": [[35, "apache-iceberg"]],
"Arrays": [[26, "arrays"]], "Arrow": [[39, null]], "Attributes": [[4,
"attributes"], [5, "attributes"], [7, "attributes"], [12, "attr [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"API Reference": [[19, null]], "Additional
Resources": [[37, "additional-resources"]], "Aggregate Functions": [[24,
"aggregate-functions"], [31, "aggregate-functions"], [33,
"aggregate-functions"]], "Aggregation": [[24, null]], "Alternative Approach":
[[20, "alternative-approach"]], "Apache Iceberg": [[35, "apache-iceberg"]],
"Arrays": [[26, "arrays"]], "Arrow": [[39, null]], "Attributes": [[4,
"attributes"], [5, "attributes"], [7, "attributes"], [12, "attr [...]
\ No newline at end of file
diff --git a/user-guide/common-operations/aggregations.html
b/user-guide/common-operations/aggregations.html
index 574cc1bf..363f99de 100644
--- a/user-guide/common-operations/aggregations.html
+++ b/user-guide/common-operations/aggregations.html
@@ -620,15 +620,15 @@ Pokemon set. Since there will be many entries of <code
class="docutils literal n
<span
class="o">+----------+--------------------------------------------------+</span>
<span class="o">|</span> <span class="n">Type</span> <span class="mi">1</span>
<span class="o">|</span> <span class="n">Type</span> <span
class="mi">2</span> <span class="n">List</span>
<span class="o">|</span>
<span
class="o">+----------+--------------------------------------------------+</span>
-<span class="o">|</span> <span class="n">Bug</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Poison</span><span class="p">,</span> <span
class="n">Grass</span><span class="p">,</span> <span class="p">]</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="n">Poison</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Ground</span><span
class="p">,</span> <span class="p">,</span> <span class="n">Flying</span><span
class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Electric</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="p">,</span> <span class="n">Steel</span><span
class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Bug</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span
class="p">,</span> <span class="n">Grass</span><span class="p">,</span> <span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="n">Poison</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Ground</span><span class="p">,</span> <span
class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Electric</span> <span
class="o">|</span> <span class="p">[,</span> <span class="n">Flying</span><span
class="p">,</span> <span class="n">Steel</span><span class="p">]</span>
<span class="o">|</span>
<span class="o">|</span> <span class="n">Fairy</span> <span
class="o">|</span> <span class="p">[]</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="n">Normal</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Fairy</span><span class="p">,</span> <span
class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Normal</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Fairy</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span
class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Ice</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="n">Grass</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span
class="p">,</span> <span class="n">Poison</span><span class="p">,</span> <span
class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Fire</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Dragon</span><span class="p">,</span> <span
class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Water</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span
class="p">,</span> <span class="n">Poison</span><span class="p">,</span> <span
class="n">Fighting</span><span class="p">,</span> <span
class="n">Ice</span><span class="p">,</span> <span class="n">Dark</span><span
class="p">,</span> <span class="p">,</span> <span class="n">Flying</span><span
class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Grass</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span
class="p">,</span> <span class="p">,</span> <span class="n">Poison</span><span
class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Fire</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="p">,</span> <span class="n">Dragon</span><span
class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Water</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span
class="p">,</span> <span class="n">Ice</span><span class="p">,</span> <span
class="n">Poison</span><span class="p">,</span> <span
class="n">Dark</span><span class="p">,</span> <span
class="n">Fighting</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Ground</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Rock</span><span
class="p">,</span> <span class="p">]</span>
<span class="o">|</span>
<span
class="o">+----------+--------------------------------------------------+</span>
<span class="n">Data</span> <span class="n">truncated</span><span
class="o">.</span>
@@ -644,15 +644,15 @@ entries entirely removed. The second is we can use the
<code class="docutils lit
<span
class="o">+----------+------------------------------------------------+</span>
<span class="o">|</span> <span class="n">Type</span> <span class="mi">1</span>
<span class="o">|</span> <span class="n">Type</span> <span
class="mi">2</span> <span class="n">List</span>
<span class="o">|</span>
<span
class="o">+----------+------------------------------------------------+</span>
-<span class="o">|</span> <span class="n">Bug</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span
class="p">,</span> <span class="n">Grass</span><span class="p">,</span> <span
class="n">Flying</span><span class="p">]</span> <span
class="o">|</span>
+<span class="o">|</span> <span class="n">Bug</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span
class="n">Grass</span><span class="p">]</span> <span
class="o">|</span>
<span class="o">|</span> <span class="n">Poison</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Ground</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="n">Electric</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Steel</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="n">Electric</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Steel</span><span class="p">]</span>
<span class="o">|</span>
<span class="o">|</span> <span class="n">Normal</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Fairy</span><span class="p">]</span>
<span class="o">|</span>
<span class="o">|</span> <span class="n">Ice</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="n">Grass</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span
class="p">,</span> <span class="n">Psychic</span><span class="p">]</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="n">Fire</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Dragon</span><span class="p">]</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="n">Water</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Fighting</span><span
class="p">,</span> <span class="n">Poison</span><span class="p">,</span> <span
class="n">Dark</span><span class="p">,</span> <span
class="n">Psychic</span><span class="p">,</span> <span
class="n">Flying</span><span class="p">,</span> <span class="n">Ice</span><span
class="p">]</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">Rock</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Ground</span><span class="p">,</span> <span
class="n">Water</span><span class="p">]</span> <span
class="o">|</span>
+<span class="o">|</span> <span class="n">Grass</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span
class="p">,</span> <span class="n">Poison</span><span class="p">]</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="n">Fire</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Dragon</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="n">Water</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span
class="p">,</span> <span class="n">Psychic</span><span class="p">,</span> <span
class="n">Ice</span><span class="p">,</span> <span class="n">Dark</span><span
class="p">,</span> <span class="n">Fighting</span><span class="p">,</span>
<span class="n">Flying</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Rock</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Water</span><span class="p">,</span> <span
class="n">Ground</span><span class="p">]</span> <span
class="o">|</span>
<span class="o">|</span> <span class="n">Ghost</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span
class="p">]</span> <span
class="o">|</span>
<span
class="o">+----------+------------------------------------------------+</span>
<span class="n">Data</span> <span class="n">truncated</span><span
class="o">.</span>
@@ -664,14 +664,14 @@ entries entirely removed. The second is we can use the
<code class="docutils lit
<span class="o">|</span> <span class="n">Type</span> <span class="mi">1</span>
<span class="o">|</span> <span class="n">Type</span> <span
class="mi">2</span> <span class="n">List</span>
<span class="o">|</span>
<span
class="o">+----------+------------------------------------------------+</span>
<span class="o">|</span> <span class="n">Bug</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span
class="p">,</span> <span class="n">Grass</span><span class="p">,</span> <span
class="n">Flying</span><span class="p">]</span> <span
class="o">|</span>
-<span class="o">|</span> <span class="n">Poison</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Ground</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="n">Electric</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Steel</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="n">Poison</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Ground</span><span class="p">]</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="n">Electric</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Steel</span><span class="p">]</span>
<span class="o">|</span>
<span class="o">|</span> <span class="n">Fairy</span> <span
class="o">|</span> <span
class="o">|</span>
-<span class="o">|</span> <span class="n">Normal</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Fairy</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="n">Ice</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Psychic</span><span class="p">]</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="n">Grass</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span
class="p">,</span> <span class="n">Poison</span><span class="p">]</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="n">Normal</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Flying</span><span
class="p">,</span> <span class="n">Fairy</span><span class="p">]</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="n">Ice</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Psychic</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="n">Grass</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span
class="p">,</span> <span class="n">Psychic</span><span class="p">]</span>
<span class="o">|</span>
<span class="o">|</span> <span class="n">Fire</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Dragon</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">]</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="n">Water</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Poison</span><span
class="p">,</span> <span class="n">Ice</span><span class="p">,</span> <span
class="n">Flying</span><span class="p">,</span> <span
class="n">Dark</span><span class="p">,</span> <span
class="n">Fighting</span><span class="p">,</span> <span
class="n">Psychic</span><span class="p">]</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">Water</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Fighting</span><span
class="p">,</span> <span class="n">Psychic</span><span class="p">,</span> <span
class="n">Poison</span><span class="p">,</span> <span class="n">Ice</span><span
class="p">,</span> <span class="n">Flying</span><span class="p">,</span> <span
class="n">Dark</span><span class="p">]</span> <span class="o">|</span>
<span class="o">|</span> <span class="n">Ground</span> <span
class="o">|</span> <span class="p">[</span><span class="n">Rock</span><span
class="p">]</span> <span
class="o">|</span>
<span
class="o">+----------+------------------------------------------------+</span>
<span class="n">Data</span> <span class="n">truncated</span><span
class="o">.</span>
diff --git a/user-guide/common-operations/basic-info.html
b/user-guide/common-operations/basic-info.html
index fedf63c6..0071c7a1 100644
--- a/user-guide/common-operations/basic-info.html
+++ b/user-guide/common-operations/basic-info.html
@@ -456,11 +456,11 @@
<span class="o">+-----+---------+--------+--------+</span>
<span class="o">|</span> <span class="n">nrs</span> <span class="o">|</span>
<span class="n">names</span> <span class="o">|</span> <span
class="n">random</span> <span class="o">|</span> <span class="n">groups</span>
<span class="o">|</span>
<span class="o">+-----+---------+--------+--------+</span>
-<span class="o">|</span> <span class="mi">1</span> <span class="o">|</span>
<span class="n">python</span> <span class="o">|</span> <span
class="mi">649</span> <span class="o">|</span> <span class="n">A</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="mi">2</span> <span class="o">|</span>
<span class="n">ruby</span> <span class="o">|</span> <span
class="mi">991</span> <span class="o">|</span> <span class="n">A</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="mi">3</span> <span class="o">|</span>
<span class="n">java</span> <span class="o">|</span> <span
class="mi">62</span> <span class="o">|</span> <span class="n">B</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="mi">4</span> <span class="o">|</span>
<span class="n">haskell</span> <span class="o">|</span> <span
class="mi">564</span> <span class="o">|</span> <span class="n">C</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="mi">5</span> <span class="o">|</span>
<span class="n">go</span> <span class="o">|</span> <span
class="mi">4</span> <span class="o">|</span> <span class="n">B</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="mi">1</span> <span class="o">|</span>
<span class="n">python</span> <span class="o">|</span> <span
class="mi">335</span> <span class="o">|</span> <span class="n">A</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="mi">2</span> <span class="o">|</span>
<span class="n">ruby</span> <span class="o">|</span> <span
class="mi">360</span> <span class="o">|</span> <span class="n">A</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="mi">3</span> <span class="o">|</span>
<span class="n">java</span> <span class="o">|</span> <span
class="mi">681</span> <span class="o">|</span> <span class="n">B</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="mi">4</span> <span class="o">|</span>
<span class="n">haskell</span> <span class="o">|</span> <span
class="mi">906</span> <span class="o">|</span> <span class="n">C</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="mi">5</span> <span class="o">|</span>
<span class="n">go</span> <span class="o">|</span> <span
class="mi">892</span> <span class="o">|</span> <span class="n">B</span>
<span class="o">|</span>
<span class="o">+-----+---------+--------+--------+</span>
</pre></div>
</div>
@@ -471,8 +471,8 @@
<span class="o">+-----+--------+--------+--------+</span>
<span class="o">|</span> <span class="n">nrs</span> <span class="o">|</span>
<span class="n">names</span> <span class="o">|</span> <span
class="n">random</span> <span class="o">|</span> <span class="n">groups</span>
<span class="o">|</span>
<span class="o">+-----+--------+--------+--------+</span>
-<span class="o">|</span> <span class="mi">1</span> <span class="o">|</span>
<span class="n">python</span> <span class="o">|</span> <span
class="mi">649</span> <span class="o">|</span> <span class="n">A</span>
<span class="o">|</span>
-<span class="o">|</span> <span class="mi">2</span> <span class="o">|</span>
<span class="n">ruby</span> <span class="o">|</span> <span
class="mi">991</span> <span class="o">|</span> <span class="n">A</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="mi">1</span> <span class="o">|</span>
<span class="n">python</span> <span class="o">|</span> <span
class="mi">335</span> <span class="o">|</span> <span class="n">A</span>
<span class="o">|</span>
+<span class="o">|</span> <span class="mi">2</span> <span class="o">|</span>
<span class="n">ruby</span> <span class="o">|</span> <span
class="mi">360</span> <span class="o">|</span> <span class="n">A</span>
<span class="o">|</span>
<span class="o">+-----+--------+--------+--------+</span>
</pre></div>
</div>
@@ -490,28 +490,28 @@ passing them to an Arrow table, and then converting them
to a pandas DataFrame.<
<div class="highlight-ipython notranslate"><div
class="highlight"><pre><span></span><span class="n">In</span> <span
class="p">[</span><span class="mi">8</span><span class="p">]:</span> <span
class="n">df</span><span class="o">.</span><span
class="n">to_pandas</span><span class="p">()</span>
<span class="n">Out</span><span class="p">[</span><span
class="mi">8</span><span class="p">]:</span>
<span class="n">nrs</span> <span class="n">names</span> <span
class="n">random</span> <span class="n">groups</span>
-<span class="mi">0</span> <span class="mi">1</span> <span
class="n">python</span> <span class="mi">649</span> <span
class="n">A</span>
-<span class="mi">1</span> <span class="mi">2</span> <span
class="n">ruby</span> <span class="mi">991</span> <span
class="n">A</span>
-<span class="mi">2</span> <span class="mi">3</span> <span
class="n">java</span> <span class="mi">62</span> <span
class="n">B</span>
-<span class="mi">3</span> <span class="mi">4</span> <span
class="n">haskell</span> <span class="mi">564</span> <span
class="n">C</span>
-<span class="mi">4</span> <span class="mi">5</span> <span
class="n">go</span> <span class="mi">4</span> <span
class="n">B</span>
+<span class="mi">0</span> <span class="mi">1</span> <span
class="n">python</span> <span class="mi">335</span> <span
class="n">A</span>
+<span class="mi">1</span> <span class="mi">2</span> <span
class="n">ruby</span> <span class="mi">360</span> <span
class="n">A</span>
+<span class="mi">2</span> <span class="mi">3</span> <span
class="n">java</span> <span class="mi">681</span> <span
class="n">B</span>
+<span class="mi">3</span> <span class="mi">4</span> <span
class="n">haskell</span> <span class="mi">906</span> <span
class="n">C</span>
+<span class="mi">4</span> <span class="mi">5</span> <span
class="n">go</span> <span class="mi">892</span> <span
class="n">B</span>
</pre></div>
</div>
<p><a class="reference internal"
href="../../autoapi/datafusion/dataframe/index.html#datafusion.dataframe.DataFrame.describe"
title="datafusion.dataframe.DataFrame.describe"><code class="xref py py-func
docutils literal notranslate"><span class="pre">describe()</span></code></a>
shows a quick statistic summary of your data:</p>
<div class="highlight-ipython notranslate"><div
class="highlight"><pre><span></span><span class="n">In</span> <span
class="p">[</span><span class="mi">9</span><span class="p">]:</span> <span
class="n">df</span><span class="o">.</span><span class="n">describe</span><span
class="p">()</span>
<span class="n">Out</span><span class="p">[</span><span
class="mi">9</span><span class="p">]:</span>
<span class="n">DataFrame</span><span class="p">()</span>
-<span
class="o">+------------+--------------------+-------+-------------------+--------+</span>
-<span class="o">|</span> <span class="n">describe</span> <span
class="o">|</span> <span class="n">nrs</span> <span
class="o">|</span> <span class="n">names</span> <span class="o">|</span> <span
class="n">random</span> <span class="o">|</span> <span
class="n">groups</span> <span class="o">|</span>
-<span
class="o">+------------+--------------------+-------+-------------------+--------+</span>
-<span class="o">|</span> <span class="n">count</span> <span
class="o">|</span> <span class="mf">5.0</span> <span
class="o">|</span> <span class="mi">5</span> <span class="o">|</span> <span
class="mf">5.0</span> <span class="o">|</span> <span
class="mi">5</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">null_count</span> <span
class="o">|</span> <span class="mf">0.0</span> <span
class="o">|</span> <span class="mi">0</span> <span class="o">|</span> <span
class="mf">0.0</span> <span class="o">|</span> <span
class="mi">0</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">mean</span> <span
class="o">|</span> <span class="mf">3.0</span> <span
class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span
class="mf">454.0</span> <span class="o">|</span> <span
class="n">null</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">std</span> <span
class="o">|</span> <span class="mf">1.5811388300841898</span> <span
class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span
class="mf">416.7307284086452</span> <span class="o">|</span> <span
class="n">null</span> <span class="o">|</span>
-<span class="o">|</span> <span class="nb">min</span> <span
class="o">|</span> <span class="mf">1.0</span> <span
class="o">|</span> <span class="n">go</span> <span class="o">|</span> <span
class="mf">4.0</span> <span class="o">|</span> <span
class="n">A</span> <span class="o">|</span>
-<span class="o">|</span> <span class="nb">max</span> <span
class="o">|</span> <span class="mf">5.0</span> <span
class="o">|</span> <span class="n">ruby</span> <span class="o">|</span> <span
class="mf">991.0</span> <span class="o">|</span> <span
class="n">C</span> <span class="o">|</span>
-<span class="o">|</span> <span class="n">median</span> <span
class="o">|</span> <span class="mf">3.0</span> <span
class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span
class="mf">564.0</span> <span class="o">|</span> <span
class="n">null</span> <span class="o">|</span>
-<span
class="o">+------------+--------------------+-------+-------------------+--------+</span>
+<span
class="o">+------------+--------------------+-------+--------------------+--------+</span>
+<span class="o">|</span> <span class="n">describe</span> <span
class="o">|</span> <span class="n">nrs</span> <span
class="o">|</span> <span class="n">names</span> <span class="o">|</span> <span
class="n">random</span> <span class="o">|</span> <span
class="n">groups</span> <span class="o">|</span>
+<span
class="o">+------------+--------------------+-------+--------------------+--------+</span>
+<span class="o">|</span> <span class="n">count</span> <span
class="o">|</span> <span class="mf">5.0</span> <span
class="o">|</span> <span class="mi">5</span> <span class="o">|</span> <span
class="mf">5.0</span> <span class="o">|</span> <span
class="mi">5</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">null_count</span> <span
class="o">|</span> <span class="mf">0.0</span> <span
class="o">|</span> <span class="mi">0</span> <span class="o">|</span> <span
class="mf">0.0</span> <span class="o">|</span> <span
class="mi">0</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">mean</span> <span
class="o">|</span> <span class="mf">3.0</span> <span
class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span
class="mf">634.8</span> <span class="o">|</span> <span
class="n">null</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">std</span> <span
class="o">|</span> <span class="mf">1.5811388300841898</span> <span
class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span
class="mf">277.14202135367344</span> <span class="o">|</span> <span
class="n">null</span> <span class="o">|</span>
+<span class="o">|</span> <span class="nb">min</span> <span
class="o">|</span> <span class="mf">1.0</span> <span
class="o">|</span> <span class="n">go</span> <span class="o">|</span> <span
class="mf">335.0</span> <span class="o">|</span> <span
class="n">A</span> <span class="o">|</span>
+<span class="o">|</span> <span class="nb">max</span> <span
class="o">|</span> <span class="mf">5.0</span> <span
class="o">|</span> <span class="n">ruby</span> <span class="o">|</span> <span
class="mf">906.0</span> <span class="o">|</span> <span
class="n">C</span> <span class="o">|</span>
+<span class="o">|</span> <span class="n">median</span> <span
class="o">|</span> <span class="mf">3.0</span> <span
class="o">|</span> <span class="n">null</span> <span class="o">|</span> <span
class="mf">681.0</span> <span class="o">|</span> <span
class="n">null</span> <span class="o">|</span>
+<span
class="o">+------------+--------------------+-------+--------------------+--------+</span>
</pre></div>
</div>
</section>
diff --git a/user-guide/common-operations/functions.html
b/user-guide/common-operations/functions.html
index fa121deb..24b98cb5 100644
--- a/user-guide/common-operations/functions.html
+++ b/user-guide/common-operations/functions.html
@@ -566,16 +566,16 @@ In here we will cover some of the more popular use cases.
If you want to view al
<span class="o">+-------------------------------+</span>
<span class="o">|</span> <span class="n">now</span><span class="p">()</span>
<span class="o">|</span>
<span class="o">+-------------------------------+</span>
-<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">03</span><span class="o">-</span><span
class="mi">27</span><span class="n">T14</span><span class="p">:</span><span
class="mi">12</span><span class="p">:</span><span
class="mf">02.214324978</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">03</span><span class="o">-</span><span
class="mi">27</span><span class="n">T14</span><span class="p">:</span><span
class="mi">12</span><span class="p">:</span><span
class="mf">02.214324978</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">03</span><span class="o">-</span><span
class="mi">27</span><span class="n">T14</span><span class="p">:</span><span
class="mi">12</span><span class="p">:</span><span
class="mf">02.214324978</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">03</span><span class="o">-</span><span
class="mi">27</span><span class="n">T14</span><span class="p">:</span><span
class="mi">12</span><span class="p">:</span><span
class="mf">02.214324978</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">03</span><span class="o">-</span><span
class="mi">27</span><span class="n">T14</span><span class="p">:</span><span
class="mi">12</span><span class="p">:</span><span
class="mf">02.214324978</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">03</span><span class="o">-</span><span
class="mi">27</span><span class="n">T14</span><span class="p">:</span><span
class="mi">12</span><span class="p">:</span><span
class="mf">02.214324978</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">03</span><span class="o">-</span><span
class="mi">27</span><span class="n">T14</span><span class="p">:</span><span
class="mi">12</span><span class="p">:</span><span
class="mf">02.214324978</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">03</span><span class="o">-</span><span
class="mi">27</span><span class="n">T14</span><span class="p">:</span><span
class="mi">12</span><span class="p">:</span><span
class="mf">02.214324978</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">03</span><span class="o">-</span><span
class="mi">27</span><span class="n">T14</span><span class="p">:</span><span
class="mi">12</span><span class="p">:</span><span
class="mf">02.214324978</span> <span class="o">|</span>
-<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">03</span><span class="o">-</span><span
class="mi">27</span><span class="n">T14</span><span class="p">:</span><span
class="mi">12</span><span class="p">:</span><span
class="mf">02.214324978</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">03</span><span class="o">-</span><span
class="mi">27</span><span class="n">T15</span><span class="p">:</span><span
class="mi">47</span><span class="p">:</span><span
class="mf">57.414332284</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">03</span><span class="o">-</span><span
class="mi">27</span><span class="n">T15</span><span class="p">:</span><span
class="mi">47</span><span class="p">:</span><span
class="mf">57.414332284</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">03</span><span class="o">-</span><span
class="mi">27</span><span class="n">T15</span><span class="p">:</span><span
class="mi">47</span><span class="p">:</span><span
class="mf">57.414332284</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">03</span><span class="o">-</span><span
class="mi">27</span><span class="n">T15</span><span class="p">:</span><span
class="mi">47</span><span class="p">:</span><span
class="mf">57.414332284</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">03</span><span class="o">-</span><span
class="mi">27</span><span class="n">T15</span><span class="p">:</span><span
class="mi">47</span><span class="p">:</span><span
class="mf">57.414332284</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">03</span><span class="o">-</span><span
class="mi">27</span><span class="n">T15</span><span class="p">:</span><span
class="mi">47</span><span class="p">:</span><span
class="mf">57.414332284</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">03</span><span class="o">-</span><span
class="mi">27</span><span class="n">T15</span><span class="p">:</span><span
class="mi">47</span><span class="p">:</span><span
class="mf">57.414332284</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">03</span><span class="o">-</span><span
class="mi">27</span><span class="n">T15</span><span class="p">:</span><span
class="mi">47</span><span class="p">:</span><span
class="mf">57.414332284</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">03</span><span class="o">-</span><span
class="mi">27</span><span class="n">T15</span><span class="p">:</span><span
class="mi">47</span><span class="p">:</span><span
class="mf">57.414332284</span> <span class="o">|</span>
+<span class="o">|</span> <span class="mi">2026</span><span
class="o">-</span><span class="mi">03</span><span class="o">-</span><span
class="mi">27</span><span class="n">T15</span><span class="p">:</span><span
class="mi">47</span><span class="p">:</span><span
class="mf">57.414332284</span> <span class="o">|</span>
<span class="o">+-------------------------------+</span>
<span class="n">Data</span> <span class="n">truncated</span><span
class="o">.</span>
</pre></div>
diff --git a/user-guide/data-sources.html b/user-guide/data-sources.html
index a2b1c80e..e692bc86 100644
--- a/user-guide/data-sources.html
+++ b/user-guide/data-sources.html
@@ -731,22 +731,35 @@ is provided in the DataFusion repository.</p>
supports this form of organizing using the <a class="reference internal"
href="../autoapi/datafusion/catalog/index.html#datafusion.catalog.Catalog"
title="datafusion.catalog.Catalog"><code class="xref py py-class docutils
literal notranslate"><span class="pre">Catalog</span></code></a>,
<a class="reference internal"
href="../autoapi/datafusion/catalog/index.html#datafusion.catalog.Schema"
title="datafusion.catalog.Schema"><code class="xref py py-class docutils
literal notranslate"><span class="pre">Schema</span></code></a>, and <a
class="reference internal"
href="../autoapi/datafusion/catalog/index.html#datafusion.catalog.Table"
title="datafusion.catalog.Table"><code class="xref py py-class docutils literal
notranslate"><span class="pre">Table</span></code></a>. By default,
a <a class="reference internal"
href="../autoapi/datafusion/context/index.html#datafusion.context.SessionContext"
title="datafusion.context.SessionContext"><code class="xref py py-class
docutils literal notranslate"><span
class="pre">SessionContext</span></code></a> comes with a single Catalog and a
single Schema
-with the names <code class="docutils literal notranslate"><span
class="pre">datafusion</span></code> and <code class="docutils literal
notranslate"><span class="pre">default</span></code>, respectively.</p>
+with the names <code class="docutils literal notranslate"><span
class="pre">datafusion</span></code> and <code class="docutils literal
notranslate"><span class="pre">public</span></code>, respectively.</p>
<p>The default implementation uses an in-memory approach to the catalog and
schema. We have support
-for adding additional in-memory catalogs and schemas. This can be done like in
the following
+for adding additional in-memory catalogs and schemas. You can access tables
registered in a schema
+either through the Dataframe API or via sql commands. This can be done like in
the following
example:</p>
-<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="kn">from</span><span
class="w"> </span><span class="nn">datafusion.catalog</span><span class="w">
</span><span class="kn">import</span> <span class="n">Catalog</span><span
class="p">,</span> <span class="n">Schema</span>
+<div class="highlight-python notranslate"><div
class="highlight"><pre><span></span><span class="kn">import</span><span
class="w"> </span><span class="nn">pyarrow</span><span class="w"> </span><span
class="k">as</span><span class="w"> </span><span class="nn">pa</span>
+<span class="kn">from</span><span class="w"> </span><span
class="nn">datafusion.catalog</span><span class="w"> </span><span
class="kn">import</span> <span class="n">Catalog</span><span class="p">,</span>
<span class="n">Schema</span>
+<span class="kn">from</span><span class="w"> </span><span
class="nn">datafusion</span><span class="w"> </span><span
class="kn">import</span> <span class="n">SessionContext</span>
+
+<span class="n">ctx</span> <span class="o">=</span> <span
class="n">SessionContext</span><span class="p">()</span>
<span class="n">my_catalog</span> <span class="o">=</span> <span
class="n">Catalog</span><span class="o">.</span><span
class="n">memory_catalog</span><span class="p">()</span>
-<span class="n">my_schema</span> <span class="o">=</span> <span
class="n">Schema</span><span class="o">.</span><span
class="n">memory_schema</span><span class="p">()</span>
+<span class="n">my_schema</span> <span class="o">=</span> <span
class="n">Schema</span><span class="o">.</span><span
class="n">memory_schema</span><span class="p">()</span>
+<span class="n">my_catalog</span><span class="o">.</span><span
class="n">register_schema</span><span class="p">(</span><span
class="s1">'my_schema_name'</span><span class="p">,</span> <span
class="n">my_schema</span><span class="p">)</span>
+<span class="n">ctx</span><span class="o">.</span><span
class="n">register_catalog_provider</span><span class="p">(</span><span
class="s1">'my_catalog_name'</span><span class="p">,</span> <span
class="n">my_catalog</span><span class="p">)</span>
+
+<span class="c1"># Create an in-memory table</span>
+<span class="n">table</span> <span class="o">=</span> <span
class="n">pa</span><span class="o">.</span><span class="n">table</span><span
class="p">({</span>
+ <span class="s1">'name'</span><span class="p">:</span> <span
class="p">[</span><span class="s1">'Bulbasaur'</span><span
class="p">,</span> <span class="s1">'Charmander'</span><span
class="p">,</span> <span class="s1">'Squirtle'</span><span
class="p">],</span>
+ <span class="s1">'type'</span><span class="p">:</span> <span
class="p">[</span><span class="s1">'Grass'</span><span
class="p">,</span> <span class="s1">'Fire'</span><span
class="p">,</span> <span class="s1">'Water'</span><span
class="p">],</span>
+ <span class="s1">'hp'</span><span class="p">:</span> <span
class="p">[</span><span class="mi">45</span><span class="p">,</span>
<span class="mi">39</span><span class="p">,</span> <span
class="mi">44</span><span class="p">],</span>
+<span class="p">})</span>
+<span class="n">df</span> <span class="o">=</span> <span
class="n">ctx</span><span class="o">.</span><span
class="n">create_dataframe</span><span class="p">([</span><span
class="n">table</span><span class="o">.</span><span
class="n">to_batches</span><span class="p">()],</span> <span
class="n">name</span><span class="o">=</span><span
class="s1">'pokemon'</span><span class="p">)</span>
-<span class="n">my_catalog</span><span class="o">.</span><span
class="n">register_schema</span><span class="p">(</span><span
class="s2">"my_schema_name"</span><span class="p">,</span> <span
class="n">my_schema</span><span class="p">)</span>
+<span class="n">my_schema</span><span class="o">.</span><span
class="n">register_table</span><span class="p">(</span><span
class="s1">'pokemon'</span><span class="p">,</span> <span
class="n">df</span><span class="p">)</span>
-<span class="n">ctx</span><span class="o">.</span><span
class="n">register_catalog</span><span class="p">(</span><span
class="s2">"my_catalog_name"</span><span class="p">,</span> <span
class="n">my_catalog</span><span class="p">)</span>
+<span class="n">ctx</span><span class="o">.</span><span
class="n">sql</span><span class="p">(</span><span class="s1">'SELECT * FROM
my_catalog_name.my_schema_name.pokemon'</span><span class="p">)</span><span
class="o">.</span><span class="n">show</span><span class="p">()</span>
</pre></div>
</div>
-<p>You could then register tables in <code class="docutils literal
notranslate"><span class="pre">my_schema</span></code> and access them either
through the DataFrame
-API or via sql commands such as <code class="docutils literal
notranslate"><span class="pre">"SELECT</span> <span class="pre">*</span>
<span class="pre">from</span> <span
class="pre">my_catalog_name.my_schema_name.my_table"</span></code>.</p>
<section id="user-defined-catalog-and-schema">
<h2>User Defined Catalog and Schema<a class="headerlink"
href="#user-defined-catalog-and-schema" title="Link to this heading">¶</a></h2>
<p>If the in-memory catalogs are insufficient for your uses, there are two
approaches you can take
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
