This is an automated email from the ASF dual-hosted git repository.
lidavidm pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-adbc.git
The following commit(s) were added to refs/heads/main by this push:
new 59ccc45b0 feat(javascript): return Arrow Table by default in
AdbcConnection (#4091)
59ccc45b0 is described below
commit 59ccc45b07bc963ceb50ea9015c8943bbaf76c3a
Author: Kent Wu <[email protected]>
AuthorDate: Sat Mar 14 08:25:04 2026 -0400
feat(javascript): return Arrow Table by default in AdbcConnection (#4091)
This PR cleans up the AdbcConnection API so that the common usage is
more ergonomic.
The high-level connection methods now return an Arrow Table by default,
consistent with how most TypeScript database clients behave. Users who
need streaming or fine-grained control can use queryStream() or drop
down to the statement-level API.
AdbcConnection Interface changes
| Method | Before | After |
|--------|--------|-------|
| `query(sql, params?)` | `params?: RecordBatch \| Table` →
`Promise<RecordBatchReader>` | `params?: Table` → `Promise<Table>` |
| `queryStream(sql, params?)` | _(new)_ | `params?: Table` →
`Promise<RecordBatchReader>` |
| `execute(sql, params?)` | `params?: RecordBatch \| Table` →
`Promise<number>` | `params?: Table` → `Promise<number>` |
| `getObjects(options?)` | `Promise<RecordBatchReader>` |
`Promise<Table>` |
| `getTableTypes()` | `Promise<RecordBatchReader>` | `Promise<Table>` |
| `getInfo(infoCodes?)` | `Promise<RecordBatchReader>` |
`Promise<Table>` |
Closes #4090
---
javascript/README.md | 46 +++++++++++-------
javascript/__test__/error_handling.spec.ts | 76 ++++++++++++++++++++++++++++++
javascript/__test__/metadata.spec.ts | 46 +++++++++++-------
javascript/__test__/profile.spec.ts | 9 ++--
javascript/__test__/query.spec.ts | 53 +++++++++++++++++----
javascript/lib/index.ts | 28 +++++++----
javascript/lib/types.ts | 42 +++++++++++------
7 files changed, 229 insertions(+), 71 deletions(-)
diff --git a/javascript/README.md b/javascript/README.md
index 244d840f9..1e5573885 100644
--- a/javascript/README.md
+++ b/javascript/README.md
@@ -23,7 +23,8 @@ Node.js bindings for the [Arrow Database Connectivity
(ADBC)](https://arrow.apac
Built on a native [NAPI](https://nodejs.org/api/n-api.html) addon — requires
Node.js 22+ and does not
support browser or Deno environments. Bun is not officially tested.
-**Alpha: APIs may change without notice.**
+> **Alpha.** APIs may change without notice.
+> If you try this and run into issues or have feedback, please [open an
issue](https://github.com/apache/arrow-adbc/issues).
## Installation
@@ -40,13 +41,13 @@ paths for a matching ADBC driver manifest or library.
```typescript
import { AdbcDatabase } from 'adbc-driver-manager'
-// Full path to a driver shared library
-const db = new AdbcDatabase({ driver: '/path/to/libadbc_driver_sqlite.dylib' })
+// Short name (resolves from system/user paths)
+const db = new AdbcDatabase({ driver: 'sqlite' })
```
```typescript
-// Short name (resolves from system/user paths)
-const db = new AdbcDatabase({ driver: 'sqlite' })
+// Or a full path to a driver shared library
+const db = new AdbcDatabase({ driver: '/path/to/libadbc_driver_sqlite.dylib' })
```
Once you have a database, open a connection and run queries:
@@ -54,25 +55,38 @@ Once you have a database, open a connection and run queries:
```typescript
const connection = await db.connect()
-// Execute a query and iterate Arrow RecordBatches
-const reader = await connection.query('SELECT 1 AS value')
-for await (const batch of reader) {
- console.log(batch.toArray())
-}
+// Execute a query — returns an Apache Arrow Table
+const table = await connection.query('SELECT 1 AS value')
+console.log(table.toArray())
-// Or use the lower-level statement API
-const stmt = await connection.createStatement()
-await stmt.setSqlQuery('SELECT 1 AS value')
-const result = await stmt.executeQuery()
-for await (const batch of result) {
+// For large result sets, stream record batches instead
+const reader = await connection.queryStream('SELECT * FROM large_table')
+for await (const batch of reader) {
console.log(`Received batch with ${batch.numRows} rows`)
}
-await stmt.close()
+
+// DML — returns the number of affected rows
+const affected = await connection.execute('DELETE FROM my_table WHERE id = 1')
await connection.close()
await db.close()
```
+For finer-grained control, use the statement API directly:
+
+```typescript
+import { tableFromArrays } from 'apache-arrow'
+
+const stmt = await connection.createStatement()
+await stmt.setSqlQuery('SELECT * FROM my_table WHERE id = ?')
+await stmt.bind(tableFromArrays({ id: [42] }))
+const reader = await stmt.executeQuery()
+for await (const batch of reader) {
+ console.log(batch.toArray())
+}
+await stmt.close()
+```
+
## Development
### Prerequisites
diff --git a/javascript/__test__/error_handling.spec.ts
b/javascript/__test__/error_handling.spec.ts
index bb671c96c..980099b20 100644
--- a/javascript/__test__/error_handling.spec.ts
+++ b/javascript/__test__/error_handling.spec.ts
@@ -114,3 +114,79 @@ test('error: unsupported option', () => {
assert.strictEqual(error.vendorCode, undefined)
assert.strictEqual(error.sqlState, undefined)
})
+
+test('error: conn.query() invalid SQL throws AdbcError', async () => {
+ await assert.rejects(
+ () => conn.query('SELECT * FROM'),
+ (e: unknown) => {
+ assert.ok(e instanceof AdbcError)
+ assert.match(e.message, /syntax error|incomplete input/i)
+ assert.strictEqual(e.code, 'InvalidArguments')
+ return true
+ },
+ )
+})
+
+test('error: conn.query() table not found throws AdbcError', async () => {
+ await assert.rejects(
+ () => conn.query('SELECT * FROM non_existent_table'),
+ (e: unknown) => {
+ assert.ok(e instanceof AdbcError)
+ assert.match(e.message, /no such table/i)
+ assert.strictEqual(e.code, 'InvalidArguments')
+ return true
+ },
+ )
+})
+
+test('error: conn.execute() invalid SQL throws AdbcError', async () => {
+ await assert.rejects(
+ () => conn.execute('INSERT INTO'),
+ (e: unknown) => {
+ assert.ok(e instanceof AdbcError)
+ assert.match(e.message, /syntax error|incomplete input/i)
+ assert.strictEqual(e.code, 'InvalidArguments')
+ return true
+ },
+ )
+})
+
+test('error: conn.execute() table not found throws AdbcError', async () => {
+ await assert.rejects(
+ () => conn.execute('INSERT INTO non_existent_table (id) VALUES (1)'),
+ (e: unknown) => {
+ assert.ok(e instanceof AdbcError)
+ assert.match(e.message, /no such table/i)
+ assert.strictEqual(e.code, 'InvalidArguments')
+ return true
+ },
+ )
+})
+
+test('error: conn.queryStream() invalid SQL throws AdbcError', async () => {
+ await assert.rejects(
+ () => conn.queryStream('SELECT * FROM'),
+ (e: unknown) => {
+ assert.ok(e instanceof AdbcError)
+ assert.match(e.message, /syntax error|incomplete input/i)
+ assert.strictEqual(e.code, 'InvalidArguments')
+ return true
+ },
+ )
+})
+
+test('error: conn.queryStream() error during iteration throws AdbcError',
async () => {
+ // Verifies that errors surfacing through the async iterator are wrapped as
AdbcError,
+ // not just errors thrown at query time.
+ let error: unknown
+ try {
+ const reader = await conn.queryStream('SELECT * FROM non_existent_table')
+ for await (const _ of reader) {
+ }
+ } catch (e) {
+ error = e
+ }
+
+ assert.ok(error instanceof AdbcError)
+ assert.match((error as AdbcError).message, /no such table/i)
+})
diff --git a/javascript/__test__/metadata.spec.ts
b/javascript/__test__/metadata.spec.ts
index 36c4e0498..38d0fcd4d 100644
--- a/javascript/__test__/metadata.spec.ts
+++ b/javascript/__test__/metadata.spec.ts
@@ -17,8 +17,9 @@
import { test, before, after } from 'node:test'
import assert from 'node:assert/strict'
-import { createSqliteDatabase, createTestTable, dumpReader } from
'./test_utils'
+import { createSqliteDatabase, createTestTable } from './test_utils'
import { AdbcDatabase, AdbcConnection, AdbcStatement } from '../lib/index.js'
+import { Table } from 'apache-arrow'
let db: AdbcDatabase
let conn: AdbcConnection
@@ -42,13 +43,14 @@ after(async () => {
})
test('metadata: getTableTypes', async () => {
- const tableTypes = await dumpReader(await conn.getTableTypes())
+ const table = await conn.getTableTypes()
+ assert.ok(table instanceof Table)
- // Sort actual results for consistent comparison
- tableTypes.sort((a, b) => (a.table_type || '').localeCompare(b.table_type ||
''))
+ const types = Array.from({ length: table.numRows }, (_, i) =>
table.getChild('table_type')?.get(i) as string)
+ types.sort()
- assert.strictEqual(tableTypes[0].table_type, 'table')
- assert.strictEqual(tableTypes[1].table_type, 'view')
+ assert.ok(types.includes('table'))
+ assert.ok(types.includes('view'))
})
test('metadata: getTableSchema', async () => {
@@ -62,21 +64,29 @@ test('metadata: getTableSchema', async () => {
test('metadata: getObjects', async () => {
// SQLite structure: Catalog (null/main) -> Schemas (null/main) -> Tables
- const objects = await dumpReader(
- await conn.getObjects({
- depth: 3,
- tableName: 'metadata_test',
- tableType: ['table', 'view'],
- }),
- )
+ const table = await conn.getObjects({
+ depth: 3,
+ tableName: 'metadata_test',
+ tableType: ['table', 'view'],
+ })
+ assert.ok(table instanceof Table)
+ assert.ok(table.numRows > 0)
- const tables = objects[0].catalog_db_schemas[0].db_schema_tables
- assert.ok(tables.some((t: { table_name: string }) => t.table_name ===
'metadata_test'))
+ // Navigate the nested Arrow structure: catalog -> db_schemas -> tables
+ const dbSchemas = table.getChild('catalog_db_schemas')?.get(0)
+ const dbTables = dbSchemas?.get(0)?.db_schema_tables
+ const tableNames = Array.from({ length: dbTables?.length ?? 0 }, (_, i) =>
dbTables?.get(i)?.table_name as string)
+ assert.ok(tableNames.includes('metadata_test'))
})
test('metadata: getInfo', async () => {
- const info = await dumpReader(await conn.getInfo())
+ const table = await conn.getInfo()
+ assert.ok(table instanceof Table)
+ assert.ok(table.numRows > 0)
- assert.strictEqual(info[0].info_name, 0)
- assert.strictEqual(info[0].info_value, 'SQLite')
+ // Find the VendorName row (info_name === 0)
+ const infoNames = table.getChild('info_name')
+ const vendorNameRow = Array.from({ length: table.numRows }, (_, i) =>
i).find((i) => infoNames?.get(i) === 0)
+ assert.ok(vendorNameRow !== undefined)
+ assert.strictEqual(table.getChild('info_value')?.get(vendorNameRow),
'SQLite')
})
diff --git a/javascript/__test__/profile.spec.ts
b/javascript/__test__/profile.spec.ts
index 640577374..85d17821b 100644
--- a/javascript/__test__/profile.spec.ts
+++ b/javascript/__test__/profile.spec.ts
@@ -21,7 +21,6 @@ import { mkdtempSync, writeFileSync, rmSync } from 'node:fs'
import { join, isAbsolute } from 'node:path'
import { tmpdir } from 'node:os'
import { AdbcDatabase } from '../lib/index.js'
-import { dumpReader } from './test_utils.js'
const testLib = process.env.ADBC_DRIVER_MANAGER_TEST_LIB
@@ -52,10 +51,10 @@ test('profile: load database from profile:// URI', async ()
=> {
})
const conn = await db.connect()
- const rows = await dumpReader(await conn.query('SELECT id, value FROM
profile_marker'))
- assert.strictEqual(rows.length, 1)
- assert.strictEqual(rows[0].id, 42n)
- assert.strictEqual(rows[0].value, 'from_profile')
+ const table = await conn.query('SELECT id, value FROM profile_marker')
+ assert.strictEqual(table.numRows, 1)
+ assert.strictEqual(table.getChild('id')?.get(0), 42n)
+ assert.strictEqual(table.getChild('value')?.get(0), 'from_profile')
await conn.close()
await db.close()
diff --git a/javascript/__test__/query.spec.ts
b/javascript/__test__/query.spec.ts
index f38675a5b..a53c5a944 100644
--- a/javascript/__test__/query.spec.ts
+++ b/javascript/__test__/query.spec.ts
@@ -19,7 +19,7 @@ import { test, before, after } from 'node:test'
import assert from 'node:assert/strict'
import { createSqliteDatabase, createTestTable, dumpReader } from
'./test_utils'
import { AdbcDatabase, AdbcConnection, AdbcStatement } from '../lib/index.js'
-import { tableFromArrays } from 'apache-arrow'
+import { Table, tableFromArrays } from 'apache-arrow'
let db: AdbcDatabase
let conn: AdbcConnection
@@ -59,9 +59,9 @@ test('query: SELECT returns correct rows', async () => {
}
assert.strictEqual(rows.length, 3)
- assert.strictEqual(rows[0].name, 'alice')
- assert.strictEqual(rows[1].name, 'bob')
- assert.strictEqual(rows[2].name, 'carol')
+ assert.deepStrictEqual(rows[0], { id: 1n, name: 'alice' })
+ assert.deepStrictEqual(rows[1], { id: 2n, name: 'bob' })
+ assert.deepStrictEqual(rows[2], { id: 3n, name: 'carol' })
})
test('query: executeUpdate returns affected row count', async () => {
@@ -71,21 +71,55 @@ test('query: executeUpdate returns affected row count',
async () => {
const affected = await stmt.executeUpdate()
assert.strictEqual(typeof affected, 'number')
assert.strictEqual(affected, 1)
+
+ // Verify the change was applied
+ const table = await conn.query('SELECT id, name FROM query_test WHERE id =
1')
+ assert.strictEqual(table.numRows, 1)
+ assert.strictEqual(table.getChild('id')?.get(0), 1n)
+ assert.strictEqual(table.getChild('name')?.get(0), 'updated')
} finally {
await stmt.close()
}
})
-test('query: conn.query() returns correct rows', async () => {
+test('query: conn.query() returns an Arrow Table', async () => {
// id=2 (bob) is never mutated by other tests in this file
- const rows = await dumpReader(await conn.query('SELECT id, name FROM
query_test WHERE id = 2'))
+ const table = await conn.query('SELECT id, name FROM query_test WHERE id =
2')
+ assert.ok(table instanceof Table)
+ assert.strictEqual(table.numCols, 2)
+ assert.strictEqual(table.numRows, 1)
+ assert.strictEqual(table.getChild('id')?.get(0), 2n)
+ assert.strictEqual(table.getChild('name')?.get(0), 'bob')
+})
+
+test('query: conn.query() with bound params', async () => {
+ // id=2 (bob) is never mutated by other tests in this file
+ const params = tableFromArrays({ id: [2] })
+ const table = await conn.query('SELECT id, name FROM query_test WHERE id =
?', params)
+ assert.ok(table instanceof Table)
+ assert.strictEqual(table.numRows, 1)
+ assert.strictEqual(table.getChild('id')?.get(0), 2n)
+ assert.strictEqual(table.getChild('name')?.get(0), 'bob')
+})
+
+test('query: conn.queryStream() returns a RecordBatchReader', async () => {
+ // id=2 (bob) is never mutated by other tests in this file
+ const reader = await conn.queryStream('SELECT id, name FROM query_test WHERE
id = 2')
+ const rows = await dumpReader(reader)
assert.strictEqual(rows.length, 1)
+ assert.strictEqual(rows[0].id, 2n)
assert.strictEqual(rows[0].name, 'bob')
})
test('query: conn.execute() returns affected row count', async () => {
const affected = await conn.execute(`UPDATE query_test SET name =
'via_execute' WHERE id = 3`)
assert.strictEqual(affected, 1)
+
+ // Verify the change was applied
+ const table = await conn.query('SELECT id, name FROM query_test WHERE id =
3')
+ assert.strictEqual(table.numRows, 1)
+ assert.strictEqual(table.getChild('id')?.get(0), 3n)
+ assert.strictEqual(table.getChild('name')?.get(0), 'via_execute')
})
test('query: conn.execute() with bound params inserts a row', async () => {
@@ -93,9 +127,10 @@ test('query: conn.execute() with bound params inserts a
row', async () => {
const affected = await conn.execute('INSERT INTO query_test (id, name)
VALUES (?, ?)', params)
assert.strictEqual(affected, 1)
- const rows = await dumpReader(await conn.query('SELECT name FROM query_test
WHERE id = 99'))
- assert.strictEqual(rows.length, 1)
- assert.strictEqual(rows[0].name, 'bound_insert')
+ const table = await conn.query('SELECT id, name FROM query_test WHERE id =
99')
+ assert.strictEqual(table.numRows, 1)
+ assert.strictEqual(table.getChild('id')?.get(0), 99n)
+ assert.strictEqual(table.getChild('name')?.get(0), 'bound_insert')
})
test('query: empty result set', async () => {
diff --git a/javascript/lib/index.ts b/javascript/lib/index.ts
index 181a9f82d..0215b104a 100644
--- a/javascript/lib/index.ts
+++ b/javascript/lib/index.ts
@@ -34,6 +34,14 @@ const asyncDisposeSymbol = (Symbol as any).asyncDispose ??
Symbol('Symbol.asyncD
type NativeIterator = { next(): Promise<Buffer | null | undefined>; close():
void }
+async function readerToTable(reader: RecordBatchReader): Promise<Table> {
+ const batches: RecordBatch[] = []
+ for await (const batch of reader) {
+ batches.push(batch)
+ }
+ return new Table(batches)
+}
+
/**
* Converts the native result iterator into an Apache Arrow
`RecordBatchReader`.
*
@@ -141,7 +149,7 @@ export class AdbcConnection implements
AdbcConnectionInterface {
this.setOption('adbc.connection.read_only', enabled ? 'true' : 'false')
}
- async getObjects(options?: GetObjectsOptions): Promise<RecordBatchReader> {
+ async getObjects(options?: GetObjectsOptions): Promise<Table> {
try {
const opts = {
depth: options?.depth ?? 0,
@@ -152,7 +160,7 @@ export class AdbcConnection implements
AdbcConnectionInterface {
columnName: options?.columnName,
}
const iterator = await this._inner.getObjects(opts)
- return iteratorToReader(iterator as NativeIterator)
+ return readerToTable(await iteratorToReader(iterator as NativeIterator))
} catch (e) {
throw AdbcError.fromError(e)
}
@@ -171,25 +179,29 @@ export class AdbcConnection implements
AdbcConnectionInterface {
}
}
- async getTableTypes(): Promise<RecordBatchReader> {
+ async getTableTypes(): Promise<Table> {
try {
const iterator = await this._inner.getTableTypes()
- return iteratorToReader(iterator as NativeIterator)
+ return readerToTable(await iteratorToReader(iterator as NativeIterator))
} catch (e) {
throw AdbcError.fromError(e)
}
}
- async getInfo(infoCodes?: InfoCode[]): Promise<RecordBatchReader> {
+ async getInfo(infoCodes?: InfoCode[]): Promise<Table> {
try {
const iterator = await this._inner.getInfo(infoCodes)
- return iteratorToReader(iterator as NativeIterator)
+ return readerToTable(await iteratorToReader(iterator as NativeIterator))
} catch (e) {
throw AdbcError.fromError(e)
}
}
- async query(sql: string, params?: RecordBatch | Table):
Promise<RecordBatchReader> {
+ async query(sql: string, params?: Table): Promise<Table> {
+ return readerToTable(await this.queryStream(sql, params))
+ }
+
+ async queryStream(sql: string, params?: Table): Promise<RecordBatchReader> {
const stmt = await this.createStatement()
try {
await stmt.setSqlQuery(sql)
@@ -202,7 +214,7 @@ export class AdbcConnection implements
AdbcConnectionInterface {
}
}
- async execute(sql: string, params?: RecordBatch | Table): Promise<number> {
+ async execute(sql: string, params?: Table): Promise<number> {
const stmt = await this.createStatement()
try {
await stmt.setSqlQuery(sql)
diff --git a/javascript/lib/types.ts b/javascript/lib/types.ts
index bf506d12b..524d98d73 100644
--- a/javascript/lib/types.ts
+++ b/javascript/lib/types.ts
@@ -73,7 +73,7 @@ export type ObjectDepth = (typeof ObjectDepth)[keyof typeof
ObjectDepth]
* Pass a subset to `getInfo()` to retrieve only specific metadata fields.
*
* @example
- * const reader = await conn.getInfo([InfoCode.VendorName,
InfoCode.DriverVersion])
+ * const table = await conn.getInfo([InfoCode.VendorName,
InfoCode.DriverVersion])
*/
export const InfoCode = {
/** The database vendor/product name (string). */
@@ -225,9 +225,9 @@ export interface AdbcConnection {
* Get a hierarchical view of database objects (catalogs, schemas, tables,
columns).
*
* @param options Filtering options for the metadata query.
- * @returns A RecordBatchReader containing the metadata.
+ * @returns A Promise resolving to an Apache Arrow Table containing the
metadata.
*/
- getObjects(options?: GetObjectsOptions): Promise<RecordBatchReader>
+ getObjects(options?: GetObjectsOptions): Promise<Table>
/**
* Get the Arrow schema for a specific table.
@@ -243,32 +243,44 @@ export interface AdbcConnection {
/**
* Get a list of table types supported by the database.
*
- * @returns A RecordBatchReader containing a single string column of table
types.
+ * @returns A Promise resolving to an Apache Arrow Table with a single
string column of table types.
*/
- getTableTypes(): Promise<RecordBatchReader>
+ getTableTypes(): Promise<Table>
/**
* Get metadata about the driver and database.
*
* @param infoCodes Optional list of info codes to retrieve. Use the {@link
InfoCode} constants.
* If omitted, all available info is returned.
- * @returns A RecordBatchReader containing the requested metadata info.
+ * @returns A Promise resolving to an Apache Arrow Table containing the
requested metadata info.
*/
- getInfo(infoCodes?: InfoCode[]): Promise<RecordBatchReader>
+ getInfo(infoCodes?: InfoCode[]): Promise<Table>
/**
- * Execute a SQL query and return the results as a RecordBatchReader.
+ * Execute a SQL query and return all results as an Arrow Table.
*
* Convenience method that creates a statement, sets the SQL, optionally
binds
- * parameters, executes the query, and closes the statement. The reader
remains
- * valid after the statement is closed because the underlying iterator holds
its
- * own reference to the native resources.
+ * parameters, executes the query, and closes the statement.
+ * For large result sets, use {@link queryStream} to avoid loading
everything into memory.
*
* @param sql The SQL query to execute.
- * @param params Optional Arrow RecordBatch or Table to bind as parameters.
+ * @param params Optional Arrow Table to bind as parameters.
+ * @returns A Promise resolving to an Apache Arrow Table.
+ */
+ query(sql: string, params?: Table): Promise<Table>
+
+ /**
+ * Execute a SQL query and return the results as a RecordBatchReader for
streaming.
+ *
+ * Use this instead of {@link query} when working with large result sets
that should
+ * not be fully loaded into memory. The reader remains valid after the
statement is
+ * closed because the underlying iterator holds its own reference to the
native resources.
+ *
+ * @param sql The SQL query to execute.
+ * @param params Optional Arrow Table to bind as parameters.
* @returns A Promise resolving to an Apache Arrow RecordBatchReader.
*/
- query(sql: string, params?: RecordBatch | Table): Promise<RecordBatchReader>
+ queryStream(sql: string, params?: Table): Promise<RecordBatchReader>
/**
* Execute a SQL statement (INSERT, UPDATE, DELETE, DDL) and return the row
count.
@@ -277,10 +289,10 @@ export interface AdbcConnection {
* parameters, executes the update, and closes the statement.
*
* @param sql The SQL statement to execute.
- * @param params Optional Arrow RecordBatch or Table to bind as parameters.
+ * @param params Optional Arrow Table to bind as parameters.
* @returns A Promise resolving to the number of rows affected, or -1 if
unknown.
*/
- execute(sql: string, params?: RecordBatch | Table): Promise<number>
+ execute(sql: string, params?: Table): Promise<number>
/**
* Commit any pending transactions.
