Yesterday, we followed a query from SQL to worker tasks and back. Now let's put a switch in front of that path.

That sounds like a small thing. But suppose I load the extension to try one large Parquet query. Should every query on that connection suddenly need a worker? What about a query over a temporary table that only exists in the application?

I'd like to keep using DuckDB normally and opt into distribution when it makes sense. So the first rule is simple: loading the extension should leave local execution available, with distribution disabled by default.

Two different kinds of optional

There are actually two decisions here. When building the extension, do we include the networking code? When running a query, do we allow that code to participate?

Our transport uses Arrow Flight, whose C++ implementation uses gRPC. Linking it brings dependencies into the extension before any SQL runs. A runtime setting cannot remove those dependencies from the binary. The Arrow Flight documentation describes the client and server APIs this dependency provides.

For a build that can omit distributed execution, the build definition needs to group the pieces together. Here's the intended structure, in pseudocode:

build the extension's local entry point
if distributed support is included:
    find Arrow Flight and its dependencies
    compile the transport, worker, and distributed planner code
    link the networking libraries
    register the distributed functionality

Conditionally linking Flight while still compiling files that include its headers won't work. Neither will removing those files while leaving calls to their registration functions. Dependency discovery, compilation, linking, and registration have to agree.

We can start with a build that includes Flight and still make execution opt-in. That's the smaller first step. A build without Flight is a separate capability to implement and verify; setting a boolean to false doesn't establish it.

Start with two settings

At runtime, I only need two things to get started: permission to distribute and a list of workers.

These are extension-defined names. Ordinary DuckDB won't recognize them until an extension registers them.

DuckDB already has a configuration system with SET, RESET, and setting inspection, so we can put our options there. In the C++ API used for this experiment, AddExtensionOption registers a name, description, type, and default. The optimizer callback reads the effective values through the current query's ClientContext, using TryGetCurrentSetting. That avoids inventing a separate configuration file for the first two options. DuckDB's configuration documentation explains the SQL side.

Enabling distribution means “consider it.” We still need to check whether the query is supported. And setting worker addresses should only store configuration; it shouldn't start connections or send work.

A worker address needs a contract

Let's use addresses such as grpc://127.0.0.1:8815 for the first local experiment. The scheme selects the transport, the host identifies the machine, and the port identifies the listening service. Loopback refers to the current machine, so remote workers will need different hostnames or addresses.

The parser's job is to turn configuration text into usable endpoints, or explain why it can't. It should not determine whether a worker is alive.

I'd keep the accepted format narrow at first:

Trim whitespace around entries, but don't remove spaces inside a hostname to make it look valid. A nonempty list with a blank entry should report an error. Parse the complete list before accepting it, so a typo doesn't quietly drop a worker. For this first contract, I'd also reject duplicate normalized endpoints rather than accidentally treating repetition as extra capacity.

An existing URI parser can handle the structure, especially IPv6 brackets. We still need to enforce our scheme, explicit port, and absence of user information, paths, queries, or fragments. The table defines the behavior we want to check when wiring that parser into the extension.

Leave the plan alone until we can replace it

The optimizer callback now has a small decision to make before it tries to split anything:

Optimized DuckDB plan
    │
    ├─ Distribution disabled? ───────────────→ Original plan
    ├─ No configured workers? ───────────────→ Original plan
    ├─ Insertion order must be preserved? ───→ Original plan
    ├─ Query unsupported? ───────────────────→ Original plan
    │
    └─ Build and validate a candidate split
           ├─ Cannot represent it safely ───→ Original plan
           └─ Valid ────────────────────────→ Distributed plan

Our network source consumes whichever stream is ready, so this design requires preserve_insertion_order to be false. We shouldn't silently change it for the user. A final ORDER BY still requests an explicit result order, as in yesterday's query.

Keeping the original plan means avoiding destructive edits before we know the replacement works. Otherwise, a serialization failure might leave us trying to execute half a rewrite. That is a surprisingly expensive way to implement “optional.”

These decisions need no worker handshake. Ordinary planning and EXPLAIN should not register remote tasks. Reading metadata to plan a remote Parquet scan can still involve storage I/O; the boundary here is worker execution.

Try the decision before adding the network

We can exercise the decision table in an ordinary DuckDB SQL session. This models the gates; it doesn't install settings or implement the optimizer callback. The inputs assume endpoint validation has already succeeded.

WITH cases(label, enabled, workers, preserve_order, supported) AS (
    VALUES
        ('disabled',    false, 2, false, true),
        ('no workers',  true,  0, false, true),
        ('keep order',  true,  2, true,  true),
        ('unsupported', true,  2, false, false),
        ('eligible',    true,  2, false, true)
)
SELECT label,
       CASE
           WHEN NOT enabled OR workers = 0
                OR preserve_order OR NOT supported
           THEN 'local'
           ELSE 'candidate'
       END AS planning_path
FROM cases
ORDER BY label;

I ran this and got:

label        planning_path
disabled     local
eligible     candidate
keep order   local
no workers   local
unsupported  local

Only one case reaches candidate construction. Change any one of its gates and it goes back to local execution. This checks the policy in isolation; checking the actual extension will also require proving that rejected candidates leave the plan intact and make no worker calls.

An unreachable configured worker is different from an empty list. Once execution starts, a failed connection should fail the query, as we discussed yesterday. Silently ignoring that address would hide a broken deployment behind a query that happened to finish locally.

We now have a place for configuration, an endpoint contract, and a clear decision before plan rewriting. Next comes something the address alone can't tell us: whether the worker can understand the plan we're about to send. That brings the build back into the conversation.