neosqlite.collection.temporary_table_aggregation.operators_lookup module

class neosqlite.collection.temporary_table_aggregation.operators_lookup.OperatorsLookupMixin

Bases: OperatorsBaseMixin

_create_lookup_hash_table(from_collection: str, foreign_field: str | None, pipeline: list[dict[str, Any]] | None = None) → tuple[str, str]

Create a hash table (temp table with index) from a foreign collection for efficient hash join lookup.

This implements O(n+m) hash join instead of O(n*m) correlated subquery.

Parameters:

• from_collection – The collection to build hash table from

• foreign_field – The field to use as join key (None for _id)

• pipeline – Optional pipeline to run on foreign collection first

Returns:

Tuple of (hash_table_name, join_key_column)

_create_lookup_hash_table_fallback(hash_table_name: str, from_collection: str, foreign_field: str, join_key: str) → None

Fallback method to create hash table by reading documents one by one.

Used when the SQL approach fails due to malformed JSON in the data column. This method skips corrupted documents gracefully.

Parameters:

• hash_table_name – Name of the hash table to create

• from_collection – Source collection name

• foreign_field – Field to use as join key

• join_key – Name of the join key column

_estimate_collection_size(collection_name: str) → int

Estimate the size of a collection in bytes.

Uses SQLite’s table statistics to estimate size.

Parameters:

collection_name – Name of the collection to estimate

Returns:

Estimated size in bytes

_get_available_memory() → int

Get available memory for hash join operations.

Returns:

Available memory in bytes (estimated from SQLite cache or system)

_should_use_hash_join(from_collection: str, pipeline: list[dict[str, Any]] | None = None) → bool

Decide whether to use hash join or correlated subquery for $lookup.

Uses memory estimation to decide: - If foreign collection estimate < 30% of available memory: use hash join (faster) - Otherwise: use correlated subquery (lower memory, slower)

Parameters:

• from_collection – The foreign collection name

• pipeline – Optional pipeline to run on foreign collection first

Returns:

True if should use hash join, False for correlated subquery

_extract_field_value(doc: dict[str, Any], field: str) → Any

Extract field value from document, supporting dot notation.

_process_lookup_stage(create_temp: Callable, current_table: str, lookup_spec: dict[str, Any]) → str

Process a $lookup stage using hash join for O(n+m) complexity.

This method implements the $lookup aggregation stage which performs a left outer join to another collection in the same database. It uses an optimized hash join approach: 1. Creates a temporary table with an index on the foreign field (hash table) 2. Uses a single JOIN query instead of correlated subquery

This replaces the previous correlated subquery approach which was O(n*m).

Parameters:

• create_temp (Callable) – Function to create temporary tables

• current_table (str) – Name of the current temporary table containing input data

• lookup_spec (dict[str, Any]) – The $lookup stage specification containing: - “from”: The name of the collection to join with - “localField”: The field from the input documents - “foreignField”: The field from the documents of the “from” collection - “as”: The name of the new array field to add to the matching documents - “pipeline”: Optional pipeline to run on foreign collection

Returns:

Name of the newly created temporary table with lookup results added

Return type:

str

_process_lookup_correlated_subquery(create_temp: Callable, current_table: str, lookup_spec: dict[str, Any]) → str

Process $lookup using correlated subquery (O(n*m) but low memory).

This is the fallback when the foreign collection is too large for hash join.

Parameters:

• create_temp – Function to create temporary tables

• current_table – Current temp table name

• lookup_spec – The $lookup specification

Returns:

Name of the new temporary table

_process_lookup_hash_join(create_temp: Callable, current_table: str, lookup_spec: dict[str, Any]) → str

Process $lookup using hash join (O(n+m) but uses more memory).

Parameters:

• create_temp – Function to create temporary tables

• current_table – Current temp table name

• lookup_spec – The $lookup specification

Returns:

Name of the new temporary table