neosqlite.collection.expr_evaluator package

Submodules

• neosqlite.collection.expr_evaluator.constants module
  • _RemoveSentinel
    • _RemoveSentinel._instance
• neosqlite.collection.expr_evaluator.context module
  • AggregationContext
    • AggregationContext.variables
    • AggregationContext.stage_index
    • AggregationContext.current_field
    • AggregationContext.pipeline_id
    • AggregationContext.__init__()
    • AggregationContext.bind_document()
    • AggregationContext.update_current()
    • AggregationContext.get_variable()
    • AggregationContext.set_variable()
    • AggregationContext.clone()
  • _is_aggregation_variable()
• neosqlite.collection.expr_evaluator.python_evaluators module
  • PythonEvaluatorsMixin
    • PythonEvaluatorsMixin._log2_warned
    • PythonEvaluatorsMixin.evaluate_python()
    • PythonEvaluatorsMixin._evaluate_expr_python()
    • PythonEvaluatorsMixin._evaluate_logical_python()
    • PythonEvaluatorsMixin._evaluate_comparison_python()
    • PythonEvaluatorsMixin._evaluate_cmp_python()
    • PythonEvaluatorsMixin._evaluate_arithmetic_python()
    • PythonEvaluatorsMixin._evaluate_math_python()
    • PythonEvaluatorsMixin._evaluate_pow_python()
    • PythonEvaluatorsMixin._evaluate_sqrt_python()
    • PythonEvaluatorsMixin._evaluate_trig_python()
    • PythonEvaluatorsMixin._evaluate_angle_python()
    • PythonEvaluatorsMixin._evaluate_cond_python()
    • PythonEvaluatorsMixin._evaluate_ifNull_python()
    • PythonEvaluatorsMixin._evaluate_switch_python()
    • PythonEvaluatorsMixin._evaluate_array_python()
    • PythonEvaluatorsMixin._evaluate_array_transform_python()
    • PythonEvaluatorsMixin._evaluate_string_python()
    • PythonEvaluatorsMixin._evaluate_date_python()
    • PythonEvaluatorsMixin._evaluate_date_arithmetic_python()
    • PythonEvaluatorsMixin._evaluate_object_python()
    • PythonEvaluatorsMixin._evaluate_data_size_python()
    • PythonEvaluatorsMixin._evaluate_type_python()
    • PythonEvaluatorsMixin._convert_to_int()
    • PythonEvaluatorsMixin._convert_to_long()
    • PythonEvaluatorsMixin._convert_to_double()
    • PythonEvaluatorsMixin._convert_to_decimal()
    • PythonEvaluatorsMixin._convert_to_string()
    • PythonEvaluatorsMixin._convert_to_bool()
    • PythonEvaluatorsMixin._convert_to_objectid()
    • PythonEvaluatorsMixin._convert_to_bindata()
    • PythonEvaluatorsMixin._convert_to_bsonbindata()
    • PythonEvaluatorsMixin._convert_to_regex()
    • PythonEvaluatorsMixin._convert_to_bsonregex()
    • PythonEvaluatorsMixin._convert_to_date()
    • PythonEvaluatorsMixin._convert_to_null()
    • PythonEvaluatorsMixin._get_bson_type()
    • PythonEvaluatorsMixin._evaluate_literal_python()
    • PythonEvaluatorsMixin._evaluate_operand_python()
• neosqlite.collection.expr_evaluator.sql_converters module
  • SqlConvertersMixin
    • SqlConvertersMixin.data_column
    • SqlConvertersMixin._jsonb_supported
    • SqlConvertersMixin._log2_warned
    • SqlConvertersMixin._current_context
    • SqlConvertersMixin._convert_expr_to_sql()
    • SqlConvertersMixin._convert_logical_operator()
    • SqlConvertersMixin._convert_comparison_operator()
    • SqlConvertersMixin._convert_cmp_operator()
    • SqlConvertersMixin._convert_arithmetic_operator()
    • SqlConvertersMixin._convert_cond_operator()
    • SqlConvertersMixin._convert_ifNull_operator()
    • SqlConvertersMixin._convert_array_operator()
    • SqlConvertersMixin._convert_set_operator()
    • SqlConvertersMixin._build_pattern_with_options()
    • SqlConvertersMixin._convert_string_operator()
    • SqlConvertersMixin._convert_math_operator()
    • SqlConvertersMixin._convert_trig_operator()
    • SqlConvertersMixin._convert_angle_operator()
    • SqlConvertersMixin._convert_date_operator()
    • SqlConvertersMixin._convert_date_arithmetic_operator()
    • SqlConvertersMixin._convert_date_diff_operator()
    • SqlConvertersMixin._convert_object_operator()
    • SqlConvertersMixin._convert_let_operator()
    • SqlConvertersMixin._convert_data_size_operator()
    • SqlConvertersMixin._get_operator_return_type()
    • SqlConvertersMixin._get_literal_bson_type()
    • SqlConvertersMixin._convert_type_operator()
    • SqlConvertersMixin._convert_operand_to_sql()
    • SqlConvertersMixin._map_comparison_operator()
    • SqlConvertersMixin._map_arithmetic_operator()
• neosqlite.collection.expr_evaluator.type_utils module
  • _convert_to_int()
  • _convert_to_long()
  • _convert_to_double()
  • _convert_to_decimal()
  • _convert_to_string()
  • _convert_to_bool()
  • _convert_to_objectid()
  • _convert_to_bindata()
  • _convert_to_bsonbindata()
  • _convert_to_regex()
  • _convert_to_bsonregex()
  • _convert_to_date()
  • _convert_to_null()
  • get_bson_type()

Module contents

$expr operator evaluator for NeoSQLite.

This module implements the MongoDB $expr operator using a 3-tier approach: 1. Single SQL Query (fastest) - Uses SQLite JSON functions 2. Temporary Tables (intermediate) - For complex expressions 3. Python Fallback (slowest but complete) - Always available, especially for kill switch

The evaluator ensures that SQLite and Python implementations produce identical results.

MongoDB $expr Compatibility: - Comparison operators: $eq, $ne, $gt, $gte, $lt, $lte, $cmp - Logical operators: $and, $or, $not, $nor - Arithmetic operators: $add, $subtract, $multiply, $divide, $mod, $abs, $ceil, $floor, $round, $trunc - Conditional operators: $cond, $ifNull, $switch - Array operators: $size, $in, $arrayElemAt, $first, $last, $isArray - Array aggregation: $sum, $avg, $min, $max - Array transformation: $filter, $map, $reduce - String operators: $concat, $toLower, $toUpper, $strLenBytes, $substr, $trim - String regex: $regexMatch, $regexFind, $regexFindAll - Date operators: $year, $month, $dayOfMonth, $hour, $minute, $second, $dayOfWeek, $dayOfYear - Date arithmetic: $dateAdd, $dateSubtract, $dateDiff - Type operators: $type, $convert, $toString, $toInt, $toDouble, $toBool - Object operators: $mergeObjects, $getField, $setField - Other: $literal, $let - Trigonometric: $sin, $cos, $tan, $asin, $acos, $atan, $atan2 - Hyperbolic: $sinh, $cosh, $tanh, $asinh, $acosh, $atanh - Logarithmic: $ln, $log, $log10, $log2 - Exponential/Sigmoid: $exp, $sigmoid - Angle conversion: $degreesToRadians, $radiansToDegrees

Note: NeoSQLite extends MongoDB with $log2 (base-2 log) operator.

class neosqlite.collection.expr_evaluator.ExprEvaluator(data_column: str = 'data', db_connection=None)

Bases: SqlConvertersMixin, PythonEvaluatorsMixin

Evaluator for MongoDB $expr operator.

Supports 3-tier evaluation: - Tier 1: Direct SQL WHERE clause using JSON functions - Tier 2: Temporary tables for complex expressions - Tier 3: Python fallback (always available for kill switch)

JSON/JSONB Support: - Automatically uses jsonb_* functions when supported for better performance - Falls back to json_* functions when JSONB is not available - Detects SQLite 3.51.0+ features (jsonb_each, jsonb_tree) for maximum performance

__init__(data_column: str = 'data', db_connection=None)

Initialize the expression evaluator.

Parameters:

• data_column – Name of the column containing JSON data (default: “data”)

• db_connection – Optional SQLite database connection for JSONB detection. If provided, JSONB support will be auto-detected. If None, json_* functions will be used (safe fallback).

property json_function_prefix: str

Get the appropriate JSON function prefix (json or jsonb).

property json_each_function: str

Get the appropriate json_each function name (json_each or jsonb_each).

property json_group_array_function: str

Get the appropriate json_group_array function name.

evaluate(expr: dict[str, Any], tier: int = 1, force_python: bool = False) → tuple[str | None, list[Any]]

Evaluate a $expr expression.

Parameters:

• expr – The $expr expression dictionary

• tier – Complexity tier (1=SQL, 2=TempTable, 3=Python)

• force_python – Force Python evaluation (kill switch)

Returns:

Tuple of (SQL WHERE clause, parameters) or (None, []) for Python evaluation

_evaluate_sql_tier1(expr: dict[str, Any]) → tuple[str | None, list[Any]]

Tier 1: Convert simple expressions to SQL WHERE clauses using JSON functions.

Supports basic operators and field comparisons.

_evaluate_sql_tier2(expr: dict[str, Any]) → tuple[str | None, list[Any]]

Tier 2: Use temporary tables for complex expressions.

This tier is used when: - Expressions are too complex for Tier 1 (single SQL WHERE clause) - Multiple intermediate calculations are needed - The expression can benefit from pre-computed field extractions

Parameters:

expr – The $expr expression

Returns:

Tuple of (SQL expression, parameters) or (None, []) for Python fallback

evaluate_for_aggregation(expr: Any, context: AggregationContext | None = None, as_alias: str | None = None) → tuple[str, list[Any]]

Evaluate expression for aggregation pipeline.

This method evaluates expressions for use in aggregation pipeline stages like $addFields, $project, $group, etc. Unlike the evaluate() method which generates WHERE clause expressions, this method generates SELECT clause expressions with optional aliases.

Parameters:

• expr – Expression to evaluate. Can be: - Dict: Expression like {“$sin”: “$angle”} - Str: Field reference like “$field” or variable like “$$ROOT” - Literal: Number, string, boolean, None, array, or dict

• context – Aggregation context for variable scoping. If None, a new context will be created.

• as_alias – Optional alias for SELECT clause (e.g., “AS field_name”)

Returns:

Tuple of (SQL expression, parameters). The SQL expression will include the alias if as_alias is provided.

Raises:

NotImplementedError – If the expression operator is not supported in SQL tier for aggregation context

Examples

>>> evaluator = ExprEvaluator()
>>> evaluator.evaluate_for_aggregation({"$sin": "$angle"})
("sin(json_extract(data, '$.angle'))", [])

>>> evaluator.evaluate_for_aggregation({"$sin": "$angle"}, as_alias="sin_val")
("sin(json_extract(data, '$.angle')) AS sin_val", [])

>>> evaluator.evaluate_for_aggregation("$field")
("json_extract(data, '$.field')", [])

>>> evaluator.evaluate_for_aggregation(42)
("?", [42])

_convert_operand_to_sql_agg(operand: Any, context: AggregationContext) → tuple[str, list[Any]]

Convert an operand to SQL for aggregation context.

This method handles different types of operands: - Expressions: Recursively evaluate using _convert_expr_to_sql - Field references: Convert to json_extract calls - Aggregation variables: Handle $$ROOT, $$CURRENT, etc. - Literals: Convert to parameterized SQL

Parameters:

• operand – The operand to convert

• context – Aggregation context for variable scoping

Returns:

Tuple of (SQL expression, parameters)

Raises:

NotImplementedError – If the operand type is not supported

_handle_aggregation_variable(var_name: str, context: AggregationContext) → tuple[str, list[Any]]

Handle aggregation variable references.

Parameters:

• var_name – Variable name (e.g., “$$ROOT”, “$$CURRENT”)

• context – Aggregation context

Returns:

Tuple of (SQL expression, parameters)

Raises:

NotImplementedError – If the variable is not supported

build_select_expression(expr: Any, alias: str | None = None, context: AggregationContext | None = None) → tuple[str, list[Any]]

Build SELECT clause expression for aggregation (SQL Tier 1 optimized).

This method is similar to evaluate_for_aggregation() but optimized for SQL tier usage. It handles field aliasing and context tracking for multi-stage pipelines.

Parameters:

• expr – Expression to evaluate. Can be: - Dict: Expression like {“$sin”: “$angle”} - Str: Field reference like “$field” or variable like “$$ROOT” - Literal: Number, string, boolean, None, array, or dict

• alias – Optional alias for SELECT clause (e.g., “AS field_name”)

• context – Aggregation context for variable scoping. If None, creates a new context. This allows tracking computed fields across pipeline stages.

Returns:

Tuple of (SQL expression, parameters). The SQL expression will include the alias if alias is provided.

Raises:

NotImplementedError – If the expression operator is not supported in SQL tier for aggregation context

Examples

>>> evaluator = ExprEvaluator()
>>> evaluator.build_select_expression({"$sin": "$angle"})
("sin(json_extract(data, '$.angle'))", [])

>>> evaluator.build_select_expression({"$sin": "$angle"}, alias="sin_val")
("sin(json_extract(data, '$.angle')) AS sin_val", [])

>>> evaluator.build_select_expression("$field")
("json_extract(data, '$.field')", [])

>>> evaluator.build_select_expression(42)
("?", [42])

build_group_by_expression(expr: Any, context: AggregationContext | None = None) → tuple[str, list[Any]]

Build GROUP BY clause expression for aggregation (SQL Tier 1 optimized).

This method is optimized for grouping operations. It’s similar to build_select_expression() but doesn’t include aliases since GROUP BY clauses reference expressions directly.

Parameters:

• expr – Expression to evaluate for GROUP BY

• context – Aggregation context for variable scoping

Returns:

Tuple of (SQL expression, parameters)

Examples

>>> evaluator.build_group_by_expression("$category")
("json_extract(data, '$.category')", [])

>>> evaluator.build_group_by_expression({"$toLower": "$category"})
("lower(json_extract(data, '$.category'))", [])

build_having_expression(expr: Any, context: AggregationContext | None = None) → tuple[str, list[Any]]

Build HAVING clause expression for post-aggregation filtering.

HAVING expressions are evaluated after aggregation, so they can reference aggregate results.

Parameters:

• expr – Expression to evaluate for HAVING clause

• context – Aggregation context for variable scoping

Returns:

Tuple of (SQL expression, parameters)

Examples

>>> evaluator.build_having_expression({"$gt": ["$total", 100]})
("(json_extract(data, '$.total') > ?)", [100])

_handle_aggregation_variable_sql_tier(var_name: str, context: PipelineContext | None = None) → str

Handle aggregation variable references for SQL tier.

This is a SQL-tier-specific version that works with PipelineContext instead of AggregationContext.

Parameters:

• var_name – Variable name (e.g., “$$ROOT”, “$$CURRENT”)

• context – Pipeline context for tracking fields

Returns:

SQL expression string

Raises:

NotImplementedError – If the variable is not supported

class neosqlite.collection.expr_evaluator.AggregationContext

Bases: object

Manages variable scoping for aggregation expressions.

Aggregation expressions have different variable contexts than query expressions. This class manages the lifecycle of aggregation variables like $$ROOT, $$CURRENT, and $$REMOVE throughout pipeline execution.

variables

Dictionary mapping variable names to their values

stage_index

Current stage index in the pipeline

current_field

Name of the field being computed (for context)

pipeline_id

Unique identifier for the pipeline (for temp table correlation)

__init__() → None

Initialize aggregation context with default variables.

bind_document(doc: dict[str, Any]) → None

Bind document to context.

Called at the start of pipeline execution to initialize $$ROOT and $$CURRENT with the input document.

Parameters:

doc – The document to bind

update_current(doc: dict[str, Any]) → None

Update current document after stage processing.

Called after each stage that modifies the document to update the $$CURRENT variable.

Parameters:

doc – The updated document

get_variable(name: str) → Any

Get variable value.

Parameters:

name – Variable name (e.g., “$$ROOT”, “$$CURRENT”)

Returns:

Variable value or None if not found

set_variable(name: str, value: Any) → None

Set variable value.

Parameters:

• name – Variable name

• value – Value to set

clone() → AggregationContext

Create a shallow copy of the context for nested scoping.

Returns:

A copy of the context.

Return type:

AggregationContext

class neosqlite.collection.expr_evaluator._RemoveSentinel

Bases: object

Sentinel value for $$REMOVE in $project stage.

When a field is set to this value, it should be removed from the output document. This is a singleton pattern - only one instance should exist.

_instance = $$REMOVE

neosqlite.collection.expr_evaluator._is_expression(value: Any) → bool

Check if value is an aggregation expression.

An expression is a dict with exactly one key starting with ‘$’ that is not a reserved field name.

Parameters:

value – Value to check

Returns:

True if value is an expression, False otherwise

Examples

>>> _is_expression({"$sin": "$angle"})
True
>>> _is_expression({"$field": "value"})  # Reserved
False
>>> _is_expression("$field")
False
>>> _is_expression(42)
False

neosqlite.collection.expr_evaluator._is_field_reference(value: Any) → bool

Check if value is a field reference.

Field references start with ‘$’ but are not expressions (i.e., they’re simple strings like “$field” or “$nested.field”).

Parameters:

value – Value to check

Returns:

True if value is a field reference, False otherwise

Examples

>>> _is_field_reference("$field")
True
>>> _is_field_reference("$nested.field")
True
>>> _is_field_reference("$$ROOT")
False
>>> _is_field_reference({"$sin": "$angle"})
False

neosqlite.collection.expr_evaluator._is_aggregation_variable(value: Any) → bool

Check for aggregation variables.

Aggregation variables start with ‘$$’ (e.g., $$ROOT, $$CURRENT).

Parameters:

value – Value to check

Returns:

True if value is an aggregation variable, False otherwise

Examples

>>> _is_aggregation_variable("$$ROOT")
True
>>> _is_aggregation_variable("$$CURRENT")
True
>>> _is_aggregation_variable("$field")
False

neosqlite.collection.expr_evaluator._is_literal(value: Any) → bool

Check if value is a literal (not an expression or field reference).

Literals include: numbers, strings, booleans, None, arrays, and plain dicts.

Parameters:

value – Value to check

Returns:

True if value is a literal, False otherwise

Examples

>>> _is_literal(42)
True
>>> _is_literal("string")
True
>>> _is_literal(True)
True
>>> _is_literal(None)
True
>>> _is_literal([1, 2, 3])
True
>>> _is_literal("$field")
False

neosqlite.collection.expr_evaluator._convert_to_int(value: Any) → Any

Convert value to int.

neosqlite.collection.expr_evaluator._convert_to_long(value: Any) → Any

Convert value to long (64-bit int).

neosqlite.collection.expr_evaluator._convert_to_double(value: Any) → Any

Convert value to double (float).

neosqlite.collection.expr_evaluator._convert_to_decimal(value: Any) → Any

Convert value to decimal (float, as SQLite lacks Decimal128).

neosqlite.collection.expr_evaluator._convert_to_string(value: Any) → Any

Convert value to string.

neosqlite.collection.expr_evaluator._convert_to_bool(value: Any) → Any

Convert value to bool.

neosqlite.collection.expr_evaluator._convert_to_objectid(value: Any) → Any

Convert value to ObjectId.

neosqlite.collection.expr_evaluator._convert_to_bindata(value: Any) → Any

Convert value to Binary (binData).

neosqlite.collection.expr_evaluator._convert_to_bsonbindata(value: Any) → Any

Convert value to Binary (bsonBinData).

neosqlite.collection.expr_evaluator._convert_to_regex(value: Any) → Any

Convert value to regex pattern.

neosqlite.collection.expr_evaluator._convert_to_bsonregex(value: Any) → Any

Convert value to regex pattern (bsonRegex).

neosqlite.collection.expr_evaluator._convert_to_date(value: Any) → Any

Convert value to date (returns as-is; proper conversion requires parsing).

neosqlite.collection.expr_evaluator._convert_to_null(value: Any) → None

Convert any value to None.

neosqlite.collection.expr_evaluator.get_bson_type(value: Any) → str

Get BSON type name for a value.

Parameters:

value – The value to check

Returns:

BSON type name (e.g., ‘null’, ‘bool’, ‘int’, ‘double’, ‘string’, ‘array’, ‘object’)

neosqlite.collection.expr_evaluator.parse_json_path(field: str) → str

Convert dot notation with array indexing to JSON path syntax.

Supports: - Simple fields: “name” -> “$.name” - Nested fields: “address.street” -> “$.address.street” - Array indexing: “tags[0]” -> “$.tags[0]” - Nested array access: “orders.items[2].name” -> “$.orders.items[2].name” - Complex paths: “a.b[0].c[1].d” -> “$.a.b[0].c[1].d”

Parameters:

field (str) – The field path in dot notation with optional array indices

Returns:

Properly formatted JSON path

Return type:

str

neosqlite.collection.expr_evaluator.build_json_extract_expression(data_column: str, field_path: str) → str

Build a complete json_extract SQL expression with properly formatted JSON path.

Parameters:

• data_column (str) – The name of the JSON data column (e.g., “data”)

• field_path (str) – The field path in dot notation (e.g., “name”, “address.street[0]”)

Returns:

Complete json_extract expression, e.g., “json_extract(data, ‘$.field’)”

Return type:

str

neosqlite.collection.expr_evaluator.supports_jsonb(db_connection) → bool

Check if the SQLite connection supports JSONB functions.

This function tests whether the SQLite installation has JSONB support by attempting to call the jsonb() function. Results are cached to avoid redundant queries.

Parameters:

db_connection – SQLite database connection to test

Returns:

True if JSONB is supported, False otherwise

Return type:

bool

neosqlite.collection.expr_evaluator.supports_jsonb_each(db_connection) → bool

Check if the SQLite connection supports jsonb_each() table-valued function.

The jsonb_each() and jsonb_tree() functions were added in SQLite 3.51.0. This function tests whether they’re available by attempting to use them. Results are cached to avoid redundant queries.

Parameters:

db_connection – SQLite database connection to test

Returns:

True if jsonb_each is supported, False otherwise

Return type:

bool

neosqlite.collection.expr_evaluator._get_json_function_prefix(jsonb_supported: bool) → str

Get the appropriate JSON function prefix based on JSONB support.

Parameters:

jsonb_supported – Whether JSONB functions are supported

Returns:

“jsonb” if JSONB is supported, “json” otherwise

Return type:

str

neosqlite.collection.expr_evaluator._get_json_each_function(jsonb_supported: bool, jsonb_each_supported: bool) → str

Get the appropriate json_each function name based on support.

Parameters:

• jsonb_supported – Whether JSONB functions are supported

• jsonb_each_supported – Whether jsonb_each is supported (SQLite 3.51.0+)

Returns:

“jsonb_each” if supported, “json_each” otherwise

Return type:

str

neosqlite.collection.expr_evaluator._get_json_group_array_function(jsonb_supported: bool) → str

Get the appropriate json_group_array function name based on support.

Parameters:

jsonb_supported – Whether JSONB functions are supported

Returns:

“jsonb_group_array” if supported, “json_group_array” otherwise

Return type:

str