Feature Engineering Without the Overhead
From Python to Petabytes, instantly
Introducing LLM-as-UDF
Custom Transformations at Scale
Apply your own Python functions directly to multimodal datasets without exporting or duplicating data.
Distributed Processing
Run UDFs across large datasets in parallel using Ray or Spark, cutting feature generation time from days to hours.
Inline Feature Creation
Generate, update, and store new features right inside LanceDB for immediate use in training or analytics.
table.add_columns({
"title_frame": extract_key_frame("video", 0),
"description": img2txt("title_frame"),
"embedding": embed("description")
})Key Features
Declarative Pipelines
Define workflows once and scale to 100K+ cores. No code changes needed to scale.
Feature DocsSchema Evolution
Add or change features without re-ingesting. Iterate on models with minimal engineering.
Feature DocsBuilt-in Orchestration
Preemption and cheeckpointing come standard. Keep training resilient without extra work.
Feature DocsGP Scheduling
Run jobs when GPUs are free or underused. Optimize for cost and throughput.
Feature Docs