Delta Lake

data lake, acid transactions, time travel, file format, storage
Open jobs

705

Companies looking for Delta

597

Back in the wild west of big data, when data lakes were less shimmering oases and more swampy messes, arose the need for some semblance of order. Delta Lake, the lovechild of Databricks and open-source ambition, promised to bring ACID transactions to the chaotic world of Apache Spark and data lakes. It aimed to solve the classic problems of unreliable data pipelines and concurrent writes – a noble goal, indeed.

The reality? It’s quite clever, really, though one does wonder if the complexity always justifies the benefit. It layers a transaction log on top of your data lake, essentially creating a reliable table abstraction. Compared to Parquet files alone, it's a significant upgrade. Versus traditional data warehouses? Still a different beast, leaning towards schema-on-read flexibility. Today, it’s become a cornerstone of the modern data stack, embraced by anyone serious about building robust data pipelines, particularly within the Spark ecosystem. Though, let’s be honest, it adds another layer of operational overhead to an already complex world.

Used together with Delta

Jobs (this month)

705

Companies with Jobs

597

Jobs in using Delta Lake for but please no

data scientist

Staff GenAI Research Scientist @ databricks

US | 2025-12-27 | USD 192000 - 260000 / year
This role is a tantalizing promise of cutting-edge AI research, but the description lacks specificity on the actual technologies, tools, or deliverables expected. The job emphasizes collaboration... read more »
Databricks, AI/ML, LLM, Go, GenAI, PhD, Analytics, Data Lakehouse, Spark, Delta
data engineer

Data Engineer II - QuantumBlack, AI by McKinsey (Critical Industries) @ quantumblack

US | 2025-12-27
Data Engineer II role promises to design, build, and optimize modern data platforms powering analytics and AI, with a tour through streaming and batch pipelines across aerospace, utilities, and... read more »
Analytics, AI/ML, R, Data Streaming, Vector DB, RAG, Agile/Scrum, C, Computer Science, Data Engineering, Python, Scala, Java, SQL, Cloud Computing, AWS, GCP, Azure, Oracle, Snowflake, BigQuery, Redshift, Delta, Databricks, AWS Glue, dbt, Spark, Flink, Kafka, Kinesis, Airflow, Dagster, Prefect, CI/CD, Terraform, CloudFormation, DataOps, Datadog, Prometheus, Amazon SageMaker, MLOps, GenAI, LLM, Management
promoted

BigQuery Cost Optimization Guide by Masthead Data

The BigQuery Cost Optimization Guide is a valuable resource that offers a deep dive into understanding and managing the complexities of BigQuery costs. It provides practical guidelines and strategies to help you optimize both storage and compute expenses, ensuring you get the most value out of...

Learn More
chief

Director, Operations Analytics & Workforce Optimization @ spring-health

US | 2025-12-26 | USD 164900 - 214450 / year
Spring Health seeks a Director of Operations Analytics & Workforce Optimization to lead a team that applies data to improve operational performance. The role offers a unique advantage in being the... read more »
Microsoft, Delta, Management, Analytics, AI/ML, KPI, Dashboard, BI, Looker, SQL
data engineer

Data Engineer II - QuantumBlack, AI by McKinsey @ quantumblack

US | 2025-12-22
Data Engineer II role promises high impact across clients and a culture of continuous learning, but the job description reads like a curriculum vitae for a consulting fortress: heavy emphasis on... read more »
Analytics, AI/ML, R, Data Streaming, Vector DB, RAG, Agile/Scrum, C, Computer Science, Data Engineering, Python, Scala, Java, SQL, PySpark, Cloud Computing, AWS, GCP, Azure, Oracle, Snowflake, BigQuery, Redshift, Delta, Databricks, AWS Glue, dbt, Spark, Flink, Kafka, Kinesis, Airflow, Dagster, Prefect, CI/CD, Terraform, CloudFormation, DataOps, Datadog, Prometheus, Amazon SageMaker, MLOps, GenAI, LLM, Management