Writing business

Upsolver SQLake makes creating a pipeline for Data in Motion as easy as writing an SQL query

SAN FRANCISCO–()–Upsolver, the company dedicated to making data in motion accessible to all data practitioners, today announced the general availability of SQLake. The new service provides a self-orchestrating SQL-based data pipeline platform that ingests and combines real-time events with batch data sources for up-to-the-minute analytics. It’s available at a groundbreaking new price of $99 per TB ingested, with no transform processing fees and no minimum commitment.

With SQLake, companies are achieving a quantum leap in data freshness for use cases such as ML model training, anomaly detection, real-time BI, and data science. It also makes data easily accessible to any SQL user – not just data engineers, but also data scientists, analysts, product managers, and other data consumers.

SQLake fundamentally redefines the data pipeline development experience. As the only data-aware data pipeline platform, SQLake achieves unprecedented simplicity by automating many functions that typically require human intervention. Users no longer need to develop, test, and maintain complicated orchestration logic (DAG), optimize their data by hand, or manually scale their infrastructure. With SQLake, everything is automatic.

Crossing the chasm from the moving data lot to new insights

More recent data leads to better decisions. However, companies looking to move away from overnight batches and embrace up-to-the-minute data freshness have had only two options, each with serious challenges. On the one hand, they could attempt to lock batch processing onto data in motion, creating an orchestration nightmare marked by unmanageable complexity, high operational costs, low data observability, and mounting technical debt.

On the other hand, they could create a “Lambda architecture” that requires deploying and managing a separate streaming infrastructure alongside their batch process. In this case, they must hire big data engineers, which is expensive and creates a significant barrier to self-service for non-engineered data consumers in the enterprise. Additionally, they must adjust, optimize, and scale the streaming solution, leading to increased operational overhead, SLA violations, and data consumer frustration.

The first self-orchestration data pipeline platform

Upsolver SQLake overcomes these challenges by treating all data as data in motion. It automatically determines dependencies between each stage of the pipeline to orchestrate and optimize data flow for efficient, resilient, and high-performance delivery.

With SQLake, creating a pipeline is as easy as writing an SQL query. This creates many benefits, including:

  • Pipeline development cycles are shortened from months to weeks or days.

  • The widespread adoption of SQL provides data users with self-service pipelines for new analytics. No Java, Python, Spark or Airflow expertise is required.

  • Production pipelines are more robust because human errors are eliminated and failure scenarios are anticipated and gracefully managed.

  • The scaling of stateful operations is automatic. Unlike scaling limits in Apache Spark, SQLake’s single state store efficiently manages billions of keys.

“Customers tell us it’s extremely difficult to bridge the chasm to new data because stream processing is too complex for most users and not powerful enough to replace batch workloads,” Ori said. Rafael, CEO and co-founder of Upsolver. “SQLake is a game changer. Now anyone familiar with SQL can easily develop and deploy data pipelines that combine real-time events with historical data, at scale. »

A proven stateful stream engine at scale

SQLake leverages the same cloud-native processing engine used by Upsolver customers today, such as IronSource (mobile app user behavior), Proofpoint (network security) and Cox Automotive (mail order flow). It ingests data continuously and in batches as events, supports stateful operations such as continuous aggregations, window functions, high-cardinality joins, and UPSERTs, and provides up-to-date and optimized data to query engines, data warehouses and analytics systems.

“Peer39 is the premier provider of contextual data used to optimize the effectiveness of marketing campaigns. We use Upsolver to ingest and optimize 20 billion events per day into our data lake on AWS, making new data available in minutes and increasing data lake queries by 10 times,” says Boaz Goldstein, R&D, Data Architecture & Business Intelligence Manager at Pair39. “Upsolver’s SQLake offering will allow our data engineers and data scientists to easily develop pipelines that bring together streams of data and historical data without having to manually develop and manage complex orchestration logic or struggle to scale the infrastructure to meet our volume of data.

Write a query, get a pipeline

SQLake redefines ease of use for pipeline development. Data engineers and data consumers can create and deploy a continuous pipeline using only SQL in a few simple steps:

  1. Select a use case from the SQLake template gallery or start a pipeline from scratch.

  2. Connect to data sources and ingest data into cloud data lake staging tables. SQLake infers and automatically evolves the source schema.

  3. Inspect and profile data using real-time statistics and SQL queries of staging tables.

  4. Develop a transformation task to create analysis-ready output tables in your data lake or data warehouse. Orchestration, data management, and infrastructure scaling are automatic.

  5. Preview the results and start the pipeline.

On-demand pricing at $99 per TB ingested; Conversions are free

With the launch of SQLake, Upsolver moved to a predictable environment, value-based pricing model. Pricing is based solely on the volume of data ingested, with no limit on the number of pipelines used, making transformations free. Unlike the opaque “processing units” used by many data management solutions, Upsolver’s costs are simple to understand and tied to customer value, not supplier costs.

To make SQLake attractive for any size pipeline project, SQLake is available for $99 per TB of data ingested with no minimum commitment. This groundbreaking entry price, plus 30 day free trial, allows any data user to get started safely with SQLake. Upsolver SQLake can be purchased from AWS Marketplace.

To see a full list of Builders Hub features, video demos, and resources, visit the Upsolver website and SQLake documentation.


Upsolver is a tight-knit group of data engineers and infrastructure developers obsessed with removing the friction of building data pipelines to accelerate real-time delivery of big data to the people who need it. .

Founded in 2015 by data engineers Ori Rafael and Yoni Eini, Upsolver has grown from an Israeli adtech-focused company to a global company serving clients in many industries, including software, manufacturing, oil and gas. gas, health care and financial services. Upsolver’s platform enables a variety of high-value analytics use cases such as user behavior, IoT monitoring, and log analytics.

Upsolver is headquartered in San Francisco with R&D centered in Tel Aviv. Customers span regions and industries, such as Cox Automotive, IronSource, ProofPoint, and Wix. Its main investors include Scale Venture Partners, Vertex Ventures US, Wing Venture Capital and JVP. For more information, please visit www.upsolver.com.