Introducing — cloud-native data engineering

Introducing — cloud-native data engineering

Introducing — a high-performance, zero-compromise, cloud-native data engineering product powered by Spark for enterprise data engineering teams.

Introducing — a high-performance, zero-compromise, cloud-native data engineering product powered by Spark for enterprise data engineering teams.

Raj Bains
Assistant Director of R&D
Texas Rangers Baseball Club
September 15, 2019
April 22, 2023

Table of Contents is a high-performance, zero compromise cloud native data engineering product powered by SparkKubernetes for Enterprise Data Engineering teams. Prophecy also provides a highly automated replacement for legacy ETL products, to accelerate the journey to open source and cloud computing.

Starting in systems engineering and excited by the potential of GPUs, I moved from Microsoft Visual Studio team to become an early engineer in the CUDA team at NVIDIA, and played a key role in compiler optimizations to get high performance on GPUs. It’s a delight to see how far CUDA as come in powering deep learning and bitcoin mining. Passionate about building a startup, I moved to learn Product Marketing & Product Management, leading both functions for ClustrixDB through a hard pivot to a limited, repeatable product-market fit.

At Hortonworks, I product managed Apache Hive through the IPO. It was not fun to be in front of customers and see them struggle with Data Engineering. The Hortonworks team put in a massive effort to make Hive better & simpler — fewer configurations, faster performance, a real cost based optimizer, and a simplified stack in Hive 2.0 and beyond.

Taking the learnings of technology and the market, we’ve decided to build a product centric company with relentless focus on the customer needs and it just works! experience, coupled with a sprinkle of joy!

We’re delighted to raise our seed round from SignalFire with Ilya Kirnos joining the board & Andrew Ng working closely with us. They join our existing investors Berkeley SkyDeck (Fall 2018 cohort), and friends from Enterprise software industry who invested Angel funds! SignalFire is unique - besides knowing the technology & market well, they get their hands dirty -running Spark in-house for their Machine Learning based product Beacon for locating talent!

There are two things I want to talk about today:

  • Enterprise Data Engineering is a Journey
  • Enterprise Data Engineering needs new Interface

Enterprise Data Engineering is a Journey

Solving the real problem!

Large Enterprises have 10s of thousands of ETL workflows in production on premise, in a legacy ETL format, and they’re paying through the nose. There are compelling reasons to move to Open Source and Apache Spark - freedom, agility, talent and cost - that are well understood by the leadership of these Enterprises.

Our strong view is that putting a product on public cloud, and asking the Enterprise to figure out the transition sucks! Are 10s of thousands of workflows to be rewritten? Is every environment to have a separate scheduler, and a separate ops team? How are graphs of dependent workflows to be migrated?

We think “Legacy ETL workflows” or “dataflows” are too low a level of abstraction to focus on. We’re focused on building products for the complete “Data Engineering” journey!

Let’s look at some specific solutions:

Legacy ETL to Spark: Transpilers!

Apache Spark is ubiquitous in public and private clouds with managed services in public clouds.Transpilers (drawn as dragons since the dragon book) are our compiler based products to convert Legacy ETL assets into Prophecy with restructuring support. This includes workflows, configurations and datasets which we transpile to an open source technology stack. This is the first step to freedom!

2. Multi-Cloud: Unified Control Plane & Distributed Data Plane

As Enterprises are moving to public clouds and multiple data centers, hybrid cloud is the new-normal state:

  • Enterprise IT infrastructure will always be across private and multiple public cloud providers and multiple regions.
  • Data sovereignty and data localization laws mean slices of data will be spread and processed across many geographies

In these cases, keeping a single control plane, while distributing the data plane provides significant simplicity and cost savings. The motivation for distributing data plane might be based on regulation, performance or reliability. This requires the right abstractions in the Data Engineering product for development and production.

Enterprise Data Engineering needs new Interfaces

As we talk to Enterprise teams, we’re finding that with different roles come different preferences:

  • Some developers prefer visual drag and drop interface
  • Some developers prefer code development interface
  • Architects prefer standardized components
  • Test and support engineers prefer visual interfaces

Clearly, current interfaces are not meeting these needs, and as we look to design the right interface, let’s review the strengths of each:

We believe there is a much better way of doing this! With compiler magic we have a Unified Visual & Code Interface that provides Interactive Execution.

We’re super excited to support the journey of moving Enterprise Data Engineering to Open Source Runtimes and Cloud Native Infrastructure, while innovating on interfaces - providing a delightful user experience!

There are a few other unique features that I’ll talk about in subsequent posts such as user defined components. Stay tuned!

PS. We’re working on hard problems and looking for top engineers to help us get there! If you’re interested reach out to me at

Ready to give Prophecy a try?

You can create a free account and get full access to all features for 14 days. No credit card needed. Want more of a guided experience? Request a demo and we’ll walk you through how Prophecy can empower your entire data team with low-code ETL today.

Lastest blog posts


Event Spotlight: Prophecy at Data + AI Summit

Emily Lewis
May 26, 2023
May 26, 2023
Customer success

Hitting data driven home runs: How the Texas Rangers win by harnessing Prophecy in their data mesh architecture

Alexander Booth
May 25, 2023
May 25, 2023
Low-code SQL

Empower all business data users with interactive SQL development

Anya Bida
May 19, 2023
May 19, 2023