Authors: Raj Bains & the Prophecy Team!

Prophecy for Databricks - what'll it do for you?

Prophecy is focused on making your data engineering easy. We’re delighted to announce the release of Prophecy for Databricks - a product focused on the best experience for Databricks users!

Before we show what Prophecy can do for you, we’d like to remind you about the continuous pain in data engineering and the struggle to meet the organizational needs. So, let’s first take a step back and remind ourselves how bad we have it :)

What’s the point of Data Engineering?

To know if our data engineering is any good, we need to remind ourselves why it is important - and if we’re meeting that mandate.

Data Analytics asks questions of your data to make intelligent business decisions and keep you competitive. Business intelligence and Machine Learning both need to be supplied with data from your disparate operational systems - data that is scattered and in incompatible shapes & formats - such as the airline tickets booked, credit card transactions done and rewards used, sales closed, customer profiles, new insurance policies bought, medical devices shipped, inventory in stock, the weather and economic data.

Data Engineering is about building data pipelines to deliver this data to analytics in a timely fashion while ensuring its high quality. If you want to get good at analytics - you have to get good at data engineering - build these pipelines fast, move them to production fast and keep them up to date and healthy.

Data Engineering is hard!

Today, Data Engineering is very hard - completely unnecessarily so - let’s remind ourselves of our unhappy reality.

Building and running data pipelines - that number from a few hundred to the tens of thousands - delivering data rapidly to you analytics consumer is hard! Some challenges that the data leaders face include:

Prophecy for Databricks makes data engineering easy. No - not a component - we’re solving the problem end-to-end and making everything much, much easier.

What will “Prophecy for Databricks” do for me?

Prophecy is a Low-Code Data platform that makes data pipelines easy - easy to build, easy to deploy, and easy to manage. Your visually developed data pipelines are stored as code on git with higher quality that hand written code. There is no tradeoff when using Prophecy - it is at least 10x better that any other approach to data engineering including coding or using other visual ETL tools.

That’s all we want to say about what we’ve built, we’d rather talk about what we’ll do for you!

Visual Pipelines → More Users Enabled

Prophecy enables many more users to build pipelines with a visual drag-and-drop environment. This means more of your existing data users can build pipelines. To hire new users you have access to a larger and more affordable talent pool. If you're the data platform team, you can enable all line of business users. In Prophecy, visual pipelines are backed by 100% open-source code that matches or exceeds the code quality produced by the best programming data engineers.

Visual Pipelines → More Productivity

With Prophecy, you can get pipelines built much quicker. It will often take a couple of days instead of a couple of weeks that it does with coding. Your team can meet and exceed the business needs.

Visual Pipelines → Standardization

Visual pipelines out of the same standardized visual blocks (visual gems in Prophecy) - by using drag and drop and writing SQL expressions.  Also, there is extensibility, so you can create new visual blocks.

With standardization, any user can quickly understand, debug and extend pipelines built by another users. This helps in many situations - from quickly debugging production issues to dealing with churn in the data team.

Mission critical pipelines need best engineering practices

Your data pipelines are every bit as mission critical as the applications. We make it easy to use the same best practices used for robust software development. All pipelines are stored as code on Github with versioning, we make it easy to get high test coverage. CI & CD enable you to move pipelines from development to production quickly with high confidence.

Analytics teams & Metadata Architects have visibility

Prophecy comes with built-in search and column-level-lineage for all your pipelines. This means that if you’re a data analyst consuming a dataset (a table in the database), you can ask when and exactly how that column was computed (the formula). Your metadata architects can tag personally identifying information (PII) on columns of datasets and as they are read. So, your analytics consumers understand how each value was produced, governance folks can follow PII.


Prophecy enables you to really make your Data Engineering run really well. It’s easy to find talent to build pipelines, pipelines get built and deployed much faster and you have greater insight for the data engineers and the analytics consumers.

We’re put in tremendous work and our customers from midsize to Fortune 50 are delighted with it - check Prophecy out, you’ll be delighted you did!

How can I try Prophecy?

Prophecy is available as a SaaS product where you can add your Databricks credentials and start using it with Databricks. You can use an Enterprise Trial with Prophecy's Databricks account for a couple of weeks to kick the tires with examples. We also support installing Prophecy in your network (VPC or on-prem) on Kubernetes. Sign up for your account now:

Sign up for your free Account!

We're super excited to share our progress with you, get in touch with us - we're looking to learn!

Learn more