FOR DATA ENGINEERS

Powerhouse Data Engineering

Address the most complex and demanding data engineering tasks, efficiently managing tens of thousands of production pipelines.

Quick start

Productivity

Prophecy makes data users productive with developing data pipelines, bringing AI and Visual designer to assist in developing code for high quality data engineering.

No lock-in

Prophecy’s combination of AI and visual development together help you develop data pipelines in high-quality Spark or SQL code.

Completeness

Prophecy is a single copilot for all your cloud data platforms, and for the full data transformation lifecycle - develop, deploy and observe.

Extensibility

Prophecy enables you to develop code plugins to add data operations that are available visually, and that AI will recommend, generate or complete.

The future of

Data Transformation

Prophecy has integrated visual designer, AI and compilers to build the most advanced approach for data transformation.

The visual, drag-and-drop interface is intuitive and approachable and the transformations built visually are turned into standard, high-quality code.

AI is integrated into the visual interface making recommendations, converting natural language to business logic, completing pipelines, generating tests and documentation, and suggesting fixes for errors.

Compilers turn visual design, tests and text generated by AI to native Spark or SQL code. They ensure code correctness and performance. Extensibility relies on compiler magic to bring new code based plugins into the visual and AI layers.

Easily build Spark batch or streaming pipelines by connecting to your Apache Spark cluster and developing visually, adding sources, targets and transformations that turn to PySpark or Scala code on your git.

Connect to your SQL data warehouse, and developed transformations visually that turn to SQL code with dbt core. If you edit the SQL, the visual data pipelines will update automatically.

You can visually orchestrate your pipelines using Databricks Workflows or Apache Airflow adding triggers on data, running multiple pipelines and sending notifications.

DataOps for

Trusted Data

Move changes to production faster, with more confidence, thanks to versions, testing and data monitoring.

Git Integration

Prophecy provides a simplified visual interface to commit changes to git, with collaboration operations such as merge done in the visual designer. All major Git providers such as Github, Gitlab, and Bitbucket and cloud specific ones are supported

Tests, CI, CD

Automatic test generation increases test coverage and ensures that changes are high quality when committed and when pushed to production. You can use Prophecy or connect to your own CI, CD system such as Azure DevOps.

Documentation

Multiple data users will build, enhance and fix data pipelines over time. AI generates documentation of the pipelines, and this will include a summary of each change made by a user so that new users can get up to speed quickly and be productive.

Observability

As you deploy pipelines, you can monitor them for break-fix, and you see the errors on the visual component that failed in the pipeline and get AI based suggestions on how to fix errors. Cost and performance information is coming soon.

Extend and reuse with

Copilot Plugins

Enhance your data engineering process by using high-quality, reusable building blocks, instead of ad hoc code. Prophecy helps you to establish unified standards across your team.

Plugins have the following reusable components:

Spark Data pipelines with multiple configuration sets

Spark Subgraphs with configuration per data pipeline

SQL models that will read tables or re-compute business logic

Spark & SQL Gems are custom visual components that can connect to your internal systems as sources or targets, or transform data such as your standard way of encrypting data, or customer table cleanup you need to perform often

Airflow operators

User defined functions to transform a value

Datasets

Software Best Practices

Plugins are code projects on git with versioning, tests and proper release cycles. They are loaded as dependencies when you use them.

Plugin Hub

Plugin hub enables data users to search for and bring plugins including additional visual components (gems) on to their canvas and use them.

EBOOK

Low-code Apache Spark and Delta Lake

A guide to make data lakehouses even easier.

Get the eBook

ON-DEMAND WEBINAR

Low-code data transformations

10x productivity on cloud data platforms.

Watch the webinar