Designed to make Data Engineering Teams exceed expectations and demands.
More Data Teams made 10x Productive
Prophecy enables many more users - including visual ETL developers and Data Analysts. All you need to do is point-and-click and write a few SQL expressions to create your pipelines.
Low-Code Spark for workflow development. Low-Code Airflow for Scheduling. Metadata Search and Column Level Lineage for Management and Governance. We support complete Development & Deployment lifecycle
As you use the Low-Code designer to build your workflows - you are developing high quality, readable code for Spark and Airflow that is committed to your Git.
Prophecy gives you a gem builder - for you to quickly develop and rollout your own Frameworks. Examples are Data Quality, Encryption, new Sources and Targets that extend the built-in ones.
Unique and intuitive design environment that helps you design superior workflows on Spark & schedules on Airflow
You can develop in visual view, and inspect and edit the code for Spark and Airflow.
Generated Spark and Airflow code on Git is 100% open source format and runs without Prophecy.
The built-in sources, targets, transforms and operators can be extended by providing templates as code with blanks to be filled in visual designer.
Step-by-step interactive execution on Spark with sample data and live run of schedules on Airflow allows you to develop faster, running step-by-step to ensure it all works.
Unit tests and data quality test support is built in - making writing these tests super easy. The tests are run every time you commit code to maintain high quality.
Enterprise-grade Metadata system that gets you faster and smarter insights about your data assets.
Your workflows and datasets store information about how the values are computed. Your schedules and executions store information about when they were computed. You can search all the information. You can track at column granularity with column-level lineage.
Prophecy Hub is extensible. You can add information about the existing Entities such as Datasets (for example business definitions), or add other Entities (such as reports) so that all your information is in a single place and queryable.
Prophecy Metadata supports Data Governance by tracking PII and understanding computations of Columns.
View column level lineage at systems, project and workflow level. Per column, get the summary of transformations done on source data to produce its value.
Propagate column level tags across data engineering pipelines to ensure that your sensitive data is appropriately tagged with privacy and security restrictions.
Add lineage for existing sources by running Prophecy Crawler on your Git repositories and your data stores. We support Spark and Hive code.
Prophecy connects to your existing Spark cluster or more commonly connects to your account and spins clusters up and down as needed to run your workload.
Prophecy runs on any cloud or on-premise. We support all major clouds. The same deployment can spin up Spark clusters in different clouds though your code is in one place.
Fabric is our virtual equivalent of on-premise big data clusters. This allows you to have separate environments for test, integration and production.
Prophecy installs on Kubernetes, whether on premise or on public cloud. In public cloud, we can run within your VPC, in addition to being available as a hosted service.
Bring the speed of DevOps to Data Engineering
We make is simple to write unit tests - by generating a template to start per component. We can jump start the practice to get high coverage (90%+) rapidly and maintaining that. Every modification (git commit) and deployment runs unit tests to maintain high quality of workflows.
We accelerate data quality tests, including auto-generation of suggested constraints to jump start data quality. If you are getting a detailed report of data quality all the time from test and production systems, you can deploy new changes with confidence.
Convert legacy ETL workflows to Spark with high automation.
Prophecy has transpilers, or cross compilers for formats and languages of Legacy ETL providers that provide highly automated modernization of your existing ETL workflows. Our expertise in moving multiple Enterprises allows you to move faster, more reliably and lower risk.
