Ship trusted data on your existing infrastructure
and batch
Easily design data pipelines.
- Read data from multiple sources and transform it visually into analytics-ready data. The visual data pipelines turn into open-source Spark code (in Python or Scala) that is high-quality and runs at scale.
- Ensure pipelines are efficient and accurate with Prophecy's step-by-step execution, reusable subgraphs and configurations, and auto-generated tests.
Visually build data pipelines and monitor transformations in real-time.
- Create, deploy and share SQL models.
- Enable visual and code developers to collaborate on SQL code.
- Import existing dbt coreTM SQL projects, which you can edit visually then store as dbt core projects.
Easily schedule pipelines to run at specific times (e.g., every morning).
- Use our visual interface to effortlessly orchestrate Apache Airflow workflows, test each step interactively and deploy to production
- Enjoy out-of-the-box support for multiple Airflow operators (we can easily add new ones, too)
Use Airflow or Databricks for orchestration.
Easily design your own frameworks with Prophecy Frameworks — shareable Git projects that can be used as libraries across teams.
- Leverage pre-built visual components for standard inputs, outputs and transformations
- Add your own to read/write to internal systems and standardize common operations (e.g., security ops with encrypt, decrypt and anonymize)
All-in-one data engineering solution
DataOps
Your visual pipelines turn to 100% open-source code, enabling you to develop using software best practices like git, tests, continuous integration and deployment.
Data Quality
Define expectations for your datasets, and use ML-based anomaly detection for zero effort observation. We excel at large and complex datasets.
Search and Lineage
Search your datasets and pipelines and for any dataset, see the column level lineage. Prophecy provides lineage for our pipelines and provides an API to add others
Monitoring
Monitor your data pipelines and data quality, and set up alerts that will notify you about breakage or other issues.