OUR approach
For 15 years, Dataware Consulting has been a trusted leader in data platform design and development.
We’ve found that successful implementations are rooted in a clear understanding of each client’s needs and objectives. To get there we engage with project stakeholders, subject matter experts and technologists to achieve clarity. We uncover the “gotchas” that lurk behind every project so that the resulting plan is realistic and achievable. Dataware Consulting moves the plan into action through our expert development, architectural design, and team leadership. We always implement with consistency to ensure the data platform is straightforward, supportable and expandable into the future.
Dataware Consulting has designed data platforms for a variety of clients across industries. Our data platform development is driven by the specific requirements, data sources, and technology stack of the business. Doing this accurately and efficiently is our value proposition to each client.
Over many implementations we have learned what is foundational to the success of a data platform. That is our value proposition to you.
dataware'S History
For the first 10 years, Dataware Consulting built data platforms primarily to support enterprise data warehouses. We mastered the modeling of analytical tables (star/snowflake/3NF schemas) in data marts and optimized them in accordance with usage patterns. We developed reports/dashboards and managed user acceptance testing. The lion's share of our technical effort was spent building the ETL to load and cleanse data. Transforming staged data to meet analytical expectations required us to level up in SQL again and again. Modeling efficient stage tables was revealed to be more art than science. We learned the data design patterns and when/where/why to apply them.
Dataware Consulting expertly delivered efficient data platforms to serve curated data to business users.
5 years ago, Dataware Consulting's clients started to demand:
-
More data sources for machine learning models
-
Quick access to raw data
-
Use of cheaper cloud storage
-
Near real time data
-
A unified data platform for accessing raw and curated data
Dataware Consulting made the shift toward Big Data and cloud based data lakes. It was clear that Big Data had exploded the options at our disposal. The demands could be met with a different skillset (Python, anyone?) and mindset (no more databases). Cheap storage, flexible compute and cloud based tools were great for processing file-based streams and huge data sets.
Yet something was missing. The client's business still relied on curated data, but the data warehouse was disconnected from the Big Data in the data lake. There was no single source of truth. The data development team had two lousy options:
-
Continue managing two environments with radically different technologies.
-
Retrofit the data warehouse ETL and tables into the data lake with the cloud based toolset. While this second option was doable, the tools were not a product. Developing and managing the patchwork solution was overwhelming the data development team with DevOps. They barely wrote straight SQL anymore.
It became clear that:
-
All data must be managed in a unified data platform to encourage an integrated "single source of truth"
-
Data warehouse processes and disciplines were still vital to producing curated data in a modern data platform
-
Big Data tools and techniques were not a replacement, but a powerful addition within the modern data platform
-
A product was needed which enables both Big Data and DW development
This is why Dataware Consulting specializes in Databricks on AWS.
Databricks provides a comprehensive open source product which enables both Big Data development in Python/Scala and DW development in SQL. The Databricks environment supports a data development team's efforts to work on datasets intended for both machine learning and business users. Furthermore, the Lakehouse model enables organizations familiar with data warehousing to incorporate a data lake into their "single source of truth". It's a data platform that delivers the fresh insights and curated data the business needs.
AWS works great with Databricks as a cloud storage platform and has a vast array of tools which can further support the data platform. Dataware Consulting focuses on this pairing of technologies so that we can expertly guide your Databricks on AWS implementation.