What They Are, Why They Matter, Where to Begin
The Foundation No One Talks About Enough
Most discussions about enterprise AI focus on the appealing parts of the systems, the models and outputs, and the related business cases. What is often ignored is everything that is foundational to the output: infrastructure. Even the most advanced AI systems cannot function to the best of their ability without an adequate data infrastructure.
Data warehouse installations are how that infrastructure is built. While they may not be the most exciting part of the AI project, they may be the most important. If they are done incorrectly, it can be felt throughout the entire system in every downstream decision.
What does End to End mean?
The term ”end to end” is often misused, which is why it is important to define it. An end-to-end data warehouse installation means building out the entire process, starting at the point of data being generated and ending at an environment where that data is structured and ready to be queried.
This involves extracting data from multiple operational systems, building out transformation and standardisation pipelines, and building out the appropriate data storage infrastructure to support the AI and analytics workflows, along with the necessary governance controls.
This is a highly complex system to build out. The hardest part is the actual transformation of the data. Most entities will have data ingested into a data warehouse, but the data will not be in a consistent and reliable form.
Why Enterprise AI Distinguishes Itself from the Warehouse
With standard business intelligence, you have an idea of how to structure the questions, but the same can’t be said for an AI system. An AI system can examine endless amounts of data and information to find correlations, can build training sets for its data, and can begin to develop inferences on data without waiting for instructions.
These changes create the demand for different layouts. More significant portions of cloud storage are needed. The way partitions are organised should reflect the way AI models are likely to access the information. Many cloud storage systems are designed to allow the workload for the AI system to be scaled independent of routine reporting.
The architecture of systems designed for traditional business intelligence reporting must be rethought if they are to be of any use for machine learning. This is because architectures for business intelligence have remained stagnant, while the use cases have moved on.
Where IT Organisational Leaders Go Wrong
There are a few ways in which many businesses tend to achieve similar results when they come to using cloud storage systems.
A common issue is that the businesses are too quick to develop rigid schemas. AI systems consume data in an unpredictable manner, and if businesses build rigid systems without understanding the ways in which the AI systems consume the data, the systems will create significant bottlenecks. Maintaining the ability to change the system, especially in the staging area, is critical to a successful implementation.
The failure to consider the way in which data changes in the system is crippling. Incorporating rigorous data lineage into the systems is essential to the AI models, and the absence of lineage forces businesses to spend inordinate amounts of time completing post-hoc analyses. High quality lineage management becomes an essential practice for quickly iterating and addressing the behaviour of AI systems.
Using only the warehouse for storage tiers. Good enterprise AI architecture separates hot, warm, and cold data. In the absence of a main warehouse, there’s a place for object storage, feature storage, and operational databases.
The Installation Plan
Beginning the journey, the new organisation’s natural desire is to evaluate the range of platforms and tools. Exercise restraint for as long as possible. The more useful first step is a comprehensive audit of data. Before and data warehouse installations you should know what data you have, where it is, how often it changes, and the current state of the data.
Thereafter, the architectural decisions are taken with ease. Organisations with structured and highly transactional data are generally presented with a class of tools and options not presented to organisations dealing with unstructured or semi-structured data.
When designing data warehouse installations, the operational model is just as important a consideration as the technical model. Who will be responsible for keeping the system up to date? How will the relationship model of the data be changed? What will the system do if a data generating system changes the way it communicates with this data? These are just as important as any other technology decision.
The future
There will always be a need for enterprise AI. The challenges presented to data infrastructure will continue to increase. Organisations that build data warehouse installations with AI at the core will have a greater advantage than those that design systems built to address enterprise AI in the past.
It doesn’t attract attention in an AI program, but in silence, it plays an important role.