Source Systems - How to win friends and influence people

Before companies with new BI implementations put that data warehouse into production they should understand that this might change your IT culture.  For example, any changes in the upstream systems need to notify the BI team that those changes are on the radar.   That's really what data governance is all about.   Lets dig into this point a little bit.  What changes am I talking about?   Adding a field to the source system, Changing a field from numeric to alpha numeric., adding a flag.  The other thing that might happen is a fields meaning to change.   Lets say that you assume that the first character means A condition, and the change might be to add a field that changes that A condition.   You may not find out about this condition for days or weeks.

When you think about data governance and ensuring that data quality is high, you must be in lock step with your source systems. Ideally, I think it makes sense for your source systems and data warehouse team to be under the same leadership.   These teams should be communicating on what changes are being made and to collaborate on testing to ensure that the changes in the upstream source system are also being accounted for in the data warehouse tables.  (When I say data warehouse, you could have an ODS).  I worked at one client and the source system changes were made and the data warehouse team would only discover the change the next day.  

Lets take this one step further.  Your source system could nor only effect your data warehouse but your data warehouse could have a feed to another system.   Data warehouse groups have the information and the tools to move data from system A to system B.  This data might come from multiple sources.  So if the source system data changes, then the subsequent downstream data may change as well.  

A proper CCB or Change Control Board where every group in IT would attend and have an opportunity to understand all changes would have been necessary as a stop gap measure.  The CCB should have an opportunity to stop any change that might break anything downstream.   Again, your primary source system should have a test bed that might be shared by the data warehouse team.  Everything needs to be tested.   Lets add to this CCB an annual audit by the data warehouse team to understand what your ETL rules are and what they need to be changed to in order to meet the needs of the business.  Over time these rules might change.   This would be a good task for your data governance team.