In reference data world, other than the pure reference data business we spend a lot of time on business rules (creating rules for data quality check or changing values of attributes based of values of other attributes).
The IT team tries to automate these rules based on requirements placed by Operations team or from understanding based on new regulations & from issues with existing data or implementations. Even with so much automation we fail to create a fully automated system. There are many reasons for this: operations team not telling us all that needs to be implemented; rules are not static, they change over time; new rules come into play with every regulatory change; Old rules gets decommissioned.
Part of IT team is always working on maintaining business rules. However, what’s not automated are always taken care by the operations team. I feel it is crucial to know what operations team is doing to understand the changes IT teams should bring to help them maintain the system. It is easier said than done, basically because these systems are built for performance and not meant for data mining activities.
To support such capability we need to augment these systems with user (operation team’s) activity mining capability. That’s the first step. Once done it can provide the insight required by IT team to propose rules proactively, even though by analyzing the data manually at this point. This should be the first step towards proactive rule generation.
After perfecting the user activity mining system, we let it prove its usability. Once we start generating value out of it, we should enhance the system to make it a rule recommender. Meaning we want to automate the value generation process.
If we classify every manual change in data separately and for every change we capture the change and all data in previous state. We should be able to build a rule recommender system.