Avoiding Virtual Data Silos

Decoupling data inputs from their isolated sources continues to be challenging for organizations. Data silos not only hinder organizational collaboration and decision making but also undermine an organization’s ability to unlock the potential of its most precious resource—its data. Advancements like cloud computing, platforms, and blockchain are giving most organizations the tools to transition away from traditional, physical data silos. However, are these tools and transitions truly eliminating data silos, or are they simply ushering in a new paradigm for maintaining them?

What Are Data Silos?

Historically, data silos were physical. A data set existed within the application (or notebook!) X, on server (or shelf) Y, in building Z. That data was accessible by a small group of individuals who could share it selectively on request. Those days are long gone. Shared drives, platforms, ETL/ELT tools, APIs, continuous data streaming, data lake houses, and adoption of the cloud are promoting data sharing, constructive collaboration, and accessibility to data like never before. Yet these have not yet solved the data silo problem fully. Why?

Data silos are no longer simply an issue of isolated data sources but of isolated meaning and understanding. This is not only a characteristic of data in many formats that are not Findable, Accessible, Interoperable, and Reusable (FAIR) but also a product of cultural silo mentalities. Such a silo mentality presents its own obstacles to a true digitization effort. A scientific data platform allows a set of data to be decoupled from the data source and made findable and accessible. But if we don’t take steps to ensure the data that crosses silo lines is interoperable, its reusability will be limited.

So, what contributes to the continued silo mentality or to virtual silos today? Many factors, but some key contributors are the following:

• Data transformation takes longer than expected.
• Cross-functional data normalization is challenging; shared understandings and a shared approach to developing an ontology takes vision, agreement, and compromise.
• Data enablement teams and data stewards hold the keys to the meanings behind the data entities. How is transparency ensured while building innovative technology if the nuances of the data meanings and intended uses are left open for interpretation by data consumers?
• Organizations tend to approach the data silo issue by looking at all data, rather than by focusing on subsets of critical data. These data products cross divisions and data sources. This dilutes the time and effort that should be spent on ensuring high-value transparency and accessibility.

Moving Away From Silos?

Organizations do not plan for silos; most transformation and digitization efforts intend to seek the opposite. But the challenges of the effort, the volume of data, and the myriad technology options chip away at the value these efforts could potentially deliver. Taking smaller slices of the organization’s data and viewing it as a data stream increases the transparency and allows identification of the critical data products. Once the first small data journey is complete, learnings can be applied, and additional critical data subsets can be added to the new system.

Key considerations in preventing a siloed data approach are:

• Focus on curated data sets, as opposed to all data, for effective data transformation efforts. An industry ontology may be a good starting point.
• Ensure there is a data lake house/warehouse integration with continuous streaming storage services to ensure real-time data findability and accessibility.
• Work to remove the cultural silo mentality while elevating data literacy understandings across the organization.
• Challenge data transformation efforts often, with questions such as:
  • What is the possibility that we are setting ourselves up to institutionalize virtual data silos? Are we taking on too much data at once? What are the most critical data sets for our objectives, and are we solely focused on them? How have we ensured a good balance of domain expertise and data handlers?

Data findability and accessibility are essential pillars that allow organizations to take advantage of forward-thinking technologies such as AI and ML. Still, the transformation efforts to get to that goal can be difficult. Careful planning and consideration reduce the challenges while building a data culture that shifts the focus from data capture and day-to-day use to the widespread view of data as an asset.  

Viewing Data as an Asset

CSols works with clients to identify high-value data sets that cross data sources. Examples of these data sets might be method development, study development, and related study testing within a contract research organization (CRO). Different groups perform each of these functions, but the CRO’s clients want a single view of all the work done for them by that CRO, and they don’t want to deal with each division’s information separately. A critical data set for the CRO would follow the individual client.

Focusing on these high-value data sets and ensuring that streaming access solutions are put in place to make the data interoperable and reusable allow organizations to quickly derive value from the process. It is important to realize that an all-or-nothing view is unproductive. This type of transformation is not a data project that ends, but rather a journey that evolves. The all-or-nothing approach poses more problems than it solves, produces virtual data silos, and delays or prevents the decision-making gains that could be achieved with an incremental approach.

The key to the transformation process is keeping the end-to-end view of the data in sight. Strategically decoupling the data from its sources allows the intent and meaning to be determined and the fit-for-purpose technology approach to be defined. 

Connect with us; we are eager to collaborate with you on your data journey. We specialize in dismantling data barriers, whether they are tangible or digital!

---

Where could unsiloing your data take you?