Meeting Big Data challenges with visual analytics The role of records management

Article Louise Lemieux, V., Gormly, B., & Rowledge, L. (2014). Meeting big data challenges with visual analytics: The role of records management. Records Management Journal, 24(2), 122-141. This paper aims to explore the role of records management in supporting the effective use of information visualisation and visual analytics (VA) to meet the challenges associated with the analysis of Big Data. Regardless of precise definition, exponential growth, speed and variety in data have created new challenges not only in the management of vast amounts of heterogeneous data, but also in how to make sense of it all. The human information processing system simply cannot hold information in working memory long enough to extract relevant patterns from the data. Visual Analytics Process has been distilled into three distinct phases: Data collection and curation Data pre-processing Analysis - a structure that emerged organically from the research data Data challenges are important and represent up to 80% of the total time needed to complete a VA project. This includes issues of data availability, fragmentation of data, data quality issues, and erroneous values and poorly formatted data, a disconnect between creation and management of data and the needs of data visual analysts, a general absence of record keeping functionality across the board int he visual analysis process. Potential Solutions to these Challenges: directly connect new VA tools to data warehouses to reduce data migration, providing support for simple data transformation in VA tools, and using well designed declarative programming languages in design of VA software. Records professionals could help tool designers and developers articulate the requirements and build record keeping functionality into these systems. This paper has discussed the data issues related to VA and suggested that a broad-based approach to information governance, alongside technical innovation, is necessary to reduce or eliminate data challenges.