Date
Data management: what it is, best practices, and platforms used
When it comes to business in the digital age, the importance of data grows day by day, making it a key asset for companies’ success. Regardless of their size or type of business, companies deal every day with an increasing volume and variety of data from an ever-more-complex range of sources.
The informational value of data holds enormous potential for the company, but it must first be extracted from its raw form, prepared, and made available for analytical applications, which—thanks to machine learning and other artificial intelligence techniques—are able to produce unprecedented results from a descriptive, predictive, and generative standpoint.
To manage data and leverage it efficiently, the know-how and experience of individual professionals are no longer sufficient. Manual processes can no longer keep pace with the volume and complexity of data, especially given the need to analyze it in real time. It is therefore necessary to develop a data management strategy that takes into account all aspects related to data, in an increasingly data-driven and customer-centric environment.
In some ways, data management represents a practical solution that is consistent with a company’s data culture. For many, this is a new concept in the digital age, but we’re talking about a process that first emerged back in the 1960s, with the spread of the first mainframe databases—systems capable of managing data according to a predefined hierarchy.
These were the precursors to relational databases, which first appeared in the 1960s and began to gain widespread adoption in the 1980s, culminating in the 1990s with the introduction of modern data warehouses, systems capable of ensuring the end-to-end data quality required for analytical applications.
In recent years, NoSQL databases have emerged, capable of handling even unstructured data, along with systems such as data lakes, which can store enormous amounts of data—even in its raw form—without having to transform it beforehand, but only when it proves useful for analytical applications.
Today, with the spread of the cloud, corporate IT systems are becoming increasingly hybrid in nature, and new concepts are being introduced—this time focused on data visibility—such as data fabric and data observability, whose applications provide a unified view of data regardless of its location, allowing it to be managed in a single virtual location.
Let’s take a look at what data management entails, what a data manager does, and what the best practices and most significant challenges are when it comes to managing data within a company.
What is meant by data management
According to the definition provided by DAMA (Data Management Association), data management consists of:“The development and implementation of architectures, policies, practices, and procedures that effectively manage the needs of an organization’s entire data lifecycle”and“the planning, execution, and oversight of policies, practices, and projects that acquire, control, protect, deliver, and enhance the value of data and information resources.”
In other words, data management consists of all activities associated with the management and governance of corporate data.
Data management is, first and foremost, a strategic endeavor aimed at generating added value from the data companies possess. To ensure that this data is analyzed efficiently, it must be managed in an organized manner, addressing a range of challenges and risks—including those related to cybersecurity and data privacy—in accordance with the provisions of the GDPR and key industry regulations.
The proliferation of IoT systems has recently made the data acquisition landscape even more diverse, through dedicated sensors, smart devices, and audio-video content continuously streamed via multimedia.
For example, when it comes to customer relationships, it is no longer sufficient to rely solely on structured data in a CRM. To gain a better understanding of customers, it is essential to draw on all data regarding interactions with the company’s communication channels: the website, e-commerce, social media, in-store displays, and all other touchpoints included in the omnichannel strategy.
The complexity of today’s scenarios requires absolutely rigorous data management, aimed at supporting analytical applications with the integrity and quality needed to provide useful answers to business needs without risking errors or issues that would undermine the efforts and investments made.
Ultimately, data management becomes an essential activity for companies that aim to acquire, integrate, cleanse, govern, archive, and optimally prepare digital content from all sources with which the business is connected.
Why it is important
To understand how proper and informed data management can help generate added value in an enterprise setting, one need only consider how disciplines such as business intelligence and business analytics can support more informed decisions in many areas, in any context where it is necessary to forecast market trends, manage supply chains, and optimize the time and costs of all processes involved in various activities.
Put simply, data management enables organizations to increase their revenue and profits by optimizing the necessary resources and making their various business activities more efficient.
A well-thought-out data management strategy ensures, both in the short and long term, practical solutions for achieving business objectives by breaking down traditional data silos, thereby fostering synergies among the various business units.
High data visibility ensures that users and the IT devices that access the data have an adequate level of usability to meet any operational need, while also facilitating the achievement of the data quality required by analytical applications to support business decisions.
Data management also fosters innovation, as it “compels” organizations to reevaluate their data strategies by integrating technological systems and established methodologies based on modern software platforms capable of processing data in real time.
Another reason why a data management strategy can no longer be postponed is related to regulatory requirements—specifically, to comply with the provisions of the GDPR and all other regulations governing data retention and processing, as well as data protection and issues related to the company’s cybersecurity.
The types of data management
Data management is an extremely multidisciplinary process that involves various technical elements, such as file management and interoperability, as well as meeting functional requirements—such as those related to archiving—starting with the selection of storage and data management systems.
It is no coincidence that, when it comes to data management, the first step is very often to define the data architecture. Data architecture is essential for managing large volumes of data and addressing the evolving complexity of that data over time, thanks to a blueprint that incorporates management systems such as databases, data lakes, and data warehouses—regardless of whether they are located on-premises or in cloud-based services.
One of the most common types of data management is database administration—systems that organize data in a way that makes it easy to update and accessible to all business applications.
Database administration involves several steps:
- Design
- Configuration
- Installation
- Update
- Maintenance
- Security
- backup / restore
- performance monitoring
Other activities that fall under the umbrella of data management include:
- Data governance: defines enterprise-wide shared policies and procedures to ensure data consistency in all planned operations
- Data quality: attributable to data preparation activities, which aim to clean data, eliminate errors and duplicates, as well as ensure the correct format for all application uses
- Data observability: aims to increase the efficiency of data governance and data quality through improved visibility of the data pipeline, achievable through automated monitoring solutions.
- Data integration: aims to collect and combine data from the different sources with which business systems are connected in order to make them available for analytical applications
- Data modeling: describes the relationships between data and how they are managed in various management systems and analytical applications
- Data fabric: represents an architecture that facilitates the end-to-end integration of various data pipelines and cloud environments through the use of platforms with automation features.
The figure of the data manager
The primary objective of a data manager is to help the companies they work for optimize the use of data in the broadest sense, in compliance with agreed-upon policies and regulations, enabling them to make decisions and take actions that maximize the benefits derived from using that data.
The three main responsibilities of a data manager include:
- Management of customer and personnel activity data;
- Account creation and assignment of database access permissions;
- Maintenance of databases and data management systems;
- Simplification of data collection, preparation and analysis procedures;
- Creation and review of documentation for all database changes;
- Resolution of any inconsistencies or anomalies that might distort analytical results;
- IT infrastructure configuration: software, hardware and data storage;
- Communication with executives and non-technical stakeholders on corporate data issues.
There are many requirements for meeting this wide range of needs. A data manager is a professional who must combine managerial skills with technical expertise in the field of data, in addition to possessing strong soft skills.
The data manager must ensure a data-driven approach throughout the company by fostering a data-driven culture within the organization. In addition to technological considerations, the data manager must be able to communicate effectively with all stakeholders—both technical and non-technical—to identify issues and facilitate their resolution by engaging the relevant company leaders.
In terms of education, data managers typically hold a degree in a data-related field, such as computer science, statistics, or data science specializations. In any case, a degree is merely a starting point, which must be supplemented by solid hands-on experience gained through taking on increasing responsibilities in a corporate setting, thereby developing practical problem-solving skills.
The data manager should also not be confused with the data protection officer (DPO), a role that companies have been required to appoint to comply with the provisions of the GDPR. The data manager focuses specifically on managing the entire data lifecycle within business processes, where compliance with the GDPR—while of crucial importance—is only one of the aspects to be taken into consideration. Similarly, the data manager should never be confused with the IT manager or the privacy manager.
Some best practices of data management
Data management is a task that requires a great deal of effort, especially during the implementation phases, and can be made easier by following a set of best practices. In fact, there are various frameworks that help structure data management within a business context, such as those defined by the aforementioned DAMA or the Data Governance Professionals Organization.
A well-known work in this field is the DAMA-DMBOK (Data Management Body of Knowledge), a text that aims to define a standard framework for the functions and methods involved in data management. Following the first edition (2009), the reference guide was most recently revised as part of a second edition, the DMBOK2 (2017)
Although they are clearly identified in the DMBOK and similar resources, best practices typically involve specialized consultants who are capable of addressing both cultural and methodological aspects. When it comes to best practices, it is advisable to refer to the following activities.
- Databases with functional performance for business needs: in particular, through the implementation of scalable databases capable of optimizing the expenditure of resources due to analytical applications, eliminating both costly waste and performance bottlenecks.
- Fostering data automation and reuse: in this context, the data manager must choose software platforms that can automate data transformation while fostering both data quality and the reuse of data itself in multiple applications.
- Fostering the adoption of AI and ML tools: applications based on machine learning techniques enable the monitoring of data management systems to levels not even conceivable with traditional systems, ensuring high levels of performance and an exhaustive understanding of what is happening on the systems themselves in real time.
- Use of converged databases: These are modern databases designed with native support for all types of data—both structured and unstructured—making them easier to use in various applications. Converged databases facilitate the integration of many emerging technologies into business processes. In addition to the aforementioned machine learning, the contributions of IoT and blockchain cannot be overlooked.
- Implementation of a discovery layer to identify data: allows analysts and data scientists to efficiently identify datasets and make them available for their business intelligence and business analytics applications
- Provision of common query layers to manage data storage: data management platforms have tools that make various storage systems interact in a way that is transparent to the end user, regardless of their physical location.
The platforms used
In the context of enterprise data management, data management platforms (DMPs) are commonly used; these are software applications designed to collect, organize, and analyze data in order to generate value through its use.
As mentioned in the introduction, data collection draws on many channels: websites, social media, email, sales history, etc. Their use has become particularly well-known in the field of marketing, where they were first implemented and have since evolved into the customer data platform (CDP).
Among the leading DMPs are:
- Microsoft Customer Insights and Marketing
- SAS Data Management
- Snowflake
- Oracle Advertising and Customer Experience
- Adobe Audience Manager
- Google Audience Center
Challenges and risks of data management
When defining a data strategy, we inevitably encounter a series of problems that stem largely from the heterogeneity of the data itself: structured, semi-structured, and unstructured. When deciding to implement a data management strategy, we often inherit fragile architectures—the result of successive layers of services and solutions—that have created a series of data silos, making communication between various company departments problematic.
The first challenge in the context of data management is to ensure that datasets are accurate and consistent across all management systems, with a particular focus on improving visibility.
One of the most complex aspects of data management is knowing what data is available and where it is available. This issue is made even more complex by cloud computing, which allows organizations to use managed services offered by different providers on different systems. To facilitate this task, data management platforms provide tools that use metadata for cataloging.
Migrating an on-premises database to the cloud involves adopting next-generation technologies that are not always compatible with traditional systems. It is therefore necessary to evaluate the most appropriate solutions on a case-by-case basis, without necessarily migrating everything from the outset.
If you want to be guided by us, find out about our services
