What are In-memory Data Fabrics?


In recent years, there has been confusion regarding the terms “data grid” and “data fabric,” and one of the main reasons is that neither of them has been well defined even in the tech industry. This article attempts to provide a definition for each so that data fabrics can be defined separately and differentiated from data grids. A great starting point of this discussion would be presenting a general definition for each, starting with the in-memory data grid.

An in-memory data grid is a high-throughput, low-latency computing solution designed for acceleration and scaling of services and applications. It minimizes access to disk for maximum data processing speed, making it a cost-effective solution that’s simple to deploy and scale. It reduces data movement across the network and allows both the application and its data to collocate in the same memory space. Scaling an in-memory data grid is also as simple as adding new nodes to the computer cluster.

An in-memory data fabric, on the other hand, refers to the natural evolution of in-memory computing as a technology. It groups the multitude of in-memory computing use cases into a collection of independent components that work together to create an efficient whole. An in-memory data grid is a typical component of a data fabric, which also comes with an in-memory file system, a compute grid, CEP streaming, and more.

In-memory Data Grid Features

The simplest way to define an in-memory data grid for a broad audience is to view it as a distributed cache with extra features. It helps companies process big data so they can be transformed into actionable insights that can be used to make informed business decisions. By running specialized software within a computer network or “grid,” computers are able to interact with each other to process large jobs that would be difficult or impossible to process using a single computer. An in-memory data grid combines the processing power of several computers, even if these computers are physically spread out in different geographical locations. Despite geographical differences, computers within a data grid can share data and resources with each other while keeping data synchronized to maintain data integrity, a process commonly referred to as grid computing. Accomplishing large, complex tasks is made easier through the use of a data grid.

In contrast to a distributed cache, an in-memory data grid also provides the following features:

  • Distributed queries
  • Distributed transactions
  • Collocation of application and its data

An in-memory data grid is a common implementation for businesses that handle large amounts of data because it ensures speed, availability, and reliability through the use of a distributed cluster. Extremely large amounts of data are processed quickly because they are processed against the full dataset, a process commonly known as “persistent store.” This also allows for the optimization of data so that it can reside on both disk and memory; this is beneficial because the most frequently used data can be stored in memory instead of on disk, minimizing data movement to and from disk storage. The persistent store capability also allows the amount of data to exceed the amount of memory.

In-memory Data Fabric Features

In-memory data fabrics take a holistic approach to computing in that all in-memory computing components can be used independently while they are integrated with each other. Similar to an in-memory data grid, a data fabric can simplify the integration of data management and processing across cloud and on-premise systems, which help in the digital transformation of an organization. This hybrid approach allows for data visibility and actionable insights, helps in overall data access and control, and ensures data security and protection.

Although a data fabric uses multiple database management systems deployed across different platforms, it helps manage data access, processing, security, and integration by providing a consolidated data management platform. Data may be disparate and always moving, but a data fabric helps manage them using a single platform. As new technologies arise and existing ones evolve, new types of data and new platforms will be borne out of necessity and further add to the complexity of modern data management. Depending on an organization’s needs and current data management systems, the changes that need to be implemented to enhance these systems will vary and can lead to approaches that are fragmented by different practices. This reactive strategy is disruptive and unsustainable, requiring a high investment of time, effort, and money. A data fabric is a highly adaptable data management environment that will help organizations catch up as new technologies are developed and will make the overall digital transformation of an organization easier.

An in-memory data fabric typically offers the following features:

  • Unified data management that provides a single framework across multiple deployments
  • Unified data access that provides a single point of access to all data, regardless of structure and deployment platform
  • Cloud mobility that helps speed up migration from one cloud platform to another
  • Consolidated data protection that provides built-in data backup, security, and disaster recovery
  • Centralized service-level management that provides a common process for all types of data and deployment platforms

The Future is “In-memory”

Big data brings with it some big problems—but also bigger technologies. Moving forward, data complexity can be solved not by fighting fire with fire, but by leveraging existing and upcoming in-memory technologies to simplify and speed up data processing and analysis. In-memory computing can process data 800 times faster than disk-based solutions, and this is just the first step toward digital transformation. In the long-term, an in-memory data fabric will allow for horizontal scalability through its distributed architecture, with the added benefit of addressing capacity limits of disk. As data grows bigger and more complex, the best approach to data management now and in the future is through in-memory computing solutions.

Was it worth reading? Let us know.