8 reference architecture designs for data engineering

Just a few short years ago, data processing and storage at scale were the major challenges facing organizations working to leverage data engineering solutions. Most of these challenges have been met by current cloud offerings, so now, data engineering focuses on high-level problems and how to use the tooling and solutions to leverage that data at scale.

For the past few years, Red Hat's Portfolio Architecture team has been developing reference architectures based on customers' real-world use cases in various industries. We have multiple criteria for developing and vetting an architecture collection before we publish it, which you can read about in my intro article about Portfolio Architectures.

We're publishing these architectures for anyone's use on our Red Hat Portfolio Architecture Center and community Portfolio Architecture Examples repository site.

This article presents architectures centered around data engineering. There are currently eight architectures in this collection. I'll provide a short overview of each and allow you to explore them in-depth on your own.

• Cloud to edge: Providing a consistent infrastructure experience from cloud to edge and enabling modern containerized applications at the edge
• Datacenter to edge: Bringing computing closer to the edge by monitoring for potential issues with physical infrastructure (edge)
• Edge manufacturing efficiency: Boosting manufacturing efficiency and product quality with artificial intelligence (AI) and machine learning (ML) out to the edge
• Edge medical diagnosis: Accelerating medical diagnosis using condition detection in medical imagery with AI/ML at medical facilities
• Intelligent Data-as-a-Service: Building and delivering systems and platforms in a secure and scalable manner while driving data needs for moving towards consumerization in healthcare
• Payments: Enabling financial institutions to offer customers fast, easy to use, and safe payment services available anytime, anywhere
• Retail data framework: Creating a framework for access to retail data from customers, stock, stores, and staff across multiple internal teams
• SCADA interface modernization: Providing interfaces with SCADA systems that are compliant with NERC regulations, creating different layers of API gateways to protect business services depending on the network zones

[ Learn how to  build, maintain, and manage data-intensive intelligent applications in a hybrid cloud blueprint. ]

In the Red Hat Portfolio Architecture Center, you'll find a table of contents outlining the technologies used in each architecture, several example schematic diagrams with descriptions, and a link to open the diagrams directly into the online tooling in your browser.

Cloud-to-edge architecture

The cloud-to-edge architecture is for providing a consistent infrastructure experience from cloud to edge and enabling modern containerized applications at the edge.

This project is a new architecture and is currently a work in progress. I'll share one of the schematic architecture diagrams, and you can monitor the project for updates as it progresses to completion.

This architecture brings cloud-like capabilities to edge locations.

Datacenter-to-edge architecture

The datacenter-to-edge architecture covers energy and utility infrastructure companies that operate across a vast geographical area that connects upstream drilling operations with downstream fuel processing and delivery to customers. These companies need to monitor their pipelines' condition and other infrastructure for operational safety and optimization.

The architecture brings computing closer to the edge by monitoring for potential issues with gas pipelines.

Edge manufacturing efficiency architecture

The edge manufacturing efficiency architecture offers additional functionality to manufacturing. The manufacturing industry consistently uses technology to fuel innovation, production optimization, and operations. By combining edge computing, AI, and ML, manufacturers can benefit from bringing processing power closer to data. This helps them take action faster on concerns like errors and predictive maintenance.

This architecture boosts manufacturing efficiency and product quality with AI and ML out to the edge.

[ Learn how open source platforms, including Kubernetes and Ceph, can help you increase data agility for AI/ML. ]

Edge medical diagnosis architecture

The edge medical diagnosis architecture services the healthcare industry. The use case accelerates medical diagnosis using condition detection in medical imagery with AI/ML at medical facilities.

Intelligent Data-as-a-Service architecture

The intelligent Data-as-a-Service (iDaaS) architecture is about building and delivering systems and platforms in a secure and scalable manner while driving data needs for moving towards consumerization in healthcare.

Payments architecture

An offering of (near) real-time payments lets businesses, consumers, and even governments send and accept funds that provide both availability to the recipient and instant confirmation to the sender.

[ Learn how to Modernize your IT with managed cloud services. ]

The payments architecture enables real-time (or at least faster) payments that improve the speed of online payment experiences to customers. This has the potential to give banks a greater opportunity to win, serve, and retain their customers. By building solutions that capture real-time payment business, banks can also drive higher payment volumes, ideally at lower costs, and engage new customer segments.

This architecture helps financial institutions enable customers with fast, easy to use, and safe payment services available anytime, anywhere.

Retail data framework architecture

Retail is the process of selling consumer goods or services to customers through multiple distribution channels to earn a profit. A retail data framework handles managing enterprise retail data. The framework or system sets the guidelines and rules of engagement for business and management activities, especially those dealing with or resulting in data creation and manipulation.

The retail data framework architecture creates a framework for access to retail data about customers, stock, stores, and staff across multiple internal teams.

SCADA interface modernization architecture

The supervisory control and data acquisition (SCADA) interface modernization architecture targets energy providers in North America seeking compliance with North American Electric Reliability Corporation (NERC) regulations by modernizing interfaces between their business applications and their SCADA systems. This modernization also provides better information consumption that can be combined with AI/ML and decision-management tools to address customer needs more effectively.

The architecture provides interfaces with SCADA systems that comply with NERC regulations, creating different layers of API gateways to protect business services based on the network zones.

[ Learn more by downloading event-driven architecture for a hybrid cloud blueprint. ]

Learn more

These are eight of the many reference architectures Red Hat's Portfolio Architects have published, and we'll continue to publish them as we complete them. If you are interested in more architecture solutions like these, feel free to explore the Red Hat Portfolio Architecture Center.

This article originally appeared on Eric D. Schabell's blog and is republished with permission.