Engineering Resilience for One of the Most Demanding Environments

How a classified organization rethought its storage architecture to keep critical data pipelines running.

The Decision

In environments where systems are fully isolated from the outside world, infrastructure decisions carry extra weight, and there’s little room for error.
A classified organization operating large-scale data pipelines on Kubernetes began encountering growing performance and stability challenges as its workloads scaled. The platform relied on Apache NiFi running on OpenShift, with storage provided via a CSI driver connected to an external storage array that was not designed for this level of demand.

As the system grew, so did the friction. Storage latency increased, recovery times slowed, and the platform became harder to operate under load.

To keep pace with its requirements, the organization needed a more reliable and efficient way to handle data without compromising the constraints of its air-gapped environment.

The Architecture

The existing architecture relied on storage accessed through a CSI driver backed by an external array, which introduced delays and added complexity when workloads scaled or nodes went down.

To address this, the focus shifted to tightly integrating storage with the Kubernetes environment itself. This meant reducing latency, improving recovery behavior, and ensuring the system could continue operating smoothly under pressure.

The Platform

TeraSky worked directly with the Portworx team, part of Everpure, to redesign the storage layer to operate within the Kubernetes platform rather than alongside it.

By leveraging local NVMe drives on each node and combining them into a shared, distributed system, the platform achieved faster data access and built-in resilience. Data could be replicated across nodes, ensuring availability even when infrastructure components failed. This approach also allowed applications to run closer to their data, improving efficiency and reducing unnecessary data movement across the cluster.

TeraSky also improved how workloads are scheduled relative to their data, enabling the system to recover faster and operate more efficiently. The platform team gained greater control over how storage is allocated and used, making it easier to tune performance to workload needs, thanks to a storage architecture purpose-built for performance, resilience, and strict isolation.

Technology Stack

Red Hat OpenShift • Apache NiFi • Portworx Enterprise • NVMe-based distributed storage

The Impact

The new architecture transformed how the platform handles data under real-world conditions.
Where performance bottlenecks and slow recovery once limited the system, the platform now operates with greater consistency and resilience – even under demanding workloads. 6Teams can process data faster, recover more quickly from failures, and maintain tighter control over how the system behaves.
As a platform engineering leader involved in the project described, it:
“The Portworx solution implemented by TeraSky significantly improved the stability, performance, and availability of our platform, with measurable gains across the board. From the initial design through deployment and ongoing operations, the collaboration with both teams was outstanding, and their expertise made the project a success.”