Iscriviti al feed

Starting in Red Hat Ceph Storage 3.0, Red Hat added support for Containerized Storage Daemons (CSD) which allows the software-defined storage components (Ceph MON, OSD, MGR, RGW, etc) to run within containers. CSD avoids the need to have dedicated nodes for storage services thus reducing both CAPEX and OPEX by co-located storage containerized daemons. 

Ceph-Ansible provides the required mechanism to put resource fencing to each storage container which is useful for running multiple storage daemon containers on one physical node. In this blog post, we will cover strategies to deploy RGW containers and their resource sizing guidance. Before we dive into the performance, let's understand what are the different ways to deploy RGW.

Co-located RGW

  • Does not require dedicated nodes for RGWs (can reduce CAPEX & OPEX).

  • A single instance of Ceph RGW container is placed on storage nodes co-resident with other storage containers.

  • Starting in Ceph Storage 3.0, this is the preferred way to deploy RGWs.

Figure 1: Co-resident RGW deployment strategy

Figure 1: Co-resident RGW deployment strategy

Multi Co-located RGW

  • Does not require dedicated nodes for RGWs (can help reduce CAPEX & OPEX).

  • Multiple Ceph RGW instances (currently tested 2x instances per storage node) co-resident with other storage containers.

  • Our testing showed this option delivers the highest performance without incurring an additional cost.

Figure2: Multiple co-resident (2 x RGW Instance) deployment strategy

Figure 2: Multiple co-resident (2 x RGW Instance) deployment strategy

Standalone RGW

  • Need dedicated nodes for RGWs.

  • Ceph RGW component get deployed on a dedicated physical / virtual node.

  • Starting in Ceph Storage 3.0, this is no longer the preferred way to deploy RGWs.

Figure 3: Standalone RGW deployment strategy

Figure 3: Standalone RGW deployment strategy

Performance Summary 

(I) RGW deployment and sizing guidelines

In the last section, we looked at different ways to deploy Ceph RGW. We will compare the performance differences between each of these methods. To gauge the performance we executed multiple tests by modulating RGW deployment strategy as well as RGW CPU core count across small and large, write and read workloads. The results are as follows.

100% Write Workload

As shown in Chart 1 and Chart 2

  • Co-located (1x) RGW instance outperformed standalone RGW deployment for both small and large object sizes.

  • Similarly, multiple co-resident (2x) RGW instance outperformed co-resident (1x) RGW instance deployment. As such, Multiple co-resident (2x) RGW instances delivered 2328 Ops and 1879 MBps performance for small and large object size respectively.

  • Across multiple tests, the 4 CPU Core / RGW instance was found to be the optimal ratio between the CPU resources to RGW instance. The allocation of more CPU cores to the RGW instance did not deliver higher performance.

Chart 1: Small Object 100% write test

Chart 1: Small Object 100% write test

Chart 2: :Large Object 100% write test

Chart 2: Large Object 100% write test

100% Read Workload Performance

Interestingly for read workload, increasing CPU cores per RGW instance did not deliver performance improvements for both small and large object sizes. As such results from 1 CPU core per RGW instance found to be almost similar to results from 10 CPU core per RGW instance. 

In fact, based on our previous testing we observed similar results that read workloads do not consume a significant amount of CPU, perhaps because Ceph makes use of systematic erasure coding and does not need to decode chunks during read. As such, we found that if the RGW workload is read-intensive, over-allocating CPU does not help.

Comparing results from Standalone RGW to Co-located (1x) RGW test found to be very similar. However just by adding one more co-located RGW (2x) performance improved by ~200% in case of small object and ~90% in the case of large object size.

As such, if the workload is read-intensive, running multiple co-located (2x) RGW instances could boost overall read performance significantly.

Chart 3: Small Object 100% Read test

Chart 3: Small Object 100% Read test

Chart 4: Large Object 100% Read test

Chart 4: Large Object 100% Read test

(II) RGW thread pool sizing guidelines

One of the RGW tuning parameters which are very relevant while deciding CPU core allocation to RGW instance is rgw_thread_pool_size which is responsible for the number of threads spawned by Beast corresponding to HTTP requests. This effectively limits the number of concurrent connections that the Beast front end can service.

To identify the most appropriate value for this tunable we ran tests by varying the rgw_thread_pool_size together with CPU Core count pre RGW instance. As shown in chart-5 and chart-6, we found that setting rgw_thread_pool_size to 512 delivers maximum performance at 4 CPU core pre RGW instance. Increasing both CPU core count as well as rgw_thread_pool_size did not do any better. 

We do acknowledge that this test could have been better if we had run a few more rounds of tests with lower rgw_thread_pool_size than 512. Our hypothesis is that since the Beast web server is based on an asynchronous c10k web server, it does not need a thread per connection and hence should perform as good with lower threads. Unfortunately, we're not able to test but will try to take care of this in the future.

As such multi-collocated (2x) RGW instance with 4 CPU Core per RGW instance and rgw_thread_pool_size of 512 delivers the maximum performance.
Chart 5: Small Object 100% Write test

Chart 5: Small Object 100% Write test

Chart 6: Large Object 100% write test

Chart 6: Large Object 100% write test

Summary and up next

In this post, we learned that multi-collocated (2x) RGW instance with 4 CPU Core per RGW instance and rgw_thread_pool_size of 512 delivers the maximum performance without increasing the overall hardware cost. In the next post, we will learn what is / how to achieve maximum object storage performance from a fixed size cluster.


Sull'autore

UI_Icon-Red_Hat-Close-A-Black-RGB

Ricerca per canale

automation icon

Automazione

Novità sull'automazione IT di tecnologie, team e ambienti

AI icon

Intelligenza artificiale

Aggiornamenti sulle piattaforme che consentono alle aziende di eseguire carichi di lavoro IA ovunque

open hybrid cloud icon

Hybrid cloud open source

Scopri come affrontare il futuro in modo più agile grazie al cloud ibrido

security icon

Sicurezza

Le ultime novità sulle nostre soluzioni per ridurre i rischi nelle tecnologie e negli ambienti

edge icon

Edge computing

Aggiornamenti sulle piattaforme che semplificano l'operatività edge

Infrastructure icon

Infrastruttura

Le ultime novità sulla piattaforma Linux aziendale leader a livello mondiale

application development icon

Applicazioni

Approfondimenti sulle nostre soluzioni alle sfide applicative più difficili

Original series icon

Serie originali

Raccontiamo le interessanti storie di leader e creatori di tecnologie pensate per le aziende