Finding block and file OCP application contents in ODF: Creating the project

As covered in part one of this article series, it can be difficult to map application object locations in OpenShift Data Foundation (ODF) clusters. My goal is to make this process easier for you by demonstrating how to establish a troubleshooting container and then how to use it for both block and file storage mapping.

Be sure to read part one first to understand the environment and necessary toolkit. It's time to start by creating the demonstration project.

Create the project

Next, create two simple test applications, one that uses the block SC and the other that uses the file SC, to perform the tasks of mapping where the data of these applications are being stored in the ODF cluster.

First, create a project named ocs-block-app to host the application:

[alexon@bastion ~]$ oc new-project ocs-block-app
Now using project "ocs-block-app" on server "https://api.example.com:6443".

You can add applications to this project with the new-app command. For example, to build a new example application in Ruby, try:

oc new-app rails-postgresql-example

Or use kubectl to deploy a simple Kubernetes application:

kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname

Then, use one of the templates made available by OCP that makes use of block. It's a good idea is to use the template rails-pgsql-persistent:

[alexon@bastion ~]$ oc get templates -n openshift -o custom-columns=NAME:.metadata.name | grep ^rails | head -1

rails-pgsql-persistent

In the parameters available in the template, it's possible to see that the volume request size is customizable:

[alexon@bastion ~]$ oc process --parameters -n openshift rails-pgsql-persistent | grep -i volume

VOLUME_CAPACITY             
Volume space available for data, e.g. 512Mi, 2Gi                                                                               
1Gi

Run this new application and adjust the volume request size to 5GB:

[alexon@bastion ~]$ oc new-app rails-pgsql-persistent -p VOLUME_CAPACITY=5Gi

--> Deploying template "openshift/rails-pgsql-persistent" to project ocs-block-app

     Rails + PostgreSQL
     ---------
     An example Rails application with a PostgreSQL database. For more information about using this template, including OpenShift considerations, see https://github.com/sclorg/rails-ex/blob/master/README.md.

     The following service(s) have been created in your project: rails-pgsql-persistent, postgresql.
     
     For more information about using this template, including OpenShift considerations, see https://github.com/sclorg/rails-ex/blob/master/README.md.

     * With parameters:
        * Name=rails-pgsql-persistent
        * Namespace=openshift
        * Memory Limit=512Mi
        * Memory Limit (PostgreSQL)=512Mi
        * Volume Capacity=5Gi
        * Git Repository URL=https://github.com/sclorg/rails-ex.git
        * Git Reference=
        * Context Directory=
        * Application Hostname=
        * GitHub Webhook Secret=ahNYBvvbrEFhRjFEX28XdXn04CvMgkbQuABBKruJ # generated
        * Secret Key=m2aq81igaa74gh1shh8vawcywvsxjqjqubywwkcadymj0b1va0krkgsog000ksdusf7h6gyyhoxyh6dcfhw2swsen85o8sq43vod1hvlbtvu7n6x14sn7k4vcs0uwxr # generated
        * Application Username=openshift
        * Application Password=secret
        * Rails Environment=production
        * Database Service Name=postgresql
        * Database Username=userWXH # generated
        * Database Password=pOxlVrav # generated
        * Database Name=root
        * Maximum Database Connections=100
        * Shared Buffer Amount=12MB
        * Custom RubyGems Mirror URL=

--> Creating resources ...
    secret "rails-pgsql-persistent" created
    service "rails-pgsql-persistent" created
   
route.route.openshift.io "rails-pgsql-persistent" created
   
imagestream.image.openshift.io "rails-pgsql-persistent" created
   
buildconfig.build.openshift.io "rails-pgsql-persistent" created
    deploymentconfig.apps.openshift.io "rails-pgsql-persistent" created
   
persistentvolumeclaim "postgresql" created
    service "postgresql" created
   
deploymentconfig.apps.openshift.io "postgresql" created
--> Success
    Access your application via route 'rails-pgsql-persistent-ocs-block-app.apps.example.com' 
    Build scheduled, use 'oc logs -f buildconfig/rails-pgsql-persistent' to track its progress.
    Run' oc status' to view your app.

After a few moments, verify that the application is functional:

[alexon@bastion ~]$ oc status

In project ocs-block-app on server https://api.example.com:6443

svc/postgresql - 172.30.176.29:5432
  dc/postgresql deploys openshift/postgresql:12-el8 
    deployment #1 deployed 4 minutes ago - 1 pod

http://rails-pgsql-persistent-ocs-block-app.apps.example.com (svc/rails-pgsql-persistent)
 
dc/rails-pgsql-persistent deploys istag/rails-pgsql-persistent:latest <-
   
bc/rails-pgsql-persistent source builds https://github.com/sclorg/rails-ex.git on openshift/ruby:2.6-ubi8 
    deployment #1 deployed 3 minutes ago - 1 pod

View details with 'oc describe <resource>/<name>' or list resources with 'oc get all'.

[alexon@bastion ~]$ oc get pods

NAME READY STATUS     
RESTARTS AGE
postgresql-1-deploy 0/1 Completed  
0 4m49s
postgresql-1-k6t47 1/1 Running    
0 4m46s
rails-pgsql-persistent-1-build 0/1    
Completed 0 4m50s
rails-pgsql-persistent-1-deploy 0/1    
Completed 0 3m9s
rails-pgsql-persistent-1-dgfkq 1/1    
Running 0 2m57s
rails-pgsql-persistent-1-hook-pre 0/1    
Completed 0 3m6s

Check the PVC created by the application that it's using the ODF block SC, which in this case is Ceph RBD:

[alexon@bastion ~]$ oc get pvc

NAME STATUS  
VOLUME                                    
CAPACITY ACCESS MODES STORAGECLASS AGE
postgresql  
Bound   
pvc-371faec8-2017-43b4-8416-7003a0d539a9 5Gi       
RWO           
ocs-storagecluster-ceph-rbd  
8m35s

In the PV specifications used by the PVC, within the CSI field, there's an attribute that provides the image name created for the application within the Ceph block pool. You can extract the name from this image as follows:

[alexon@bastion ~]$ oc get pv pvc-371faec8-2017-43b4-8416-7003a0d539a9 -o jsonpath="{.spec.csi.volumeAttributes.imageName}{'\n'}"

csi-vol-24624906-bccb-11eb-9cab-0a580a81023f

With the name of the image in hand, access the toolbox again and list the existing pools:

[alexon@bastion ~]$ oc rsh -n openshift-storage $toolbox

sh-4.4$ ceph df

RAW STORAGE:
    CLASS SIZE       
AVAIL USED RAW USED %RAW USED 
    ssd 1.5 TiB 1.3 TiB    
252 GiB 255 GiB 16.63 
    TOTAL 1.5 TiB    
1.3 TiB 252 GiB 255 GiB 16.63 
 
POOLS:
    POOL                                          
ID STORED OBJECTS USED       
%USED MAX AVAIL 
   
ocs-storagecluster-cephblockpool 1 84 GiB 22.39k    
252 GiB 19.37 350 GiB 
   
ocs-storagecluster-cephfilesystem-metadata 2    
1.4 MiB 25 4.2 MiB 0      
350 GiB 
    ocs-storagecluster-cephfilesystem-data0 3 0 B 0 0 B 0      
350 GiB 

Remember that I mentioned that the ocs-storagecluster-cephblockpool pool is used for blocks? See if you can find your image inside:

sh-4.4$ rados -p ocs-storagecluster-cephblockpool ls | grep csi-vol-24624906-bccb-11eb-9cab-0a580a81023f

rbd_id.csi-vol-24624906-bccb-11eb-9cab-0a580a81023f

There's your image. Display some more information about it:

sh-4.4$ rbd -p ocs-storagecluster-cephblockpool info csi-vol-24624906-bccb-11eb-9cab-0a580a81023f

rbd image 'csi-vol-24624906-bccb-11eb-9cab-0a580a81023f':
           size 5 GiB in 1280 objects
           order 22 (4 MiB objects)
           snapshot_count: 0
           id: 926554e4aba49
           block_name_prefix: rbd_data.926554e4aba49
           format: 2
           features: layering
           op_features: 
           flags: 
           create_timestamp: Mon May 24 20:03:43 2021
           access_timestamp: Mon May 24 20:03:43 2021
           modify_timestamp: Mon May 24 20:03:43 2021

See that it has the same size previously defined for the creation of PVC. And what happens if you increase the capacity of PVC? Increase from 5GB to 10GB:

[alexon@bastion ~]$ oc get pvc

NAME        
STATUS VOLUME                                    
CAPACITY ACCESS MODES STORAGECLASS AGE
postgresql  
Bound   
pvc-371faec8-2017-43b4-8416-7003a0d539a9 5Gi       
RWO           
ocs-storagecluster-ceph-rbd 17m

[alexon@bastion ~]$ oc patch pvc postgresql -n ocs-block-app --type json --patch '[{ "op": "replace", "path": "/spec/resources/requests/storage", "value": "10Gi" }]'

persistentvolumeclaim/postgresql patched

[alexon@bastion ~]$ oc get pvc

NAME        
STATUS VOLUME                                    
CAPACITY ACCESS MODES STORAGECLASS AGE
postgresql  
Bound   
pvc-371faec8-2017-43b4-8416-7003a0d539a9 10Gi      
RWO           
ocs-storagecluster-ceph-rbd 19m

Access the toolbox pod again. The image size has also been modified, reflecting the resize of the PVC:

[alexon@bastion ~]$ oc rsh -n openshift-storage $toolbox

sh-4.4$ ceph df

RAW STORAGE:
    CLASS SIZE       
AVAIL USED RAW USED %RAW USED 
    ssd 1.5 TiB 1.3 TiB    
253 GiB 256 GiB 16.64 
    TOTAL 1.5 TiB    
1.3 TiB 253 GiB 256 GiB 16.64 
 
POOLS:
    POOL ID STORED     
OBJECTS USED %USED MAX AVAIL 
   
ocs-storagecluster-cephblockpool 1 84 GiB 22.41k    
253 GiB 19.39 350 GiB 
   
ocs-storagecluster-cephfilesystem-metadata 2    
1.4 MiB 25 4.2 MiB 0      
350 GiB 
   
ocs-storagecluster-cephfilesystem-data0 3 0 B 0 0 B 0      
350 GiB 

sh-4.4$ rbd -p ocs-storagecluster-cephblockpool info csi-vol-24624906-bccb-11eb-9cab-0a580a81023f

rbd image 'csi-vol-24624906-bccb-11eb-9cab-0a580a81023f':
           size 10 GiB in 2560 objects
           order 22 (4 MiB objects)
           snapshot_count: 0
           id: 926554e4aba49
           block_name_prefix: rbd_data.926554e4aba49
           format: 2
           features: layering
           op_features: 
           flags: 
           create_timestamp: Mon May 24 20:03:43 2021
           access_timestamp: Mon May 24 20:03:43 2021
           modify_timestamp: Mon May 24 20:03:43 2021

What if you want to know which device and host the image is mapped on? The Ceph client retrieves the latest cluster map. The CRUSH algorithm calculates how to map the object to a placement group and then calculates how to assign the placement group to an OSD dynamically. To find the object location, all you need is the object name and the pool name:

sh-4.4$ ceph osd map ocs-storagecluster-cephblockpool csi-vol-24624906-bccb-11eb-9cab-0a580a81023f

osdmap e405 pool 'ocs-storagecluster-cephblockpool' (1) object 'csi-vol-24624906-bccb-11eb-9cab-0a580a81023f' -> pg 1.ecb58a2b (1.b) -> up ([1,0,2], p1) acting ([1,0,2], p1)

Now I know that my object is in a PG that has OSD ID 1 as its primary device, as well as replicas in OSDs ID 0 and 2. On which node in my cluster is the OSD 1 daemon? See below:

sh-4.4$ ceph osd status
+----+------------------------------+-------+-------+--------+---------+--------+---------+-----------+
| id |            
host | used | avail | wr ops | wr data | rd ops | rd data | state |
+----+------------------------------+-------+-------+--------+---------+--------+---------+-----------+
| 0 | ip-10-0-171-63.ec2.internal | 86.7G | 425G | 50 | 517k 
| 0 |    
0 | exists,up |
| 1 | ip-10-0-143-192.ec2.internal | 86.7G | 425G | 109  
| 1224k |   
0 | 0  
| exists,up |
| 2 | ip-10-0-154-20.ec2.internal | 86.7G | 425G | 78 | 1048k 
| 2 |  
106 | exists,up |
+----+------------------------------+-------+-------+--------+---------+--------+---------+-----------+

You can see above that it resides in the node ip-10-0-143-192.ec2.internal. And which device is being used on that node? See below:

sh-4.4$ ceph osd tree

ID CLASS WEIGHT TYPE NAME                                           
STATUS REWEIGHT PRI-AFF 
 -1 1.50000 root default                                                                
 -5 1.50000 region us-east-1                                                         
 -4 0.50000 zone us-east-1a                                                     
 -3 0.50000 host ocs-deviceset-gp2-csi-1-data-085b8h                         
  1 ssd 0.50000 osd.1 up 1.00000 1.00000 
-10      
0.50000 zone us-east-1b                                                     
 -9 0.50000 host ocs-deviceset-gp2-csi-2-data-0n9lkb                         
  2 ssd 0.50000 osd.2 up 1.00000 1.00000 
-14      
0.50000 zone us-east-1c                                                      
-13      
0.50000 host ocs-deviceset-gp2-csi-0-data-0gvt22                         
  0 ssd 0.50000 osd.0 up 1.00000 1.00000 

Now I know which node and device (host ocs-deviceset-gp2-csi-1-data-085b8h) my image is on. But I still don't know what path within that node it is. For this, you'll see another attribute of the PV that will give you this information:

[alexon@bastion ~]$ oc get pv pvc-371faec8-2017-43b4-8416-7003a0d539a9 -o jsonpath="{.spec.csi.volumeHandle}{'\n'}"
0001-0011-openshift-storage-0000000000000001-24624906-bccb-11eb-9cab-0a580a81023f

So, with the node name, device and volume handle information in hand, access the node and get your image. You can do this either by grepping for the image name on the node's current mount points or the name of the volume handle:

[alexon@bastion ~]$ oc debug node/ip-10-0-143-192.ec2.internal
Starting pod/ip-10-0-143-192ec2internal-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.143.192
If you don't see a command prompt, try pressing enter.

sh-4.4# mount | grep 24624906-bccb-11eb-9cab-0a580a81023f

/dev/rbd2 on /host/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-371faec8-2017-43b4-8416-7003a0d539a9/globalmount/0001-0011-openshift-storage-0000000000000001-24624906-bccb-11eb-9cab-0a580a81023f type ext4 (rw,relatime,seclabel,stripe=16)

You can see that the image is mounted on the following path with an EXT4 filesystem by the device/dev/rbd2:

/host/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-371faec8-2017-43b4-8416-7003a0d539a9/globalmount/0001-0011-openshift-storage-0000000000000001-24624906-bccb-11eb-9cab-0a580a81023f

Look at the contents of this directory:

sh-4.4# ls /host/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-371faec8-2017-43b4-8416-7003a0d539a9/globalmount/0001-0011-openshift-storage-000000000000001-24624906-bccb-11eb-9cab-0a580a81023f

lost+found userdata

sh-4.4# ls /host/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-371faec8-2017-43b4-8416-7003a0d539a9/globalmount/0001-0011-openshift-storage-000000000000001-24624906-bccb-11eb-9cab-0a580a81023f/userdata/

PG_VERSION global pg_dynshmem pg_logical pg_replslot pg_stat pg_tblspc pg_xact postmaster.opts
base log pg_hba.conf pg_multixact pg_serial pg_stat_tmp pg_twophase postgresql.auto.conf postmaster.pid
current_logfiles 
pg_commit_ts pg_ident.conf pg_notify    
pg_snapshots pg_subtrans pg_wal postgresql.conf

And as you can see above, these are the contents of your database of the block application you created. 

[ Learn the basics of using Kubernetes in this free cheat sheet. ]

Wrap up

At the start of this article (part two in the series), you created a demonstration project to work with. You also looked at block application mapping within an ODF cluster by using the Rook toolbox and OpenShift commands.

Be sure to read part three, as it contains additional mapping and troubleshooting ideas centered around file storage and mapping.