S3 and MPI

Over a year passed since I last posted here! From now on, I’ll try to post at least monthly.

Since my PhD at KTH Royal Institute of Technology started, I had the opportunity to work more with cloud computing and high-performance computing, which is very much similar to what I did during my Masters. Now I am working mostly with cloud technologies, with a focus in container orchestration and object storage.

My current work with the latter deals with interfacing the MPI IO library (HPC domain) with the S3 standard (from cloud computing) in order to access object storage systems (i.e., in our work we used Ceph). The paper was accepted to HPCAsia 2023, in which I will be presenting next month. The camera ready version can be seen here.

The work in a nutshell: we essentially interface by using PMPI and replace the write/read for put/get that were derived from AWS SDK. Metadata is saved in YAML files. Tested in three applications, results are not promising due to the hardware we used. Also our library is full of limitations in which I should work another day.