OpenFOAM Parallel I/O¶
OpenFOAM (Open-source Field Operation And Manipulation) is a free, open-source CFD software package with an extensive range of features for solving anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics and electromagnetics.
Using OpenFOAM in HoreKA.¶
OpenFOAM simulations can be done in parallel. For large cases, that won't fit in the memory of a single node this is crucial. However, to run OpenFOAM first the domain of the simulation should be partitioned. After that each process can write and read to its data independently from others.
The decomposition is done using an OpenFOAM utility called decompoesPar. After the decomposition is finished the main simulation can start. All the processes perform the I/O into their own local files. Where each physical property is stored in a unique file, for large scale simulations with thousands of processes this can result into millions of files being created and accessed. This can result into excessive parallel file system load, possibly causing issues to other users as well.
To overcome that issue we suggest doing the decomposition into a job local file system like BeeOND
Script adaptation¶
OpenFOAM workflows are structured as following:
...
$ decomposeParHPC
...
$ mpirun .....
...
$ reconstructParHPC
...
...
$ decomposeParBeeOND
...
$ mpirun .....
...
$ reconstructParBeeOND
...
One very important thing to emphasize, usually users use separate jobs for each phase. That's not possible with our solution since the local file system is available only during the job runtime.
For that reason all the phases should be run in a single job.
Data loss!
Since this approach uses the local file system for storing the processor data, if a job exits prematurely all job data is lost.
For that reason we suggest copying those data manually. For that you could simply login to one of the allocated nodes, from there on copy the data to the case directory using the following command:
BEEOND_DIR="/mnt/odfs/$SLURM_JOB_ID/stripe_default"
CASE_DIR=path_to_foam_case
$ parallel --eta rsync -auv {} $CASE_DIR ::: $BEEOND_DIR/processor*