LArge Scale IASI data procEssing (LASSIE)

Project ID: LASSIE

Introduction

The MUSICA IASI processing chain has been developed within several third-party funded projects during the last ten years. Here we propose to make this chain much more efficient in order to be able to process data in a timely and reliable manner and to reprocess data at a vast scale. The improved chain will guarantee a sustained, timely and reliable supply of highly consistent MUSICA IASI data. We expect the project to cause a substantial impact on atmospheric water and greenhouse gas cycle research, which are two very decisive fields of climate research (e.g., https://www.wcrp-climate.org/grand-challenges/grand-challenges-overview). In addition, the project will serve as a testbed for the processing of spectra from the upcoming Metop-SG mission, which will involve even more computational challenges (increased number of observations and higher resolution of measured spectra).

Using the reprocessing mode of the efficient chain, we will be able to document the dependency of the computational performance on different elements (code efficiency, processor architectures, usage of CPUs or GPUs, workflow management). This can serve as a guideline for other communities where large-scale data processing is essential and continuous performance improvements are crucial.

Project description 

Satellite-based spectrometers enable us to observe the atmosphere’s composition on a global scale. Nowadays, even operational meteorological satellites are equipped with high-quality spectrometers. A prominent example is the instrument IASI (Infrared Atmospheric Sounding Interferometer) aboard the currently three EUMETSAT (European Organisation for the Exploitation of Meteorological Satellites) satellites Metop-A, -B, and -C. In the framework of the European Research Council project MUSICA (Multi-platform remote sensing of Isotopologues for investigating the Cycle of Atmospheric water, 2011-2016), we have developed an algorithm for retrieving the atmospheric composition of H2O, HDO/H2O ratio, CH4, N2O and HNO3 from IASI spectra measured under cloud-free conditions. The algorithm has been continuously improved during MUSICA follow-up projects (Schneider et al., 2021a). MUSICA IASI data products are of excellent quality (e.g., Wiegele et al., 2014; Borger et al., 2018; García et al., 2018; Diekmann et al., 2021a; Schneider et al., 2021b). This makes them crucial for advanced international water and greenhouse gas cycle studies (e.g., Dahinden et al., 2021; Diekmann et al., 2021b; Toride et al., 2021; Tu et al., 2021), and the scientific community is highly interested in the large scale generation of MUSIC IASI data (see letter attached in the Appendix).

The Metop/IASI mission combines high spectral resolution and low measurement noise with high spatial and temporal coverage (12km ground pixel, global, twice daily). The IASI measurements started in 2007, and in the context of an already approved successor mission (Metop-SG/IASI-NG), observations are guaranteed until the 2040s. These conditions offer unprecedented potential for climate and weather research; however, the total processing of the IASI observations is a severe computational challenge. First, the number of observations that have to be individually processed is very high (the currently three orbiting IASI instruments provide about 3.8 million spectra every 24h). Second, each highly resolved spectrum processing involves large matrices that have to be inverted several times (due to non-linearities of the radiative transfer calculations, several iterations are necessary, e.g. Rodgers 2000).

A flow chart of the MUSICA IASI processing chain is shown in Fig. 1. The chain has already been applied on ForHLR I/II and HoreKa (HPC projects GLOMIR, GLOMIR+, and the extension of GLOMIR+); however, it is still not compatible with the requests of the scientific community of a sustained and quasi-operational global processing and the possibility of reprocessing data at large scales. In this project, we will address this deficit by significantly improving the efficiency of the MUSICA IASI processing on HPC systems of the latest generation and thus adequately support the scientific community.