High-quality remotely sensed satellite data series are important for many ecological and environmental applications. Unfortunately, irregular spatiotemporal samples, frequent image gaps and inevitable observational biases can greatly hinder their application. As one of the most effective gap filling and noise reduction approaches, the harmonic analysis of time series (HANTS) method has been widely used to reconstruct geographical variables; however, when applied on multi-year time series over large spatial areas, the optimal harmonic formulas are generally varied in different locations or change across different years. The question of how to choose the optimal harmonic formula is still unanswered due to the deficiency of appropriate criteria. In this study, an adaptive piecewise harmonic analysis method (AP-HA) is proposed to reconstruct multi-year seasonal data series. The method introduces a cross-validation scheme to adaptively determine the optimal harmonic model and employs an iterative piecewise scheme to better track the local traits. Whenapplied to the satellite-derived sea surface chlorophyll-a time series over the Bohai and Yellow Seas of China, the AP-HA obtains reliable reconstruction results and outperforms the conventional HANTS methods, achieving improved accuracy. Due to its generic approach to filling missing observations and tracking detailed traits, the AP-HA method has a wide range of applications for other seasonal geographical variables.

Satellite remote sensing is a useful tool for obtaining global-scale datasets synoptically [

In the last two decades, considerable effort has been concentrated on developing smoothing and reconstruction techniques for time series, which have mainly examined on seasonal terrestrial vegetation indexes (e.g., normalized difference vegetation index (NDVI)) [

In this study, we propose an adaptive piecewise harmonic analysis (AP-HA) approach for reconstructing long-term satellite time series with various seasonal periodicities. First, a global harmonic fitting approach embedding a cross-validation scheme is used to determine the optimal harmonic order adaptively and provide initial estimates of the missing observations. Second, an iterative piecewise fitting procedure is applied to improve the capability of the harmonic method in tracking local intra-annual and inter-annual traits of the profile. We examine the use of the AP-HA method for the reconstruction of a satellite-derived sea surface chlorophyll-a dataset and achieve significant improvements in performance when compared with conventional HANTS methods.

The HANTS algorithm was primarily developed based on the Fourier transform to deal with irregularly spaced time series and to remove contaminated values in terrestrial NDVI time series [_{r}, Y and ε are the reconstructed, original and error series, respectively. T is the characteristic fundamental period (which is typically one year), t_{i} is the time when Y is observed and _{k}_{k}_{j} is the coefficient of each polynomial term.

The AP-HA algorithm was mainly designed based on the principles of conventional HANTS algorithms [

The original data series of a pixel contains both existing and missing observations. Before reconstruction, 20% of the existing observations in the target data series were randomly selected as the testing dataset and the remaining 80% of the existing observations were the training dataset. The training dataset was mainly used for model fitting, while the testing dataset was used for progressive cross-validation of the fitted results.

In this step, a global HANTS fitting process (shown in Equation (1)) was preliminarily carried out on the training dataset to construct a simple average annual model for the target data series (see

To better account for local intra-annual and inter-annual traits in the multi-year data series, the piecewise fitting approach could be a potential scheme solution, as proposed in AG and DL methods [

The local piecewise fitting and subsequent global merging scheme is illustrated in _{r1} and Y_{r2} are the local functions of subseries from T_{1} to T_{3} and T_{2} to T_{4}, respectively; α is the weight of each observation.

In fact, the average annual pattern constructed by the global HANTS fitting algorithm in step 2 is not strictly equivalent to the actual profile during a specific year, which might result in unrealistic biases between piecewise fitted values (obtained in step 3) and the original values of existing observations. In order to make the fitted values approach the original values of existing observations as closely as possible, an iterative fitting process based on the cross-validation was designed. After the first piecewise fitting process, the training dataset was updated, with the missing observations being replaced by the new fitted values, then the RMSE between the original and piecewise fitted values for the testing dataset was calculated and recorded. Then, we ran another piecewise fitting procedure on the updated training time series and calculated the RMSE for the testing time series from the new piecewise fitted values. This process iterates as the RMSE gradually decreases. The iterations stop when the RMSE starts to increase, then the final reconstructed data series can be obtained as the piecewise fitted data series from the last iteration, which will have the lowest RMSE value (see

The main objective of this study was to construct an improved version of the conventional HANTS algorithms to achieve greater accuracy and to allow adaptive optimization when applied for multi-year time series reconstruction; therefore, to assess the performance of the AP-HA method, it was directly compared with conventional HANTS approaches, which are generally implemented with empirical parameters (i.e., polynomial degree and harmonic order). In this study, the candidate HANTS models were HA31, HA53, HA75 and HA97, with polynomial degrees (L in Equation (1)) and harmonic orders (N in Equation (1)) set to 3 and 1, 5 and 3, 7 and 5 and 9 and 7, respectively. HA-CV refers to the initial global HANTS model obtained through cross-validation in step 2. As stated in

In this study, the weekly composite satellite-derived sea surface chlorophyll-a (CHL) data over the Bohai and Yellow Seas (BYS) of China (_{rs}) images from the moderate-resolution imaging spectroradiometer (MODIS) on the Aqua satellite with local area coverage (LAC) during the period from 4 July 2002 to 31 December 2018 were downloaded for the geographic area of 117–127°E by 31–41°N from the OceanColor website of the Goddard Space Flight Center (_{rs} images were remapped onto a common 1/24° × 1/24° grid based on linear interpolation for convenience and to reduce the data volume for data analysis. A regional statistical GAM (generalized additive model) algorithm was used to calculate the daily sea surface CHL based on _{rs} datasets, which have improved accuracy in both magnitude and seasonality when compared with the standard OC3M (ocean CHL three-band algorithm for MODIS) algorithm [

The AP-HA method and other candidate methods were applied to the CHL dataset pixel-by-pixel to reconstruct long-term complete data series of each pixel. In this section, the AP-HA method was first illustrated on a selected CHL time series as an example. Then, the reconstruction performance of all methods was quantitatively evaluated and visually inspected.

The panels in

The multi-year CHL time series reconstructed by candidate methods were plotted against different years in order to visually compare their capabilities in tracking the intra-annual and inter-annual traits of the original time series. Two original CHL time series associated with the reconstructed profiles are given as examples in

To evaluate and compare the spatial performance of the candidate harmonic-based techniques for CHL data reconstruction, we visually inspected the examples of original CHL and reconstructed images, as shown in

We developed an improved harmonic-based fitting approach to reconstruct long-term time series using MODIS satellite-derived weekly CHL data series for the BYS as an example. To our knowledge, this is the first experiment on piecewise harmonic fitting to reconstruct multi-year incomplete data series. As the results show, the AP-HA method was demonstrated to be an adaptive and reliable technique that is capable of reconstructing multi-year CHL time series with various features, including gap distributions and local profiles traits. The improvement of the AP-HA method was mainly due to the following aspects.

First, a preliminary global HANTS fitting method was employed to prefill the missing observations. In fact, various prefilling algorithms had been adopted prior to application of harmonic fitting in previous studies, confirming that the prefilling method is indeed an efficient way to improve accuracy, as well as to promote the stability of the HANTS fitting method, particular for data series with substantial and continuous gaps [

Second, the cross-validation was first introduced as an objective criterion to determine the optimal harmonic formulas in the HANTS fitting method in this study. In practice, the HANTS fitting based on cross-validation RMSE values achieved the ultimate objective of adaptively determining the optimal harmonic model with a combination of polynomial degrees and harmonic orders. As a consequence, the cross-validation scheme was demonstrated to be an effective criterion for optimal model selection in the HANTS algorithm. Furthermore, the iterative process meant the harmonic fitted values gradually approach the detail local traits of the original data series and quantitatively increased the accuracy of the reconstructed results.

In addition, for piecewise-defined AG and DL algorithms, it is essential to identify the local maximum and minimum points in order to segment the complete data series into several subseries [

At present, there are no continuous field survey plots in the BYS that can provide concurrent CHL time series information for the study period (2002–2018) that could be used for a direct or an indirect validation of the results from this study; therefore, it is impossible to establish one-to-one relations between the reconstructed values and those measured from field-based observations. In this study, the cross-validation method was adopted to select the best model and to evaluate the reconstructed accuracy. These schemes assumed that the satellite-derived CHL values were unbiased, which is generally not true for optically complex coastal waters such as the BYS. In particular, when a testing dataset contains many noisy observations, the outcome is likely to be highly uncertain. Regardless, for missing data reconstruction purposes, the proposed AP-HA method clearly achieved the stated goals of adaptive parameter selection, robust performance and improved accuracy.

The pixel-wise global HANTS fitting approach to constructing an average annual model provides prefilled values for gaps in any location. The representativeness of the constructed average annual model depends on the inter-annual variability of the time series; that is to say, large inter-annual variability implies a poorer fit of the average annual pattern because the real series generally vary over different years. As such, this method is unsuitable for regions with frequent changes of annual shapes in different years. The method used in this study was performed only on data series of a single pixel; therefore, no spatial coherence was considered, although the reconstructed CHL examples generated using the AP-HA approach were spatially coherent and natural looking (as shown in

As a self-adaptive method, the AP-HA method relaxes the constraints on the artificial setting of harmonic orders that occur in conventional HANTS methods, making the implementations more objective; however, several parameters must be predefined before the AP-HA method is implemented, such as the maximum polynomial degree and harmonic order in the initial models. In this respect, further studies are required to solve these issues, or at least to allow a sensitivity analysis of these parameters. Regardless, the AP-HA method is undoubtedly more objective and accurate than the conventional harmonic-based methods. While being substantially more adaptable to a greater variety of practical applications, the AP-HA technique also involves significantly more mathematical overhead owing to the processes involved in optimal model selection and iterative piecewise fitting. In our view, and clearly in the view of others who have previously used such techniques, the added computational effort is well worth the investment.

The HANTS method has been widely used in the smoothing and reconstruction of time series. Appropriate modifications to this method could improve its performance for specific applications. In this study, we proposed an adaptive piecewise harmonic-based fitting approach (AP-HA) that can search for the optimal model needed to reconstruct high-quality multi-year cyclic profiles from incomplete satellite data series. By applying the newly developed method to a weekly satellite-derived sea surface CHL product and by comparing the results with those of the conventional HANTS methods, we found that the AP-HA method has the following advantages: (1) the cross-validation process was adopted to choose the optimal parameters in harmonic models, which is more objective than the widely used empirically predetermined method; (2) the iterative piecewise fitting scheme significantly improved the capability of the harmonic models in tracking the intra-annual and inter-annual traits of multiyear time series; (3) the prefilling of missing values using the preliminary global HANTS means the proposed method is suitable for various data gap conditions.

The outputs of the AP-HA are long-term time series without temporal or spatial data gaps. This method is an improvement of the conventional HANTS method for long-term reconstruction, which can effectively restore cyclic patterns of seasonal variables. Although the AP-HA method has only been examined for CHL data in the ocean, due to its nature, this method is expected to be suitable for a wide range of other datasets with cyclic annual patterns.

Conceptualization, Y.W. and J.N.; methodology, Y.W. and Z.G.; software, Y.W.; formal analysis, Y.W. and J.N.; writing—original draft preparation, Y.W.; writing—review and editing, Y.W., J.N. and Z.G.; funding acquisition, Y.W. All authors have read and agreed to the published version of the manuscript.

This work was supported by the National Natural Science Foundation of China (grant numbers 41706134 and 42030402), and the Open Fund of the CAS Key Laboratory of Marine Ecology and Environmental Sciences, Institute of Oceanology, Chinese Academy of Sciences (grand number: KLMEES202005).

The authors would like to thank the Ocean Biology Processing Group of NASA for providing the MODIS dataset (

The authors declare no conflict of interest.

Flowchart showing the reconstruction process of a time series using the AP-HA method.

Graphical illustration of the two successive local fittings and their merging. Y_{r1} and Y_{r2} are the two local harmonic functions from T1 to T3 and T2 to T4, respectively. Y_{r}* is their merged function for the overlapped period from T2 to T3.

Location of the study area overlaid with a composite image of satellite-derived sea surface chlorophyll-a concentrations during 10–16 September 2013.

Example of a CHL time series and the outputs in the different steps of the proposed AP-HA method. Data series result from the three steps: (

Example of an original unimodal CHL time series associated with the reconstructed time series, which was processed using different methods: (

Example of an original bimodal CHL time series associated with the reconstructed time series, which was processed using different methods: (

Time series of weekly CHL images for weeks 13 to 24 for the year 2011.

Time series of weekly CHL images for week 13 for different years.

Statistics for the RMSE values for the candidate HANTS methods.

Dataset | HA31 | HA53 | HA75 | HA97 | HA-CV | AP-HA |
---|---|---|---|---|---|---|

Training | 0.224 | 0.198 | 0.192 | 0.188 | 0.195 | 0.155 |

Testing | 0.229 | 0.207 | 0.281 | 0.303 | 0.197 | 0.188 |