Resources

Sustainable Development Goal 6: Synthesis Report 2018 on Water and Sanitation (United Nations, 2018).The Millennium Development Goals Report 2015 (United Nations, 2015).Progress on Household Drinking Water, Sanitation and Hygiene 2000–2017: Special Focus on Inequalities (UNICEF and WHO, 2019).Global Health Observatory Data Repository (WHO, accessed 9 June 2022); https://www.who.intMontgomery, M. A. & Elimelech, M. Water and sanitation in developing countries: including health in the equation. Environ. Sci. Technol. 41, 17–24 (2007).Article 

Google Scholar 
Combating Waterborne Disease at the Houshold Level (WHO, 2007).Results of Round II of the WHO International Scheme to Evaluate Household Water Treatment Technologies (WHO, 2019).Chu, C., Ryberg, E. C., Loeb, S. K., Suh, M.-J. & Kim, J.-H. Water disinfection in rural areas demands unconventional solar technologies. Acc. Chem. Res. 52, 1187–1195 (2019).CAS 
Article 

Google Scholar 
McGuigan, K. G. et al. Solar water disinfection (SODIS): a review from bench-top to roof-top. J. Hazard. Mater. 235, 29–46 (2012).Article 
CAS 

Google Scholar 
Fisher, M. B., Keenan, C. R., Nelson, K. L. & Voelker, B. M. Speeding up solar disinfection (SODIS): effects of hydrogen peroxide, temperature, pH, and copper plus ascorbate on the photoinactivation of E. coli. J. Water Health 6, 35–51 (2008).CAS 
Article 

Google Scholar 
Shannon, M. A. et al. In Nanoscience and Technology: A Collection of Reviews from Nature Journals (ed. Rodgers, P.) 337–346 (World Scientific, 2010).Loeb, S., Li, C. & Kim, J.-H. Solar photothermal disinfection using broadband-light absorbing gold nanoparticles and carbon black. Environ. Sci. Technol. 52, 205–213 (2018).CAS 
Article 

Google Scholar 
Loeb, S. K. et al. Nanoparticle enhanced interfacial solar photothermal water disinfection demonstrated in 3-D printed flow-through reactors. Environ. Sci. Technol. 53, 7621–7631 (2019).CAS 
Article 

Google Scholar 
Wigginton, K. R. & Kohn, T. Virus disinfection mechanisms: the role of virus composition, structure, and function. Curr. Opin. Virol. 2, 84–89 (2012).CAS 
Article 

Google Scholar 
Fraise, A. P., Lambert, P. A. & Maillard, J.-Y. Russell, Hugo & Ayliffe’s Principles and Practice of Disinfection, Preservation and Sterilization (Wiley & Sons, 2008).McDonnell, G. E. Antisepsis, Disinfection, and Sterilization: Types, Action, and Resistance (Wiley & Sons, 2020).Burch, J. D. & Thomas, K. E. Water disinfection for developing countries and potential for solar thermal pasteurization. Sol. Energy 64, 87–97 (1998).Article 

Google Scholar 
Sampathkumar, K., Arjunan, T., Pitchandi, P. & Senthilkumar, P. Active solar distillation—a detailed review. Renew. Sustain. Energy Rev. 14, 1503–1526 (2010).CAS 
Article 

Google Scholar 
Wang, Z. et al. Pathways and challenges for efficient solar-thermal desalination. Sci. Adv. 5.7, aax0763 (2019).Article 
CAS 

Google Scholar 
Pang, Y. et al. Solar-thermal water evaporation: a review. ACS Energy Lett. 5, 437–456 (2020).CAS 
Article 

Google Scholar 
Results of Round I of the WHO International Scheme to Evaluate Household Water Treatment Technologies (WHO, 2016).Velmurugan, V., Gopalakrishnan, M., Raghu, R. & Srithar, K. Single basin solar still with fin for enhancing productivity. Energy Convers. Manage. 49, 2602–2608 (2008).Article 

Google Scholar 
Badran, O. O. & Abu-Khader, M. M. Evaluating thermal performance of a single slope solar still. Heat Mass Transf. 43, 985–995 (2007).CAS 
Article 

Google Scholar 
Luzi, S., Tobler, M., Suter, F. & Meierhofer, R. SODIS Manual: Guidance on Solar Water Disinfection (Eawag, 2016).Loeb, S. K. et al. The technology horizon for photocatalytic water treatment: sunrise or sunset? Environ. Sci. Technol. 53, 2937–2947 (2019).CAS 
Article 

Google Scholar 
Hirayama, H., Tsukada, Y., Maeda, T. & Kamata, N. Marked enhancement in the efficiency of deep-ultraviolet AlGaN light-emitting diodes by using a multiquantum-barrier electron blocking layer. Appl. Phys. Express 3, 031002 (2010).Article 
CAS 

Google Scholar 
Shur, M. S. & Gaska, R. Deep-ultraviolet light-emitting diodes. IEEE Trans. Electron Devices 57, 12–25 (2009).Article 
CAS 

Google Scholar 
Khan, A., Balakrishnan, K. & Katona, T. Ultraviolet light-emitting diodes based on group three nitrides. Nat. Photonics 2, 77–84 (2008).CAS 
Article 

Google Scholar 
Zhang, X. et al. Global sensitivity analysis of environmental, water quality, photoreactivity, and engineering design parameters in sunlight inactivation of viruses. Environ. Sci. Technol. 54, 8401–8410 (2020).CAS 
Article 

Google Scholar 
Haag, W. R. & Yao, C. D. Rate constants for reaction of hydroxyl radicals with several drinking water contaminants. Environ. Sci. Technol. 26, 1005–1013 (1992).CAS 
Article 

Google Scholar 
Brown, J. & Clasen, T. High adherence is necessary to realize health gains from water quality interventions. PLoS ONE 7, e36735 (2012).CAS 
Article 

Google Scholar 
Trimmer, J. T. et al. Re-envisioning sanitation as a human-derived resource system. Environ. Sci. Technol. 54, 10446–10459 (2020).CAS 
Article 

Google Scholar 
UN-Water Global Analysis and Assessment of Sanitation and Drinking-Water (GLAAS) 2019 Report: National Systems to Support Drinking-Water, Sanitation and Hygiene: Global Status Report 2019 (WHO, 2019).The United Nations World Water Development Report 2019: Leaving No One Behind (United Nations Educational, Scientific and Cultural Organization, 2019).Enger, K. S., Nelson, K. L., Rose, J. B. & Eisenberg, J. N. The joint effects of efficacy and compliance: a study of household water treatment effectiveness against childhood diarrhea. Water Res. 47, 1181–1190 (2013).CAS 
Article 

Google Scholar 
Hijnen, W., Beerendonk, E. & Medema, G. J. Inactivation credit of UV radiation for viruses, bacteria and protozoan (oo)cysts in water: a review. Water Res. 40, 3–22 (2006).CAS 
Article 

Google Scholar 
Evaluating Household Water Treatment Options: Health-Based Targets and Microbiological Performance Specifications (WHO, 2011).Kohn, T. & Nelson, K. L. Sunlight-mediated inactivation of MS2 coliphage via exogenous singlet oxygen produced by sensitizers in natural waters. Environ. Sci. Technol. 41, 192–197 (2007).CAS 
Article 

Google Scholar 
Guidelines for Drinking-Water Quality 4th edn (WHO, 2011).National Primary Drinking Water Regulations: Long Term 2 Enhanced Surface Water Treatment Rule; Final Rule (US EPA, 2006).Loeb, S., Hofmann, R. & Kim, J.-H. Beyond the pipeline: assessing the efficiency limits of advanced technologies for solar water disinfection. Environ. Sci. Technol. Lett. 3, 73–80 (2016).CAS 
Article 

Google Scholar 
Liu, B., Zhao, X., Terashima, C., Fujishima, A. & Nakata, K. Thermodynamic and kinetic analysis of heterogeneous photocatalysis for semiconductor systems. Phys. Chem. Chem. Phys. 16, 8751–8760 (2014).CAS 
Article 

Google Scholar 
Malato, S., Fernández-Ibáñez, P., Maldonado, M. I., Blanco, J. & Gernjak, W. Decontamination and disinfection of water by solar photocatalysis: recent overview and trends. Catal. Today 147, 1–59 (2009).CAS 
Article 

Google Scholar 
Cho, M., Chung, H., Choi, W. & Yoon, J. Linear correlation between inactivation of E. coli and OH radical concentration in TiO2 photocatalytic disinfection. Water Res. 38, 1069–1077 (2004).CAS 
Article 

Google Scholar 
Cho, M., Cates, E. L. & Kim, J.-H. Inactivation and surface interactions of MS-2 bacteriophage in a TiO2 photoelectrocatalytic reactor. Water Res. 45, 2104–2110 (2011).CAS 
Article 

Google Scholar 
Park, G. W. et al. Fluorinated TiO2 as an ambient light-activated virucidal surface coating material for the control of human norovirus. J. Photochem. Photobiol. B 140, 315–320 (2014).CAS 
Article 

Google Scholar 
Nelson, K. L. et al. Sunlight-mediated inactivation of health-relevant microorganisms in water: a review of mechanisms and modeling approaches. Environ. Sci. Process. Impacts 20, 1089–1122 (2018).CAS 
Article 

Google Scholar 
DeRosa, M. C. & Crutchley, R. J. Photosensitized singlet oxygen and its applications. Coord. Chem. Rev. 233–234, 351–371 (2002).Article 

Google Scholar 
Dobrowsky, P. et al. Efficiency of microfiltration systems for the removal of bacterial and viral contaminants from surface and rainwater. Water Air Soil Pollut. 226, 33 (2015).Article 
CAS 

Google Scholar 
Dobrowsky, P., Carstens, M., De Villiers, J., Cloete, T. & Khan, W. Efficiency of a closed-coupled solar pasteurization system in treating roof harvested rainwater. Sci. Total Environ. 536, 206–214 (2015).CAS 
Article 

Google Scholar 
Abraham, J., Plourde, B. & Minkowycz, W. Continuous flow solar thermal pasteurization of drinking water: methods, devices, microbiology, and analysis. Renew. Energy 81, 795–803 (2015).Article 

Google Scholar 
Spinks, A. T., Dunstan, R., Harrison, T., Coombes, P. & Kuczera, G. Thermal inactivation of water-borne pathogenic and indicator bacteria at sub-boiling temperatures. Water Res. 40, 1326–1332 (2006).CAS 
Article 

Google Scholar 
Sanciolo, P. et al. Pasteurisation for Production of Class A Recycled Water: A Report of a Study Funded by the Australian Water Recycling Centre of Excellence Report No. 1922202665 (Australian Water Recycling Centre of Excellence, 2015).Parry, J. & Mortimer, P. The heat sensitivity of hepatitis A virus determined by a simple tissue culture method. J. Med. Virol. 14, 277–283 (1984).CAS 
Article 

Google Scholar 
Hewitt, J., Rivera‐Aban, M. & Greening, G. Evaluation of murine norovirus as a surrogate for human norovirus and hepatitis A virus in heat inactivation studies. J. Appl. Microbiol. 107, 65–71 (2009).CAS 
Article 

Google Scholar 
Maheshwari, G., Jannat, R., McCormick, L. & Hsu, D. Thermal inactivation of adenovirus type 5. J. Virol. Methods 118, 141–146 (2004).CAS 
Article 

Google Scholar 
Strazynski, M., Krämer, J. & Becker, B. Thermal inactivation of poliovirus type 1 in water, milk and yoghurt. Int. J. Food Microbiol. 74, 73–78 (2002).Article 

Google Scholar 
Fujino, T. et al. The effect of heating against Cryptosporidium oocysts. J. Vet. Med. Sci. 64, 199–200 (2002).Article 

Google Scholar 
Fayer, R. Effect of high temperature on infectivity of Cryptosporidium parvum oocysts in water. Appl. Environ. Microbiol. 60, 2732–2735 (1994).CAS 
Article 

Google Scholar 
Harp, J. A., Fayer, R., Pesch, B. A. & Jackson, G. J. Effect of pasteurization on infectivity of Cryptosporidium parvum oocysts in water and milk. Appl. Environ. Microbiol. 62, 2866–2868 (1996).CAS 
Article 

Google Scholar 
Jarroll, E. L., Hoff, J. C. & Meyer, E. A. in Giardia and Giardiasis (eds Erlandsen, S. L. & Meyer, E. A.) 311–328 (Springer, 1984).Ongerth, J. E., Johnson, R. L., MacDonald, S. C., Frost, F. & Stibbs, H. H. Back-country water treatment to prevent giardiasis. Am. J. Public Health 79, 1633–1637 (1989).CAS 
Article 

Google Scholar 
Schaefer, F. W., Rice, E. W. & Hoff, J. C. Factors promoting in vitro excystation of Giardia muris cysts. Trans. R. Soc. Trop. Med. Hyg. 78, 795–800 (1984).Article 

Google Scholar 
Global Solar Atlas 2.0 (World Bank Group, 2020); https://globalsolaratlas.info/R Core Team. R: A language and environment for statistical computing (R Foundation for Statistical Computing, 2021).Campolongo, F., Cariboni, J. & Saltelli, A. An effective screening design for sensitivity analysis of large models. Environ. Model. Softw. 22, 1509–1518 (2007).Article 

Google Scholar 
Saltelli, A. Sensitivity analysis for importance assessment. Risk Anal. 22, 579–590 (2002).Article 

Google Scholar 
Sobol, I. M. Sensitivity analysis for non-linear mathematical models. Math. Modell. Comput. Exp. 1, 407–414 (1993).
Google Scholar 
Saltelli, A., Tarantola, S., Campolongo, F. & Ratto, M. Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models Vol. 1 (Wiley Online Library, 2004).Zhang, T. et al. A global perspective on renewable energy resources: NASA’s prediction of worldwide energy resources (power) project. In Proc. ISES World Congress 2007 Vol. 1–Vol. 5 (eds Goswami, D. Y. & Zhao, Y.) 2636–2640 (Springer, 2009).Stackhouse, P. Jr. et al. Surface Meteorology and Solar Energy (SSE) Release 6.0 Methodology version 3.2.0 (NASA, 2016).Stackhouse, P. Jr. et al. Supporting energy-related societal applications using NASA’s satellite and modeling data. In Proc. 2006 IEEE International Symposium on Geoscience and Remote Sensing (ed. Tsang, L.) 425–428 (IEEE, 2006).World Development Indicators (World Bank, accessed 9 June 2022); https://datacatalog.worldbank.org/dataset/world-development-indicatorsHaitz, R. H., Craford, M. G. & Weissman, R. H. In Handbook of optics Vol. 2 (ed. Bass, M.) 121–129 (Optical Society of America, 1995).García-Gil, Á., Abeledo-Lameiro, M. J., Gómez-Couso, H. & Marugán, J. Kinetic modeling of the synergistic thermal and spectral actions on the inactivation of Cryptosporidium parvum in water by sunlight. Water Res. 185, 116226 (2020).Article 
CAS 

Google Scholar  More

Adjusting GEFS forecasts to local climatologyWhat amount of correction is required for GEFS forecasts to align with CHIRPS local climatology? The amount of correction varies widely across the globe and throughout the year. Figure 1a shows annual mean bias for GEFS reforecast 15-day totals. In this figure, wetter-than-CHIRPS climatology and systematic over-prediction of 15-day totals by GEFS is indicated by positive mean bias values, while the opposite is indicated by negative values. GEFS forecast mean bias was calculated for each month and then averaged across rainy season months, to focus aggregate results on the rainfall seasons, when precipitation forecasts are relevant. Monthly dry masks excluded locations with a monthly average of less than 10 mm, according to CHIRPS climatology. In general, one consistent result from Fig. 1a is a tendency to increase precipitation in many mountainous tropical and subtropical regions. By design, orographic precipitation enhancements in such regions are represented fairly well in CHIRPS, and these are carried through to CHIRPS-GEFS precipitation forecasts. The CHIRPS-GEFS bias-correction process reduces systematic errors (Fig. 1b), with the overall mean absolute bias error going from 24.1 mm for GEFS to 19.7 mm for CHIRPS-GEFS, an ~18% reduction.Fig. 1Annual mean bias and global error characteristics for GEFS reforecast data compared to CHIRPS, based on 15-day precipitation totals from Day 1, 6, 11, and 16 of each month during 2000–2019. Annual mean bias (a) shows the annual average of differences in GEFS reforecast and CHIRPS monthly means. Annual average error (b) shows the distribution of GEFS reforecast and CHIRPS-GEFS errors (product – CHIRPS). Both panels are based on in-season pixels, which are defined by monthly average CHIRPS  > 10 mm.Full size imageFigure 1a through Fig. 5 are based on GEFS reforecast, CHIRPS, and CHIRPS-GEFS data for the 5-day or 15-day periods beginning on the 1st, 6th, 11th, and 16th day of the month. All these exclude dry season months. Figure 1b shows the corresponding global distribution of annual average error for the GEFS reforecast and CHIRPS-GEFS, and is discussed later.GEFS has a large annual average positive bias of higher-than 40 mm in some areas of the globe, including in central Mexico, Central America, northern South America, the Andes and Himalayan Mountain ranges, and in southern China, Papua New Guinea, and localized areas of central Africa, the Ethiopian Highlands, and the western montane United States (Fig. 1a). GEFS has positive bias, by more than 5 mm for the annual average 15-day period, across the northern United States including in the Midwest, from Mexico’s northern mountains through most of Central America, in northern South America, the Andes range, eastern Brazil, in parts of central Europe, central and northern Asia, in the area from southern China to Myanmar and Thailand, and in northeastern and western India. GEFS has positive bias in portions of East Africa (Rwanda, Burundi, Tanzania, western Ethiopia), West Africa (Cameroon, Gabon), and Southern Africa (Zambia, central Angola, northern Zimbabwe, eastern South Africa). GEFS has negative bias, by more than 5 mm on average, in parts of central and northern Africa, Senegal, northern Australia, central South America, western India, the Yucatan peninsula, and the United States Gulf Coast.GEFS’ systematic bias changes throughout the year, as shown by the monthly mean bias in January, April, July, and October (Fig. 2). This is unsurprising, given that drivers of weather change too, but higher bias in particular months can be problematic for forecast users. In Ethiopia, for example, GEFS overestimates by large amounts during the Kirempt season (e.g., in July) and in October in the southwest. In central Brazil, the bias changes markedly by season, from a high negative bias in October to an expansive wet bias in April. In the Midwestern and northern United States, GEFS also shows a more expansive wet bias in April than in January, July, or October. In some areas, like in southern China and the Andes mountains, GEFS means are higher than CHIRPS means throughout the year.Fig. 2Monthly mean bias for GEFS reforecast data compared to CHIRPS, based on 15-day precipitation totals from Day 1, 6, 11, and 16 of each month during 2000–2019. Mean bias for January (a), April (b), July (c), and October (d) shows the difference in GEFS reforecast and CHIRPS monthly means. Shown for in-season pixels, which are defined by monthly average CHIRPS  > 10 mm.Full size imageThe CHIRPS-GEFS downscaling procedure corrects for systematic errors in GEFS forecasts that vary spatially and temporally. To assess the efficacy of the CHIRPS-GEFS approach, we began by calculating the per-pixel difference between GEFS and CHIRPS, and CHIRPS-GEFS and CHIRPS for 15-day periods. These were calculated for each month, for in-season pixels, and then averaged across the year. We then looked at the histogram of the resulting differences (Fig. 1b), to identify the distribution of annual average errors in these two products. CHIRPS-GEFS errors are shown as gray bars and GEFS errors are overlaid as hollow red bars. A desirable pattern is more small errors (higher bars close to 0 mm) and fewer large magnitude errors (lower bars at larger precipitation values). As shown in Fig. 1b, the bias-correction procedure has this effect, and results in CHIRPS-GEFS having overall lower errors for global rainy seasons compared to GEFS. GEFS 15-day errors more commonly involve over prediction of observed amounts than under prediction, as shown by the higher proportion of positive versus negative moderate to large positive errors. Part of this is due to the lower limit of under prediction being zero precipitation, while over prediction can range from marginal precipitation amounts to very high amounts. As shown in Fig. 1b, the CHIRPS-GEFS bias correction particularly reduces GEFS forecast errors for moderate-to-high rainfall amounts, and it results in a global 15-day error distribution that has a higher proportion of small errors, e.g., errors within −10 mm to 10 mm of CHIRPS values (51% for CHIRPS-GEFS and 43% for GEFS). Errors in categories ranging from 10 mm to 40 mm occur less often in CHIRPS-GEFS, globally, with probabilities in those categories reduced by around 15 and 25 percent at 10 mm to 20 mm and 20 mm to 30 mm, respectively, and by around 30 percent to 40 percent for errors that are higher than 40 mm.Next, we show performance of the 5-day and 15-day CHIRPS-GEFS precipitation forecasts by correlations and mean absolute errors for the historical record, compared to CHIRPS data for these periods. As described in Data Records, multiple outlets use forecast amounts for these periods. In the Usage Notes section, probability of detection scores for 15-day CHIRPS-GEFS in Africa are presented while describing an operational application of the CHIRPS-GEFS for seasonal monitoring. In that discussion we also examine the performance of 5-day forecasts during the 2020–2021 season in key regions of Kenya, Angola, Zambia, Zimbabwe, and Madagascar.Pearson correlation coefficients for 5-day and 15-day CHIRPS-GEFS, compared to CHIRPS (Fig. 3), indicate the ability of forecasts to predict deviations from average. It should be noted that correlations are nearly entirely driven by the information coming from the GEFS forecasts. The conversion to CHIRPS-GEFS adjusts the GEFS values to make them more “CHIRPS-like,” while also approximating the historical context of the GEFS forecast. Wet extremes forecasted by GEFS translate into wet extremes in CHIRPS-GEFS. Areas with very low correlations (R  0.7) are the United States, Western Europe, and Eastern Europe, southeastern South America, southern Central Asia, eastern China, parts of East and Southern Africa, and Australia. Globally, correlations are higher in January, April, and October than in July, which indicates generally higher forecast accuracy in those months. Exceptions are in eastern China, southern Brazil, eastern Mexico, northeastern Ethiopia, and central and southern Australia, where July correlations are not substantially lower. 15-day forecasts also have high correlations in some areas, including in the Western and Midwestern United States in January, in central and northern Australia in April, and in eastern Brazil in January and October.Fig. 3CHIRPS-GEFS 5-day and 15-day Pearson correlation coefficients, as compared to CHIRPS, for January, April, July, and October. (Validation data: CHIRPS 5-day and 15-day totals from the 1st, 6th, 11th, and 16th of the month, for 2000 to 2019. Shown for in-season pixels, which are defined by monthly average CHIRPS  > 10 mm.Full size imageIn Africa, a region where CHIRPS data is actively used by the Famine Early Warning System Network (FEWS NET) and other organizations for seasonal monitoring and drought early warning, forecast correlations indicate moderate to good 5-day and 15-day forecast performance in areas of East Africa, Southern Africa, and western North Africa during rainy season months. Some of the highest 15-day correlations in Africa are during important rainy season months, for example, in northeastern Ethiopia in July and April, in Kenya in April, in Zimbabwe and southern Mozambique in January, and in the Sudanian zone of West Africa in October. Very low correlations indicate low forecast skill in the Sahel, coastal West Africa, and in Central Africa in the DRC, Republic of the Congo, and Gabon.Mean absolute error of the bias-corrected GEFS forecasts highlight the areas where forecast amounts have historically been less reliable (Fig. 4). These indicate non-systematic errors associated with rains not materializing in the forecast location in the forecast period, which can be from GEFS model deficiencies and the inherent challenges of weather forecasting. Extreme precipitation events and warm season, deep moist convection-driven precipitation are notorious challenges for numerical weather prediction systems48,49, and CHIRPS-GEFS data are not immune to this problem. Remotely sensed data, including CHIRPS, also struggle with estimating extreme high rainfall amounts13,50, though since we are comparing CHIRPS-GEFS to CHIRPS, the main source of the large errors shown here would be the GEFS reforecast.Fig. 4CHIRPS-GEFS 5-day and 15-day mean absolute errors, as compared to CHIRPS, for January, April, July, and October. Validation data: CHIRPS 5-day and 15-day totals from the 1st, 6th, 11th, and 16th of the month, for 2000 to 2019. Shown for in-season pixels, which are defined by monthly average CHIRPS  > 10 mm.Full size imageAs shown in Fig. 4, the magnitude of errors follows climatology, with 5-day errors typically under 10 mm for drier rainy season months. In wetter months and locations errors are typically between 10 mm and 20 mm. With higher rainfall magnitude there is greater potential for larger errors. The 15-day forecast errors exhibit a similar spatial pattern to the 5-day errors, and error magnitudes correspond to the three-times larger accumulation interval as well as expected lower skill at longer lead time. Figure 4 shows especially large 15-day mean absolute errors in January near northern Mozambique and Madagascar, in July and October in parts of Central America, in April in central Kenya and southwestern Tanzania, in July in India’s Western Ghats Mountains and in the Himalayas, and in the Maritime Continent. In southeast China, while the 15-day correlations indicated decent skill at forecasting the sign of precipitation anomalies, large 15-day errors indicate the influence of poorly forecast large storms, which unbiasing cannot correct for. In the Amazon rainforest, many areas with low correlations also have high forecast errors, underscoring poor forecast performance there. More

Availability per capitaThe APC water stress indicator represents the state of physical water scarcity. The total population under a certain level of water scarcity is called the stressed population. We calculated the water-stressed population and compared it to earlier estimates for validation. We found that the total population percentage (calculated using the ensemble mean discharge) facing acute physical water stress calculated using the APC of 500 m3/capita/year will vary as to ({54.9}_{-1.7}^{+1.1} %;({47.6}_{-2.5}^{+2.1} % )), ({66.6}_{-3.3}^{+2.8} %;({59.8}_{-6.1}^{+5.6} % )), and ({55.6}_{-1.8}^{+4.2} % ;({47.0}_{-2.6}^{+5.7} %)) (+/− values show the maximum variation considering discharge using single GCM to the ensemble mean discharge) at the end of the century (i.e., the year 2099) under the SSP1–RCP2.6, SSP3–RCP7.0, and SSP5–RCP8.5 scenarios, respectively, representing different socioeconomic and climate conditions considering the MY19 (JO16) future population dataset (methods for scenarios and datasets details). By contrast, 44.5% (45.1%) of the global population faced acute physical water stress at the beginning of the century (i.e., the year 2000). The above percentages correspond to 3.5 (3.3), 7.9 (7.5), and 3.9 (3.4) billion populations for the SSP1–RCP2.6, SSP3–RCP7.0, and SSP5–RCP8.5 scenarios and 2.68 (2.75) billion for the historical scenario (i.e., beginning of the century). The historical value is consistent with the value of 2.7 billion previously reported by Hoekstra et al.13 and 2.4 billion mentioned by Oki and Kanae1.APC enhanced with GDP per capita—country-scale assessmentNext, we analysed the relationship between APC and GDP per capita. First, to revisit the findings of Oki et al.19, we conducted country-level analyses for the beginning (i.e., the year 2000) and end (i.e., the year 2099) of the century. To compare the absolute change for a longer period with the constant exchange rate, we used GDP-PPP per capita (USD 2005) due to its availability and defined water stress (physical and economic water scarcity) for both past and future scenarios using the same threshold line (see “Methods” section). The consistency in the results in terms of distribution of countries (Fig. 1 and Supplementary Fig. 1) for both historical (GPWv4 and HYDE3.2) and future (MY19 and JO16) population datasets confirm the similarity in aggregated country-level population data. We did not find any countries below the threshold line at the end of the century, whereas we found Somalia, Western Sahara, Yemen, and Niger below the threshold line at the beginning of the century (Fig. 1, Supplementary Table 1 for base scenario experimental settings, results for other combinations of climate and population dataset are provided as Supplementary Fig. 1). The comparison of per capita water availability (APC) for countries below the threshold line for this study and additional analysis considering various climate forcing data with the same socioeconomic data showed substantial differences. These arid region countries have less runoff and considerable sensitivity towards the metrological data, causing the large difference in availability per capita (APC) of freshwater (Supplementary Table 2 for comparison of values considering different climate forcing data). Additionally, the quality of socioeconomic data contains uncertainty due to political instability23,24,25,26, defying the hypothesis for these countries. We confirmed that although a few countries can contradict, the hypothesis of Oki et al.19 remains valid for various scenarios considered.Fig. 1: Country-level scatter plot for APC vs GDP-PPP per capita and density plot considering the number of countries for various socioeconomic and climate scenarios.Each circle corresponds to a country, and the circle’s size corresponds to the country’s population. CHN, ESH, IND, NER, SOM, USA, and YEM represent China, Western Sahara, India, Niger, Somalia, United States of America and Yemen, respectively. Yellow, green, red, and blue colours represent the historical, SSP1–RCP2.6, SSP3–RCP7.0, and SSP5–RCP8.5 scenarios, respectively, and the dashed line represents the threshold value for physical and economic water scarcity. The analysis was performed considering the GPWv4 dataset for the historical, i.e., the year 2000 population and MY19 for the future, i.e., the year 2099 population.Full size imageAPC enhanced with GDP per capita—grid-scale assessmentNext, we proceeded with grid-level analyses. We confirmed the existence of locations in the world facing the challenges of economic and physical water scarcity identified at 0.5° resolution (Fig. 2, results of SSP1–RCP2.6 and SSP5–RCP8.5 are shown in Supplementary Fig. 2 and Supplementary Fig. 3). The total population and spatial distribution facing challenges (i.e., grids below the threshold line defined by Eq. 1) differed in the different scenarios.Fig. 2: Grid-level scatter plots for APC vs GDP-PPP per capita and density plot considering the number of grids.(a) Historical-GPW, (b) Historical-HYDE, (c) Future370-MY19, and (d) Future370-JO16 scenarios. Grid values are represented as circles, and the dashed line represents the threshold line proposed by Oki et al.19. The density plot includes dotted coloured lines (lime and red) for the median and dark shading for the interquartile range (first and third quartiles). The white circle represents the grid size of 20 million population. e Boxplot for the total population facing physical and economic water scarcity (grids below the threshold of Eq. 1) for all considered scenarios. Legend symbols represent the analysis using the discharge considering various GCMs and the ensemble mean of discharge considering all GCMs. *analysis for Historical-GPW and Future-MY scenarios, **analysis for Historical-HYDE and Future-JO scenarios (Supplementary Table 1 for scenarios/ experiment settings, and Supplementary Table 5 for water-scarce population and uncertainty values).Full size imageIt can be observed from the density plots in Fig. 1 and Fig. 2 that there is a rightward shift in the peak and a significant increase in the mean and median values of the GDP-PPP per capita for the future scenarios compared to the historical scenario. The density plot for the APC for the future follows a trend similar to the trend of the past (i.e., a similar frequency distribution of APC at the grid scale), with an increase (decrease) in the median values observed for the future scenarios for MY19 (JO16) at the grid level (Supplementary Table 3 and Supplementary Table 4 for results of all statistical analyses considering future and historical datasets).We calculated the population facing hardship due to both physical and economic water scarcity (i.e., grids below the threshold line defined by Eq. 1). As a result, at the end of the 21st century (i.e., the year 2099), ({0.32}_{+0.00}^{+0.68}) (({234}_{-10}^{+24})) million people were estimated to face hardship under the SSP1–RCP2.6 scenario when using an urban-concentrated, i.e., MY19 (dispersed, i.e., JO16) population dataset. The estimated populations facing hardship under the SSP3–RCP7.0 and SSP5–RCP8.5 scenarios were ({327}_{+35}^{+202}) (({665}_{-67}^{+181})) and ({6.9}_{-1.1}^{+1.2},left({176}_{-3}^{+36}right)) million respectively (+/− values show the maximum variation in the global population facing water scarcity, calculated considering discharge using single GCM and the ensemble mean discharge), compared to 327 (358) million at the beginning of this century (i.e., the year 2000) (Fig. 2e, Supplementary Table 5). Analysis considering MY19 and JO16 population datasets yield three orders of difference in the stressed population at maximum. The total number of water-stressed populations would decrease in the future (except for the SSP3-RCP7.0 with JO16 population distribution i.e., Future370-JO16 experiment) due to an increase in income.Analysis considering various scenarios (Supplementary Table 1 for scenarios) shows that the uncertainty associated with the SSP–RCP scenarios (i.e., maximum, and minimum difference in the population facing scarcity considering any two scenarios among SSP1-RCP2.6, SSP3-RCP7.0, and SSP5-RCP8.5 for the ensemble mean discharge) and global climate models (GCMs) (i.e., maximum and minimum difference in the population facing scarcity considering any two GCMs for a particular SSP-RCP scenario) were in the range of 6.58–489 and 0.03–248 million, respectively (Supplementary Table 5).We found that the population distribution uncertainty (i.e., maximum and minimum difference in the population facing scarcity considering MY19 and JO16 gridded population distribution for a particular SSP-RCP scenario) for the end century (i.e., the year 2099) followed a similar trend and was in the range of 169.1–338 million (Supplementary Table 5). At the same time, the uncertainty at the beginning of the century (i.e., the year 2000) was within ~10 %, considering GPWv4 and HYDE3.2 gridded population datasets, confirming the high accuracy in estimation of historical population and their distribution. The maximum range value is brought by SSP3-RCP7.0, which is attributed to the large dispersion of population distribution in the SSP3. The grid-level analyses revealed that the future prediction includes large uncertainty due to the spatial distribution of within-country population along with the SSP–RCP paths of global sustainability (SSP1–RCP2.6), regional rivalry (SSP3–RCP7.0), and economic optimism (SSP5–RCP8.5) taken by the world (Fig. 3). The number of water-scarce grids (i.e., grids below the threshold line) in the future will increase or decrease compared to the past and depend mainly on the spatial distribution of population and GDP compared to freshwater availability.Fig. 3: Physical and economic water-scarce regions.a Historical (2000) considering the GPWv4 dataset, (b) Historical (2000) considering the HYDE3.2 population dataset, (c) Future (2099) considering the MY19 population dataset, and (d) Future (2099) considering the JO16 population dataset scenarios. The future (2099) case shows the possible combinations of scenarios with different colours; the values inside the circular legend show the number of people (in millions) facing scarcity with ranges representing the minimum and maximum values considering scenarios combination.Full size imageFactor decompositionThe spatial distribution of grids below the threshold line of various historical and future scenarios (Fig. 3) showed that there would be an emergence of new water-scarce grids in the future, i.e., new grids facing water scarcity in future scenarios but were not facing water scarcity in the historical scenarios. These grids will face water scarcity either due to the decrease in freshwater availability (climate change) or GDP-PPP (socioeconomic change) or an increase in the population (socioeconomic change) among the considered variables for the analysis. Fig. 4 presents the boxplot distributions of absolute values for freshwater (mm/year), population density (capita/km2), and GDP-PPP (USD/year) for newly identified water-scarce grids (grids facing scarcity in the future but not facing it in the past), comparing the values for the historical and future scenarios. The freshwater availability (mean and median values) does not change significantly over time for the new water-scarce grids, i.e., the difference between the future scenarios (SSP1–RCP2.6, SSP3–RCP7.0, and SSP5–RCP8.5) and the historical scenario is negligible. Compared to freshwater, there is a significant increase in population density for all considered scenarios and a less significant increase (decrease) of GDP-PPP of the grids (regions) for the MY19 (JO16) population datasets (Supplementary Table 6 for statistical analysis), suggesting that the primary reason for the water scarcity in these areas will be population growth.Fig. 4: Comparison of new water-scarce grids (i.e., grids facing physical and economic water scarcity in the future but not in the past).Box plots comparing the absolute values of (a), (d) freshwater availability (mm/year); (b), (e) population density (capita/km2); and (c), (f) GDP-PPP. The analysis for (a), (b), and (c) was performed considering the Future-MY and Historical-GPW Experiment settings, and that of (d), (e), and (f) was performed considering the Future-JO and Historical-HYDE experiment settings (Supplementary Table 1). The error bars show the 100% confidence interval (i.e., 0th and 100th percentile), the bottom and top of the box are the 25th and 75th percentiles, and the line inside the box is the median (50th percentile).Full size imageThe global water scarcity analysis considering various future scenarios (SSP1–RCP2.6, SSP3–RCP7.0, and SSP5–RCP8.5) identify various possible water stress regions (grids below the threshold line) of the world affecting the different number of populations. The common water-scarce grids recognised in all these scenarios (grids showing water scarcity for the SSP1–RCP2.6, SSP3–RCP7.0, and SSP5-RCP8.5 scenarios simultaneously) have the highest possibility (certainty) of facing water scarcity in the future. We compared the sensitivity analyses (methods for the approach adopted and Supplementary Table 7 for sensitive analysis experiment settings) results with the base scenario (Supplementary Table 1) values to know the major factor causing water stress among the considered variables for the grids with the highest possibility of water scarcity. The water-scarce population, which can be simultaneously identified in all future scenarios, will be in the range of 0.46–1.82 (156–393) million (range shows the minimum and maximum population affected considering all three future scenarios), considering the historically available freshwater for future scenarios, i.e., Historical-MY (Historical-JO) experiments. Similarly, the population affected considering the historical population for the future scenarios, i.e., Future-GPW (Future-HYDE) experiments, was determined to be 13 (10–16) million; considering the historical GDP-PPP for the future scenarios, i.e., Future-MY-TG, (Future-JO-TG) experiments, the result was 1514–2928 (1466–3132) million (Supplementary Fig. 4 and Supplementary Fig. 5, Supplementary Table 7 for experiment settings). The comparison of all sensitive analysis scenarios values with the base scenario value of 0.0 (110–269) million (Fig. 3c, d) showed that the effects of the different variables were in the order of GDP  > population > climate for the regions with the highest chances of facing water scarcity in future.Even though the overall water availability on the globe per capita are 6525.16 (6434.99) m3/capita/year for historical (i.e., the year 2000) and 6960.63 (6375.98) m3/capita/year, and 3894.64 (3671.39) m3/capita/year, 6821.34 (6459.90) m3/capita/year for future (i.e. the year 2099) considering SSP1-RCP2.6, SSP3-RCP7.0, and SSP5-RCP8.5 scenarios respectively (values in bracket consider HYDE3.2 and JO16 population datasets), more than 70% of world population faces the physical water scarcity defined using a threshold value of 1700 m3/capita/year of APC15 for all the scenarios (Supplementary Table 8, and Supplementary Note 1). Estimation of population facing severe water stress considering physical aspect only (i.e., APC of 500 m3/capita/year) is 2.7 billion for historical scenarios and 3.9–7.9 (3.3–7.5) billion for the future scenario, whereas considering both physical and economic aspects (i.e., threshold line defined by Oki et al.19) is 301 (333) million for the historical scenarios and 0.33–325 (176–665) million for the future scenarios (Supplementary Fig. 6 and Supplementary Fig. 7). These values show a substantial difference in the water-stressed population when considering only physical aspects and accounting for both physical and economic factors. It also indicates that a few rich (i.e., grids with high GDP-PPP per capita) physical water-scarce regions (water-stressed regions identified using APC) can ease water scarcity by water management and technological measures.The overall analysis revealed the possibility of underestimation (or overestimation) of the population facing scarcity in the future due to large differences associated with the population and GDP data distribution within the country for the SSP scenarios. The spatial distribution of the future population and GDP within and outside a country can be affected by many factors, such as water availability27,28, job opportunities, disaster adaptation and mitigation capability of a location, migration of people29, and different policies, which can be directly and indirectly associated with climatic29,30 and socioeconomic factors27,29. Hence, it would be preferable for the projection of population and GDP to consider the feedback from the hydrological and hydrodynamic models to increase their reliability based on various climate phenomena, such as water availability, floods, and droughts, in addition to simple approaches such as the statistical model limited to roads and other infrastructure for auxiliary variables by Murakami and Yamagata21 and the gravity-based model by Jones and O’Neill22. More

Summary Progress Update 2021: SDG 6 — Water and Sanitation for All (United Nations, 2021).Lu, L. et al. Wastewater treatment for carbon capture and utilization. Nat. Sustain. 1, 750–758 (2018).Article 

Google Scholar 
Li, W.-W., Yu, H.-Q. & Rittmann, B. E. Chemistry: reuse water pollutants. Nature 528, 29–31 (2015).Article 

Google Scholar 
McCarty, P. L., Bae, J. & Kim, J. Domestic wastewater treatment as a net energy producer — can this be achieved? Environ. Sci. Technol. 45, 7100–7106 (2011).Article 

Google Scholar 
Transforming our World: The 2030 Agenda for Sustainable Development (UN Department of Economic and Social Affairs — Sustainable Development, 2015).He, M. et al. Critical impacts of pyrolysis conditions and activation methods on application-oriented production of wood waste-derived biochar. Bioresour. Technol. 341, 125811 (2021). Evaluates the critical impact of pyrolysis temperature on physicochemical properties of pristine and activated biochar.Article 

Google Scholar 
IPCC. Strengthening and implementing the global response. Special Report on Global Warming of 1.5 °C (eds Masson-Delmotte, V. et al.) Ch. 4 (WMO, 2018).Lehmann, J., Gaunt, J. & Rondon, M. Bio-char sequestration in terrestrial ecosystems — a review. Mitig. Adapt. Strateg. Glob. Chang. 11, 403–427 (2006).Article 

Google Scholar 
Wiedner, K. & Glaser, B. in Biochar for Environmental Management: Science, Technology and Implementation 2nd edition (eds Lehmann, J. & Joseph, S.) 14–32 (Routledge, 2015).Wang, H. et al. Phosphorus recovery from the liquid phase of anaerobic digestate using biochar derived from iron-rich sludge: a potential phosphorus fertilizer. Water Res. 174, 115629 (2020).Article 

Google Scholar 
Chen, S. S. et al. Designing sustainable drainage systems in subtropical cities: challenges and opportunities. J. Clean. Prod. 280, 124418 (2021).Article 

Google Scholar 
Shaheen, S. M. et al. Wood-based biochar for the removal of potentially toxic elements in water and wastewater: a critical review. Int. Mater. Rev. 64, 216–247 (2019).Article 

Google Scholar 
Yang, Q. et al. Prospective contributions of biomass pyrolysis to China’s 2050 carbon reduction and renewable energy goals. Nat. Commun. 12, 1698 (2021).Article 

Google Scholar 
Lehmann, J. et al. Biochar in climate change mitigation. Nat. Geosci. 14, 883–892 (2021). Highlights that biochar is a carbon-negative material for environmental and energy applications.Article 

Google Scholar 
Euronews.green. How is the €3 billion biochar industry transforming green energy in Sweden? https://www.euronews.com/green/2021/06/14/how-is-the-3-billion-biochar-industry-transforming-green-energy-sweden (2021).Inkwoodresearch. Global biochar market forecast 2020–2028. https://www.inkwoodresearch.com/reports/global-biochar-market/# (2021).State of the Biochar Industry 2015 (International Biochar Initiative (IBI), 2015).Kumar, M. et al. Critical review on biochar-supported catalysts for pollutant degradation and sustainable biorefinery. Adv. Sustain. Syst. 4, 1900149 (2020).Article 

Google Scholar 
Godlewska, P., Ok, Y. S. & Oleszczuk, P. The dark side of black gold: ecotoxicological aspects of biochar and biochar-amended soils. J. Hazard. Mater. 403, 123833 (2021). Reviews the potential risks of biochar application, which need further investigation.Article 

Google Scholar 
Li, H. B. et al. Mechanisms of metal sorption by biochars: biochar characteristics and modifications. Chemosphere 178, 466–478 (2017).Article 

Google Scholar 
Lee, J., Kim, K. H. & Kwon, E. E. Biochar as a catalyst. Renew. Sust. Energ. Rev. 77, 70–79 (2017).Article 

Google Scholar 
Xiao, X., Chen, B. L., Chen, Z. M., Zhu, L. Z. & Schnoor, J. L. Insight into multiple and multilevel structures of biochars and their potential environmental applications: a critical review. Environ. Sci. Technol. 52, 5027–5047 (2018). Reviews how the transformation of organic and inorganic phases with increasing temperature determines the properties of biochar and its potential applications.Article 

Google Scholar 
Dai, Y. J., Zhang, N. X., Xing, C. M., Cui, Q. X. & Sun, Q. Y. The adsorption, regeneration and engineering applications of biochar for removal organic pollutants: a review. Chemosphere 223, 12–27 (2019).Article 

Google Scholar 
Wang, J. L. & Wang, S. Z. Preparation, modification and environmental application of biochar: a review. J. Clean. Prod. 227, 1002–1022 (2019).Article 

Google Scholar 
Zhao, Y., Yuan, X., Li, X., Jiang, L. & Wang, H. Burgeoning prospects of biochar and its composite in persulfate-advanced oxidation process. J. Hazard. Mater. 409, 124893 (2021).Article 

Google Scholar 
Ren, S. et al. Hydrochar-facilitated anaerobic digestion: evidence for direct interspecies electron transfer mediated through surface oxygen-containing functional groups. Environ. Sci. Technol. 54, 5755–5766 (2020).Article 

Google Scholar 
Wu, J., Lu, T., Bi, J., Yuan, H. & Chen, Y. A novel sewage sludge biochar and ferrate synergetic conditioning for enhancing sludge dewaterability. Chemosphere 237, 124339 (2019).Article 

Google Scholar 
Whitman, T. & Lehmann, J. Biochar — one way forward for soil carbon in offset mechanisms in Africa? Environ. Sci. Policy 12, 1024–1027 (2009).Article 

Google Scholar 
Woolf, D., Amonette, J. E., Street-Perrott, F. A., Lehmann, J. & Joseph, S. Sustainable biochar to mitigate global climate change. Nat. Commun. 1, 56 (2010).Article 

Google Scholar 
Chen, W., Meng, J., Han, X., Lan, Y. & Zhang, W. Past, present, and future of biochar. Biochar 1, 75–87 (2019).Article 

Google Scholar 
Kwak, J.-H. et al. Biochar properties and lead(ii) adsorption capacity depend on feedstock type, pyrolysis temperature, and steam activation. Chemosphere 231, 393–404 (2019).Article 

Google Scholar 
Dutta, S., He, M., Xiong, X. & Tsang, D. C. W. Sustainable management and recycling of food waste anaerobic digestate: a review. Bioresour. Technol. 341, 125915 (2021).Article 

Google Scholar 
Xiao, X. & Chen, B. A direct observation of the fine aromatic clusters and molecular structures of biochars. Environ. Sci. Technol. 51, 5473–5482 (2017).Article 

Google Scholar 
Li, S., Harris, S., Anandhi, A. & Chen, G. Predicting biochar properties and functions based on feedstock and pyrolysis temperature: a review and data syntheses. J. Clean. Prod. 215, 890–902 (2019).Article 

Google Scholar 
Crittenden, J. C., Trussell, R. R., Hand, D. W., Howe, K. J. & Tchobanoglous, G. MWH’s Water Treatment: Principles and Design 3rd edition (Wiley, 2012).Enaime, G., Bacaoui, A., Yaacoubi, A. & Luebken, M. Biochar for wastewater treatment — conversion technologies and applications. Appl. Sci. 10, 3492 (2020).Article 

Google Scholar 
Thompson, K. A. et al. Environmental comparison of biochar and activated carbon for tertiary wastewater treatment. Environ. Sci. Technol. 50, 11253–11262 (2016).Article 

Google Scholar 
Cheng, N. et al. Adsorption of emerging contaminants from water and wastewater by modified biochar: a review. Environ. Pollut. 273, 116448 (2021).Article 

Google Scholar 
Huggins, T. M., Haeger, A., Biffinger, J. C. & Ren, Z. J. Granular biochar compared with activated carbon for wastewater treatment and resource recovery. Water Res. 94, 225–232 (2016).Article 

Google Scholar 
Activated Carbon Market by Type (Powdered, Granular, Others (Pelletized, Bead)), Application (Liquid Phase (Water Treatment, Foods & Beverages, Pharmaceutical & Medical), Gaseous Phase (Industrial, Automotive)), Region — Global Forecast to 2021 (MarketsAndMarkets, 2017).Chen, Z., Zhang, W., Wang, D., Ma, T. & Bai, R. Enhancement of activated sludge dewatering performance by combined composite enzymatic lysis and chemical re-flocculation with inorganic coagulants: kinetics of enzymatic reaction and re-flocculation morphology. Water Res. 83, 367–376 (2015).Article 

Google Scholar 
Shewa, W. A. & Dagnew, M. Revisiting chemically enhanced primary treatment of wastewater: a review. Sustainability 12, 5928 (2020).Article 

Google Scholar 
Tao, S. et al. Enhanced sludge dewaterability with sludge-derived biochar activating hydrogen peroxide: synergism of Fe and Al elements in biochar. Water Res. 182, 115927 (2020).Article 

Google Scholar 
Yang, X. et al. Enhanced sludge dewaterability by a novel MnFe2O4-biochar activated peroxymonosulfate process combined with tannic acid. Chem. Eng. J. 429, 132280 (2022).Article 

Google Scholar 
Wu, Y. et al. Possibility of sludge conditioning and dewatering with rice husk biochar modified by ferric chloride. Bioresour. Technol. 205, 258–263 (2016).Article 

Google Scholar 
Wu, Y. et al. Combined sludge conditioning of micro-disintegration, floc reconstruction and skeleton building (KMnO4/FeCl3/biochar) for enhancement of waste activated sludge dewaterability. J. Taiwan. Inst. Chem. Eng. 74, 121–128 (2017).Article 

Google Scholar 
Hu, P. et al. The influence of hydrophobicity on sludge dewatering associated with cationic starch-based flocculants. J. Environ. Manage. 296, 113218 (2021).Article 

Google Scholar 
Useviciute, L. & Baltrenaite, E. Methods for determining lignocellulosic biochar wettability. Waste Biomass Valoriz. 11, 4457–4468 (2019).Article 

Google Scholar 
Li, H. et al. Enhanced sludge dewaterability by Fe-rich biochar activating hydrogen peroxide: co-hydrothermal red mud and reed straw. J. Environ. Manage. 296, 113239 (2021).Article 

Google Scholar 
Liang, J., Luo, L., Li, D., Wang, H. & Wong, J. W. C. Conductive materials supplement alters digestate dewaterability during anaerobic co-digestion of food waste and sewage sludge and promotes follow-up indigenous peroxides activation. Chem. Eng. J. 431, 133875 (2021).Article 

Google Scholar 
Wang, C. et al. Role of biochar in the granulation of anaerobic sludge and improvement of electron transfer characteristics. Bioresour. Technol. 268, 28–35 (2018).Article 

Google Scholar 
Zhao, Z., Li, Y., Quan, X. & Zhang, Y. Towards engineering application: potential mechanism for enhancing anaerobic digestion of complex organic waste with different types of conductive materials. Water Res. 115, 266–277 (2017).Article 

Google Scholar 
Fagbohungbe, M. O. et al. The challenges of anaerobic digestion and the role of biochar in optimizing anaerobic digestion. Waste Manage. 61, 236–249 (2017).Article 

Google Scholar 
van Dijk, E. J. H., Pronk, M. & van Loosdrecht, M. C. M. A settling model for full-scale aerobic granular sludge. Water Res. 186, 116135 (2020).Article 

Google Scholar 
de Kreuk, M. K., Kishida, N. & van Loosdrecht, M. C. M. Aerobic granular sludge — state of the art. Water Sci. Technol. 55, 75–81 (2007).Article 

Google Scholar 
Wang, X. et al. Rapid aerobic granulation using biochar for the treatment of petroleum refinery wastewater. Pet. Sci. 17, 1411–1421 (2020).Article 

Google Scholar 
Ming, J. et al. Bioreactor performance using biochar and its effect on aerobic granulation. Bioresour. Technol. 300, 122620 (2020).Article 

Google Scholar 
Sohn, W. et al. A review on membrane fouling control in anaerobic membrane bioreactors by adding performance enhancers. J. Water Process. Eng. 40, 101867 (2021).Article 

Google Scholar 
Wang, Z., Wu, Z. & Tang, S. Extracellular polymeric substances (EPS) properties and their effects on membrane fouling in a submerged membrane bioreactor. Water Res. 43, 2504–2512 (2009).Article 

Google Scholar 
Sima, X.-F. et al. Robust biochar-assisted alleviation of membrane fouling in MBRs by indirect mechanism. Sep. Purif. Technol. 184, 195–204 (2017).Article 

Google Scholar 
Shimabuku, K. K. et al. Biochar sorbents for sulfamethoxazole removal from surface water, stormwater, and wastewater effluent. Water Res. 96, 236–245 (2016).Article 

Google Scholar 
Suresh Kumar, P., Korving, L., Keesman, K. J., van Loosdrecht, M. C. M. & Witkamp, G.-J. Effect of pore size distribution and particle size of porous metal oxides on phosphate adsorption capacity and kinetics. Chem. Eng. J. 358, 160–169 (2019).Article 

Google Scholar 
Zhang, M. et al. Formation of disinfection byproducts as affected by biochar during water treatment. Chemosphere 233, 190–197 (2019).Article 

Google Scholar 
Kwarciak-Kozłowska, A. in Industrial and Municipal Sludge (eds Narasimha Vara Prasad, M. et al.) 337–360 (Butterworth-Heinemann, 2019).Gopinath, A. et al. Conversion of sewage sludge into biochar: a potential resource in water and wastewater treatment. Environ. Res. 194, 110656 (2021).Article 

Google Scholar 
Chen, Y.-d et al. Production, properties, and catalytic applications of sludge derived biochar for environmental remediation. Water Res. 187, 116390 (2020).Article 

Google Scholar 
Yu, J. et al. Magnetic nitrogen-doped sludge-derived biochar catalysts for persulfate activation: Internal electron transfer mechanism. Chem. Eng. J. 364, 146–159 (2019).Article 

Google Scholar 
Wan, Z. et al. Critical impact of nitrogen vacancies in nonradical carbocatalysis on nitrogen-doped graphitic biochar. Environ. Sci. Technol. 55, 7004–7014 (2021).Article 

Google Scholar 
Yan, L. et al. ZnCl2 modified biochar derived from aerobic granular sludge for developed microporosity and enhanced adsorption to tetracycline. Bioresour. Technol. 297, 122381 (2020).Article 

Google Scholar 
Ding, X., Chen, H., Yang, Q., Wei, J. & Wei, D. Effect of sludge property on the synthesis, characterization and sorption performance of sludge-based biochar. Bioresour. Technol. Rep. 7, 100204 (2019).Article 

Google Scholar 
Barbusiński, K., Parzentna-Gabor, A. & Kasperczyk, D. Removal of odors (mainly H2S and NH3) using biological treatment methods. Clean. Technol. 3, 138–155 (2021).Article 

Google Scholar 
Talaiekhozani, A., Bagheri, M., Goli, A. & Talaei Khoozani, M. R. An overview of principles of odor production, emission, and control methods in wastewater collection and treatment systems. J. Environ. Manage. 170, 186–206 (2016).Article 

Google Scholar 
Hwang, O. et al. Efficacy of different biochars in removing odorous volatile organic compounds (VOCs) emitted from swine manure. ACS Sustain. Chem. Eng. 6, 14239–14247 (2018).Article 

Google Scholar 
Choudhury, A. & Lansing, S. Biochar addition with Fe impregnation to reduce H2S production from anaerobic digestion. Bioresour. Technol. 306, 123121 (2020).Article 

Google Scholar 
Hao, X. et al. Environmental impacts of resource recovery from wastewater treatment plants. Water Res. 160, 268–277 (2019).Article 

Google Scholar 
Fang, L. L., Valverde-Pérez, B., Damgaard, A., Plósz, B. G. & Rygaard, M. Life cycle assessment as development and decision support tool for wastewater resource recovery technology. Water Res. 88, 538–549 (2016).Article 

Google Scholar 
Zheng, Y. et al. Reclaiming phosphorus from secondary treated municipal wastewater with engineered biochar. Chem. Eng. J. 362, 460–468 (2019).Article 

Google Scholar 
He, M. et al. A critical review on performance indicators for evaluating soil biota and soil health of biochar-amended soils. J. Hazard. Mater. 414, 125378 (2021).Article 

Google Scholar 
Yang, F. et al. Metal chloride-loaded biochar for phosphorus recovery: noteworthy roles of inherent minerals in precursor. Chemosphere 266, 128991 (2021).Article 

Google Scholar 
Zheng, M., Xie, T., Li, J., Xu, K. & Wang, C. Biochar as a carrier of struvite precipitation for nitrogen and phosphorus recovery from urine. Int. J. Environ. Eng. 144, 4018101 (2018).
Google Scholar 
Medeiros, D. C. C. d. S. et al. Pristine and engineered biochar for the removal of contaminants co-existing in several types of industrial wastewaters: a critical review. Sci. Total Environ. 809, 151120 (2021).Article 

Google Scholar 
Mohan, D., Sarswat, A., Ok, Y. S. & Pittman, C. U. Organic and inorganic contaminants removal from water with biochar, a renewable, low cost and sustainable adsorbent — a critical review. Bioresour. Technol. 160, 191–202 (2014).Article 

Google Scholar 
Ahmad, Z. et al. Removal of Cu(ii), Cd(ii) and Pb(ii) ions from aqueous solutions by biochars derived from potassium-rich biomass. J. Clean. Prod. 180, 437–449 (2018).Article 

Google Scholar 
Xu, Z., Xu, X., Zhang, Y., Yu, Y. & Cao, X. Pyrolysis-temperature depended electron donating and mediating mechanisms of biochar for Cr(vi) reduction. J. Hazard. Mater. 388, 121794 (2019).Article 

Google Scholar 
Heo, J. et al. Enhanced adsorption of bisphenol A and sulfamethoxazole by a novel magnetic CuZnFe2O4–biochar composite. Bioresour. Technol. 281, 179–187 (2019).Article 

Google Scholar 
Choudhary, M., Kumar, R. & Neogi, S. Activated biochar derived from Opuntia ficus-indica for the efficient adsorption of malachite green dye, Cu2+ and Ni2+ from water. J. Hazard. Mater. 392, 122441 (2020).Article 

Google Scholar 
Tao, Y. et al. Efficient removal of atrazine by iron-modified biochar loaded Acinetobacter lwoffii DNS32. Sci. Total. Environ. 682, 59–69 (2019).Article 

Google Scholar 
Xu, X. Y. et al. Removal of Cu, Zn, and Cd from aqueous solutions by the dairy manure-derived biochar. Environ. Sci. Pollut. Res. 20, 358–368 (2013).Article 

Google Scholar 
Xu, Z., Xu, X., Tsang, D. C. W. & Cao, X. Contrasting impacts of pre- and post-application aging of biochar on the immobilization of Cd in contaminated soils. Environ. Pollut. 242, 1362–1370 (2018).Article 

Google Scholar 
Xu, X. Y., Cao, X. D. & Zhao, L. Comparison of rice husk- and dairy manure-derived biochars for simultaneously removing heavy metals from aqueous solutions: role of mineral components in biochars. Chemosphere 92, 955–961 (2013).Article 

Google Scholar 
Pei, L. et al. Further reuse of phosphorus-laden biochar for lead sorption from aqueous solution: isotherm, kinetics, and mechanism. Sci. Total. Environ. 792, 148550 (2021).Article 

Google Scholar 
Klüpfel, L., Keiluweit, M., Kleber, M. & Sander, M. Redox properties of plant biomass-derived black carbon (biochar). Environ. Sci. Technol. 48, 5601–5611 (2014). Reveals considerable redox reactivity on biochar due to its surface functionality.Article 

Google Scholar 
Xu, Z. et al. Direct and indirect electron transfer routes of chromium(vi) reduction with different crystalline ferric oxyhydroxides in the presence of pyrogenic carbon. Environ. Sci. Technol. 56, 1724–1735 (2022).Article 

Google Scholar 
Xu, Z. et al. Electroactive Fe-biochar for redox-related remediation of arsenic and chromium: distinct redox nature with varying iron/carbon speciation. J. Hazard. Mater. 430, 128479 (2022).Article 

Google Scholar 
Zhong, D. et al. pH dependence of arsenic oxidation by rice-husk-derived biochar: roles of redox-active moieties. Environ. Sci. Technol. 53, 9034–9044 (2019).Article 

Google Scholar 
Liu, J. et al. Highly efficient removal of thallium in wastewater by MnFe2O4–biochar composite. J. Hazard. Mater. 401, 123311 (2021).Article 

Google Scholar 
Ruan, X. et al. Formation, characteristics, and applications of environmentally persistent free radicals in biochars: a review. Bioresour. Technol. 281, 457–468 (2019).Article 

Google Scholar 
Liang, J. et al. Different mechanisms between biochar and activated carbon for the persulfate catalytic degradation of sulfamethoxazole: roles of radicals in solution or solid phase. Chem. Eng. J. 375, 121908 (2019).Article 

Google Scholar 
Sun, T. et al. Rapid electron transfer by the carbon matrix in natural pyrogenic carbon. Nat. Commun. 8, 14873 (2017). Emphasizes the importance of graphitic structures for the electron transfer capacity of high-temperature biochar.Article 

Google Scholar 
Wan, Z. et al. A sustainable biochar catalyst synergized with copper heteroatoms and CO2 for singlet oxygenation and electron transfer routes. Green Chem. 21, 4800–4814 (2019).Article 

Google Scholar 
Dou, J. et al. Biochar co-doped with nitrogen and boron switching the free radical based peroxydisulfate activation into the electron-transfer dominated nonradical process. Appl. Catal. B 301, 120832 (2022).Article 

Google Scholar 
Liu, W.-J., Jiang, H. & Yu, H.-Q. Emerging applications of biochar-based materials for energy storage and conversion. Energy Environ. Sci. 12, 1751–1779 (2019).Article 

Google Scholar 
Yao, F. et al. Synergistic adsorption and electrocatalytic reduction of bromate by Pd/N-doped loofah sponge-derived biochar electrode. J. Hazard. Mater. 386, 121651 (2020).Article 

Google Scholar 
Yao, F. et al. Effective adsorption/electrocatalytic degradation of perchlorate using Pd/Pt supported on N-doped activated carbon fiber cathode. J. Hazard. Mater. 323, 602–610 (2017).Article 

Google Scholar 
Zhao, Z. et al. Enhanced removal of Cu-EDTA in a three-dimensional electrolysis system with highly graphitic activated biochar produced via acidic and K2FeO4 treatment. Chem. Eng. J. 430, 132661 (2022).Article 

Google Scholar 
Zhang, T. et al. Ti–Sn–Ce/bamboo biochar particle electrodes for enhanced electrocatalytic treatment of coking wastewater in a three-dimensional electrochemical reaction system. J. Clean. Prod. 258, 120273 (2020).Article 

Google Scholar 
Sun, C. et al. Biochar cathode: reinforcing electro-Fenton pathway against four-electron reduction by controlled carbonization and surface chemistry. Sci. Total Environ. 754, 142136 (2021).Article 

Google Scholar 
Liu, W.-J., Jiang, H. & Yu, H.-Q. Development of biochar-based functional materials: toward a sustainable platform carbon material. Chem. Rev. 115, 12251–12285 (2015). Reviews how biochar-based functional materials can be used for various sustainable applications.Article 

Google Scholar 
Ng, Y. H., Ikeda, S., Matsumura, M. & Amal, R. A perspective on fabricating carbon-based nanomaterials by photocatalysis and their applications. Energy Environ. Sci. 5, 9307–9318 (2012).Article 

Google Scholar 
Wang, Z., Murugananthan, M. & Zhang, Y. Graphitic carbon nitride based photocatalysis for redox conversion of arsenic(iii) and chromium(vi) in acid aqueous solution. Appl. Catal. B 248, 349–356 (2019).Article 

Google Scholar 
Lisowski, P. et al. Dual functionality of TiO2/biochar hybrid materials: photocatalytic phenol degradation in the liquid phase and selective oxidation of methanol in the gas phase. ACS Sustain. Chem. Eng. 5, 6274–6287 (2017).Article 

Google Scholar 
Zhai, Y. et al. Novel biochar@CoFe2O4/Ag3PO4 photocatalysts for highly efficient degradation of bisphenol a under visible-light irradiation. J. Colloid Interface Sci. 560, 111–121 (2020).Article 

Google Scholar 
Tang, R. et al. π–π stacking derived from graphene-like biochar/g-C3N4 with tunable band structure for photocatalytic antibiotics degradation via peroxymonosulfate activation. J. Hazard. Mater. 423, 126944 (2022).Article 

Google Scholar 
Mian, M. M. & Liu, G. Recent progress in biochar-supported photocatalysts: synthesis, role of biochar, and applications. RSC Adv. 8, 14237–14248 (2018).Article 

Google Scholar 
Colmenares, J. C., Varma, R. S. & Lisowski, P. Sustainable hybrid photocatalysts: titania immobilized on carbon materials derived from renewable and biodegradable resources. Green. Chem. 18, 5736–5750 (2016).Article 

Google Scholar 
Shi, J. On the synergetic catalytic effect in heterogeneous nanocomposite catalysts. Chem. Rev. 113, 2139–2181 (2013).Article 

Google Scholar 
Wang, W., Serp, P., Kalck, P. & Faria, J. L. Visible light photodegradation of phenol on MWNT–TiO2 composite catalysts prepared by a modified sol–gel method. J. Mol. Catal. A Chem. 235, 194–199 (2005).Article 

Google Scholar 
Matos, J., Hofman, M. & Pietrzak, R. Synergy effect in the photocatalytic degradation of methylene blue on a suspended mixture of TiO2 and N-containing carbons. Carbon 54, 460–471 (2013).Article 

Google Scholar 
Wan, D. et al. Photogeneration of reactive species from biochar-derived dissolved black carbon for the degradation of amine and phenolic pollutants. Environ. Sci. Technol. 55, 8866–8876 (2021).Article 

Google Scholar 
Fu, H. et al. Photochemistry of dissolved black carbon released from biochar: reactive oxygen species generation and phototransformation. Environ. Sci. Technol. 50, 1218–1226 (2016).Article 

Google Scholar 
Yang, F. et al. Effects of biochar-dissolved organic matter on the photodegradation of sulfamethoxazole and chloramphenicol in biochar solutions as revealed by oxygen reduction performances and free radicals. Sci. Total. Environ. 781, 146807 (2021).Article 

Google Scholar 
Farhadi, S., Aminzadeh, B., Torabian, A., Khatibikamal, V. & Alizadeh Fard, M. Comparison of COD removal from pharmaceutical wastewater by electrocoagulation, photoelectrocoagulation, peroxi-electrocoagulation and peroxi-photoelectrocoagulation processes. J. Hazard. Mater. 219-220, 35–42 (2012).Article 

Google Scholar 
Zaied, B. K. et al. A comprehensive review on contaminants removal from pharmaceutical wastewater by electrocoagulation process. Sci. Total. Environ. 726, 138095 (2020).Article 

Google Scholar 
An, X. et al. Integrated co-pyrolysis and coating for the synthesis of a new coated biochar-based fertilizer with enhanced slow-release performance. J. Clean. Prod. 283, 124642 (2021).Article 

Google Scholar 
Krasucka, P. et al. Engineered biochar — a sustainable solution for the removal of antibiotics from water. Chem. Eng. J. 405, 126926 (2021).Article 

Google Scholar 
Zhang, Y. et al. Regulation of biochar mediated catalytic degradation of quinolone antibiotics: Important role of environmentally persistent free radicals. Bioresour. Technol. 326, 124780 (2021).Article 

Google Scholar 
Nidheesh, P. V. et al. Potential role of biochar in advanced oxidation processes: a sustainable approach. Chem. Eng. J. 405, 126582 (2021).Article 

Google Scholar 
Hynes, N. R. J. et al. Modern enabling techniques and adsorbents based dye removal with sustainability concerns in textile industrial sector -a comprehensive review. J. Clean. Prod. 272, 122636 (2020).Article 

Google Scholar 
Yu, K. L. et al. Adsorptive removal of cationic methylene blue and anionic Congo red dyes using wet-torrefied microalgal biochar: equilibrium, kinetic and mechanism modeling. Environ. Pollut. 272, 115986 (2021).Article 

Google Scholar 
Yu, F. et al. ZnO/biochar nanocomposites via solvent free ball milling for enhanced adsorption and photocatalytic degradation of methylene blue. J. Hazard. Mater. 415, 125511 (2021).Article 

Google Scholar 
Medha, I. et al. (3-Aminopropyl)triethoxysilane and iron rice straw biochar composites for the sorption of Cr (vi) and Zn (ii) using the extract of heavy metals contaminated soil. Sci. Total. Environ. 771, 144764 (2021).Article 

Google Scholar 
Xu, Z. et al. Interaction with low molecular weight organic acids affects the electron shuttling of biochar for Cr(vi) reduction. J. Hazard. Mater. 378, 120705 (2019).Article 

Google Scholar 
Wang, T. et al. Novel Bi2WO6 loaded N-biochar composites with enhanced photocatalytic degradation of rhodamine B and Cr(vi). J. Hazard. Mater. 389, 121827 (2020).Article 

Google Scholar 
Kicińska, A. & Wikar, J. Ecological risk associated with agricultural production in soils contaminated by the activities of the metal ore mining and processing industry — example from southern Poland. Soil Tillage Res. 205, 104817 (2021).Article 

Google Scholar 
Shi, J., Huang, W., Han, H. & Xu, C. Pollution control of wastewater from the coal chemical industry in China: environmental management policy and technical standards. Renew. Sust. Energ. Rev. 143, 110883 (2021).Article 

Google Scholar 
Xu, X. et al. Indispensable role of biochar-inherent mineral constituents in its environmental applications: a review. Bioresour. Technol. 241, 887–899 (2017). Highlights the indispensable role of biochar’s inorganic phase in environmental applications, including pollutant removal, carbon sequestration, and soil quality improvement.Article 

Google Scholar 
Xu, Z. et al. Unraveling iron speciation on Fe-biochar with distinct arsenic removal mechanisms and depth distributions of As and Fe. Chem. Eng. J. 425, 131489 (2021).Article 

Google Scholar 
Xu, Z. et al. Participation of soil active components in the reduction of Cr(vi) by biochar: differing effects of iron mineral alone and its combination with organic acid. J. Hazard. Mater. 384, 121455 (2020).Article 

Google Scholar 
Nguyen, T. T. N. et al. The effects of short term, long term and reapplication of biochar on soil bacteria. Sci. Total. Environ. 636, 142–151 (2018).Article 

Google Scholar 
Lau, A. Y. T. et al. Surface-modified biochar in a bioretention system for Escherichia coli removal from stormwater. Chemosphere 169, 89–98 (2017).Article 

Google Scholar 
Sun, Y. et al. Waste-derived compost and biochar amendments for stormwater treatment in bioretention column: co-transport of metals and colloids. J. Hazard. Mater. 383, 121243–121243 (2020). Shows the promising potential of biochar for stormwater harvesting in sustainable drainage systems.Article 

Google Scholar 
Lehmann, J. et al. Biochar effects on soil biota — a review. Soil. Biol. Biochem. 43, 1812–1836 (2011).Article 

Google Scholar 
Zhang, S., Lin, Z., Zhang, S. & Ge, D. Stormwater retention and detention performance of green roofs with different substrates: observational data and hydrological simulations. J. Environ. Manage 291, 112682 (2021).Article 

Google Scholar 
Tirpak, R. A. et al. Conventional and amended bioretention soil media for targeted pollutant treatment: a critical review to guide the state of the practice. Water Res. 189, 116648 (2021).Article 

Google Scholar 
Tian, J. et al. A pilot-scale, bi-layer bioretention system with biochar and zero-valent iron for enhanced nitrate removal from stormwater. Water Res. 148, 378–387 (2019).Article 

Google Scholar 
Marcińczyk, M. & Oleszczuk, P. Biochar and engineered biochar as slow- and controlled-release fertilizers. J. Clean. Prod. 339, 130685 (2022).Article 

Google Scholar 
Danish, A. et al. Reusing biochar as a filler or cement replacement material in cementitious composites: a review. Constr. Build. Mater. 300, 124295 (2021).Article 

Google Scholar 
Llovet, A. et al. Fresh biochar application provokes a reduction of nitrate which is unexplained by conventional mechanisms. Sci. Total. Environ. 755, 142430 (2021).Article 

Google Scholar 
Mohanty, S. K., Cantrell, K. B., Nelson, K. L. & Boehm, A. B. Efficacy of biochar to remove Escherichia coli from stormwater under steady and intermittent flow. Water Res. 61, 288–296 (2014).Article 

Google Scholar 
Valenca, R. et al. Biochar selection for Escherichia coli removal in stormwater biofilters. Int. J. Environ. Eng. 147, 1843 (2021).
Google Scholar 
Xu, Z., He, M., Xu, X., Cao, X. & Tsang, D. C. W. Impacts of different activation processes on the carbon stability of biochar for oxidation resistance. Bioresour. Technol. 338, 125555 (2021). Reveals how aggressive modification of biochar might lead to a decrease in carbon stability, which needs further consideration.Article 

Google Scholar 
Ulrich, B. A., Loehnert, M. & Higgins, C. P. Improved contaminant removal in vegetated stormwater biofilters amended with biochar. Environ. Sci. Water Res. Technol. 3, 726–734 (2017).Article 

Google Scholar 
Ashoori, N. et al. Evaluation of pilot-scale biochar-amended woodchip bioreactors to remove nitrate, metals, and trace organic contaminants from urban stormwater runoff. Water Res. 154, 1–11 (2019).Article 

Google Scholar 
Spokas, K. A. et al. Physical disintegration of biochar: an overlooked process. Environ. Sci. Technol. Lett. 1, 326–332 (2014).Article 

Google Scholar 
Wang, L. et al. Biochar aging: mechanisms, physicochemical changes, assessment, and implications for field applications. Environ. Sci. Technol. 54, 14797–14814 (2020). Highlights how ageing processes might have a strong impact on the long-term performance of biochar.Article 

Google Scholar 
Yang, X., Pan, H., Shaheen, S. M., Wang, H. & Rinklebe, J. Immobilization of cadmium and lead using phosphorus-rich animal-derived and iron-modified plant-derived biochars under dynamic redox conditions in a paddy soil. Environ. Int. 156, 106628 (2021).Article 

Google Scholar 
Beiyuan, J. et al. (Im)mobilization and speciation of lead under dynamic redox conditions in a contaminated soil amended with pine sawdust biochar. Environ. Int. 135, 105376 (2020).Article 

Google Scholar 
Beckers, F. et al. Impact of biochar on mobilization, methylation, and ethylation of mercury under dynamic redox conditions in a contaminated floodplain soil. Environ. Int. 127, 276–290 (2019).Article 

Google Scholar 
Tong, M., He, L., Rong, H., Li, M. & Kim, H. Transport behaviors of plastic particles in saturated quartz sand without and with biochar/Fe3O4-biochar amendment. Water Res. 169, 115284 (2020).Article 

Google Scholar 
Chen, M. et al. Facilitated transport of cadmium by biochar–Fe3O4 nanocomposites in water-saturated natural soils. Sci. Total. Environ. 684, 265–275 (2019).Article 

Google Scholar 
Song, B., Chen, M., Zhao, L., Qiu, H. & Cao, X. Physicochemical property and colloidal stability of micron- and nano-particle biochar derived from a variety of feedstock sources. Sci. Total Environ. 661, 685–695 (2019).Article 

Google Scholar 
Gui, X. et al. Soil colloids affect the aggregation and stability of biochar colloids. Sci. Total Environ. 771, 145414 (2021).Article 

Google Scholar 
Negative Emission Technologies: What Role in Meeting Paris Agreement Targets? (European Academies’ Science Advisory Council, 2018).Hu, Q. et al. Biochar industry to circular economy. Sci. Total. Environ. 757, 143820 (2021).Article 

Google Scholar 
Maroušek, J. Significant breakthrough in biochar cost reduction. Clean. Technol. Environ. Policy 16, 1821–1825 (2014).Article 

Google Scholar 
Pourhashem, G., Hung, S. Y., Medlock, K. B. & Masiello, C. A. Policy support for biochar: review and recommendations. Glob. Change Biol. Bioenergy 11, 364–380 (2019).Article 

Google Scholar 
State and Trends of Carbon Pricing 2020 (World Bank Group, 2020).Standardized Product Definition and Product Testing Guidelines for Biochar that is used in Soil Version 2.1 (International Biochar Initiative (IBI), 2015).European Biochar Certificate — Guidelines for a Sustainable Production of Biochar Version 9.3E of 11 April 2021. (European Biochar Foundation, 2012).Shackley, S., Ibarrola Esteinou, R., Hopkins, D. & Hammond, J. Biochar Quality Mandate (BQM) Version 1.0 (British Biochar Foundation, 2014).Meyer, S. et al. Biochar standardization and legislation harmonization. Int. J. Environ. Eng. Landsc. Manage 25, 175–191 (2017).Article 

Google Scholar 
Azzi, E. S., Karltun, E. & Sundberg, C. Prospective life cycle assessment of large-scale biochar production and use for negative emissions in Stockholm. Environ. Sci. Technol. 53, 8466–8476 (2019).Article 

Google Scholar 
ESG Investing: Environmental Pillar Scoring and Reporting (OECD, 2020).Maroušek, J., Strunecký, O. & Stehel, V. Biochar farming: defining economically perspective applications. Clean. Technol. Environ. Policy 21, 1389–1395 (2019).Article 

Google Scholar 
Maroušek, J., Hašková, S., Zeman, R. & Vaníčková, R. Managerial preferences in relation to financial indicators regarding the mitigation of global change. Sci. Eng. Ethics 21, 203–207 (2014).Article 

Google Scholar 
Mašek, O., Buss, W. & Sohi, S. Standard biochar materials. Environ. Sci. Technol. 52, 9543–9544 (2018).Article 

Google Scholar 
Zhu, X. et al. Machine learning exploration of the direct and indirect roles of Fe impregnation on Cr(vi) removal by engineered biochar. Chem. Eng. J. 428, 131967 (2022).Article 

Google Scholar 
Palansooriya, K. N. et al. Prediction of soil heavy metal immobilization by biochar using machine learning. Environ. Sci. Technol. 56, 4187–4198 (2022).Article 

Google Scholar 
Marris, E. Black is the new green. Nature 442, 624–626 (2006).Article 

Google Scholar 
Lehmann, J. A handful of carbon. Nature 447, 143–144 (2007).Article 

Google Scholar 
Woods, W. I., Falcao, N. P. S. & Teixeira, W. G. Biochar trials aim to enrich soil for smallholders. Nature 443, 144–144 (2006).Article 

Google Scholar 
Chan, K. Y., Van Zwieten, L., Meszaros, I., Downie, A. & Joseph, S. Agronomic values of greenwaste biochar as a soil amendment. Aust. J. Soil. Res. 45, 629–634 (2007).Article 

Google Scholar 
Chan, K. Y., Van Zwieten, L., Meszaros, I., Downie, A. & Joseph, S. Using poultry litter biochars as soil amendments. Aust. J. Soil. Res. 46, 437–444 (2008).Article 

Google Scholar 
Sanchez, M. E., Lindao, E., Margaleff, D., Martinez, O. & Moran, A. Bio-fuels and bio-char production from pyrolysis of sewage sludge. Residuals Sci. Technol. 6, 35–41 (2009).
Google Scholar 
Cao, X. D., Ma, L. N., Gao, B. & Harris, W. Dairy-manure derived biochar effectively sorbs lead and atrazine. Environ. Sci. Technol. 43, 3285–3291 (2009).Article 

Google Scholar 
Warren, G. P., Robinson, J. S. & Someus, E. Dissolution of phosphorus from animal bone char in 12 soils. Nutr. Cycl. Agroecosyst. 84, 167–178 (2009).Article 

Google Scholar 
Yu, X. Y., Ying, G. G. & Kookana, R. S. Reduced plant uptake of pesticides with biochar additions to soil. Chemosphere 76, 665–671 (2009).Article 

Google Scholar 
Shen, Y. W., Linville, J. L., Urgun-Demirtas, M., Schoene, R. P. & Snyder, S. W. Producing pipeline-quality biomethane via anaerobic digestion of sludge amended with corn stover biochar with in-situ CO2 removal. Appl. Energy 158, 300–309 (2015).Article 

Google Scholar 
Ulrich, B. A., Im, E. A., Werner, D. & Higgins, C. P. Biochar and activated carbon for enhanced trace organic contaminant retention in stormwater infiltration systems. Environ. Sci. Technol. 49, 6222–6230 (2015).Article 

Google Scholar 
Fang, G., Liu, C., Gao, J., Dionysiou, D. D. & Zhou, D. Manipulation of persistent free radicals in biochar to activate persulfate for contaminant degradation. Environ. Sci. Technol. 49, 5645–5653 (2015).Article 

Google Scholar 
Yu, L. P., Yuan, Y., Tang, J., Wang, Y. Q. & Zhou, S. G. Biochar as an electron shuttle for reductive dechlorination of pentachlorophenol by Geobacter sulfurreducens. Sci. Rep. 5, 16221 (2015).Article 

Google Scholar 
Li, M. et al. Simultaneously promoting charge separation and photoabsorption of BiOX (X = Cl, Br) for efficient visible-light photocatalysis and photosensitization by compositing low-cost biochar. Appl. Surf. Sci. 386, 285–295 (2016).Article 

Google Scholar 
Maurer, D. L., Koziel, J. A., Kalus, K., Andersen, D. S. & Opalinski, S. Pilot-scale testing of non-activated biochar for swine manure treatment and mitigation of ammonia, hydrogen sulfide, odorous volatile organic compounds (VOCs), and greenhouse gas emissions. Sustainability 9, 929 (2017).Article 

Google Scholar 
Ayyappan, C. S., Bhalambaal, V. M. & Kumar, S. Effect of biochar on bio-electrochemical dye degradation and energy production. Bioresour. Technol. 251, 165–170 (2018).Article 

Google Scholar 
Chen, B. L., Chen, Z. M. & Lv, S. F. A novel magnetic biochar efficiently sorbs organic pollutants and phosphate. Bioresour. Technol. 102, 716–723 (2011).Article 

Google Scholar 
Nzediegwu, C., Naeth, M. A. & Chang, S. X. Feedstock type drives surface property, demineralization and element leaching of nitric acid-activated biochars more than pyrolysis temperature. Bioresour. Technol. 344, 126316 (2021).Article 

Google Scholar 
Li, B. et al. Adsorption of Cd(ii) from aqueous solutions by rape straw biochar derived from different modification processes. Chemosphere 175, 332–340 (2017).Article 

Google Scholar 
Yu, Y. et al. Synergistic role of bulk carbon and iron minerals inherent in the sludge-derived biochar for As(v) immobilization. Chem. Eng. J. 417, 129183 (2021).Article 

Google Scholar 
Sanford, J. R., Larson, R. A. & Runge, T. Nitrate sorption to biochar following chemical oxidation. Sci. Total. Environ. 669, 938–947 (2019).Article 

Google Scholar 
Sizmur, T., Fresno, T., Akgül, G., Frost, H. & Moreno-Jiménez, E. Biochar modification to enhance sorption of inorganics from water. Bioresour. Technol. 246, 34–47 (2017).Article 

Google Scholar 
Zhao, L. et al. Copyrolysis of biomass with phosphate fertilizers to improve biochar carbon retention, slow nutrient release, and stabilize heavy metals in soil. ACS Sustain. Chem. Eng. 4, 1630–1636 (2016).Article 

Google Scholar 
Cuong, D. V., Wu, P.-C., Chen, L.-I. & Hou, C.-H. Active MnO2/biochar composite for efficient As(iii) removal: insight into the mechanisms of redox transformation and adsorption. Water Res. 188, 116495 (2021).Article 

Google Scholar 
Liang, J. et al. Persulfate oxidation of sulfamethoxazole by magnetic iron-char composites via nonradical pathways: Fe(iv) versus surface-mediated electron transfer. Environ. Sci. Technol. 55, 10077–10086 (2021).Article 

Google Scholar 
Liu, L.-L. et al. Edge electronic vacancy on ultrathin carbon nitride nanosheets anchoring O2 to boost H2O2 photoproduction. Appl. Catal. B 302, 120845 (2022).Article 

Google Scholar 
Zhou, Y. et al. Sulfur and nitrogen self-doped carbon nanosheets derived from peanut root nodules as high-efficiency non-metal electrocatalyst for hydrogen evolution reaction. Nano Energy 16, 357–366 (2015).Article 

Google Scholar 
Zhang, Z. et al. A novel biochar electrode for efficient electroreduction of nitrate: selective and regulation of halogen. Chemosphere 288, 132400 (2022).Article 

Google Scholar 
Chen, P. et al. Nitrogen-doped nanoporous carbon nanosheets derived from plant biomass: an efficient catalyst for oxygen reduction reaction. Energy Environ. Sci. 7, 4095–4103 (2014).Article 

Google Scholar 
Hagemann, N. et al. Organic coating on biochar explains its nutrient retention and stimulation of soil fertility. Nat. Commun. 8, 1089 (2017).Article 

Google Scholar 
Farid, I. M. et al. Co-composted biochar derived from rice straw and sugarcane bagasse improved soil properties, carbon balance, and zucchini growth in a sandy soil: a trial for enhancing the health of low fertile arid soils. Chemosphere 292, 133389 (2022).Article 

Google Scholar 
Antonangelo, J. A., Sun, X. & Zhang, H. The roles of co-composted biochar (COMBI) in improving soil quality, crop productivity, and toxic metal amelioration. J. Environ. Manage. 277, 111443 (2021).Article 

Google Scholar 
Wang, Y., Xiao, X., Xu, Y. & Chen, B. Environmental effects of silicon within biochar (Sichar) and carbon–silicon coupling mechanisms: a critical review. Environ. Sci. Technol. 53, 13570–13582 (2019).Article 

Google Scholar 
Liang, J. et al. High oxygen reduction reaction performance nitrogen-doped biochar cathode: a strategy for comprehensive utilizing nitrogen and carbon in water hyacinth. Bioresour. Technol. 267, 524–531 (2018).Article 

Google Scholar 
Parsa, M., Nourani, M., Baghdadi, M., Hosseinzadeh, M. & Pejman, M. Biochars derived from marine macroalgae as a mesoporous by-product of hydrothermal liquefaction process: characterization and application in wastewater treatment. J. Water Process. Eng. 32, 100942 (2019).Article 

Google Scholar 
Zhao, L., Cao, X., Mašek, O. & Zimmerman, A. Heterogeneity of biochar properties as a function of feedstock sources and production temperatures. J. Hazard. Mater. 256–257, 1–9 (2013).
Google Scholar 
Xiao, X., Chen, B. & Zhu, L. Transformation, morphology, and dissolution of silicon and carbon in rice straw-derived biochars under different pyrolytic temperatures. Environ. Sci. Technol. 48, 3411–3419 (2014).Article 

Google Scholar 
Qiu, Y. et al. Contribution of different iron species in the iron–biochar composites to sorption and degradation of two dyes with varying properties. Chem. Eng. J. 389, 124471 (2020).Article 

Google Scholar 
Liu, X. N., Shen, F., Smith, R. L. & Qi, X. H. Black liquor-derived calcium-activated biochar for recovery of phosphate from aqueous solutions. Bioresour. Technol. 294, 122198 (2019).Article 

Google Scholar 
Luo, J., Yi, Y., Ying, G., Fang, Z. & Zhang, Y. Activation of persulfate for highly efficient degradation of metronidazole using Fe(ii)-rich potassium doped magnetic biochar. Sci. Total Environ. 819, 152089 (2022).Article 

Google Scholar 
Nan, H. et al. Pyrolysis temperature-dependent carbon retention and stability of biochar with participation of calcium: implications to carbon sequestration. Environ. Pollut. 287, 117566 (2021).Article 

Google Scholar  More

Receiver operator curve and area under the curveThe receiver operator curve (ROC) is used to visualise how the model performs independently of the decision threshold, providing a useful tool for visualising how well the classifier avoids false classifications32. The ROC plot shows a trade-off between the True Positive Rate (TPR) or sensitivity, the fraction of observations that are correctly classified, calculated in Eq. (1) as$${rm{TPR}} = frac{{{rm{TP}}}}{{{rm{TP}} + {rm{FN}}}}$$
(1)
where TP is True Positive and FN False Negative, and the False Positive Rates (FPR) or specificity, the fraction of observations that are incorrectly classified, calculated in Eq. (2) as$${rm{FPR}} = frac{{{rm{FP}}}}{{{rm{FP}} + {rm{TN}}}}$$
(2)
The passing of two lines corresponding to a 100% TPR and a 0% FPR = 1 (TPR versus 1−FPR) is considered a perfect discriminatory ability. This is graphically represented by the ROC curve passing the upper left-hand corner of the plot. The passing of the curve through the diagonal y = x represents a model that is no better than a random guess33. The Area Under the Curve (AUC) is an aggregated measure of performance for all classification thresholds and represents the measure of separability by describing the capability of the predictions in distinguishing between the classes. An AUC measure is returned between zero and one, with zero representing a perfectly inaccurate test and one a perfect test. In general, an AUC of 0.7 to 0.8 is considered acceptable, 0.8 to 0.9 is considered excellent, and >0.9 is outstanding34. Figure 1 shows the ROC curve for the test dataset close to the top left-hand corner and an AUC value of 0.89, suggesting the model has an excellent discriminative ability to distinguish between the classes, and the TPR and FPR appear robust enough to predict failures on the unseen test data.Fig. 1: Test data accuracy, Receiver Operator Curve (ROC) curve with Area Under the Curve (AUC) measure of performance for all classification thresholds.The red line is the ROC curve, and the grey line represents the diagonal y = x and a point where the curve is random.Full size imageThe calibration curve provides a means of observing how close the predictions are to those observed. Since the outcome in this model is the probability of failure between 0 and 1, it is appropriate to use a binning method. Binning is advantageous since it averages the probability of failure for each bin which provides a useful graphical representation of how well the model is calibrated. The mean probability is then compared to the frequency of observed failures in each bin. In this case, a fixed-width binning approach is used, where the data is partitioned into ten bins known as decile analysis, and approach used in similar studies35. A reliability curve provides a means of visualising this comparison, whereby perfectly calibrated probabilities would lie on a diagonal line through the middle of the plot. The briers score is a useful measure of accuracy for probabilistic predictions and is equivalent to the mean squared error whereby the cost function minimises to zero for a perfect model and maximises to 1 for a model with no accuracy4. The Brier’s Score (BS) is calculated in Eq. (3) as$${rm{BS}} = frac{1}{N}mathop {sum }limits_{i = 1}^N (P_i – O_i)^2$$
(3)
where N is the total number of observations, Pi is the prediction probability and Oi is equal to the event outcome failure or no failure. Figure 2 shows the calibration plot for the model and suggests the model is well calibrated for the lower and upper deciles since most bins fit the diagonal. The upper middle deciles do not fit the diagonal where the calibration curve is below or above the diagonal, suggesting the predictions have a lower probability than those seen in the data The briers score of 0.007 is low, suggesting accurate predictions overall.Fig. 2: Test data accuracy, calibration curve with Briers score.The red line is the calibration curve; the grey line represents a perfect fit.Full size imageConfusion matrix and accuracyThe confusion matrix describes the frequency of classification outcomes by explicitly defining the number of True Positives (TP, or Precision), True Negatives (TN), False Positives (FP), and False Negatives (FN). The decision to convert a predicted probability into a class label is determined by an optimal probability threshold such that the value of the response (y_i = left{ {begin{array}{*{20}{c}} {{rm{no}},{rm{failure}},{rm{if}},P_i le {rm{threshold}}} \ {{rm{failure}},{rm{if}},P_i > {rm{threshold}}} end{array}} right.). The default probability threshold within the model is 0.536. By this definition, there remains a practical need to optimise the probability threshold specifically to the behaviour of pipe failures within the imbalanced test data. An optimal probability threshold typically strikes a balance between sensitivity and specificity. However, there is a trade-off between TPR and FPR when altering the threshold, where increasing or decreasing the TPR typically results in the same for the FPR and vice versa. Probability threshold optimisation is an important step in the decision-making process and is specific to each problem. In the case of pipe replacement, expert judgement should be used by reasoning that water companies would seek to avoid unnecessarily replacing pipes that may have a longevity of several decades more, resulting in wasted maintenance effort and cost. Furthermore, only 0.5–1% of the network is typically replaced each year due to budget constraints37. It is therefore important to only identify pipes with the highest probability of failure. Considering this, the optimal threshold is set to reduce the FNs (i.e., pipes predicted to fail when they have not). This reduces the number of TPs predicted as discussed above but targets those pipes most likely to fail.A factorial experimental design was used, whereby the threshold was iterated from 0.01 through to 0.99, observing each threshold to reveal the point where the highest accuracy meets the lowest FN value. The Matthews Correlation Coefficient (MCC) was used to measure accuracy and is useful for imbalanced data since it accounts for the difference in class size and only returns a high accuracy score if all four confusion matrix categories are accurately represented. For this reason, Chicco (2017) argues that it is the correct measure for imbalanced data sets. The MCC describes the prediction accuracy as worst value = −1 and best value = +1 and is calculated as shown in Eq. (4) as follows:$${rm{MCC}} = frac{{{rm{TP times TN – FP times FN}}}}{{sqrt {left( {{rm{TP}} + {rm{FP}}} right)({rm{TP}} + {rm{FN}})({rm{TN}} + {rm{FP}})({rm{TN}} + {rm{FN}})} }}$$
(4)
Table 1 shows a small range of the thresholds for brevity. The optimal threshold in this instance has been identified firstly with the highest MCC accuracy and then the lowest FN. The MCC of 0.27 suggests the model is better than a random fit, but a low MCC value also represents a high percentage of false positives (i.e., values incorrectly identified as non-failure). The balanced accuracy is also a good measure of the accuracy for imbalanced classes, where 1 is high and 0 is low. The balanced accuracy for this model is 0.65. In practical terms, the results are helpful for water companies to target areas for further investigation and potential replacement since they focus on those pipes having the highest probability of failure, yet there are still incorrect predictions that could lead to the potential replacement of pipes unnecessarily. The model predicts 20.20% of all failures occurring in the WDN, found in 7.83% of the WDN pipe network. The results show that approximately 32.80% of the observed pipe failures were correctly predicted as failures, whilst approximately 67.20% of the observed pipe failures were falsely predicted as no failure. If desired, water companies could choose an alternative threshold, one that eliminates FN predictions, however, the number of TP predictions will also reduce.Table 1 Table of thresholds from training data.Full size tableRelative variable influenceThe relative variable influence shows the empirical improvement (I_t^2) accounted for by variable interval xj, averaged across all boosted trees as presented in Eq. (5) as follows38:$$hat J_j^2 = mathop {sum }limits_{{rm{splits}},{rm{on}},x_j} I_t^2$$
(5)
The variable influence helps understand which variables contribute more when predicting pipe failures. For GBT models, this is the summation of predictor influence accumulated over all the classifiers. Figure 3 shows the results, suggesting similar findings compared to existing literature. The most important variables are the number of previous failures and pipe length, both a proxy for pipe performance and deterioration. It is worth reiterating that both variables represent the grouped pipe and do not consider individual pipe history. Soil Moisture Deficit (SMD) is the most important weather variable being linked with shrinkage of clay soils and subsequent ground movement in AC pipe failures. Conversely, clay soils and soils shrink–swell potential, both representing ground movement, show lower influence.Fig. 3: Relative variable influence.Bar graph, ranking from highest to lowest, the importance of each variable as determined by the model output.Full size imagePipe diameter, and material are less important factors in this network than as reported in comparable studies11,20,21,39. The relative variable influence of days air frost and temperature is not as high as expected, given their correlation with high pipe failure frequency in iron pipes and the large percentage of iron pipes in the WDN. It is likely to be a result of over summarising the data to facilitate the annual prediction interval. A shorter prediction interval (week, or month) for networkwide groups of pipes is necessary to capture inter-annual variation accurately, but short prediction intervals in the authors’ experience can result in low predictive accuracy. The overall relative variable influence of soil (shrink well, soil corrosivity, Hydrology of Soil Type) is low. From literature and an engineering perspective, soil corrosion is strongly related to the deterioration of metal pipes and their ability to withstand internal and external forces3. It is possible that many pipes in this network may have been rehabilitated and protected against corrosion; however, this information was unavailable at the time of this study. Water source is the only operational variable and shows low influence compared to many other variables. The most important water source is surface water, resulting in lower temperatures during the winter due to its exposure to weather. This causes higher failure rates in metal pipes, yet compared to other variables, the influence is low. Other variables are imaginable such as installation details like bedding and backfill material, surrounding environments providing evidence on loading such as traffic loading and construction works, operational data such as pipe pressure and transients, water quality and spatial failure characteristics. These are not investigated here but will likely result in performance gains.Risk mappingFor the mapping to be effective from an asset management standpoint, the results of the weighted risk analysis should be able to separate out low, medium, and high failures. The number of high failures is expected to be small for two reasons, (1) pipes rarely fail more than once and (2) utilities are only able to allocate investments to those at the greatest risk due to budget limitations and are therefore only interested in the top 1–2% of pipes. The outcome of the weighted risk analysis is presented in Fig. 4, representing a small section of the WDN for clarity. Natural Jenks arranges the risk level into three categories, low [0; ≤0.02], medium [ >0.02; ≤0.06] and high [ >0.06; ≤0.92]. In this scenario, the length of pipe in the high-risk category is 13.9 km of the 300.7 km or 4.6% of the pipe network present in Fig. 4, a useful percentage of the network to target for management decisions. The choropleth risk map approach is an important means of visualising individual pipes or clusters of pipes with the highest risk in the WDN, evidenced in Fig. 4. Figure 4 also highlights how many pipes in this section of the network have a low risk, which is to be expected since many pipes have a low probability of failure and have small diameters, potentially causing less damage if they fail.Fig. 4: Choropleth weighted risk map categorised using Natural Jenks.Risk is calculated as a measure of the probability of pipe failure and the consequence of damage to nearest property and water lost based on pipe diameter. The map represents approximately 2% of the entire UK WDN.Full size imagePractical considerationsCreating groups of pipes was an important step given the low frequency of failures in the UK WDN dataset. Grouping pipes in this way assumes that all pipes in the group share similar failure rates, which is not the case, and thus the approach adopted here presents a suitable solution to this limitation. Grouping pipes on a lower spatial scale can capture localised influences on pipe performance, that can often be obfuscated when generalising over the whole network. However, the approach used may not be as useful for rural areas where fewer pipes are present, where smaller scales may be more appropriate (e.g., 1:100,000 is a smaller scale than 1:100). Further investigation into grouping scales is merited. Optimisation the threshold is challenging and inevitably leads to inappropriately classified failures on either side of the threshold. Optimising is even more difficult with imbalanced data sets since conventional classification methods are built to assume that all classes are equal. An alternative approach was applied in this study, which used MCC accuracy and FN to set a threshold, reducing the potential for wasting budgets replacing pipes that will not fail. In the process, the number of TPs was reduced to 32.80% of the observed pipe failures, whilst the number of FPs was 67.20% of the observed pipe failures, which may not present a good argument to professionals. Despite this, the results can be used directly in strategic planning, which sets long-term key decisions regarding maintenance and potential replacement of pipes. Predicting the probability of failure is an essential response since it enables the identification and prioritisation of risk across the network. This methodology could also be used to provide longer-term predictions to support the development of Asset Management Plan, which cover a five-year period of regulated investment.Categorising the pipes based on a weighted risk analysis and visually presenting them using Natural Jenks offers a useful method for prioritising pipes based on the consequence of their failure and is an easily assessed cartographic presentation. It extends the probability of failure into a more useful measure of risk, providing more information for decision makers. The use of distance to property in this study is a simple approach to determine flooding. To provide a realistic determination of flooding, an understanding of key geographical features for overland flow routing is required40. The list of consequences was limited in this study and could be extended when such data is available. There are potentially numerous consequences of failure inherent to each network, yet common consequences include loss of water, potential disruption, reduction in water quality, reliability, direct costs (damage to property and infrastructure and pipe repair and replacement) and indirect costs (environmental and social)8. In this study, the risk estimates were concluded by expert knowledge, and any contextual mismatch between weightings could potentially skew the outcomes. Therefore, the weightings should be considered carefully by network professionals. At an engineering level, the risk mapping can be further used to determine areas of the network leading to a high probability of failure, which can be used to take constructive pre-emptive actions towards extending the life of future pipe construction41.The economic benefits of this model will manifest when performing proactive maintenance, potentially averting associated risks that may arise from damaging properties and infrastructure. It is anticipated that the modelling approach proposed will enhance decision-making at a local level, facilitated through numerical outputs which report on the serviceability of the WDN and help meet regulatory performance targets avoiding heavy fines. Operationally, the approach will help with highlighting short pipe segments for repair and replacement though graphical outputs, these are practical lengths of pipes for operational teams that typically do not replace kms of pipe at any given time42. This approach shows similar performance to comparable GBT studies11,20, but is beneficial since the method provides reliable predictions on a shorter annual time frame. The method here is also computationally easier to develop than other more complex machine learning methods such as neural networks and Bayesian Neural Networks.The predictions rely on the quality of the data, and several challenges were presented during the cleaning and processing, most notably the location of the pipe failures, many of which were geographically displaced, and some by a considerable distance yet was necessary to retain all the failures in the dataset. These were snapped to the nearest pipe with similar characteristics, yet it is conceivable that some were incorrectly placed despite the protocols established for the snapping process. Further limitations to the study include limited data, where pressure data or other operational data may have proved useful, the advantage of which may consist of increased model accuracy and interpretability. Over-summarised local conditions can also affect the model accuracy, and in this study, the local soil conditions were presented from a soil map at 1:250,000 scale. Likewise, the weather variables were highly summarised to an annual scale from a 40 × 40 km grid source. Inevitably these limitations will affect the model, which can potentially hinder effective decision-making. There are several challenges faced when modelling pipe failures, from uncertainties in data collection and management to specific data processing solutions. There is a need to understand these holistically, and from the view of current practice for a more in-depth perspective of current challenges in practice that may hinder useful data gathering. In addition, future research aimed at understanding how practitioners understand pipe failure models, the limitations, and opportunities is beneficial, since there is often a discord between the capabilities of modelling and user expectations. This further research may help to improve pipe failure models by encouraging enhancements in the pipe failure model process that promotes quality data capture.Concluding remarksThis study considered the prediction of pipe failures using a GBT model and establishing the risk based on weighted risk analysis to prioritise pipes for proactive management. A 1 km spatial scale was included in this model when grouping the pipes, which aimed to capture localised conditions and remove the failure rate disparities shared when grouping pipes across a network. This spatial scale, together with a short prediction interval, the absence of some essential variables, and additional inherent problems with pipe failure data sets, has ultimately resulted in acceptable accuracy. However, in practical terms, when used in conjunction with expert knowledge, the results provide a useful approximation of potential failures and a better understanding of the current WDN to help plan rehabilitation and replacement efforts. Improving model accuracy may be achieved by increasing the prediction interval to five-year asset management plan, potentially accumulating more failures per pipe group from which to predict. Yet this may not be as useful to water companies where management decisions are typically annual. Furthermore, understanding the issues faced with data collection and quality from current practice may help to encourage data quantity and quality, and could potentially provide marked improvements in the final predictions.Further suggested research includes exploring different pipe grouping variations, collecting more data on the consequences of failure to enhance the weighted risk analysis and, expanding on this idea, understanding the data quantity and quality issues from current practice, and exploring feature engineering techniques to derive more valuable data sets that may improve model accuracy. More

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain
the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in
Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles
and JavaScript. More

Experimental sitesExperimental designs varied across the 39 research sites with plot size ranging from 0.04 ha to 80 ha. The size of the plot drainage areas varied accordingly from 0.02 to 56 ha. The number of site-years of available data ranging from 2 to 17 with a mean of 7 years. There were diverse soil types, five soil textural classes and soil organic carbon ranging from 0.1% to 3.7%. Corn (Zea mays) and soybean (Glycine max) were the predominant crops grown, but 23 site-years had popcorn (Zea mays everta), wheat (Triticum aestivum), forage, oats (Avena sativa), or sugar beets (Beta vulgaris).CD was practised at the greatest number (19) of sites (Fig. 2) across seven states in the Midwest and North Carolina. The research sites extended from 35.8° to 46.4° N and 76.7° to 96.9° W. The majority of sites (30) were on private farm (cooperator) fields through a lease or collaborative arrangement, with the remaining 9sites on university-owned and managed research farms. The USDA soil drainage class for the dominant soil type at each site ranged from somewhat poorly drained to very poorly drained11. The subsurface drainage of all sites consisted of 102 mm-diameter perforated corrugated tubing except MN_Clay sites (76 mm diameter tubing) and included both CD and free drainage (FD) treatments. Tile depth ranged from 0.61 m to 1.22 m, and tile spacing varied from 6 m to 36 m with median 13.7 m. All sites had similar drain spacings across treatments except IA_di4 and IA_Washington. These two sites varied tile spacing and/or tile depth. IA_di4 tile spacing differed with 27 m and 36 m for FD and CD plots, respectively. While at IA_Washington, tile spacing was 12 m in the shallower drainage treatment compared to 18 m spacing in the conventional drainage treatment. Seven sites had replicated drainage treatments with an average drainage area of 1.1 ha. Sites that did not include replications were larger farm fields with an average drainage area of 10.5 ha, except one university research field with a drainage area of 1.8 ha.Fig. 2Availability of key variables published in the Transforming Drainage data. Number of site-years shown (a) by drainage water conservation practices, and (b) by year measurement occurred.Full size imageDWR research was conducted at seven sites across the Midwest. Individual research site locations ranged from 39° to 46° N and 83° to 96° W. The treatments at the sites included DWR utilizing controlled drainage with sub-irrigation or controlled drainage with on-surface drip irrigation. In addition, there was a comparison treatment of FD with no irrigation. The three Ohio sites included wetland monitoring in addition to drainage water recycling as part of the Wetland Reservoir Subirrigation System (WRSIS) project12.Eight SB sites were monitored as part of this project, seven in Iowa and one in Minnesota. One of the Iowa sites included the first SB installed in the US4. Five sites categorized as ‘Other’ included monitored drainage practices slightly different from the previously described categories. The IN_Tippecanoe site was a wetland with future drainage water recycling planned but not implemented during this period. MN_Clay1 was a conventionally drained farm, MN_Clay3 was an undrained farm with only surface drainage, MN_Redwood2 was an undrained prairie area and ND_Richland had controlled drainage and a sub-irrigated area utilizing a sump pump lift station for water management.Data collected at each siteThe data describes crop and field management, soil physical characteristics, water quality and quantity time series, drainage system design and specific practice variables for the 39 research sites. Weather data, primarily precipitation and air temperature, were also available for each site. However, other data collected varied since the measurement protocols were not coordinated before research was initiated at many sites. Cumulatively, more than 90 in-field variables were measured across all sites to characterize the performance of these alternative agricultural water management strategies. Water quality and quantity time series (drain flow, water table depth, nitrate-N concentration, and precipitation) were considered essential data for temporal robustness and accuracy regarding the hydrological response.Precipitation (39 sites) and drain flow or discharge (36 sites) were the most reported variables, followed by nitrate-N concentration (32 sites) and load (30 sites) (Fig. 2). Other common water quality variables are summarized in Fig. 3. In addition, soil moisture time series collected at varying depths were reported for 16 sites.Fig. 3Type of water quality data in the Transforming Drainage data. Number of site-years per variable shown with type of drainage practice denoted by colour. Ortho P, Total N, and Total P are defined by whether the sample was filtered prior to analysis to remove suspended (solid) content from the aqueous fraction.Full size imageIn addition to the water quantity and quality variables that provide a direct measure of treatment impact to water sustainability, other variables including crop yield, crop and field management and soil characteristic data are important for evaluating inter-site variability. For example, differences in nutrient application with fertilizer and nutrient removal through crop uptake will influence the water quality impact of different treatments. Soil texture (reported for 21 sites), crop yield (29 sites), tillage (27 sites) and fertilizer application (31 sites) were considered most essential of these site characteristic variables13. Along with crop yield, sites reported additional variables that assisted in quantifying plant water, nutrient and carbon uptake, including grain moisture content (13 sites), final plant population (end of season plant density; 9 sites), grain total N (8 sites) and grain biomass (6 sites). Whole plant, vegetative and cob biomass, and whole plant, vegetative, cob and grain N and C contents, forage biomass and leaf area index were reported for five or fewer sites.Sixteen sites reported soil organic carbon and total N, in addition to basic soil texture information. In addition, 31 other soil parameters were reported for a subset of sites; the most common are summarized in Table 1. Soil organic matter, infiltration, lime index, sodium concentration or amount, sodium absorption ratio, neutralizable acid and salinity were reported for five or fewer sites.Table 1 Most reported soil variables and number of sites.Full size tableSummary of measurement methodsMost experiments were not coordinated when the data collection project was initiated; hence research data collected, length of experimentation, years of available data, and protocols varied. Methods for each research site are provided in the data to document differences in measurement schedule, sample size, sample collection frequency, and equipment precision. Here, we summarize methods for determining drain flow, nitrate-N concentration and load, water table, soil properties and weather data due to the variability across sites within these key metrics. Crop yield is not summarized here despite its importance as a metric due to more consistent methods typically used across sites. Inter-site sampling methods for water measurements varied more than methodology for measuring other parameters. This variability is due to differing infrastructure at each site that required different measurement methods and the financial resources available for monitoring.Drain flow measurement and reportingDrain flow or discharge data were reported for 36 sites, including 19 CD, eight SB, six DWR and three with other practices (e.g., wetland). For all CD, three DWR and two wetland sites, drain flow was reported in mm/day (drainage discharge normalized by the drainage area). For all other sites, volumetric drainage discharge was reported in m3/day. Two of the sites (MN_Clay3 and MN_Redwood2) were undrained control sites that did not report drain flow or discharge. A third site (MO_Shelby) focused on the agronomic impact of subsurface drainage practices and did not monitor drain flow.Drain flow was measured hourly or sub-hourly at more than 80% of the sites, followed by aggregation to daily flow measurements. Subsurface drainage flow rates were determined as a function of the water head measured using pressure transducers installed inside drainage control structures or at the drain outlet for approximately two-thirds of the sites. The water head was measured upstream of V-notch or rectangular weirs and empirical equations that depend on the weir dimensions were used to determine drain flow, which was measured and recorded hourly or sub-hourly. For IN_Tippecanoe drain flow was estimated as a function of water head using an empirical rating curve. At three sites, drain flow was measured using inline flow meters and recorded by data loggers. The advantage of this method is that flow could be recorded in either direction, valuable for sites experiencing backflow in the drainage system due to high downstream water levels14. At ND_Richland, drainage was collected at a sump where a current sensor was used to measure pumping frequency to calculate drainage flow15. For an additional three sites, drainage discharge was measured using a depth-velocity meter installed at the outlet of the drainage pipe or a drainage ditch. The drainage discharge was calculated as the product of the flow velocity and the area of flowing water. Only one site (MN_Redwood3) had manual measurements of drain flow that were collected two to three times per week.Measured drain flow data exhibited variable frequency and duration gaps due to instrumentation malfunctioning, particularly with the automated monitoring systems that provide near-continuous data. Missing data and their non-uniform distribution created problems in statistical analyses when comparing aggregated drain flow and loads from different locations. A systematic approach was used to infill missing drain flow data utilizing variables available at all sites (precipitation, temperature, drain flow) and replicate plots where available. The method consisted of the following three phases and completed in progression, when applicable.Phase 1, fill in zero flow.
During most winters in the northern states, the soil is frozen to the depth of the tile, and no subsurface drain flow is expected. Such periods were identified based on expert judgment by researchers at each site, relying on soil and air temperature information and local knowledge of the drainage system’s response to these conditions. If no drainage measurements were available due to frozen soil, the corresponding gaps in the data record were infilled with zero.

Phase 2, predict using replicate plots.
Regression-based estimation was used to infill missing data at three sites which had replicated plots or adjacent fields with available data. Due to the seasonal nature of subsurface drainage from croplands, individual linear regression models were developed for each season: winter (Jan, Feb, Mar), spring (Apr, May, Jun), summer (Jul, Aug, Sep) and autumn (Oct, Nov, Dec). Regression r2 values ranged from 0.66 to 0.94 based on the site and season, although mean across-site values were similar: winter (0.80), spring (0.82), summer (0.80), and autumn (0.83).

Phase 3, populate based on precipitation and drain flow from the preceding day.
The remaining missing daily drain flow data at 11 sites were filled as described below, based on the assumption that drain flow occurs on a given day only if (a) precipitation occurred on that day or (b) the drain continued to flow from the day before.

a.

For days with precipitation, a two-day moving average was calculated to account for the time lag between rainfall and resulting drain flow. A linear regression model was fitted to non-zero drain flow and two-day moving average precipitation for each season, with the model’s intercept fixed to zero. We used these models to predict the missing drain flow data for days with non-zero precipitation. The predicted drain flow values were limited to the drainage system’s capacity by replacing predictions greater than the site’s drainage coefficient (depth of water the drainage system could remove within 24 hours) with the coefficient’s value.

b.

For days with zero precipitation, missing drain flow was calculated from the previous day’s observed flow using the following first-order recession equation

$${Q}_{i}={Q}_{i-1}{e}^{k}$$where Q is daily drain flow, k is the average recession coefficient of falling limbs calculated as a linear slope of ln(Q), and i indicates day. The recession coefficient was calculated as a linear slope between the peak and inflection point of log-transformed daily drain flow data. The coefficient was calculated for all falling limbs of drain flow data, and the average seasonal values were calculated as their arithmetic mean.
The regression model between on-site precipitation and peak flow and recession equation were only applied to the original (pre-gap-filled) drain flow data. Predictions were not made when the number of missing drainage days exceeded 152 (5 months) within a calendar year; therefore, approx. 18% of the drain flow data remain missing. Both the original and filled data are included in the published data.
Nitrate-N concentration and load measurement and reportingNitrate-N (NO3) concentrations were reported for 32 sites, including 15 CD, eight SB, six DWR, and three sites with other practices (e.g., wetland). The three sites not reporting drain flow (MN_Clay3 and MN_Redwood2, MO_Shelby) did not report NO3 concentrations. Two sites (MO_Knox1 and MO_Knox3) provided NO3 load along with discharge in place of reporting the concentration of individual water samples. Two sites (OH_Hardin2 and OH_Henry) did not report NO3 concentrations or load due to limited water sample collection at these sites.Six sites collected flow-proportional samples, in which a sample is collected every time a given volume of water passes through the drainage system. The flow-proportional sampling methods at the sites varied. At NC_Washington, a portion of flow was diverted continuously into a composite sample which was collected fortnightly (or more frequently under high flows). At IA_di4, a proportional sample was collected each time the drainage system was pumped. At MN_Redwood1, flow proportional samples were collected during storm and baseflow conditions. These samples were not composited but rather kept discrete. Seven sites used automated samplers to collect time-proportional samples. Five of these sites composited samples daily, while one site (IN_Randolph) collected samples hourly, then combined samples into approximately weekly composites. One site (IN_Tippecanoe) collected weekly grab samples prior to 2016 but then switched to automated, time-proportional sampling composited weekly in March 2016. Sites that used automated samplers typically switched to manual sampling (every two days to weekly frequency) in winter to protect automated samplers from freezing. Twelve sites collected weekly grab samples, another collected samples 2–3 times per week. One site collected biweekly grab samples, and four sites collected grab samples approximately monthly. Regardless of the collection method, all samples were either frozen or refrigerated (4–5 °C) upon return to the laboratory until analysis.The sampling strategy primarily affects the frequency and compositing strategy of the water samples. Automated samplers permit more complex sampling strategies, such as flow-proportional or sub-daily sampling. However, the disadvantages of this method are the high initial expense of sampling equipment and the propensity for equipment malfunction at below-freezing air temperatures. The potential for equipment failure prompted sites using automated samplers to switch to a manual sampling in winter while drains remained flowing. Manual sampling frequency varied among sites due to differences in site accessibility or personnel availability. Both automated and manual water samples were often composited following collection, and sample compositing frequency ranged from daily to biweekly. Although sample collection frequency and compositing strategy affect the uncertainty of loading measurements, a collection frequency between 3 to 17 days is generally sufficient to reach ± 10% accuracy for annual nitrate load estimation for tile-drained landscapes in the Midwest16.For nitrate-N analysis, 12 sites reported a cadmium reduction followed by a sulfanilamide reaction (equivalent to EPA 353.2). However, there was a slight methodological variation depending on the equipment, either Lachat QuikChem 8000 Flow-Injection Analyzer or SEAL AQ2 Discrete Analyzer. The resulting nitrate-N concentrations calculated via cadmium reduction were directly comparable regardless of the instrument used. At one site, SD_Clay, ion chromatography (EPA 300.1) was used to measure nitrate-N in 2015 but was subsequently switched to a cadmium reduction method. Samples at the seven IA sites were analysed by second-derivative spectrophotometry17.Daily nitrate loads were calculated by multiplying nitrate concentration by drain flow and were therefore available for 32 sites for which both values were reported. Load calculation methods differed slightly in terms of determining the volume of water associated with each concentration. Typically, linear interpolation was used to determine the daily nitrate concentration at sites which collected “grab” water samples following precipitation events or on a schedule spanning two days or more. One variation used assumed the measured concentration was representative of adjacent days (prior and post), hence no interpolation was done. One site (OH_Delaware) used a midpoint approach to determine the time interval in which measured concentrations were associated with, while another site (IA_di4) assumed measured concentrations represented all water drained before the sample was collected.Water table measurement and reportingThe water table was measured at 16 sites including nine CD sites, three SB sites, three DWR sites and one wetland site. Documenting water table fluctuation is key to experimental and modelling research investigating crop production systems on artificially drained soils. In a tile-drained field, the water table is used as an input parameter in estimates of drain flow, evapotranspiration, and soil hydraulic conductivity14,18,19. In controlled drainage, the water table is used to determine CD effectiveness and guide water management in the field for different crop stages. For DWR practice, the water table, particularly the midpoint water table, is used to evaluate sub-irrigation performance, such as uniformity and efficiency20. In a saturated buffer field, the water table is the most important factor used to indicate a field’s saturation status21.The field water table was typically measured at the midpoint between the subsurface drains. Some field studies also measured the water table at two locations, one near the drain tile and the other at the midpoint between two drain tiles. The water table was commonly measured and recorded hourly or sub-hourly using pressure transducers installed inside 1.5–2.5 m deep wells of perforated PVC pipes. The water table depth was calculated using the measured water pressure above the transducer, and the in-situ water temperature and barometric pressure measured in a nearby field and periodically adjusted with manually measured water tables. If there were any discrepancies, all previous water table depth data were moved up or down correspondingly.Differences across sites spanned the type of pressure transducers used, depth of measurement (1.5 to 2.4 m), data collection frequency (0.17 hr (10 min) to 6 hr), location of the measurement, and the length of the screened section of the pipe. The selection of the transducer type was due to individual choice and cost, while differences in the water table depth measurements were affected by the soil types and drain depth. The frequency of data collection was based on data logger capacity, water table variations, and the purpose of the measurements. The length of the perforated (“screened”) section of the pipe, in which the transducer was installed, also varied. For a typical tile-drained field, the pipe was screened beginning 0.3 m below the soil surface while for a saturated buffer, the pipe was screened beginning at the soil surface16. The data collection frequency for the saturated buffer area was every 6 hr since the water table variations were minimal across time. Within the field experiments, data were collected every hour at 10 sites and every 0.17 hr (10 min) at two sites.Soil physicochemical variable measurement and reportingPotentially important soil physical and chemical properties that might affect or be affected by soil drainage were collected from 19 experiments across six states. Data included 17 total variables, continuous and categorical. Soil physical variables included bulk density, hydraulic conductivity (saturated), moisture (water) content, temperature, texture, and soil water retention data (used to form a water retention curve). The remaining 11 variables were chemical properties. The five most common chemical variables characterized were nitrate, total nitrogen, soil organic carbon, pH and cation exchange capacity with several sites using similar methods22.There was large variability of soil sampling depth among the studies and within specific variables at a site, and in a few cases ( More

As NDBC publish their historical and real time in situ wave and meteorological data in multiple online locations, USACE developed a methodology to combine these data sources and develop a unique USACE QCC Measurement Archive that is fully self-describing. This required merging the manually quality controlled data that is stored on the NDBC website with the lower quality netCDF data with metadata files for the same stations that are stored at NCEI. The NOAA DODS source was not included as those data are exact copies of what is found within the NDBC historical station pages.As mentioned, the NDBC website historical station pages contain the cleanest data that has been subjected to manual QA/QC by NDBC Mission Control data analysts. Data collected during service periods (when the buoys were physically on board ships for maintenance) were removed during the manual QA/QC, and are typically not present within the NDBC website data. However this data source contains no metadata other than date and time. This lack of metadata allows for the erroneous inclusion of unidentifiable data from historical time periods where the moored buoys were adrift (inaccurate wave readings, wind, temperature etc.). Additionally, although NDBC switched to a redundant meteorological sensor paradigm during the last decade, only single variable values are available per time stamp per station on the NDBC website. This is because NDBC toggles the release of primary and secondary sensor data to ensure that the highest quality data are published. However, the NDBC website contains no associated metadata indicating when these data release switches occur and hence instrumentation usage is indeterminable. Users often need these sensor details, for example wind sensor height above sea level to extrapolate wind speed at additional heights above the moored buoy. The NDBC website also does not store uncorrected non-directional spectral energy estimates (({c}_{11}^{m})).Conversely, the NDBC netCDF data stored at NCEI includes metadata such as time-stamped GPS positions, instrumentation metadata, data quality flags1, and data release flags (indicating which data were released to the real time stream). These GPS positions allow for the identification of data that was collected while NDBC moored buoys were adrift. For ease of data source identification, these NDBC netCDF files stored at NCEI will be referred to as NCEI netCDF data below. However, readers should remember that these are all NDBC data, with time-paired values that are collected from the same, unique sensor.This NCEI netCDF data source also includes both the primary and secondary redundant meteorological sensor outputs, with metadata, as well as uncorrected non-directional spectral energy estimates (({c}_{11}^{m})). These primary and secondary sensor variables are only found within these NCEI netCDF datasets. However, since 2011, these netCDF data are pulled from the NDBC real-time data stream, which is only subjected to automated QA/QC protocols that flag but do not remove suspect data1. Prior to 2011, the NDBC data were stored in an encoded Trusted Data Format (TDF), but these data were converted into netCDF format in early 2020.Of note is that the NCEI netCDF structures differ for data stored before and after the 2011 switch to netCDF file usage. Throughout the historical netCDF dataset, the netCDF file structures contain non-uniform netCDF formats that are dependent on the data collected during file-specific time periods. Additionally, the pre-2011 netCDF files contain a nominal, fixed deployment position that is repeated for each date/time stamp within the datasets. Furthermore, these pre-2011 netCDF files contain erroneous spectral wave frequency bands that are not included in the NDBC website datasets (and do not match any wave instrumentation frequencies that NDBC has historically deployed). Both formats include instrumentation metadata that are not only inconsistent throughout the years, but within individual netCDF file’s group attributes.Therefore, to mitigate these identified data source issues10, the USACE QCC Measurement Archive process utilized a methodology (Fig. 1) that combines each dataset’s advantages to develop a best available historical NDBC measurement dataset. For example, the GPS data included within the post-2011 NCEI netCDF files were used to detect data that fell outside a reasonable radius of the moored buoy. Conversely, the NDBC website data were used to isolate which primary or secondary sensor data were released to the public – achieved by matching the individual NDBC variable values to the equivalent primary or secondary NCEI netCDF values, therefore identifying the correct netCDF metadata. Additional outlier QA/QC variable checks, station and metadata verification (provided by literature reviews and historical NDBC buoy deployment log books) allowed for the development of a best available, self-described USACE QCC Measurement Archive.Fig. 1Flowchart of the USACE QCC Measurement Archive methodology. This flowchart outlines input data sources, station and metadata verification, selected ‘best’ data sets and output netCDF files.Full size imageThe USACE QCC Measurement Archive methodology process consists of two phases. The first phase of the project processes the historical data, while a second phase annually appends newly available data to the historical database. The data archive routine involves a six step process (Fig. 1) for each buoy station: (1) download, (2) concatenation, (3) metadata verification, (4) comparison, geographical QA/QC and metadata attachment, (5) best dataset selection, and (6) netCDF data file creation. Finally these netCDF files are uploaded to the buoy section of the USACE CHL Data server.These steps were automated using scripts developed in R software11. Where necessary, each script was subset to handle the particular idiosyncrasies10 of the NDBC and NCEI netCDF data archives. To process all of the historical NDBC data (1970–2021), steps two to five in phase one required ~ 400k cpu hours at the Department of Defense (DOD) Supercomputing Resource Center.The following steps outline the methodologies utilized within this USACE QCC Measurement Archive development. For more detailed information, please see the USACE QCC Measurement Archive Standard Operating Procedure (SOP) document that is stored in the Archive GitHub (https://github.com/CandiceH-CHL/USACE_QCC_measurement_archive.git).

1.

Step 1: Download. Historical NDBC data for all NDBC stations are downloaded from the NDBC website and the NCEI archives. Source-specific archive download links are listed in the USACE QCC Measurement Archive SOP. Data from the storage specific files types (detailed below) are extracted for concatenation in step 2.
The NDBC website stores data in zipped yearly and monthly files as standard meteorological (stdmet), spectral wave density (swden), spectral wave (alpha1) direction (swdir), spectral wave (alpha2) direction (swdir2), spectral wave (r1) direction (swr1), and spectral wave (r2) direction data (swr2). These files require unzipping. Included within the NDBC stdmet datasets are collected meteorological and bulk wave data in the following structure: wind direction (°), wind speed (m/s), wind gusts (m/s), significant wave height (m), dominant wave period (seconds), average wave period (seconds), mean wave direction (°), air pressure at sea level (hPa), air temperature (°C), water temperature (°C), dew point temperature (°C), visibility (miles) and tide (ft). Visibility and tide are no longer collected by NDBC, and are disregarded.
The NCEI website stores monthly NDBC files per year in netCDF format. All available data and metadata are extracted from these netCDF files. These files contain the same NDBC data as listed above, but also include additional wave spectral parameters such as uncorrected spectral energy wave data (({c}_{11}^{m})), spectral wave co- and quad-spectra, and four wave data quality assurance parameters that are produced by the NDBC wave processing procedure12.
The NCEI netCDF file formats differ significantly before and after January 2011. After January 2011, these netCDF structures varied throughout the years as NDBC buoy structures and netCDF creation procedures changed. Each format requires format-specific code to extract the data from the variable fields.
For example, the pre-2011 netCDF files consistently contain all variables directly within the main file directory. However, the post-2011 netCDF files are structured by ‘payload’, with subset sensor fields (e.g. ‘anemomenter_1’), which in turn have their own subset variable fields (e.g. wind_speed, ‘wind_direction’) with associated quality control and release flags. Therefore users have to navigate through the payload and sensor subfields to discover the variable data with their associated metadata.
Importantly, these ‘payload’ fields do not always refer to the on-board computer system that serves the sensor suites, e.g. NDBC’s Automated Reporting Environmental System13 (ARES), but also delineate between sensor suites with available primary and secondary sensor data (e.g. ‘payload_1’, ‘payload_2’). Conversely these primary and secondary sensor data (e.g. ‘air_temperature_1’ and ‘air_temperature_2’) may be subset within a single ‘payload’. Of note is that these multiple payloads often contain duplicated data.
These ‘payload’ fields are also important when extracting data captured by NDBC Self-Contained Ocean Observations Payloads (SCOOP), as these netCDF files resemble the physical structure of the buoy stations with their modular sensor assembly. For example, the NCEI netCDF July 2020 data file for station 41009 includes 5 payload subsections. ‘payload_1’ contains an ‘anemometer_1’ sensor suite, which contains subset wind variables and data flags; ‘barometer_1’, with subset air pressure variables and flags; and a ‘gps_1’ sensor suites, with subset lat, lon variables, etc. ‘payload_2’ contains a second ‘anemometer_1’, ‘barometer_1’, ‘gps_1’, ‘air_temperature_sensor_1’, and ‘humidity_sensor_1’ suites. Payload 3 contains a single ‘gps_1’ fields (lat and lon variables with flags), while payloads 4 and 5 house ‘wave_sensor_1’ and ‘ocean_temperature_sensor_1’ sensor suites respectively, both with their own ‘gps_1’ data. In this example, ‘payload_1’ represents an R.M. Young sensor, while ‘payload_2’ is listed as a MetPak Weather Station instrument in the netCDF sensor suite attributes.
NDBC is in the process of redesigning these netCDF file formats to be more user friendly. However, they do not plan to reformat their archive datasets. For more details on the NDBC and NCEI netCDF file formats and code extraction descriptions, please see the USACE QCC Measurement Archive SOP within the Archive GitHub.

2.

Step 2: Concatenation. This step merges each yearly and monthly data files to produce a single time series of concatenated stdmet data, and time series files for each individual spectral wave variable. The concatenated stdmet data format mirrors the NDBC website data formats. To handle the NDBC data, this step allows for the management of differing yearly file formats and spectral frequencies; the concatenation of multiple date and time columns into one field; and the removal of redundant date, time and tide columns in stdmet data. This step allocates the spectral data into the standard NDBC 38 frequencies (old wave sensors), and 47 frequencies (new wave sensors). Finally, this step converts the NDBC r1 and r2 values to their correct units (NDBC r1 and r2 data are scaled by 100 to reduce storage requirements, so these data should be multiplied by 0.01).
To handle the NCEI data, this step allows for the concatenation of stdmet data to create a dataset that matches the NDBC website data nomenclature. This step also removes data that were flagged as erroneous by automated NDBC QA/QC protocols. As unit standards vary between the NCEI and NDBC website archives, this step converts the NCEI netCDF pressure units to match the NDBC units (Pa to hPa), and converts the air, water and dew point temperatures from Kelvin to degree Celsius to match NDBC data. This step also performs outlier QA/QC, where it removes zero (‘0‘) wind gust values when no wind speed values are present; direction values greater than 360 °; obvious variable outliers; and duplicated netCDF data points that are ~5–10 seconds apart. To handle the erroneous netCDF spectral frequency data, the code advances through the spectral data and matches the available spectral frequency data to the appropriate 38 frequencies (old wave sensors) or 47 frequencies (new wave sensors).

3.

Step 3: Verify metadata. This step is applied solely to the NCEI netCDF data files to validate the netCDF metadata with NDBC-sourced, buoy specific metadata spreadsheets. These metadata spreadsheets were constructed from the NDBC database and original NDBC service technician log books, and provide accurate station and sensor information. Scripts verify or insert missing hull type, payload and mooring type; and verify or insert missing instrument processing systems (for wave data only), instrumentation names and sensor deployment heights. If none are available, metadata fields are augmented with pre-set hull-specific instrumentation specifications that were sourced from online references (for hull-specific instrumentation specifications, please see the USACE QCC Measurement Archive SOP).

4.

Step 4: Compare, geographically QA/QC and attach metadata. Compare: Although these data originate from the same sensor, storage protocols resulted in different time stamps for each within their various archives. This step compares the NDBC and NCEI sourced data by matching the datasets by nearest date and time (to the minute), after which geographical data are appended to the NDBC datasets.
As the NDBC data is manually QA/QC’d and does not contain data collected during buoy maintenance operations, these data were considered as a date/time reference to quality control the fixed positions of the pre-2011 netCDF datasets. In other words, if data were present within the NCEI dataset, but not within the NDBC dataset, those NCEI data records were removed.
Of interest are the datasets within the NCEI netCDF files that pre-date any data published on the NDBC website. These data are likely from sensor and processing tests conducted during deployments that were intentionally not released to the public. These early data are included in the USACE QCC Measurement Archive but have quality control (QC) flags that rate them as unreliable. For more information on these earlier datasets, please reference the technical note on utilizing NDBC data, ERDC/CHL CHETN-I-10010.
Geographically QA/QC: Each dataset is filtered to remove GPS positions and associated data that are not within a one (1) degree radius (~60 nautical miles) of the NDBC station watch circles (the surface area through which a buoy can travel while tethered to specific location by a mooring). This radius allows for fluctuations in NDBC deployment locations over the decades, as tests showed that radii of less than one degree significantly removed viable data (see Fig. 2 in the Technical Validation section). Users may wish to further filter their specific datasets to remove additional data points that are outside their target deployment locations; a task now easily achievable with the fully-described, verified metadata included within this USACE QCC Measurement Archive13.
Two methods are used to geographically QA/QC these data: 1) a sorted table of value occurrences to find the most common latitude and longitude positions (using the assumption that the buoy held its correct station for the majority of its life cycle); 2) a manual confirmation and insertion of the primary station locations that were sourced from NDBC buoy specific metadata spreadsheets. This manual step was relevant for buoys that did not consistently hold their stations due to high vandalism rates or strong currents.
Assign metadata: Once the data are geographically QA/QC’d, this step assigns verified metadata (from step 3) to the NDBC stdmet datasets as follows. Station-specific hull type, water depth, payload and mooring type are appended to the NDBC stdmet datasets from the NDBC-sourced, buoy specific metadata spreadsheets. These NDBC Buoy Metadata Spreadsheets and the verified NCEI netCDF metadata are then used to assign the correct primary or secondary sensor designation, which includes metadata such as instrument processing systems (for waves) and instrumentation information (names, deployment heights etc.), to the NDBC stdmet datasets by matching the time paired NDBC variable values with the exact NCEI values.

5.

Step 5: Create best dataset. This step selects a combination of the geographically QA/QC datasets that were created in step 4 above. These best available, self-describing datasets (Fig. 1) include:

NDBC website wind direction, wind speed, wind gust, air pressure at sea level, air temperature, sea surface temperature, significant wave height, dominant and peak periods, mean wave direction, spectral c11, alpha1, alpha2, r1, r2, with their now fully-described, verified metadata.

NCEI netCDF spectral ({c}_{11}^{m}). These data are retained within the USACE QCC Measurement Archive to allow for bulk wave parameter re-calculations without the influences of NDBC shore-side processing protocols.

Verified station metadata obtained from the NDBC Buoy Metadata Spreadsheets.

NCEI netCDF data for the above variables that pre-date the NDBC datasets (where applicable).

6.

Step 6: Create netCDF data files. This step creates monthly netCDF NDBC data files that collate all of the best available data variables that were selected in step 5 above. For easy access by the USACE and user community, these month-long netCDF data files are stored on the USACE CHL Data Server and are updated annually. A static copy of the historical data (1970–2021) is located within the USACE Knowledge Core Library Datasets13. More