Development of Data Acquisition and Management System for GEOSS Evaluation

Zhu, J.^{1,2  Liu}, Y. H.³  Fan, J. L.^{4*  Liu}, C.⁵  Wu, J. J.⁶  Adugna, T.⁷

1. National Satellite Meteorological Center (National Centre for Space Weather), Beijing 100081, China;

2. Innovation Center for FengYun Meteorological Satellite (FYSIC), Beijing 100081, China;

3. Aerospace Hongda Information Technology Co., LTD., Beijing 100089, China;

4. Faculty of Geographical Sciences, Beijing Normal University, Beijing 100875, China;

5. Institute of Geographic Sciences and Natural Resources Research, Chinese Academy of Sciences, Beijing 100101, China;

6. Institute of Aerospace Information Innovation, Chinese Academy of Sciences, Beijing 101408, China;

7. Yangtze Delta Region Institute (Huzhou), University of Electronic Science and Technology of China, Huzhou 313001, China

Abstract: The Group on Earth Observations (GEO) is an intergovernmental organization establi­sh­ed in 2005 that aims to enhance our understanding of the Earth system and provide information services for decision-making. The principal objective of the organization is to develop a Global Earth Observation System of Systems (GEOSS) to the benefit of mankind. The GEO work plan is the driving force in the implementation of the GEO Strategy plan for the New Decade (2016?C2025). The progress of the GEO work plan and the development of the Earth observation directly mark the construction process of GEOSS. However, GEOSS is a complex multi-level, multi-disciplinary, and multi-field integrated system. Obtaining information about GEO work development and progress via traditional methods is extremely challenging. For this reason, with the support of the Earth observation satellite database operated by the WMO (World Meteorological Organization), CEOS (Committee on Earth Observation Satellites), GEO library, GEOSS literature, and related documents of Earth observation conferences, this paper presented the GEOSS crowdsourced big data acquisition and management system integrated a suite of crowdsourced big data acquisition technology. The system allows us to obtain text messages, pictures, reports, audio and video, and scientific articles related to Earth observation. In this paper, the system structure and development was reported and the system may provide effective support for the strategic evaluation of GEOSS progress.

Keywords: GEOSS; big data; crowdsourced observation; GEOSS progress evaluation; GEO

DOI: https://doi.org/10.3974/geodp.2024.04.03

CSTR: https://cstr.escience.org.cn/CSTR:20146.14.2024.04.03

1 Introduction

The Group on Earth Observations (GEO), an intergovernmental international organization, was established in 2005^[1,2] after three Earth observation Ministerial Summits held in 2003, 2004 and 2005, respectively. The organization was founded in response to an urgent request for coordinated observations of the state of the earth at the 2002 World Summit on Sustainable Development in Johannesburg, South Africa, and the declaration of the G8 Summit held in France in 2003, which stated that Earth observation should be an important and priority action. With the mission to support informed decisions through integrated, coordinated, and sustainable earth observations, enhance understanding of the earth system, and provide information services for decision-making. GEO primarily aims to develop a Global Earth Observation System of Systems (GEOSS).

To achieve its objective, the organization designed two strategic implementation plans for the GEOSS spanning 20 years, from 2005 to 2025. The first 10-year strategic implementation plan (2006?C2015) was formulated in the early days of GEO. The project was a     development framework that identified 9 Societal Benefit Areas (SBA)^[1] such as disasters, health, energy, climate, water resources, weather, ecosystems, agriculture and biodiversity. Following the conclusion of the first plan, the second GEOSS strategic plan (2016?C2025) was launched at the fourth GEO Ministerial Summit held in Mexico City in November 2015. This project selected 8 SBAs to work in. These are biodiversity, ecosystem management, disaster prevention and mitigation, energy and mineral resource management, food security and sustainable agriculture, infrastructure, and transport system management, public health monitoring, sustainable urban development, and water resources management. Furthermore, GEO has identified the United Nations 2030 Agenda for Sustainable Development, the Paris Agreement on Climate Change, and the Sendai Framework for Disaster Reduction as its top three working priorities.

The GEO work plan (2016?C2025) is an important working document to promote the implementation of the GEO new decade strategic plan^[3] where the GEO puts profound importance on executing the GEO work plan and advancing the development of GEOSS. In this regard, many Chinese scholars have played notable roles in contributing to the success of the organization??s plan in several ways. Chinese researchers have constantly been revising the international development trend^[4?C8] and carrying out follow-up research studies evaluating and commenting on GEOSS^[9] data distribution system, Asia and Oceania regional Earth observation system plan^[10], China??s use of foreign facilities^[11] to build Earth observation system, applications in the field of social benefit^[12,13], and scientific data sharing policies^[14?C16].

However, GEOSS is a complex multi-level, multi-disciplinary, and multi-field comprehensive system^[17]. It involves Earth observation systems in developed countries and developing countries^[18?C20], hardware systems and software systems, and even regional, language, human geography and other issues, both public welfare and commercial. Consequently, it is extremely daunting task to obtain information on the implementation and progress of GEOSS using the traditional approaches^[21?C25].

This paper, therefore, discusses some new ideas and techniques, especially the use of crowdsourced big data technology to collect Earth observation progress information to support GEOSS evaluation.

2 System Structure and Function

GEOSS big data acquisition and management system focuses on the main idea of ??building environment, gathering data, developing applications, and providing service support??,   creating an application evaluation system based on GEOSS application data, deeply mining the value of data, and improving the evaluation ability of GEOSS application progress. The system comprises 7 parts: infrastructure layer, data resource layer, application service  support layer, business application layer, user interaction layer, security guarantee system, and standard specification system. The overall framework and topological structure of the system are shown in Figure 1 and Figure 2, respectively. The infrastructure layer includes network and basic hardware, which is used to support the stable operation of various application systems. The network environment of the system mainly relies on cloud servers. Basic hardware includes servers, network and storage devices, and others.

As an information resource center, the data resource layer provides comprehensive data services to ensure system operation. Data is stored in a structured and distributed manner. The stored data includes spatial database data, information materials, conference information, interview materials, and other relevant materials.

The application service support layer includes, data engine, workflow engine, interface service, model base construction technology, web crawler technology, and so on.

Under the support of the security system, the business application layer carries out  business applications on the basis of infrastructure, database, shared components, and data center, including data management, evaluation model management, and GEOSS application progress evaluation.

Figure 1  The overall framework of the system

The user interaction layer is the interface between the system and users, where users can process, generate, and obtain the required information.

The security guarantee system is the legal basis and institutional guarantee for the normal operation of the system, including laws, regulations and standards, as well as information security, system security organization and management system.

The standard and specification system comprised of the unified national standards and specifications, the technical specifications adopted, and the unified format of data that are followed and implemented during system development and application.

Figure 2  The topological structure of the system

The function of GEOSS big data acquisition and management system is to integrate CEOS/ESA MIM spatial database, WMO OSCAR (Observing Systems Capability Analysis and Review Tool) spatial database, GEO Library and crowdsourced big data by information technology to obtain the information and materials related to GEOSS. The information includes text messages, pictures, reports, audio and video, scientific paper data. All the information was acquired and managed by the Earth Observation Big Data Platform.

3 System Design

GEOSS big data acquisition and management system adopts a component-based design idea and B/S structure based on the general web browser specifications. It is compatible with Google browser and 360 browser by default. The system provides functional modules for sustainable loading and maintenance services and can continuously improve and expand system functions.

The data of the GEOSS big data acquisition and management system mainly includes CEOS/ESA MIM spatial database, WMO OSCAR spatial database, GEO library information materials, earth observing-related text messages, pictures, reports, audio and video, scientific paper materials. These data can be obtained from various sources including GEO work plan symposia, GEO plenary and Earth Observation Ministerial Summits, AOGEO, EUROGEO, AmeriGEO, AfriGEO, CEOS, the China-Europe Dragon Program, SPIE Earth Observation satellite session, IGARSS Earth Observation satellite session and other Earth observation conference. For data storage, MySQL database is used to store structured data, whereas, HBase database is employed for unstructured data.

3.1 Data Architecture Design

The data storage function is the core and foundation of the platform to provide external services. In the development of software systems, data storage design should follow the necessary design principles and theories so as to reduce data redundancy and ensure the integrity and correctness of data. The design scheme is directly related to the efficiency of system execution and the stability of the system. In order to construct a solid, reliable, and high-performance information service system, the data storage design follows the following basic principles:

(1) The principle of centrality

The system realized system centralized, data centralized, processing centralized under the unified design, and following the unified standards in the development and application.

(2) The principle of advanced and mature

The system chooses a certain advanced representative level and relatively mature technology to build an information service system by adopting the latest and most common hardware platform and database engine. In addition, it manages the data specification and authority of the system through the management and maintenance of the database, ensures the stability and maturity of the system, and maintains a certain advancement.

(3) Reliability and availability principle

The system fully considers the capacity of strain, fault tolerance, and error correction, and adopts highly reliable technology for development to ensure the stable operation and reliable safety of the system.

(4) The principle of benefit and practicability

The design and development of this system comprehensively consider the economic and social benefits of the system.

(5) The principle of prospectivity and expansibility

The system design possesses innovative features to ensure that the system is still advanced and stable in a long period of time. Additionally, the system has good expandability and upgradeability, which allow seamless upgrades to a new generation of equipment and technology platform.

(6) The principle of security and confidentiality

The security and confidentiality of the system focuses on the design and development of equipment security, network security, and data security from multiple angles to ensure the security of business information.

(7) Standard and normative principles

The database construction of the system adheres to the relevant national standards and norms, in which data stratification, classification and coding, accuracy, symbols and other references to the existing relevant national standards were considered.

(8) Overall accuracy principle

The database content design of this system is as comprehensive as possible to ensure that the type and length of the fields in the database can meet the needs of business applications at the current as well as in the future.

(9) Principle of loose coupling

Each subsystem of the system follows the principle of loose coupling, that is, no mandatory constraint relationship is set between each subsystem. The connection between subsystems is established by re-input, query, program default filling, and so on. And, the associated fields between subsystems are redundant storage.

The database stores text messages, pictures, reports, audio and video, and other data by accessing other business data platform through data interface. The preprocessing includes data decoding, data combination, and rearrangement, format conversion, and so on.

The various types of data were accessed from different websites, as shown in Table 1, and the list of data along with the associated data sources, types, and database is illustrated in Table 2.

Table 1  List of data sources

Table 2  List of data sources and data types

3.2 Interface Design

The interface design generally follows the design principle of high cohesion and low coupling, which is also an important principle to implement in the process of software design. The procedure helps to reduce the coupling degree between the systems and the modules within the system. In addition, it lessens the complexity of the operation, ensures the universality of the system, improves the system reusability and scalability.

The specific principles are as follows:

(1) Main principles

1) All the interface design follows the project construction regulations and interface specifications;

2) Consider the component-based design idea of SOA to achieve loose coupling between systems.

(2) Other principles

1) Easy to use, fast, versatility, high reliability;

2) Full consideration of the application expansion for the various system interfaces, and flexibly support of the changing demands;

3) The consistency of the interface data among various systems;

4) The confirmed process after the transmission and the reception in the process of data interaction.

Interface implementation methods include messages, API, and shared directories.

In this system, the message queue mode is mainly used for the transmission of monitoring information between the monitoring system, and the transmission of job scheduling instructions between each system. Its design incorporates several functions such as message   encapsulation, automatic routing, reliable message conversion and message transmission. It also has the function of encryption and fault tolerance for special messages.

In order to maintain the compatibility of interfaces between multiple systems, it is necessary to provide a variety of API for other subsystems to obtain data from the main system. The API interface design has the following attributes: independent package of logical processing function interface, convenient integration with front-end C, JAVA, PYTHON, and other programs, API management function, high reliability and efficiency of connection with the server, and complete logging function. Besides, it has the function of configurable connection parameters with the server.

The shared directory is employed to exchange file data within the system. The shared directory interface is designed as follows: interactive file storage directories in a unified manner, efficient and reliable directory management strategy, configurable directory of read and write permissions control, reliable trigger mechanism for data file arrival and processing, identification of completed or incomplete status for data file copy, identification of being processed or unprocessed status for data file.

The shared directory interface mainly realizes the interaction of data files between systems by means of shared read-write storage. When files are read or written, it is realized through the copy mechanism of directory files.

3.3 Performance and Non-functional Design

3.3.1 Performance Optimization Design

The performance of this system is mainly reflected in timeliness, stability, easy operation, and scalability. In order to ensure the realization of performance indicators, it is necessary to optimize the design of database servers, database access, application programs, and others.

3.3.2 Optimization Design of Database Server

Data Block: the data file of the database is stored in blocks of appropriate size to obtain the best data storage space and efficient access.

Parallel Processing: Data query and insert, modify, delete and other operations make full use of the parallel processing capability of the database.

Raw Device: the raw device is used as the data storage medium (database) to improve the performance of the database with frequent read or write operations.

3.3.3 Database Optimization Design

Sub-database: Based on the distributed database principle, the sub-database design ensures that the data is evenly distributed on each underlying database instance, and can be efficiently queried and accessed.

Field Index: In considering the requirements of data storage and application, the appropriate data field index is designed.

3.3.4 Application Optimization Design

Cache: Parameter tables or metadata repeatedly used are cached in the memory to improve data application efficiency.

Paging: The system query results are displayed in a reasonable paging mode to avoid large result sets, and effectively improve the system response speed and performance.

Parallel processing: It makes full use of the computing power of multi-core CPU by the parallel processing of dense data to improve the efficiency of system operation.

3.3.5 Reliability Design

In order to ensure the reliable and stable operation of the system, the following strategies are adopted in the design scheme.

(1) High availability policy: Adopt the high availability policy, the system services are deployed on the primary and secondary servers. The primary and secondary services detect the system availability through heartbeat monitoring, and find the system crash phenomenon in time, so as to ensure that the whole system can still run normally without crash or data loss in case that a server fails.

(2) High efficiency strategy: With the continuous richness of data acquisition, the requirements for system service capability, data access capability (IO capability), and computing power will become higher and higher. Therefore, the use of the current more advanced big data technology, such as distributed database architecture, parallel computing, parallel scheduling, distributed computing, efficient shared file storage, and other technical means, ensure that the system runs efficiently under the pressure of large data volume processing.

(3) Software complexity control: The system has many modules and complex logic, in order to reduce the sensitivity of software defects to the input environment and decrease the probability of software failures. The object-oriented design method is adopted in software design, and the length and logic complexity of the program is controlled. Consequently, the software has the characteristics of encapsulation, abstraction, and inheritance, the object is relatively independent, and the internal elements of the object are closely related, thus forming a ??high cohesion, low coupling?? software system.

The error rate estimate range of the program is generally 0.04%?C7%, for less than 100 statements of the program. The number of lines of the source code and the error rate are linearly related. The error rate increases in a non-linear way with the increase of the program statements. In this software package, the length of code for the method design in the object is controlled to an average of about 60 lines.

The complexity of the program flow is controlled. The system analyzes the number of program branches and the number of cycles. If the complexity is more than 10 programs, it decomposes into smaller programs to reduce the program error rate.

(4) Error correction mechanism: This system sets the system running log to record the key steps and abnormal information in the system operation. The system running log with its extension of ??.log?? is a collection of log records arranged in chronological order, in the form of a text file composed of lines of text. Once a system failure occurs, such as a system crash, one can query the system running log to find out the exact reason for the system failure. In the design of the decision scheme evaluation and optimization subsystem, JAVA exception handling mechanism is applied to all kinds of system running exceptions, such as null pointer errors.

3.4 Design of Data Acquisition Subsystem

Data acquisition subsystem has GEO application data collection, processing, classification, archiving, warehousing and other functions. The data acquisition and entry subsystem is composed of automatic data acquisition module, data entry module and data archiving module.

According to the data type listed in Table 2, the automatic data acquisition module realizes the collection of various types of business data required by the project.

The data entry module realizes the manual entry of relevant data, and stores the data in the database after the data verification.

(1) The collected data will be scientifically classified, and the data entry verification standard will be set and recorded by the data collector.

(2) For text data, the file is saved in .docx format after manual input.

(3) For table data, saved in .xlsx format in order to query and analysis.

(4) For photos or multimedia data, saved separately in .jpg format or general streaming media format.

The data archiving module is meticulously designed to achieve categorized data and search through metadata, intermediate data, business data, mining data, and other data, and to improve the accuracy and efficiency of the user retrieval, invocation, and mining of the data. As a result, users can automatically classify the data by presetting classification rules, or manually define new classification rules to categorize the data. Finally, the comprehensive interface of the system is developed as the Figure 3.

Figure 3  System interface

4 Conclusion

The GEOSS is a complex multi-layered, multi-disciplinary, and multi-field integrated system. What progress has GEOSS made in its second decade of development, and how can it be objectively assessed? These are the critical questions of the whole earth observation community. Based on the documents of relevant Earth observation conferences, the authors used crowdsourced data acquisition technology to retrieve information related to GEOSS development progress, including text message, picture, report, audio and video information, and scientific paper. After evaluating the existing systems, an advanced GEOSS big data acquisition and management system was developed to play a unique supporting role for the evaluation of GEOSS??s strategic progress.

Author Contributions

Fan, J. L. and Liu, Y. H. made the overall design of the system; Zhu, J., Liu, C. and Wu, J. J. put forward optimization suggestions for the system design; Fan, J. L., Zhu, J. and Adugna, T. wrote the paper.

Conflicts of Interest

The authors declare no conflicts of interest.

References

[1]     Feng, J., Gao, F., Huang, X. Y. Constructing an integrated global Earth observation system: three international Earth observation summits and GEOSS [J]. Advances in Earth Science, 2005, 20(12): 7. DOI: 10.3321/j.issn:1001-8166.2005.12.008.

[2]     Xiao, Y. The world will strengthen cooperation in Earth observation [J]. Space International, 2005(4): 1. DOI: CNKI:SUN:GJTK.0.2005-04- 003.

[3]     Fan, J. L., Liu, C., Wu, J. J., et al. Analysis on the characteristics of the eighth phase plan (2023?C2025) of the Group on Earth Observation (GEO) [J]. Journal of Global Change Data & Discovery, 2023,7(3): 233‒241. https://doi.org/10.3974/geodp.2023.03.01. https://cstr.escience.org.cn/CSTR:20146.14.2023.03.01.

[4]     Li, M. X. Earth observation decade execution plan and enlightenment [J]. Science & Technology Industry of China, 2007(7): 4. DOI: 10.3969/j.issn.1002-0608.2007.07.018.

[5]     Li, M. X., Zhang, S. M. Earth observation field international science and technology cooperation status,trends and enlightenment [J]. Remote Sensing Information, 2009(2): 4. DOI: 10.3969/j.issn.1000-3177.2009.02.022.

[6]     Zhang, D. F. The global Earth observation system of portal [D]. Beijing: Beijing Forestry University, 2016. DOI: CNKI:CDMD:2.1016.145077.

[7]     Zhao, L. M., Miao, C., Xing, J. et al. Earth observation knowledge hub: connotation, key technology and prospect [J]. National Remote Sensing Bulletin, 2024, 28(6): 1375?C1394. DOI: 10.11834/jrs.20222302.

[8]     Li, G. Q, Zhang, H. Y, Zhang, L. C., et al. Tendency of the development of Earth observation data sharing [J]. National Remote Sensing Bulletin, 2016, 20(5): 12. DOI: 10.11834/jrs.20166173.

[9]     Fan, J. L. Earth observation satellite data distribution system development review [J]. Advances in Earth Science, 2012, 27(7): 712‒716. DOI: 10.13769/j.cnki.cn50-1011/d.2014.07.024.

[10]  Gu, X. F., Niu, Z., Yu, T. et al. Asia Pacific regional integrated Earth observation system plan progress [J]. National Remote Sensing Bulletin, 2018, 22(4):14. DOI: 10.11834/JRS.20188065.

[11]  Liu, C., Wen, H. T., Zhao, L. C. et al. EOS MODIS in our country present situation, problems and countermeasures of construction of ground station [J]. Remote Sensing Information, 2003(4): 42‒47. DOI: 10.3969/j.issn.1000-3177.2003.04.011.

[12]  Fan, J. L., Zhang, M. W., Cao, G. Z., et al. Global satellite monitoring drought plan [J]. Advances in Meteorological Science and Technology, 2014, 4(5): 4. DOI: 10.3969/j.issn.2095-1973.2014.05.006.

[13]  Li, S. J. International cooperation mechanism for space disaster reduction (IV) introduction of Group on Earth Observations (GEO) mechanism [J]. Disaster Reduction in China, 2018, 330(15): 52?C53. DOI: CNKI: SUN:ZGJI.0.2018-15-018.

[14]  Sun, S., Zhang, X. E., Guo, Z. Y. et al. The scientific data sharing policy report [J]. China Basic Science, 2002(5): 3. DOI: 10.3969/j.issn.1009-2412.2002.05.009.

[15]  Jing, G, F. Open science to analysis the influence of the global integrated construction of Earth observation system [J]. Remote Sensing for Land and Resources, 2020, 32(4): 7. DOI: 10.6046/gtzyyg.2020.04.01.

[16]     Chu, W. B., Uhlir, P. F. Summary of symposium on data sharing, data management solutions and best cases in Group on Earth Observations (GEO) [J]. Journal of Global Change Data & Discovery, 2017, 1(4): 486?C489. https://doi.org/10.3974/geodp.2017.04.21.

[17]  Ma, C. L., Gao, Y. B. System integration of the global Earth observation system and the global ocean observation system [J]. Ocean Technology, 2006(3): 41?C44+50. DOI: 10.3969/j.issn.1003-2029.2006.03.010.

[18]  Yang, F. The European Union to strengthen the construction of ability of Earth observation [J]. Chinese Journal of Space Science, 2018, 38(1): 1. DOI: CNKI:SUN:KJKB.0.2018-01-009.

[19]  Li, M. X., Townshend, J. R., Wu, B. F. China??s contribution to the global Earth observation system [J]. National Remote Sensing Bulletin, 2010(3): 8. DOI:10.3724/SP.J.1011.2010.01138.

[20]  Miao, C., Liu, Z. C., Zhang, J., et al. The composition and practice of China??s integrated Earth Observation System [J]. Remote Sensing Information, 2024, 39(2):12‒18.

[21]  Sun, K., Chen, Y. W., Li, J. F., et al. Earth observation satellite system performance evaluation index system research [C]. The harmonious development and systems engineering, systems engineering society of China the 15th annual conference proceedings, 2008. DOI: ConferenceArticle/5aa0852fc095d722207612c6.

[22]  Sun, K., Chen, Y. W., Li, J. F., et al. Earth observation satellite system performance index and its application simulation study [J]. Computer Simulation, 2009(4): 4. DOI: 10.3969/j.issn.1006-9348.2009.04.015.

[23]  Xiang, L., Meng, X., Zhang, X. C., et al. Earth observation satellite system effectiveness evaluation based on HLA study [J]. Computer Simulation, 2010(7): 5. DOI: 10.3969/j.issn.1006-9348.2010.07.008.

[24]  He, D. L., Li, G. P., Hou, Y. K. Space-based Earth observation system effectiveness evaluation study [J]. Chinese Space Science and Technology, 2014, 34(1): 8. DOI: 10.3780/j.issn.1000-758X.2014.01.003.

[25]  Sun, W. W., Yang, G., Chen, C., et al. China??s current situation of the development of Earth observation satellite remote sensing and literature analysis [J]. National Remote Sensing Bulletin, 2020, 24(5): 32. DOI: CNKI:SUN:YGXB.0.2020-05-001.