Development of Parallel Subsurface Multiphase Flow Simulation Capability, 20-R8087
Principal Investigators
Hakan Basagaoglu
Will L. Arensman
Stephen W. Cook
Inclusive Dates: 01/01/09 – 01/01/11
Background — Multiphase subsurface flows are important to radioactive waste repository assessments. SwRI researchers have acquired considerable experience in multiphase subsurface flow analyses, which often require computationally demanding simulations. Prior to this project, researchers developed an in-house numerical code, xFlo, for continuum-scale simulations of non-isothermal, multiphase flows in deep geological fractured-porous domains. Multiphase subsurface flows are also of interest in a variety of other applications including geological sequestration of carbon dioxide, geothermal energy production, compressed air energy storage in aquifers, groundwater recharge and contamination assessments, bioremediation, studies of permafrost and investigations of Mars and other solid bodies such as the moons of Jupiter and Saturn. Parallelization of xFlo, based on multicore computers arranged in clusters, was needed to take advantage of existing computing hardware. Hence, the main objective of the project was to parallelize xFlo for computationally efficient multiphase subsurface flow simulations that could be used in diverse applications.
Approach — Parallelization of xFlo is complicated by the need for fully implicit timestepping in the solution algorithm and the use of unstructured grids in model step-up. Researchers proposed using domain decomposition parallelization implemented with one-sided message passing interface (MPI) communication, which was added to the serial version of the xFlo code. In this approach, a preprocessor was used to decompose the data set into smaller, yet similarly constructed, data sets. Each smaller data set (associated with different subdomains) was assigned to different processors. With this approach, the original xFlo source was largely kept the same, with code introduced at key points to perform ghost cell exchanges between processors. Updates of the ghost cells (which collectively form a subdomain halo where information exchange between processors occurs) were delayed until the end of each timestep. Ghost cells were used in domain decomposition to ensure flux continuity between adjacent subdomains. By delaying halo updates until the end of a timestep, interdomain flux information was not communicated during Jacobian matrix construction.
Accomplishments — A newly proposed parallelization approach based on the domain decomposition method via one-sided MPI communication was successfully implemented in the xFlo code. In demonstrative one-dimensional simulations, the proposed parallelization approach resulted in 1.79 to 2.37 times speed-up on four processing cores, indicating that the proposed approach is computationally efficient. External coupling of the parallelized xFlo code with the existing and widely used standalone geomechanical (e.g., FLAC3D) and/or geochemical (e.g., Geochemist's Workbench) software was proposed to the U.S. Nuclear Regulatory Commission (NRC) as part of business development. The NRC agreed that coupled models would offer a new computational tool simulating coupled thermal, hydrological, mechanical, and/or chemical processes that the NRC staff can use in their independent analyses for evaluating the performance of proposed deep-geological formations for high-level waste disposal. In fiscal year 2012, development of coupled models will be conducted under the NRC Integrated Spent Nuclear Fuel Regulatory Activities program.
