SAMRAI has been used in a variety of SAMR application development and algorithm research efforts at LLNL and other institutions. This page describes some efforts currently underway. More details about these codes and algorithms may be found in our Publications.

Application Development at LLNL

ALE-AMR

The ALE-AMR project is a research effort in CASC that uses SAMRAI to explore algorithms that couple ALE and AMR methods. ALE (Arbitrary Lagrangian-Eulerian) methods are used widely at LLNL for many shock hydrodynamics problems. The goal of this effort is to combine the best features of ALE and AMR methods to enable more detailed three-dimensional simulations of problems important to the Laboratory, such as detonation fronts with detailed chemistry.

The CASC ALE-AMR effort is the principal driver of programmatic interest in SAMRAI at LLNL. Several efforts to add SAMRAI AMR capabilities to existing codes are underway at LLNL.

Figure 1. An example ALE-AMR multi-material calculation. The boxes with black borders represent the finest grid patches on the moving, deforming ALE mesh.
Figure 1. An example ALE-AMR multi-material calculation. The boxes with black borders represent the finest grid patches on the moving, deforming ALE mesh.

Multiscale Materials and Fragmentation

A three-dimensional parallel computer code, NIF-ALE-AMR, based on the ALE-AMR effort is being developed to perform advanced modeling of different targets for the National Ignition Facility. The AMR code, built on SAMRAI, provides a true multiscale simulation that allows for different physical models on different scales. NIF-ALE-AMR uses a variety of material models to model fragmentation and dismantling of NIF target elements following the intense laser pulse. The three-dimensional moving adaptive mesh allows us to track particles and wave fronts in the flow and predict the effect on NIF's debris shields and diagnostic tools. The code is developed in the Center for Applied Scientific Computing at the Lawrence Livermore National Laboratory in collaboration with researchers from the University of California, San Diego, and the Chemistry, Material and Life Sciences directorate at LLNL. NIF-ALE-AMR is part of the NIF facility modeling group suite of codes.

Electronic Structures LDRD Project

Quantum simulations of finite size systems such as molecules or atomic clusters, in vacuum or surrounded by a continuum solvent model, can benefit from AMR. A fine mesh is used close to the atoms, where the quantum electronic wave functions vary rapidly. A coarser mesh can be used away from the system to represent the smooth decay of the wave functions and electrostatic potential. We have developed a quadratic Finite Element approach to discretize the Kohn-Sham equations (Density Functional Theory) on structured non-uniform meshes. A multigrid FAC preconditioner is used to iteratively solve the equations by an accelerated steepest descent scheme. The method is implemented using SAMRAI.

Figure 2. An electronic structure calculation of Be4 cluster on a composite AMR mesh. The two structured mesh levels and their decomposition in 8 patches are shown. An isosurface of the electronic density is plotted, surrounding the 4 Be atoms represente
Figure 2. An electronic structure calculation of Be4 cluster on a composite AMR mesh. The two structured mesh levels and their decomposition in 8 patches are shown. An isosurface of the electronic density is plotted, surrounding the 4 Be atoms represented by spheres.

Phase Field Modeling LDRD Project

A new LDRD project is underway at LLNL to develop realistic, microstructure-aware constitutive models that apply to phase transitions of interest. Specifically, the goal is to extend highly successful molecular dynamics simulations of nucleated microstructure to hydrodynamic length- and time-scales. To accomplish this, a new continuum Phase Field Modeling (PFM) capability is being developed that couples to models that include deformation. Of particular interest is the solidification of metals and alloys under rapid compressive deformation and phase transformations during shock loading. In such problems, changes in thermodynamic state (P, V, T) occur on time-scales comparable to the kinetics of the transformation (nanoseconds to microseconds). The goal is that by deriving a phase field energy functional directly from molecular dynamics simulations, we will obtain the first reliable calculation of microstructure and subsequent material response under the conditions of interest to LLNL. This PFM with deformation capability will employ a real-space representation of the time-dependent Ginzburg-Landau equations in a scalable simulation code using AMR based on SAMRAI.

Application Development at Other Institutions

Cardiac Simulation

Boyce Griffith, Charles Peskin, and David McQueen of the Courant Institute of Mathematical Sciences at New York University are using SAMRAI to develop parallel AMR computational methods for cardiac physiology simulation. They aim to perform detailed three-dimensional simulations of cardiac mechanics (including blood, muscle, and valve mechanics) coupled to cardiac electrophysiology. The theoretical and numerical framework for coupling both aspects of heart function within a single model is based on the immersed boundary (IB) method. The IB method was pioneered by Peskin as an approach to fluid-structure interaction problems, such as cardiac fluid mechanics. Peskin and Griffith have recently extended it to also describe cardiac electrophysiology.

The implementation of the adaptive IB method for problems of fluid structure interaction is built on SAMRAI. This code called IBAMR is publicly available.

The IBAMR software is also being used in other research at NYU and elsewhere. These efforts include:

  • Ph.D. student Pilhwa Lee is using IBAMR to develop adaptive numerical methods for the advection-diffusion of solutes in fluid-solute-structure interaction. Such methods could be used to perform detailed simulations of neural or cardiac electrophysiology.
  • Postdoctoral researcher Yongsam Kim is exploring the penalty IB method for simulating elastic boundaries with mass. Kim is using IBAMR to develop adaptive simulations of a flapping filament in a soap film and simulations of the dynamics of parachutes, windsocks, and flags.
  • Aaron Fogelson, Professor of Mathematics and Adjunct Professor of Bioengineering at the University of Utah, is developing IBAMR-based immersed boundary simulations of the formation of blood clots under flow.
  • Xiaoyu Luo, Reader in the Department of Mathematics University of Glasgow, along with Postdoctoral Research Fellow Chunlei Liang and Ph.D. student Min Yin, are using IBAMR to perform adaptive immersed boundary simulations to improve the design of a prosthetic two-leaflet mitral valve using dynamic analysis.
  • John Stockie, Associate Professor of Mathematics at Simon Fraser University, along with M.Sc. student Badri Ratnam, are using IBAMR to investigate the flow of suspensions of flexible fibers using the immersed boundary method.
Figure 3. Diastolic filling of the right ventricle (blue markers) in a three-dimensional AMR simulation of cardiac fluid dynamics in the human heart.
Figure 3. Diastolic filling of the right ventricle (blue markers) in a three-dimensional AMR simulation of cardiac fluid dynamics in the human heart.

Implicit AMR Methods

Bobby Philip and Luis Chacon of Los Alamos National Laboratory and Michael Pernice of Idaho National Laboratory are developing discretization schemes and solvers for radiation diffusion and magneto-hydrodynamic problems on SAMR meshes. The solver capabilities and interfaces to other solver libraries in SAMRAI have largely resulted from this collaboration. For more information, contact Bobby Philip.

Lagrangian-Eulerian Hydrodynamics

The CleverLeaf mini-application is using SAMRAI to create a lightweight Eulerian hydrodynamics application. Developed at the University of Warwick and AWE, and released as part of Mantevo mini-app collection, CleverLeaf is being used as a proxy to investigate the performance of AMR for structured Lagrangian-Eulerian hydrodynamics at scale.

Figure 4. An example CleverLeaf calculation, showing two interacting shock waves after multiple reflections at the domain boundaries. The black boxes show the finest level patches.
Figure 4. An example CleverLeaf calculation, showing two interacting shock waves after multiple reflections at the domain boundaries. The black boxes show the finest level patches.

Firespread Modeling

The Combustion and Reactions Simulation (CRSIM) group at the University of Utah ASCI Center (CSAFE) is using SAMRAI to develop adaptive algorithms to model the effects of fire on containers holding high explosives. This problem of fire simulation spans length scales ranging from about 1 km to about 10^-6 m, and also a large range of associated time scales. This variation in length and time scales cannot be resolved using a single grid for the entire problem. While high resolution is needed near the base and in the center of the fire, the edges of the fire can be coarsely resolved.

Dislocation Dynamics

Kevin Chu of Princeton University is using SAMRAI to develop a parallel (and eventually adaptive) dislocation dynamics simulation framework based on a level set representation of dislocation lines. This framework is being used to study mechanisms of dislocation interactions in bulk materials and thin-films and the structure and motion of low-angle grain boundaries. This work involves a collaboration with David Srolovitz (Yeshiva University) and others.

Simulation of Subsurface Contaminant Transport

A new DOE Office of Science effort to develop simulation tools for modeling movement of subsurface contaminants will use SAMRAI.

Helicopter Wake Dynamics

The DoD HPC Institute for Advanced Rotorcraft Modeling and Simulation (HI-ARMS) at the Army's Aeroflightdynamics Directorate at NASA Ames Research Center, Moffett Field, CA is using SAMRAI to develop simulation capabilities that will predict wake dynamics of helicopters. The new CFD code interfaces NASA's ARC3D CFD code with SAMRAI for parallel adaptive gridding. This high-order Cartesian AMR code is further being coupled to the unstructured NSU3D turbulent Navier-Stokes CFD solver to capture near-blade flow physics using an overset approach. The combination of these two codes - NSU3D to capture the turbulent unsteady physics near the blade surface and SAMRAI-ARC3D to capture the far-field tip vortices - provides a powerful tool to predict helicopter vibrational loading, noise, and accident scenarios. For more information, contact Andy Wissink.

Figure 4. An example calculation illustrates the use of AMR to resolve tip vortices of an isolated V-22 rotor.
Figure 5. An example calculation illustrates the use of AMR to resolve tip vortices of an isolated V-22 rotor.