The laboratory owns an HPC cluster dedicated to numerical simulations. The cluster was designed and it is maintained by the laboratory members in collaboration with the department of informatics of the University of Pavia. This cluster enters in the framework of the iCardioCloud project.
The domain of expertises, using both academic and commercial codes, are:
- Computational Fluid Dynamics (CFD) for biomedical applications using the finite elements library LifeV, in collaboration with Alessandro Veneziani (Emory University);
- Isogeometric analysis using FEAP, in collaboration with Robert L. Taylor (UC Berkeley), and LS-DYNA;
- Non-linear materials using FEAP, with a focus on Shape Alloyed Memory materials.
Particular care is taken by the staff members to improve the parallel performance of the mathematical software on the cluster, as most lab application are time critical and would benefit from such optimizations. One on-going research activity is to modify the LIFEV suite to implement hybrid parallelism, and to make the resulting software available to a broad scientific audience.
The cluster is composed of 4 computational nodes combining 256 cores and 1Tera of RAM. In addition, a login node with 32 cores and 252Giga of RAM is also used for development purposes. A third dedicated server with 24 cores and 66Giga of RAM is used for FEAP.
In detail
- 1 Master node Dell PowerEdge R820 (Tomato):
- Cpus: 4x Intel Xeon E5-4620 @ 2.20 GHz:
- 8 cores per cpu
- Total number of cores: 32
- Ram: 252Gb
- Total disk capacity: 10Tb
- Cpus: 4x Intel Xeon E5-4620 @ 2.20 GHz:
- 1 Computing node Dell PowerEdge R740:
- Cpus: 2x Intel Xeon Gold 6248 @ 2.50 GHz:
- 20 cores per cpu
- Total number of cores: 40
- Ram: 252Gb
- Total disk capacity: 4Tb
- Cpus: 2x Intel Xeon Gold 6248 @ 2.50 GHz:
- 4 Computing nodes Dell PowerEdge R815:
- Cpus: 4x AMD Opteron 6272 @ 2.1GHz:
- 16 cores per cpu
- Total number of cores per node: 64
- Ram: 252Gb
- Total disk capacity: 292Gb
- Cpus: 4x AMD Opteron 6272 @ 2.1GHz:
- 1 Gigabit Switch that manages the local network of the 4 computing nodes
Software set-up
-
Debian GNU/Linux 7.5 (Wheezy).
-
NFS: Network File System used to share a directory of the Master node to all others computing nodes. This directory provides data, binaries and a common workspace for the cluster.
-
Mpi: message-passing interface used for parallel applications within the cluster. Two versions are used: mpich and openmpi.
-
Torque: distributed resource manager used to manage batch jobs over the computing nodes.
-
Trilinos: open source software libraries for scientific applications.
-
Octave: A matrix laboratory.
-
VMTK: the Vascular Modeling Toolkit.
This HPC cluster is dedicated to ANSYS ™ CFD and Mechanical. It includes two machines:
- Lenovo P520 with Xeon W-2175 (2.5GHz, 4.3GHz Turbo, 14C, 19MB Cache (140W)) and 256 GB DDR4 2666MHz
- Dell Precision Tower 5820 with an Intel Xeon W-2123 (3.6GHz, 3.9GHz Turbo, 4C, 8.25MB Cache, HT, (120W)) DDR4-2666) and 192 GB DDR4 2666MHz RDIMM ECC.
This cluster allows us to have 18 physical cores, 36 threads and a memory of 448 GB available for complex CFD, structural, and fluid-structure interaction analyses. Job’s queues are managed using ANSYS ™ Remote Service Manager (RSM). Moreover, this cluster is widely employed to run topology optimization of 3D printed components and perform Laser Powder Bed Fusion (LPBF) process simulations of critical components produced by ProtoLab, our 3D printing Lab.
This is a standalone Dell R740 machine equipped with 2xIntel Xeon Gold 6248 (2.5GHz, 3.90GHz Turbo, 20C, 27.5MB Cache (150W)), 8x64Gb RDIMM. 2933MT/s, 2x960Gb SSD e 1×1.8Tb HHD. This node is dedicated to vascular simulations using ABAQUS ™. In particular, nonlinear, contact analysis of stent implants are performed.
References
- A. Finotello, S. Morganti, F. Auricchio. Finite element analysis of TAVI: Impact of native aortic root computational modeling strategies on simulation outcomes, Medical Engineering and Physics, vol. 47 (SI), pp. 2-12 (2017)
- C. Spadaccio, L. Mazzocchi, I. Timofeva, L. Macron, C.N. De Cecco, S. Morganti, F. Auricchio, F. Nappi. Bioengineering Case Study to Evaluate Complications of Adverse Anatomy of Aortic Root in Transcatheter Aortic Valve Replacement: Combining Biomechanical Modelling with CT Imaging, Bioengineering, vol. 7 (4), pp. 1-10 (2020)
Wiki:
List of tutorial presentations for a correct use of the server and how to access it using Windows and Linux OS.
- CompMech server architecture presentation: Overview of the server architecture and some useful Windows applications to utilize the server from a Windows machine.
- Tutorial batch file: A basic example of batch file to run simulations on our cluster.
- Bash on Win10: Short presentation for a quick Ubuntu installation on Windows 10 (not a virtual machine!) to access the cluster via ssh.
- HPC@ProtoLab slides: Tutorial slides for BigBlackMama cluster@ProtoLab.
Lab Members:
- Ferdinando Auricchio, Eng., PhD, Prof. (UniPV)
- Alessandro Reali, Eng., PhD, Prof. (UniPV)
- Michele Conti, Eng., PhD, Prof. (UniPV)
- Simone Morganti, Eng., PhD, Prof. (UniPV)
- Massimo Carraturo, PhD (UniPV)
Acknowledgements:
-
The lab activities are kindly supported also by:
-
Cariplo Foundation through the Project iCardioCloud no. 2013-1179, Regione Lombardia through the Project no. E18F13000030007 and Ministero dell’Istruzione, dell’Università e della Ricerca through the Project no. 2010BFXRHS;
-
European Research Council (ERC) through the ISOBIO project;
-
Cariplo Foundation through the project 2009-2822.
-