#Amazon forecasting model full#
#Amazon forecasting model series#

The compute, storage and networking requirements are defined in the ParallelCluster configuration file. With AWS ParallelCluster, the user just needs to create a simple text configuration file to model the resources of the cluster which is used to provision compute, storage and networking capabilities in an automated manner.ДЇor running Harmonie on AWS, we launched a cluster using a custom Amazon Machine Image (AMI) configured with all the necessary components to launch and run Harmonie. This is an AWS-supported open-source cluster management tool that makes it easy to deploy and manage HPC clusters on AWS. AWS deployment architectureДЇigure 1: The AWS Architecture for HarmonieДЄs shown in Figure 1, we used AWS ParallelCluster to launch the cluster of Amazon EC2 instances. Both systems use Cray proprietary Aries interconnect and Cray Sonexion storage systems for the Lustre file-systems.ДЄs NWP models benefit from high-speed networking, we evaluated the Harmonie model performance on C5n.18xlarge instances with EFA on AWS and compared it to two Cray machines, Cray XC40 and Cray XC50 both with Cray Aries interconnect. The Cray-XC40 system used in this study consist of 36-core Intel (R) Xeon (R) CPU E5-2695-V4 2.10GHz and the Cray-XC50 system consist of 36-core Intel(R) Xeon (R) Gold 6140 CPU 2.30GHz nodes. C5n.18xlarge (36 physical cores, Intel Skylake based instance) support Elastic Fabric Adapter (EFA), that enables running applications with high levels of inter-node communication using MPI at scale on AWS. These instances are ideal for network intensive HPC workloads. The C5n instances leverage fourth generation of custom Nitro card and Elastic Network Adapter (ENA) device to deliver 100 Gbps of network throughput to a single instance.

#Amazon forecasting model series#

Instances evaluatedДЄmazon EC2 C5 instances are built on the AWS Nitro system and are powered by AWS-custom Intel(R) Xeon(R) Platinum 8000 series processors. The success metric was to provide a 1 hour forecast in 50 seconds. For the purposes of this evaluation, we reduced the forecast length to 1H. The target time for forecasting is 40min with additional post-processing required for an hourly run cycle. A typical production forecast length is 48H with hourly output. This workload is currently being developed as a joint effort towards a common setup for Numerical Weather Prediction for the United Weather Centre West. Harmonie testcaseДЇor our evaluation we used a production size workload with a Grid size of 1920x1620x90 at 2km resolution. The report also talks about the increasing relevance of Cloud computing for operational NWP and the numerous use-cases of cloud computing in the fields of weather forecasting and climate research.

#Amazon forecasting model full#

We have recently published a report in the ACCORD newsletter which provides a description on the porting experience and execution of a full NWP cycling dataflow with the Harmonie model to AWS. All 10 members of the United Weather Centre and the larger NWP model development community ACCORD relies on the same underlying NWP model which we here refer to as the Harmonie model. DMI is a member of the United Weather Centre and the NWP workflow studied here stems from the United Weather Centre West joint effort towards a common setup for Numerical Weather Prediction for the United Weather Centre West. The DMI activities which includes both national obligations and research are centered around numerical weather prediction (NWP). The Danish Meteorological Institute (DMI) is the National Met Service in Denmark, responsible for running atmospheric, climate and ocean models covering the kingdom of Denmark. We also present these as-is timing results together with as-is timings attained on the supercomputing systems based on Cray XC40 and Intel Xeon based Cray XC50. In this blog post, we expand on that report to present the initial timing results from running the forecast component of Harmonie model on AWS. You can find a report of the porting and operational experience in the ACCORD community newsletter. We worked together with the DMI to port and run a full numerical weather prediction (NWP) cycling dataflow with the Harmonie Numerical Weather Prediction (NWP) model to AWS. The Danish Meteorological Institute (DMI) is responsible for running atmospheric, climate and ocean models covering the kingdom of Denmark. This post was contributed by Jacob Poulsen, Senior HPC Researcher at DMI, and Karthik Raman, Senior HPC Solutions Architect at AWS.