Benchmarking Amazon AWS vs Microsoft Azure for Low-End Linux Virtual Machines

When you create a virtual machine in AWS or Azure, you are giving some basic specs to choose from, but how do you really know what you are getting performance-wise, and how does a VM from one service compare to another? In this post I’m going to attempt to benchmark the performance of low-end Amazon AWS EC2 and Microsoft Azure virtual machines. My goal is to find out which is the best value.
 

Methodology

For the comparison, I chose to use the two least expensive virtual machine instance types from both Microsoft and Amazon. I created each of the instances using the default values, with the exception of disk size, which I increased to 20 GiB for AWS instances. I ran sysbench on a freshly provisioned Ubuntu 16.04 LTS virtual machine. No tuning or optimization was done by me. To keep it as simple as possible, I only ran the benchmark one time (on Sunday, Feb 26, 2017, at around 12:00PM ET). I realize that a temporary condition might affect the results, but I did go back and re-run the benchmarks again to make sure there was no significant difference in the results.

After creating each instance, I logged in via SSH and executed the following commands:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install sysbench
sudo reboot

After rebooting, I logged back in and executed the following commpands:

lsb_release -a
df
sysbench --test=cpu --cpu-max-prime=20000 run
sysbench --test=fileio --file-total-size=5G prepare
sysbench --test=fileio --file-total-size=5G --file-test-mode=rndrw --init-rng=on --max-time=300 --max-requests=0 run
sysbench --test=fileio --file-total-size=5G cleanup
cat /proc/cpuinfo
cat /proc/meminfo

 

Specs

Instance Type CPU Memory (GiB) Disk (GiB) Price per Hour (USD)
AWS Nano (t2.nano) 1 vCPU 0.50 20 0.0059
AWS Micro (t2.micro) 1 vCPU 1.00 20 0.0120
Azure Basic A0 1 Core (Shared) 0.75 30 0.0180
Azure Basic A1 1 Core 1.75 30 0.0230

Price shown is the listed hourly price for the virtual machine only (It is the on-demand price for AWS) as of 2/26/2017 and does not include any storage or bandwidth costs.
 

Results

CPU Disk I/O
Instance Type Total Time (s) ms per Request Price / Performance Total # of Events ms per Request Price / Performance
AWS Nano 29.60 3.06 0.018 420,748 0.82 0.005
AWS Micro 30.48 3.01 0.036 483,253 0.77 0.009
Azure Basic A0 129.91 21.07 0.379 8,400 66.92 1.205
Azure Basic A1 98.61 17.08 0.393 13,100 50.12 1.153

Price/Performance is requests per second divided by price per hour. Lower is better.

CPU Benchmark Results

Disk I/O Benchmark Results

Price/Performance Results
 

Data

In case you are interested, here is the console output of each benchmarking session:

 

Conclusion

AWS t2.nano instances are a great value if you need to host something that is not too resource intensive. The two Azure VMs I benchmarked just did not stack up well against supposedly comparable VMs from AWS. In fact, the difference was so great, that I even tried running a benchmark for a slightly more expensive Azure Standard A1 instance (standard tier instead of basic tier), but the results were not much better.

The difference in CPU performance might be due to the “burstable” nature of AWS t2 instances, allowing an idle instance to accrue CPU credits that can be used to burst CPU performance when needed (see CPU Credits for more info on that).

The difference in Disk I/O performance is likely due to AWS using SSD storage by default as opposed to Azure which uses HDD by default. Azure does allow you to opt for Premium Storage but that is not something enabled by default when you create a VM, so was not part of this benchmark. If I do another benchmark, I may try using premium storage on Azure.

I also wonder if the Azure benchmarks would be better for Windows virtual machines, and if I get motivated, I may run some follow-up tests to find out. For Ubuntu virtual machines, however, the results clearly show that the AWS instances provide not only the best performance and value, but do so at a lower cost.

1 Comment

  • Ricky says:

    The reason that I searched to reach your post because I felt a significant difference of installation time of a piece of Linux software called guacamole on Amazon t2.micro and Azure Basic A0. Thanks for making the benchmark test and I agree with your result. Thanks.

1 Trackback

Leave a Reply

Your email address will not be published. Required fields are marked *