Running BSD on AWS Julien Simon Nicolas David

Transcript

All right. Good afternoon, everyone. So I'm Nicolas. This is Julien, just so you won't mix us up. I'm a technical trainer based in Paris, delivering trainings for AWS all over EMEA. Julien is our beloved technical evangelist for the French region, but he's not really in France. He's all over the world. Today, we're going to discuss running BSD, mostly FreeBSD and a little bit of OpenBSD, on AWS. We thought we'd have some fun with these two operating systems. So, this is a little bit about us. I'll start with myself and then hand the mic over. I discovered OpenBSD in the early 2000s, and the Sparc station you can see here is the first machine I installed OpenBSD on. This dates back a long time. Shortly after that, I was one of the few French people learning about OpenBSD, so I started the OpenBSD France community, which I ended earlier this year due to lack of time. I've been learning Unix in general, starting from my OpenBSD background. Hi, I'm Julien. I'm an older guy, so I want to tell you when I started with open source. It makes me feel bad. You can guess by the age of my CD-ROM collection over there. I guess most of you were not even born, which is a terrible thought. Back in 1996, I think, I translated a book, which is probably in your library, the French version of Kirk McCusick's book on BSD. I know he was here a few days ago, and it's a tragedy that I couldn't even meet him because I was traveling. So, hi Kirk. Me again. All right. And... No shit. This guy. Stick around, all right? It's the first time we meet. Wow. I know I cannot do this presentation anymore. I feel so worthless. You should be filming this guy. Yeah. I didn't bring it, but that's okay. You made my day. He's a legend. He's a proper legend. All right. So now we should be really good, right? Yeah. Okay. Okay, so here's the agenda for now. We're going to talk about the AWS infrastructure just for a bit, to show you what kind of architecture we have. Then, Nicolas is going to talk about instances, virtual machines, and operating systems. We'll start some benchmarks, right? Because it's important to see how fast things are running. We'll spend some time talking about building Amazon Machine Images with BSD. It's an interesting process, and we'll talk about automation quite a bit. Then we'll look at the results of our benchmarks, which hopefully will be complete. Don't be too fast. And then we'll take your Q&A. We decided we would love to take your Q&A during the session to make it more interactive. If there's anything that doesn't make sense, anything you disagree with, or if you want to throw stuff at us, that's okay. We can take it. We've seen worse. Okay, a word about infrastructure then. As you probably know, our infrastructure is spread across 16 regions worldwide. These regions are broken into availability zones, which are infrastructure partitions that are close enough to allow for replication, etc., but far enough so that if one of them explodes or if a volcano erupts in Dublin, the others should keep running. We have a whole lot of edge locations, and I'm sure this number is already false. These are the locations for the CloudFront service, our content delivery network, which spans the globe. A region is a number of data centers, and when I say "number," the number is larger than one. Some competitors like to call regions one data center. For us, it's much more because we think redundancy and high availability are really important. We have multiple availability zones that are fully connected with a very low latency network, usually less than one millisecond, which allows us to replicate data, storage, databases, etc., even synchronously. So, that's one of the regions, and you probably know we're going to open a region in France this year. Yeah, this year, right? So it's between now and December 31st, 11:59 PM, 59 seconds. People keep asking me for the day on Twitter all the time. Yeah, this year, right? If you want to know more about this stuff, there's a brilliant presentation from re:Invent by James Hamilton, one of our infrastructure gurus. I cannot recommend it highly enough. It's legendary too. Inside an AZ, we have multiple data centers. Let's talk about the French AZs that are coming. Each AZ will be at least one data center, but in fact, it's always more. The largest AZs could have six or seven data centers in the US, which are older and at a higher scale. Each AZ is going to be a number of data centers. They are close enough to allow for synchronous operations but distant enough so that if one is broken or has a power failure, fire, or disaster, the others should keep running. When it comes to network latency, they are very close, much less than a millisecond. This shows that we have multiple levels of redundancy. We could lose a data center and not lose an AZ, lose an AZ and not lose a region, or lose a region. God forbid, it has never happened. If you had a multi-region design, you would probably be okay. I guess we'll find out when we lose a region, right? Not soon, I hope. Inside the data center, we have racks and servers. That's not really original, except it's all custom stuff. If you want to know more about that, please look at James Hamilton's keynote. He goes into some detail about the servers, routing, and network equipment. It's all custom, 100%. We think this is where we have an advantage from a technology and price point of view. We decided not to have very large data centers. We could have more than 50k servers inside a data center, but if they get too big, the impact on the rest of the system if one dies is terrible. So we stick to mid-sized data centers and build many of them next to one another. Okay? Makes sense? All right, so let's talk about instances, those virtual machines in the cloud and operating systems. I'll start by saying that we call them EC2. There will never be any EC3 or EC1. EC2 stands for Elastic Cloud Compute. These machines are virtual machines based on the Xen hypervisor for now. Some things are going to evolve. We signed a partnership with VMware last year, and things have grown really fast. This hypervisor is now available in one of the regions in the US, US East 1, around Washington. To run these machines, you need hardware and software. This is where it gets very interesting. You have multiple places to grab something called an AMI, an Amazon Machine Image, which is an OS template containing your OS plus some more stuff. By default, AWS gives you access to AMIs for Windows, various Linuxes, and some BSDs, like FreeBSD for now. We're hoping to have OpenBSD soon, very soon. These AMIs are pretty basic, with the latest kernel, latest patches, and all that stuff. But you might want something a little more custom, like adding your own layer of security tools, a custom kernel, and so on. So you might want to create your own AMI. This is one of the things we'll be attempting to do on OpenBSD using some of the stuff Antoine Jacuto has been doing. You can create your own AMI, put all your stuff in there, and then eventually share it with the entire world, which becomes really interesting when you want to distribute your software. In the past, it used to be floppies, CDs, DVDs, downloads, and now you can run your own server with the application already pre-installed, so no one can tell you, "Well, you've done it wrong." The software editor or the person granting access to this AMI ensures that the AMI has the right configuration and tools, so when you boot your instance, everything is running fine. You might want to make some money with this AMI at some point. To do that, we have a place called the Marketplace. The Marketplace allows you to sell, by the hour, the license to your AMI or your software installed on the AMI. You can either pay a little bit per hour or per agreement, with the one-year term being the most common. Four places to get your AMI. Inside this AMI, you have the software and the OS. You want to instantiate that on some hardware. We have a combination of hardware options, which you'll see in the next slides. Until now, you had to pay by the hour for these resources: compute, storage, and network. Starting October 2nd, you will pay on a per-second basis, which is very interesting because most of the stuff you run doesn't require an entire hour. Maybe you need five minutes and don't want to pay for an entire hour. So you'll pay for one minute, the boot time, and then the running time of your instance. This is coming up on October 2nd and is only for Linux. The Windows license might have some things to do with it. You might think, "This VM is cheaper elsewhere." It's not really the case. You might want to compare apples to oranges. We have a very broad geographical coverage and a bunch of services running with those EC2 instances. There are about 100 services now, ranging from basic compute, storage, and network to IoT, machine learning, Elasticsearch, and more. Inside the region, as Julien said, we also have multiple availability zones allowing you to have high availability for your workloads and synchronize your data between different availability zones. Let's look at those instance types. The naming scheme is pretty easy: the family, the generation, and then the size, just like t-shirts. We made it pretty simple, except sometimes the size goes really big. Like this one. You might want to buy sheets instead of a shirt that size. Toga party, maybe. We have GPUs in some instance families, like the G3 and P2 families. The P2 is quite exceptional, with 16 GPUs, NVIDIA Quadros with about 12 gigs of RAM each, giving you many possibilities for heavy computation. If you're looking at memory-optimized instances, the R family or the X family, which is the biggest we have so far, has 128 cores and 4 terabytes of RAM with a 25 gig network interface. We're extending it soon to 16 terabytes of RAM. If you're running in-memory databases or caches, this is probably one of the sweetest instances to run your stuff on. The smallest instance is one core, half a gig of RAM, and the biggest is 128 cores, 4 terabytes of RAM. In these instance families, you'll choose something that fits your needs. One thing we'll highlight with Julien is that size doesn't matter most of the time. Right? Don't give away the results. Size doesn't matter. Right? The largest instance size to the most broad family, the T2 family. You can see the CPUs here are all Intel Xeons. On these Intel CPUs, you can tap into a lot of things, like instruction sets, P-states, and C-states control, to tweak your application. Today, Julien will be performing a few things on the i3 family, the C4 family, and the X1 family. The i3 family uses Intel E7 Haswell processors with a 25 gig network interface. The C4 instance is the second family Julien will be using, with Haswell processors at 2.9 gigahertz instead of the usual 2.3 maximum. This means we have custom CPUs. Intel is one of our partners, and we buy a lot of CPUs from them. At some point, we wanted to make a market differentiator, so we have the same CPU as everyone else but at a higher frequency, about 30% more performance than regular CPUs, and it's only available on AWS. The X1 is a multiprocessor architecture with a NUMA architecture, where each socket has dedicated memory. This is how you get 4 terabytes and such large amounts. If the OS supports it, we can migrate frequently accessed pages to the closest CPU. If CPU1 is accessing memory from CPU0 a lot, we can migrate that to the closest CPU, but it requires OS support. Thank you, Julien. The last family I want to talk about is the i3 family. i3 stands for IO, and we mean a ton of IO with this family. This is the newest generation of instances using NVMe storage, which is quite fast, and you can get up to 3.3 million IOPS. This is incredibly fast. The same 25 gig ENI elastic network interface is available on this instance, with half a terabyte of RAM and 64 cores. Again, custom CPU for this one as well, promising excellent performance. We have storage, compute, and memory. We need to know about the storage, which is where we'll store or instantiate our AMI. We have two main families of storage: classical standard magnetic hard drives and SSD drives. For EBS volumes, we have some advantages. For magnetic hard drives, we focus on throughput: 250 megs per second on cold drives and 500 megs per second on throughput-optimized drives. For SSD drives, we have two types: general purpose and provisioned IOPS. General purpose can burst up to 10,000 IOPS, and you can merge or RAID one, two, three, or four of them to go up to 40,000 IOPS. For databases requiring a lot of IOPS, we have provisioned IOPS, which deliver a constant 20,000 IOPS, and you can RAID two or more to get up to 40,000 IOPS. This is likely to be RAID 0 at minimum. Behind each block of these storage types are multiple physical blocks, so if we lose a hard drive, you don't lose your data. One question I often get is how we destroy hard drives. Do we have a special process? We do have an actual process from the US Department of Defense. It's a three-step process. First, demagnetize. Second, drill holes at regular intervals. Third, shred them. This way, you can grow new hard drives or separate metal parts from the rest. EBS storage is something you have to pay for, but there's another option that is free. We talked about the 3.3 million IOPS for the i3 family. These are the hard drives attached to the hypervisor. For the i3 family, you get 3.3 million IOPS and up to 15.2 terabytes of storage, and these hard drives are free. They are really performant, but the storage is not persistent. When you start your instance, it starts on a hypervisor. If you stop and start your instance again, you have more chances of winning the lottery than running the instance on the same hypervisor. For security and privacy, we won't copy data from one hypervisor to another, so you will lose this data. However, if you can work with that, this storage can be used for temporary files, video, or other cool stuff. One of our customers, Netflix, uses more than 100,000 instances on AWS and doesn't use EBS drives because of the costs and performance compared to instance storage. Instance storage is really fast and free, but you have to work with the non-persistence. Okay, and thanks for all this information on instances. They have very different specs, and we want to know how fast they are on real-life workloads. Benchmarking can be useful up to a point, but at the end of the day, you want to run a real workload and see what happens. So, we're going to do that. I've picked the largest C4, the largest X1, and the largest i3. Here are the specs and the setup. We're going to build the world on FreeBSD and see what happens. I'm using the 11.1 release, which is the AMI available in the marketplace. I think it's faster with the latest versions, but the AMI is still 11.1. I wanted to run a test that anyone can replay in five minutes or so. The C4 has a bit of memory, a few cores, network storage, and I'm using provisioned IOPS for a reliable IO level. I'm using UFS as the file system. The X1 has a ton of memory, a ton of cores, and instant store, so I have about 4 terabytes of local SSD to build on. For the i3, I have quite a bit of memory, a few cores, and the new generation of SSD called NVMe. Since I have eight volumes, I figured I might give ZFS a try. I'm madly in love with ZFS, so that's my excuse for trying it. We're going to build and see what happens. I also ran these numbers with a RAM disk, and we'll see what happens there. We'll talk about the hourly price of each instance. Just a few more details before we do this. I'm building on these three instance types. For the X1, I have two local SSDs, so I'm using one for USRSRC and one for usr-obj. I'm going to use all available cores to build. On the C4, I'm putting both directories on the same network volume with 10k IOPS. That's the simplest way to do it, and I'm using all cores. For the i3, I'm creating two ZFS pools for src and obj and using all cores. Let's do the setup quickly, launch the benchmarks, and then Nicolas will explain how to build AMIs. Hold on one second. I want to see your terminal. It's all black. Probably so. No, not the region. Ah, there we go. So, this is my EC2 console. As you can see, I'm running my three instances. I'm going to SSH to each of them, which should already be the case. Here's my C4, here's my X1, and here's my i3. In the interest of time, I have all the commands ready. But I don't think you'll learn anything here. Most likely, you'll fix my commands. For the C4, I'm just extracting sources, which I think I downloaded already, and building world on 36 cores. Let's make sure this is the right instance. Yeah, that should be quite fast. On the X1, I can see my two instant store volumes here, XBD1 and XBD2. I'm just going to format them, mount them. That should be fast. That's the X1, right? Okay. Okay, and yeah. Go C4. Alright. Okay. Now I can do the same thing. Extract sources and build. Maybe you want to see that actually happening. Of course. All right. Okay. So C4 is starting to build. On the i3, same thing. I can see my volumes here, my 8 NVMe volumes. I'm going to quickly build my pools. Okay. Alright, so X1 is building to... Okay, and same thing here. Okay, so I can see my two pools. We're ready to go. Extract the sources and build. Okay, so this is going to run for some minutes. I just want to make sure this one is actually starting before handing the mic back. So we're on our way here. This one is starting too, right? So all three instances are building. Let's show you those specs once again so you can decide which one you think will be the fastest. Don't lie to yourself. Pick one and don't change your mind. Let's take some bets by raising your hand. Who's going for the C4? Just by raising your hand. Yeah, come on. It's going to be closer than you think, right? So, okay. Who goes for X1? All right. Who goes for the i3? Okay. So you guys don't believe too much in the C4, right? Okay. And the rest is pretty much split. Most people are split between X1 and i3. Okay, but the brave soul, I said the C4 would win. There's always a brave soul. Okay, so that's fine. We need brave souls in this silly world. All right, so it's going, right? Okay, so let's switch back to slides. We can check the results at the end. Nicolas is going into the process of building OpenBSD AMIs. I'm a FreeBSD guy, like you understood, but come on, we're brothers, right? So your turn. All right. Thank you, Julien. So while Julien's build world is going on for all those instances, we'll talk about building BSD AMIs. Laurent in the back of the room, raise your hand. He had a presentation yesterday on how to build stuff with Consul and OpenBSD. Part of the stuff you use could probably be taken care of from the marketplace, where there's a lot of things available. But what I want to talk about today is using a few tools other than just the console, maybe in complement, like Packer, which is from the same company. Maybe some other tools like the CLI, which is the command line interface, or the shell CLI, which I really like about AWS, or eventually Aminator to build your own AMI. There's one template we've shared to build your own CI-CD pipeline. The idea behind what I want to show you is how we can bring some of the stuff we do manually on OpenBSD up to CI-CD and speed up some things, check more stuff, and maybe use managed services. For those who don't know what managed services are, it's the same thing as what you're doing by hand but with no hands. It's usually cheaper, as secure if not more secure, and follows a pay-as-you-go model. If you consume it, you pay for it, like turning on the light in a room. If you turn off the light, you don't pay for it anymore. That's the idea behind most managed services on AWS. We're going to build an OpenBSD AMI factory. We'll have a host running OpenBSD with about 12 gigs available, some room for the AMI we're going to create, plus about 4 gigs of temporary files. We'll also use the Create AMI script from Antoine, which brings everything together on a local file system and then, with a little bit of magic, pushes it to a storage service on AWS called S3. S3 stands for Simple Storage Service, one of the oldest AWS services, about 12 or 13 years old. In the Ireland region, it's one of the coolest services I use. It's about 2.2 cents per gig and can trigger notifications upon the arrival of a file. If something happens, I can use this notification to run some code, maybe modify my infrastructure, just as Laurent showed in his previous presentation. Here's what I want to do. I want to commit my code and then trigger a service called Lambda. Lambda is a container-managed service that runs your code in Java, JavaScript, Python, or C# upon notification. It runs between 100 milliseconds and five minutes. The first million executions of Lambda are free forever. The second million costs $0.20. Pretty cheap, right? Here, I just need to run this for maybe a split second to notify my OpenBSD host to create a new AMI that includes the code I committed and then notify me that the new AMI is ready, maybe using CodePipeline. CodePipeline, if you know Jenkins, is similar but managed and very API and AWS developer services oriented. Once we have this notification, CodePipeline can trigger other services, like CloudFormation. CloudFormation is one of my favorite services. It allows you to describe your infrastructure using either JSON or YAML, whichever is your preference. CloudFormation will then take all this information, scan it, identify the resources needed to be created first, like network and security, and then the resources that can be created in parallel. You can create your entire infrastructure really fast. This can be used for many things, like deploying your application in UAT or setting up your DR with all the necessary components. The cool thing about CloudFormation is that every time you create something, it creates a stack that can be updated with different behaviors or, the main advantage, creates idempotent stuff. The same thing every time, which humans are not always as good at as services. Once this is done, CloudFormation will deploy the application in UAT, and you can run some tests, maybe security and compliance tests using Inspector, a managed version of Nessus. We have a different set of tests, including PCI DSS compliance. You might want to do load tests, and I really like the name of this tool: Bees with Machine Guns. It's from News Corp. You might also want to test load and security simultaneously to see if your application behaves the same way under full or unexpected load. You might test features to see if they still work and if manual intervention is needed or can be automated. With this, you get some results and can feed them back to the developer, saying, "Hey, you missed something here," or "The percentage of comments versus code is not good enough." Eventually, things will move from UAT to production. The goal is to use blue-green deployment methods, with the blue being the existing environment and the green being the new environment, to switch from one to another without interrupting the customer's experience. The goal is to have everything working smoothly. So, that said, I think it's this one. Yes, it's where I start to do some demos. All right. So did I do enough chickens today so that my demos will run well? Yeah, it's going to work. Okay, so let's take this one. And then this. Move on to here. And then SSH to my OpenBSD host. As you can see, this is 6.1. And then, sorry, DA minus edge. A little bit of stuff going on here. I haven't cleaned my stuff since this morning. Actually, since an hour ago. Yeah. Yeah, it's on time. Just in time delivery. Absolutely. One minute before. I could do this automatically, but I want you guys to see how things are working. Do you have this open in Atom already? Sorry about this. No, this is yours. There we go. So there's a lot of information we want you to see. There we go. And this is the stuff I will be loading. I'll be exporting some stuff and then adding some. I'm showing my keys. I don't like that. There we go. I'm going to set a mirror for Ireland as my machine is running in the Ireland region. Then I'm going to add some cool stuff, clone my repository, make some modifications, and then generate the AMI. So let's cut and paste. There we go. There we go. Sorry about this. Please don't take a photo of my keys. That would help. It's on Twitter already. Really? Yeah, okay. Let's remove those keys real fast. And we're not live, so... So let's run this. I'm cloning some stuff, getting the script, and creating the AMI. You've probably seen this already. Creating the storage, creating all the stuff I want to. Once this is done, I will notify the rest of my applications in the pipeline via a service called SNS, for Simple Notification Service. It can send many types of notifications. The one I'm going to use is to signify the end of a task to Lambda so that Lambda can run the rest of it. As we're building, this is something none of you have seen before, right? Except my keys. As I'm building, this is one of the things I want to draw your attention to. This process is working, and it's a great process. However, it takes a little bit of time. It takes a little bit of time because some of the tools we're using are not up to date, and some of the drivers might not be giving their best. This is one of the things we might want to ask for your help to make it better on AWS. We have a slide later on for FreeBSD as well as some of the stuff that we're needing some help on. Yeah, there's a lot of people running NetBSD in Australia on AWS. And there's in the US and in Europe, in Canada as well, we're starting to see a lot of OpenBSD. Thanks to Antoine and Rake. Yes, that helped us. All right. So this is being built. Let me go back to the slides real quick. There we go. This is it. So we're at this stage right here. We're pushing the notification here and then we're building this. Once this is done, the notification will go here. CodePipeline will trigger, launch CloudFormation and deploy the application. CloudFormation is quite easy, actually, once you get used to it. But this is only for AWS. One of the services is dedicated for AWS. However, if you want something that is more platform-agnostic, you guys have probably heard of a tool called Terraform. And this is pretty cool because you use one DSN and then you plug whatever you want behind it and start running about the same thing. Again, apples and oranges in those different environments. I've seen a lot of customers doing this with VMware because it was already there and you bought those very, very costly licenses. So you need to maybe use them at some point. And so, yeah, by API interactions, by CloudFormation, you can do stuff inside of AWS or outside of AWS. I could be talking about Chef, maybe, or Puppet, or Ansible, or Salt, where once your application is deployed, you have different configurations between UAT and production. So maybe you put everything that is most stable, quote-unquote, or most non-moving parts into and then the moving parts, you can add them later on. Once your AMI has been baked, once your AMI has been used to deploy and create new instances in your environments, then maybe you can modify the configuration. And then those configuration management tools are really good to do this at a large scale. Because let's face it, I'm doing it with a few instances here, but maybe you could do it for hundreds, a time, 100,000 instances, maybe at the scale of some of our largest customers. Yep. Yep, you can do that. Absolutely. That's a great question. Again, most of the stuff that I'm doing is just a one-shot thing just to show you how it would run the first time. But later on, you will have to maybe maintain more AMIs, maybe more versions of your application. And think about it. One AMI per version of your application, the boot time of a machine is what? One to three minutes for most of the UNIXs? Quite fast. So you can boot that and deploy that with the template that was the version of your application equals a template. So you have a template and an AMI. You can deploy that very easily. And each and every one of the AMIs that you create has a unique ID. So it's AMI dash something, something, something. Something hard to remember most of the time. So you'd have to build some tools, maybe with automation, with the CLI, with some scripts, to have some kind of management of those AMIs. And also, good point, you pay for the storage. So as you create more and more AMIs, you may want to have some automated way of recreating those AMIs very quickly and make the decision for cost as well. Because storage or image generation time is going to take more or less money. So you have to take something that is tailored to your needs. Does that make sense? All right. So while this image is building, it's taking a little bit of time, a little bit too much time actually. This is why I was asking for some help. Right now it's about 20 to 25 minutes to build an OpenBSD AMI. It's still reasonable by automation. It's pretty good, but we can make it faster. We can make it a lot faster. So for that, drivers and tools are your best friends. Back to what I was saying earlier. Once you've committed, you back the AMI, you notify your teams and code pipeline, you deploy a new AT environment and use this new AMI, you test your application, and then once everything is satisfactory, then you move on to the next stage. Yep? Can you create your AMI locally and then upload it to storage? This is what we're doing. Maybe Laurent's technique was a little bit faster than mine. From your experience, how long does it take to build an AMI from Vagrant? With Packer. With Packer, yes, sorry. If the OS is already installed. So it's going to take maybe like 10 minutes, that's a good one. So with different tools, see, we can split the building time in half. I'm sure we can go a lot faster. I'm really sure. Yep. Okay. So different techniques, maybe different results, just like... You probably need, you know, you don't have to pick one or the other. I mean, most of the time you want to be on a stable OS and you just want to maybe rebuild the AMI and add extra stuff. And when there's a new version coming out, then yeah, maybe you want to rebuild completely from scratch. So it's a combination, right? It's a combination. And most of the time, you're just going to be deploying your app anyway on the latest AMI that you have. So that's going to be really fast. It could be just deploy the app, just do what you were doing, start from a stable OS and build my AMI, or rebuild it completely. All three make sense at some point in your development process. All right. So the takeaways from this is that DevOps is for AMIs, but it's also maybe for containers. You've seen probably the process resembling some other stuff like maybe Docker or things like that. Try to use services instead of servers, right? So to make up some more time to experiment more things, more services, right? And this is clearly going towards DevOps, I know. But this is the way you're using AWS, most agile way to use AWS. Security, again, is something very important. Again, when you're granting those services to access different parts, we have a service called IAM that I didn't show, sorry, lack of time, Identity and Access Management, which handles users, groups, policies, and roles, which can be assumed by different services or different resources to talk to each other. And along the way, when you're going to build this CI-CD pipeline, you're going to have to use roles and make sure that you use the least amount of privileges for the right amount of actions. And then last but not least, one of the advantages is clearly to pay by the usage of what you need. One of the services that I didn't show you is called CodeBuild that can build your code and you pay by the time of execution and the number of builds. So quite interesting as well. Instead of maintaining everything together, Managed Services can help you with that. With that, I'll give you the remote. Before looking at the benchmark results, this is how you can help. So if you're involved in the FreeBSD, sorry, the OpenBSD community, then helping us improve the speed of those scripts is definitely top of the list. So please get in touch. And if you made the right choice in your life and you're actually using FreeBSD, then... Oh, it is the FreeBSD room here, I told you. Yeah, Colleen is our hero. So yeah, thank you so much for the hard work on building those AMIs. And is part of the FreeBSD core team, as you know. And he needs your help on testing FreeBSD on AWS. He needs your help on writing documentation, which is always so important and sometimes the most difficult part of open source. So please help out. And any help that you can provide also on having one-click instant everything for FreeBSD would be very nice. Today we have the AMIs, but we would love to have proper packages, proper AMIs, ready to go with WordPress and whatever people like to run on FreeBSD. So there's plenty of ways. Yeah, everybody loves WordPress. And so anything that you can do there would be much appreciated. So he's there. Talk to him. That's the email address. Float him. He needs your help. And, you know, we want to see FreeBSD much more on AWS. Okay, let's look at the benchmark results. So, okay. All right. So, let's just look at the numbers. I ran those tests again yesterday. Hopefully, it's complete now. Please, don't worry. C4 is 11 minutes 42 seconds. Right? So keep that one in mind. This one is X1 1138. And now all the guys say, yeah, I won. I won. I knew the X1 was faster. And I3 is under 11 minutes. Right? So, and it's fun because it's pretty much exactly the numbers I did yesterday. So this is what we get, right? So I3 wins. And that goes to show a few things. That goes to show NVMe storage is just insane. I knew it was going to be fast, but it's blazingly fast. And you would think, my guess when I did this test was X1 is going to destroy everything. Because building is all about CPU, blood and fire and flames and skulls. It's just the biggest, baddest CPU wins. But no. And my guess is, when I actually spent a few hours looking at this build process all over again, and don't take offense in any way, but the FreeBSD build process is just not parallel enough to actually leverage those 128 CPUs. There are lots of sequential steps that are just running on a single core and you waste a lot of time doing that. But I don't think, I'm not sure it can be helped, but you can actually see most of the time, you just don't see parallelism on a lot of steps. And I think that's where you would win. That's where you would make a lot of speed. Yeah, but that's exactly my point. We're not using them. You know, I could have... No, no. is, no, so we're using, I don't know, I don't have what the max, I don't know what the maximum number is, but, you know, we probably end up using, I don't know, at any given time, maybe 40 or 50 cores in parallel, but we never go as high as 128, right? Because, you know, we, yeah, please. So I actually ran these benchmarks this morning because I knew there were going to be tests here. Actually, the X1, if you only run with 64 parallelism, it takes 10 minutes and 39 seconds. We actually have issues with kernel locking. We're just spending too much time with contention. There's work in head which improves that with a head kernel compiling 11.1, we can do it in 8 minutes and 31 seconds. So there's definitely progress happening there, but it's a scalability issue in the kernel, not just with a build process. So that's what I referred to earlier. I mean, I'm using 11.1 release because it's the official AMI right now, but yeah, it's going to get faster. So it's pretty interesting to see that not the biggest instance wins actually. So I run those same tests, exact same parameters on RAM disk. And this is what I get. So I have a minor improvement on C4. I have a small improvement on X1. And I run this repeatedly. And actually, I get slower with the RAM disk on I3. And my only conclusion is ZFS is just brilliant. But that's probably my troll for today. And the last thing is, how much does this cost? You know, we pay per hour and soon we're going to be paying per second. So here are the prices. Yeah. So again, it goes to show one thing is that Performance is very nice, but at the end of the day, even if you're going to use i3, right, would you be willing to pay four, yeah, almost, yeah, four to five times the hourly price just to gain a few seconds? I don't think so. And so that's the advice we give to customers all the time, again and again and again, and Laurent will agree with that, that when they ask us, I've got this disk workload, what instance size, what instance family, what instance size should I pick? The only reasonable answer is please run your benchmarks. Please run the actual application and figure it out. And then you get that performance level and you decide how much you want to pay for that. So if absolute speed matters, and maybe not for building, right? Let's agree on this. It could be another application. You could say, yeah, every second counts. So, okay, I'm paying that premium here. But I guess the right price point here for this use case would be C4. So, run your benchmarks. Again, synthetic benchmarks are nice, but the real testing should happen with the real workload. And then you can see what happens. As a conclusion, we talked about BSD today. But actually, you know, this is just the OS, right? It's important, but it's just the OS. And then all of our customers on top of that, all of our users, they run a crazy amount of open source, right? And, you know, from databases to NoSQL to Hadoop to, yeah, Puppet and Chef and Jenkins and all the CI, CD tools, et cetera. And actually, all of these, one way or the other, work really well on AWS. Some of our services are even based on those pieces of technology, like Amazon RDS for relational databases, where you can pick from MySQL and Postgres, etc. MarioDB and so on. So, you know, all of those, one way or the other, we help you run them on your OS. Okay? And let's face it, yes, sometimes it's running Linux. We help BSD run better on AWS, but we don't just stop there. We really want to have as many open source projects running very well on AWS. So keep that in mind and feel free to ask questions later on or get us on Twitter. And pretty much that's my conclusion. So thank you very much for listening. And we're really looking forward to having more FreeBSD and maybe a little bit of OpenBSD and NetBSD run as well on AWS. Thanks again. And if you have questions, we'll hang around. So please show up. Thank you.