Interested in how DevOps, IT Modernization and Agile practices can positively impact customer experience?
One of our primary goals at Flux7 Labs is to help our clients reduce their AWS costs. In fact, our product VyScale is based entirely on cost optimization using Spot instances. We inform our clients when it makes economic sense for them to buy instance reservations because reservations for periods of unexpected minimum usage can be beneficial. Reserved instances function exactly like on-demand instances, except that you pay an upfront fee to gain cheaper hourly rates. Reserved Instance Pricing There are several levels of reservations. Higher ones allow you to pay more up front in order to achieve a lower hourly cost. The table below shows the rates for various types of reserved and on-demand instances for m1.large instances. Upfront Hourly in cents On-demand 0 24 Light Util 243 13.6 Med Util 554 8.4 Heavy Util 676 5.6 At some levels of utilization it makes sense to purchase reservations, rather than to rely solely upon on-demand instances. By factoring in upfront costs, we can determine which levels warrant purchasing reservations. What’s surprising is that those levels are fairly low, especially in the case of light reservations. Even at 30% utilization, light-reservation costs start to break even with those of on-demand instances. The following graph shows equivalent hourly costs for reservations at various utilization levels To better understand these numbers in terms of total cost, as opposed to incremental cost, the figure below shows total annual expenditures at various levels. For a 100% utilization of an instance, one can reduce costs to almost half of Amazon’s standard pricing for reserved instances, and that’s without any of the bulk discounts made available to high-volume AWS customers. At lower levels, reservations cost almost twice as much as those at higher levels.
In our previous post here, we detailed why Ganglia is a good tool for monitoring clusters. However, when monitoring a Hadoop cluster you often need more information about CPU, disk, memory, and nodal network statistics than the generic Ganglia config can provide. For those who need more finely tuned monitoring, Hadoop supports a framework for recording internal statistics and then for posting them to an external source, either to a file or to Ganglia. In fact, Hadoop now supports an implementation of the Metrics2 Framework for Ganglia. In this post we’ll discuss Hadoop Metrics2 Framework’s design and how it enables Ganglia metrics. Features The Hadoop Metrics2 Framework provisions multiple metrics output plugins for use in parallel. It allows dynamic reconfiguration of metrics plugins without having to restart the server, and it exports metrics via Java Management Extensions (JMX). Design Overview The Hadoop Metrics2 Framework consists of three major components: 1. The metric source is used to generate metrics. 2. The metric sink is used to consume the metrics produced by the metric sources. 3. The metric system is used to periodically poll metric sources and to pass the metric records to sink. Implementing and Configuring Components A metric source class must implement the following interface: org.apache.hadoop.metrics2.MetricsSource A metric sink must implement this interface: org.apache.hadoop.metrics2.MetricsSink The basic syntax to configure metric system components is: <prefix>.(source|sink).<instance>.<option> Here’s a sample job tracker configuration for sinking a file: jobtracker.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink jobtracker.sink.file.filename=jobtracker-metrics.out Filtering Metrics Metrics can be filtered on source, context, records, tags, and metrics themselves. Here is a filtering example : test.sink.file1.class=org.apache.hadoop.metrics2.sink.FileSink test.sink.file0.context=foo: This will filter out all the metrics within the context “foo”. Hadoop–Ganglia Integration Metrics are collected from the following: 1. JobTracker 2. TaskTracker 3. NameNode 4. DataNode 5.
I’ve been doing a lot of analysis of latency and throughput recently as a part of benchmarking work on databases. I thought I’d share some insights on how the two are related. For an overview of what these terms mean, check out Aater’s post describing the differences between them here. In this post we’ll talk about how they each relate to one another. The basic relationship between latency and throughput within a stable system—one that is not ramping up and down—is defined by a simple equation called Little’s Law: occupancy = latency x throughput Latency in this equation means the unloaded latency of the system, plus the queuing time that Aater wrote about in the post referenced above. Occupancy is the number of requestors in the system. The Water Pipe Revisited To understand the latency/throughput relationship, let’s go back to Aater’s example of the water pipe. Occupancy is the amount of water in the pipe through which we are measuring latency and throughput. The throughput is the rate at which water leaves the pipe, and the latency is the time it takes to get water from one end of the pipe to the other. If we cut the pipe in two, we halve the latency, but throughput remains constant. That’s because by halving the water pipe we’ve also halved the total amount of water in the pipe. On the other hand, if we add another pipe, the latency is unaffected and the throughput is doubled because the amount of water in the system has doubled. Increasing the water pressure also changes the system. If we double the speed at which the water travels we halve the latency, double the throughput, and the occupancy remains constant. The following figures give a graphical view of the equation.
Recently at Flux7 Labs we developed an end-to-end Internet of Things project that received sensor data to provide reports to service-provider end users. Our client asked us to support multiple service providers for his new business venture. We knew that rearchitecting the application to incorporate major changes would prove to be both time-consuming and expensive for our client. It also would have required a far more complicated, rigid and difficult-to-maintain codebase. We had been exploring the potential of using container technology, Docker, to set up Flux7 Labs’ internal development environments and, based on our findings, believed we could use it in order to avoid a major application rewrite. So we decided to use Docker containers to provide quick, easy, and inexpensive multi-tenancy by creating isolated environments for running app tier multiple instances for each provider. What is Docker? Docker provides a user-friendly layer on top of Linux Containers (LXCs). LXCs provide operating-system-level virtualization by limiting a process’s resources. In addition to using the chroot command to change accessible directories for a given process, Docker effectively provides isolation of one group of processes from other files and system processes without the expense of running another operating system. In the Beginning The “single provider” version of our app had three components: Cassandra for data persistence, which we later use for generating each gateway’s report. A Twisted TCP server listening at PORT 6000 for data ingestion from a provider’s multiple gateways. A Flask app at PORT 80 serving as the admin panel for setting customizations and for viewing reports. In the past, we’d used the following to launch the single-provider version of the application: Both code bases were hard coded inside the Cassandra KEYSPACE.
Press Release January 24, 2014—Austin, TX—Flux7, a solutions company offering cloud optimization products, cloud consulting and implementation, specializing in DevOps and AWS development services, for small businesses through Fortune 500 corporations, today announces that its VyScale spot strategy solution—the only one of its kind on the cloud market—recently earned honorable mention in a competition sponsored by Amazon Web Services [AWS]. Judges of the second annual contest shortlisted droves of applicants to reveal the top six, naming Flux7’s VyScale among them last November at AWS re:Invent 2013, considered the largest gathering of developers and technical leaders from the AWS community. “We received many impressive applications making this year’s competition even harder to judge than last year’s. There were many sophisticated and novel uses of spot instances, and VyScale was among them, “ wrote Stephen A. Elliott, Senior Product Manager, AWS Elastic Compute Cloud [EC2], to Aater Suleman, CEO, Flux7. “Given how tight the competition was, we congratulate you on your honorable mention!” Suleman first learned of the award at the AWS re:Invent developers conference, and was quite happy with the result. He showed that, overall, VyScale provides more than 60 percent in reduced costs without the need for human intervention, installations or access to sensitive data. VyScale, now in closed beta with paying customers, seamlessly combines the cost savings of spot instances with the reliability of reserved and on-demand instances by using smart placement, instance selection and AI-based prediction. “VyScale is an application that’s the result of our team’s extensive research and expertise in cloud consulting,” Suleman said. “It’s the only spot-strategy-as-a-service product on the market that delivers key strategies for the AWS spot marketplace and solutions for scaling and disaster recovery.
On January 11, Aater and I attended Data Day Texas 2014 here in Austin. Sponsored by Geek Austin, it was such a great event that I thought I’d share some highlights. Data Day Texas holds special significance for Flux7 Labs because it was at Data Day 2013 that we made our first presentation, when Aater gave a talk on the role of microservers in big data, which you can find here. What I like the most about this conference is that it provides a perfect balance between technical depth and high-level theory that appeals to everyone. Another great thing about it is that, in addition to presentations on a wide array of topics, you get to meet and talk with a broad spectrum of cutting-edge leaders and innovators in the field. It was also great getting to meet and exchange ideas with many of our Austin network friends, people we know both personally and by reputation, but with whom we don’t often get time to hang out in our everyday professional lives. It’s inspiring and rejuvenating to step back from foci on our own work and to hear what others are thinking about and working on. This year’s keynote was delivered by Paco Nathan (@pacoid). Entitled “Data 2014: The Big Picture”, Nathan covered a variety of topics, from how playing Minecraft has taught his daughter enough to be a Linux sysadmin to how our schools are failing to educate students in the particular fields of math that are most relevant to the 21st century. For instance, current engineering curricula is placing primary emphasis on calculus when in fact it’s graph theory and linear algebra that’s most needed today. He also described the big data processing pipeline from beginning to end, explaining current trends toward functional programming.
At Flux7 Labs we solve a variety of problems for our customers and often that includes guiding clients to the right tools for their needs. In our previous post on NoSQL, we discussed how NoSQL solutions offer a better alternative to RDBMSs. In this post we’ll walk you through different types of NoSQL database models and solutions and show you how different architectures and design philosophies support various features. We’ll explain how NoSQL can be a tool that better serves your needs than a one-size-fits-all tool like an RDBMS. 1. Key-Value Stores – These types of database stores data as a hash table with a unique key and a pointer to a particular item of data. Similar to traditional hash tables, it allows data storage and retrieval through keys. It’s the simplest data model and it scales incrementally. However, it’s not suitable where queries are based on the value rather than on the key. It’s also not suitable for transactions or for storing relationship details between data. 2. In-Memory Key-Value Stores – In-memory key-value stores, like Redis, deserves their own category because they’re designed to minimize latency while compromising capacity by holding data within memory. In-memory key-value stores are frequently used to cache results from larger-but-slower databases. In fact, these often aren’t RDBMS replacements, but rather work in conjunction with an existing RDBMS solution. An in-memory key-value store can still scale to high capacities using multiple nodes. For example, it’s been implemented with as many as 250 million key-value pairs. 3. Document Stores – This type of database stores, retrieves and manages data as documents. A document is basically the same thing as a row in a traditional RDBMS. Documents are schema free, i.e., different documents can have structures and schema that differ from one another.
As mentioned in part 1 of this series (Creating a LAMP Stack AMI), a common concern among most customers is to choose the right instance type. It is important to do the capacity planning. Before you choose instance type ask yourself the following three questions: Is the application Memory intensive CPU intensive or Network intensive? For this question to be meaningful put a monitoring system in place and collect the data for a few weeks of real usage. What is the expected request count at peak hours? What is the minimum number of instances you want to run in non-business hours? https://docs.google.com/a/flux7.com/spreadsheet/ccckey=0AkmUqlScRGp8dGJTNndjY0VjTURid3FTV2dNMEljOUE#gid=0 Make sure to run at least 2 servers for high availability. During times of fewer loads, choose two m1.small instances for non-business hours like weekends. This would optimize cost instead of using 2 large instances for the same request. Different instance types have different capacity levels. If the application is memory intensive choose and use m1. class. If the application is CPU intensive then choose and use c1. class. Network bandwidth varies between different instance types. If the application requires more bandwidth (ex: video streaming applications) choose higher instance types irrespective of Memory or CPU to get better bandwidth. To know more about each instance types visit the following link. http://aws.amazon.com/ec2/instance-types/ Start with micro/small instance for any workload. Do the Load run starting with the minimum number of users and increase the load gradually. Also scale the servers vertically or horizontally gradually until it reaches the maximum capacity. This would give a clearer picture of the number of instances required to serve the maximum or minimum load. Closely watch the CloudWatch graphs to understand usage statistics better.
In part 3 of the Autoscaling for LAMP on AWS series, After setting up your application autoscaling, it’s important to load run the application in order to understand the minimum and maximum number of instances required for each application. Read Part 2 to learn more about setting up autoscaling groups. First, fire the load run. For this article we’ve used HP LoadRunner because it provides more detailed results than others, but there are also other load runner tools to choose from. Be sure to run the load from multiple IP addresses or else the ELB may not distribute it equally across all instances. HP LoadRunner helps you use multiple nodes in order to generate the load. Update the Auto Scaling Group’s minimum and maximum instance counts. For the this article, we’ve set the minimum and maximum instance counts at 2 and 8 respectively. Now change your Auto Scaling Policies. A scaling policy specifies whether to scale the auto scaling group up or down, and by how much. Here we’ve chosen 2 instances for Scale Up and Scale Down policies. These policies automatically increase or decrease the instances according to your defined conditions in order to maintain performance and minimize cost. Next, increase the load gradually with LoadRunner. Once the CPU reaches 75% of the instances, Autoscaling will trigger the scale up policy to launch 2 more instances. Now decrease the load gradually and the CPU instances load should come down to less than 45%. Autoscaling will then trigger the scale down policy and delete the additional instances that were launched. Consider a scenario in which a 2,000-concurrent-request count is reached during peak hours and 500 users are reached during non-peak hours. For best load run results, try the following.
In part 2 of the Autoscaling LAMP in AWS series, let’s discuss how to create autoscaling launch configuration, autoscaling groups and how to verify the setup autoscaling. Autoscale Implementation Autoscale configuration is now available in console. AWS command lines are no longer needed for implementation. Complete the following steps as detailed in Part 1 of this series in order to set up autoscaling : Configure AMI to launch the Instances. Configure Instance type to launch the instances. (Example: m1.small,m1.large). Configure KeyPair Name to access the machines. Configure Security Group to allow the Instances to communicate with other components. Keep the ELB name readily available. Keep your availability zones ready. (Example: us-east-1a, us-east-1b). Set the minimum number of instances for Maximum and Desired Capacity. (Start with zero). Set Health Check Type. (ELB). Set Region. Change Capacity Cooldown time. Adjust for scale up and scale down. 1. Create the Autoscaling Launch Config Log in to the AWS console and navigate to Services-> EC2-> Launch Configuration. Click on “Create Autoscaling Group”. On the next screen, click on “Create Launch Configuration”. Create a new launch configuration. The name of the launch configuration must be unique within the scope of the client’s AWS account. Choose AMI: Go to My AMIs and select the LAMP AMI created. Choose Instance Type: We’ve selected a micro instance for our example. Configure Details: Give a name for the Launch Configuration Add Storage: Keep the values on default. Configure the Security Group: Select the Security group to launch the autoscaling instances. Review the details and create the Launch Configuration. In the next window, select “KeyPair” to access the instances. 2. Create the Auto Scaling Group Create a new Auto Scaling group with a specified name and other attributes.
This is part 1 of the AWS Autoscaling Tutorial for LAMP in AWS series. This autoscaling step-by-step guide would walk you through: Part 1: Creating a LAMP Stack AMI Part 2: Setting up Autoscaling Groups. Part 3: Load Test the environment Part 4: Choose instance type First, let’s discuss how to prepare the Amazon Machine Image -AMI- create an Elastic Load Balancer and Amazon RDS, verify the application and terminate the instance. Prepare AMI In AWS Autoscaling, create an AMI with all required packages installed. This AMI will be used as a template to launch instances in autoscaling. For AWS LAMP Stack, you should install and fine tune the latest versions of Apache and PHP. 1. Log in to AWS console. 2. Navigate to EC2 services. Make sure to switch to the desired region before launching instances. 3. Click on “Launch instance”. 4. Select your favorite Linux AMI in Classic Wizard. For this article we used CentOS 6.3 from the community AMIs, but OpenSuse and Ubuntu are also good choices. 5. Launch a micro instance from the selected AMI with the desired AWS Security Group and Key pair. Remember, this instance is only for creating an AMI and will be terminated once LAMP Stack is installed on it. 6. Now log in to the instance and perform the PHP and Apache installation steps. For this article, we used Cygwin to connect the Linux servers. Make sure you open port 22 to allow SSH access using security groups. We modified the “default” security group to open the SSH port. We recommend that you not completely open port 22 for SSH, but rather allow incoming requests to SSH from your specific IP address. In our case the IP address was 22.214.171.124.
At Flux7 Labs, as part of our consulting, we found ourselves solving the same problems in cost and performance optimization for our clients. We have productized some of these learnings into our product VyScale. A major feature of VyScale is its management of spot instances. We have been working closely with users of spot instances and Amazon to come up with strategies to use spot instances. As part of that, today, we hosted a Hangout on spot instances.