Automatic software verification

Infrastructure as a Service IaaS cloud computing has advanced recently, and users can use virtual resources such as virtual machines, virtual networks, virtual routers, virtual storage, and virtual load balancers on demand from IaaS service providers for example, Rackspace public cloud [ 1 ]. Users can install OS and middleware such as DBMS, Web servers, application servers, and mail servers to virtual machines by themselves and can customize virtual machines as if they were their own private servers.

Software vendors periodically issue software patches for OS and middleware deployed on virtual machines in order to protect them from security vulnerabilities or provide additional functions. In most cases of IaaS virtual machines, users manually select and install these patches to their virtual machines.

Therefore, users need to distribute patches to their virtual machines and verify the health of their systems by themselves. With existing shared hosting services, only service providers configure OS or middleware. Meanwhile, in the case of IaaS cloud computing, users can customize virtual machine OS or middleware. Thus, no service provider currently verifies patch normality after a patch distribution to user virtual machines.

The service model is such that users pay optional service fees for patch verifications to providers. Our proposed method replicates user virtual environments, extracts verification test cases for user virtual environments from a test case database DB , distributes patches to virtual machines on the replicated environments, and executes those test cases automatically on the replicated environments. We implemented the proposed method on OpenStack [ 3 ] using Jenkins and confirmed the feasibility of automatic selection and execution of test cases based on user virtual environments.

Using the implementation, we evaluated the effectiveness of test case creation efforts by the idea of two-tier abstraction. We also evaluated the automatic verification performance. The rest of this paper is organized as follows. In Section Problems with existing technologies, we introduce IaaS platforms such as OpenStack, review existing automatic test tools, and clarify problems of virtual machine patch verification for service providers. In Section Proposal of automatic verification technology of virtual machines patches, we propose automatic software patch verification technology for user virtual machines and describe a design to solve the problems of existing methods.

In Section Evaluation of automatic verification technology of virtual machines patches, we explain how we implemented the proposed method, confirmed its feasibility, and evaluated test case creation costs and automatic verification performance.

We compare our work to other related work in Section Related work and summarize the paper in Section Conclusion. When the providers verify software patches for OS or middleware, they only repeat the same regression tests because there are only pre-known configuration settings, and verification efforts are minimal.

Therefore, OS and middleware of virtual machines can be customized by users, and users need to apply patches by themselves. This paper targets patches for virtual machines on IaaS cloud. The basic idea of our proposed method is independent from the IaaS platform. For the first step, however, we implement a prototype of the proposed method on OpenStack see Section Evaluation of automatic verification technology of virtual machines patches.

Therefore, we use OpenStack as an example of an IaaS platform in this section. For example, our method uses Heat [ 7 ], which is a template deployment technology of OpenStack; Amazon Web Services have a similar deployment function called Amazon CloudFormation [ 8 ]. OpenStack is composed of plural function blocks. Some function blocks provide coordinate functions such as authentication, orchestration and monitoring of other function blocks.

Figure 1 shows a diagram of OpenStack function blocks. Neutron manages virtual networks. OVS Open Virtual Switch [ 9 ] and other software switches can be used as virtual switches, and Neutron controls to create these virtual switches or virtual routers.

Nova manages virtual machines. KVM Kernel based Virtual Machine [ 10 ], Xen [ 11 ], and others can be used as hypervisors, and Nova controls to create virtual machines on these hypervisors. OpenStack provides two storage management function blocks: Cinder for block storage and Swift for object storage. Both types of storage are used for retaining data. Glance manages image files for virtual machines.

Heat orchestrates these function blocks and provisions multiple virtual resources according to a template text file. Keystone is a function block that enables single sign-on authentication among other OpenStack function blocks. Ceilometer is a monitoring function of virtual resource usage. Major versions of OpenStack are released once every six months; the latest version is called Juno. Regarding Linux patches, distributors such as RedHat confirm function degradation when they release a patch or upgraded version, and users can adopt a stable software version provided by distributors.

However, distributors only confirm functions of OS and do not check middleware behavior on Linux. Therefore, users need to verify middleware behavior on Linux to check whether a Linux patch affects it. It is also said that distributors do not check the performance of each patch, so system performance degradation checks are necessary after patch distributions.

For example, to check the transaction performance of a Web three-tier model, it is better to run a TPC-C Transaction Processing Performance Council benchmark test on a user virtual environment. Some tools enable automatic tests, for example, Jenkins [ 12 ] and Selenium [ 13 ].

Jenkins is a tool to support Continuous Integration and is useful not only for building software but also for executing regression test cases for software that is changed during the software life cycle.

Selenium is a tool to enable automatic Web tests; it captures actions of Web browsers and repeats captured Web actions or conducts Web actions described by Selenium IDE scripts.

However, the objectives of these tools are recurrent executions of the same regression test cases. There are two problems with IaaS virtual machine patch verifications. However, the paper [ 2 ] targeted IaaS platform development, and regression tests of user virtual machines deployed on an IaaS platform are out of scope. The paper [ 2 ] also describes that three to five times of the amount of work are needed for automatic test case creations using Jenkins and Selenium compared with manual regression test executions.

In Automatic verification steps, we explain the automatic verification steps. The figure shows OpenStack, but OpenStack is not a precondition of the proposed method. In Test case extraction method, we explain the process of selecting automatic test cases, which is a core process of the verification steps. Figure 2 shows the processing steps of automatic verification when a software patch is released. For example, we consider a case in which a patch was issued for a Windows server.

Service providers extract users who would like to verify the Windows patch for their virtual machines from the customer DB. The automatic verification steps when a patch is released from a software vendor are as follows. Operators specify a patch and a user tenant logical space for each user where virtual resources are deployed to which a patch is distributed to AVFs. A user is extracted from the customer DB.

A tenant is a logical space for each user where virtual resources such as virtual machines, virtual routers, and volumes are deployed. We assume both use cases of a manual verification start or automatic verification start.

When verifications are handled for many users, a provider prepares a script program that extracts verification target users from a customer DB, manages released patches to be verified, schedules orders of each user tenant verification, and calls the AVF API to start verifications. AVFs replicate a user virtual environment.

Note that the current OpenStack Heat cannot extract a template from a user tenant directly; we use complementary technology to Heat [ 14 ] for OpenStack tenant replication cases. Second, AVFs request the IaaS controller to deploy an extracted template with the target tenant ID; then the IaaS controller provisions virtual resources of the user tenant on the specified tenant.

When volumes are replicated, volumes data such as installed software are extracted as a RAW image file; then the image file data are copied to a volume on the specified replicated tenant. Replicated virtual resources are deployed on tenants managed by service providers so as not to charge users.

Our technology main targets are users who do not have sufficient skills in using OpenStack, or sufficient resources for verifications. Therefore, we extract a template from an actual user environment in this step.

However, there are some users who can utilize OpenStack Heat templates sufficiently. If these users would like to verify patches by themselves, our technology does not support them, but if they would like to use our automatic verifications, AVFs skip a process of template extraction and receive their own templates to build test tenants, then we can help them to verify patches based on their templates.

AVFs acquire environmental data of installed software. Specifically, the data of the software that is installed on each virtual machine is acquired from replicated virtual environments.

AVFs select test cases for patch verifications from the test case DB. Test cases are executed after patch distributions to virtual machines, but some test cases may need to set verification data before distributing patches.

To select test cases, virtual resources structure template information step 2 , and software environmental data step 3 are used. This is a core step of automatic verification; thus, we explain it in detail in Test case extraction method. AVFs distribute a specified patch to replicated virtual machines. Existing patch distribution methods corresponding to virtual machine software can be used.

Here, we explain an example of windows update case. As a prerequisite, cygwin module is installed on a windows virtual machine and patches are stored in a server which can be accessed from a windows virtual machine. Have questions? Your clients depend on you for access to accurate, reliable information in their time of need. But keeping hundreds or thousands of resource records up-to-date? Using this tool, send emails to the contacts you have listed for your resource records and ask them to verify the information you have on file.

Deselect any resources you do not want to send in the request by unchecking them. Please note, as soon as one Resource record is deselected, Setup Method 2 will apply to the automated verification request see below.

Note: Only one query can be used per automated verification request. This is an example of where Custom Fields may be an option. As soon as you deselect any displayed Resource, you are now using Method 2.

Important to note , the query you use to create a Method 1 automated verification request will be saved, and repeated at the time the request is actually sent. If any changes happen to your Resources between the initial creation of the automated verification request, and when it is sent, all records that are found in the same query will be sent.

However, you end up busy and unable to send the automated verification request that day, and decide to delay sending it, until Friday. When the automated verification request is sent on Friday, all records that match the initial query will be included, including those that were not included when the automated verification request was first created on Monday. You want to send an automated verification request to just this agency to inquire about the status of their programs.

Setting 2 tells the automated verification request what template you want to use when sending the request. Setting 3 tells the automated verification request in what order to look for email addresses.

Cimatti, A. Griggio, M. Keremoglu, R. Heizmann , J. Hoenicke, A. Bounded Model Checking, k-Induction A. Biere, A. Cimatti, E. Clarke, O. Strichman, Y. Zhu: Bounded model checking, Advances in computers 58 11 , Academic Press, Clarke, D.

Kroening, K. Donaldson, L. Haller, D. Kroening, P.