The Real Deal with ASE and Linux Disk I/O, Part One
January 15, 2007 8:57 AM
Filed Under: Adaptive Server Enterprise (ASE) Linux Kernel
Okay – so perhaps you administer ASE on Linux and were lured in by the title. Maybe you are Linux curious and want to learn more. Maybe you couldn't care less about Linux but you find ASE disk I/O highly exciting. Perhaps you are really into Frisbee and found this post because it uses the work 'disk' a lot. Regardless, you've come to the right post (well, except for the Frisbee people – try again in your search engine).
Asynchronous I/O on Linux has been a pain for ASE going back some time. In the past we've addressed this with a myriad of traceflags and linking strategies. I'm pleased to say that we believe to have finally put the issue to bed by implementing a pure kernel asynchronous disk i/o strategy in Linux.
Allow me to provide some background on asynchronous disk i/o (AIO). I'll begin with an explanation of what, AIO really means.
A read from, or write to, a disk is normally a blocking operation. This means that the program that issues the read() or write() system calls will be put to sleep until the i/o completes, which in turn means that the program can't do anything until the i/o is done. For most programs this doesn't really matter. For a multi-tasking server program like ASE, it matters a great deal. ASE is very much like an operating system in that it has a lot of tasks that it manages and must schedule. It cannot pause operations because one task has issued a read – that would kill throughput (imagine if your o/s stalled anytime a process did a disk i/o).
In order to maintain throughput, ASE uses asynchronous disk i/o. In the AIO model, the read or write call essentially queues an i/o request for another context to perform. This allows the calls to return immediately without blocking. Because the blocking portion of the i/o is done in a different context, the program can continue doing other work while the i/o is “in-flight.” In terms of ASE, when a task (spid) issues an AIO, say to read a page into cache, the task is put to sleep until the AIO completes, and another task is run. Scheduling tasks while another task is sleeping on an asynchronous disk i/o allows ASE to maintain throughput. When ASE has determined that the AIO has completed, the issuing task is woken and scheduled to run. (As an aside, there are several mechanisms by which the issuing program can determine when an AIO has completed. These vary by platform but usually consist of a signal method, a polling method, and a blocking method. ASE typically uses a polling method).
So in short: synchronous i/o blocks the caller, where as asynchronous i/o allows the caller to do other work while the i/o is in-flight. Does this make an asynchronous i/o faster than a vanilla (synchronous) i/o? No it doesn't. The individual i/o is no faster using AIO, and in fact there may be a CPU penalty for all the work to setup the separate context and retrieve the completion of the I/O. What AIO does is allow a multi-tasking system, such as ASE, to maintain throughput by working while i/os are in-flight.
Now that we know what AIO is and why we want it, let's tackle the “Kernel” part of Kernel Asynchronous I/O, which from this point forward will be referred to as KAIO. Remember that with AIO, a separate context does the real work of the i/o. KAIO uses a context within the O/S kernel to perform the i/o (kernel thread). Its counterpart, Posix AIO, uses a user space thread as the AIO context. KAIO tends to be more lightweight and efficient because it is implemented directly within the O/S kernel whereas Posix AIO is implement in a run-time library. This means that Posix AIO will create user space threads to handle the AIO – these threads must be scheduled to run just like the main ASE thread.
ASE's issues with Linux AIO tend to revolve around the use of KAIO vs. Posix AIO. Prior to the true KAIO implementation in ASE, even when ASE was attempting to use KAIO, it did it through a bridge library, librtkaio. This library was, in theory, very nice: because KAIO availability varies by kernel, the library will use KAIO when it is available and fall back to Posix AIO when it isn't. A major issue appeared when librtkaio was no longer available on distributions based on the Linux 2.6 kernel (SLES 9, RHEL 4). With librtkaio gone, there was no longer a “bridge” between ASE and KAIO. This resulted in Posix AIO being used, with its additional threads, context switches, and other overheads.
Well, if you made it this far you must be pretty into Linux disk i/o (I think we definitely weeded out the Frisbee folks by now). In my next post I'll explain the ASE KAIO implementation and tuning considerations surrounding it.
Cheers,
Dave
Posted by David Wein on January 15, 2007 8:57 AM
Comments
Eli Perl email -
What's the latest with vxfs? Do you still need to install quick IO and create those special links?
David Wein email -
Hi Eli. I confess I don't know the specifics of vxfs support. You might want to try the sybase.public.ase.linux newsgroup, or contacting Symantec.
-Dave
