Tuesday, May 24, 2005

The Complexity of Modern SoftwareTest Automation

This will be a recurring theme here so get used to it. I have a lot of time on my hands right now as I'm on temporary hiatus.

The issue of automation in all its forms has always been exciting to me. I started out as a Mechanical Engineer hoping to enter into the field of Robotics - boy what a hard field to break into.Anyway, since I went to college in the Seattle area - HOW THE HELL DID I END UP THERE??? - I decided, or my financial necessity decided to give Microsoft a try - boy do I miss the campus.
I started out in USB\FireWire for Windows 98 and to this day say that I have the distinction of being the FIRST PERSON in the world to actively test WDM ( Windows 98 drivers on Windows 2K). Anyway after hearing that my manager - who went on to become Manager of WQHL - actually started out as a contractor and became the writer of the WORLD'S FIRST USB driver for OSR2, I began to rekinde my desire to write code - being 40 I actually programmed one of those old fashioned terminals with the printer and phone connection to a mainframe.

I guess it could have been punch cards!!

After a few years in HW I moved to Windows Management. My first project was consolidating and maintaining batch files for SAF\HelpCenter BVTs in WinME. I finished that pretty quick and noticed that it only downloaded the bits and ran the install. That meant the tester still had to reboot to DOS and run the batch file after downloading from the network i the GUI.

Can you say "COULD BE BETTER?"

I then proceeded to take my limited VB knowledge and apply it to creating a UI. For my first one it was pretty good. Of course error handling was close enough to being non-existent that it was a royal pain in the ass.

Oh no not the dreaded 'Runtime Error -5".

It had a pretty rich feature set though. It enabled downloads from all available build labs in all languages; created an autoexec.bat and unattend.inf on the fly; rebooted the machine after prompting for the floppy; autoloaded the test binaries and ran the appropriate HelpCenter page; recorded the time each test mod was run. My previous manager "commended" me on the fact that only the ME setup team completed their BVTs ahead of us.
A rousing success. And with VB 6 UI elements and VBScript logic, no less!!!

I then left automation for awhile as I volunteered myself onto a team with a major hemorrhoid for a manager. After months of dead ends and expletives carefully deleted, I went back to automation where I encountered the greatest conflict ever, THAT OF CMD LINE VS UI test harnesses. The SHELL gets a bad rap for being a resource hog and slower than csrss, but after using C#, I can say that if the CMD line is faster - OH MY GOD!!! - I get nearly instantaneous execution of some pretty complex event based tests.

Anyway this conflict caused me no end of grief. As a former tester, I grew to hate long param lists that - get this - HAD TO BE ORDERED so I felt that I shoul dgo the extra mile to try to get as much speed as possible with more emphasis on ease of use. I WISH I HAD READ CODE COMPLETE then. Unfortunately, my minimal experience with C\C++ gave me little access to "convincing arguments." Even more unfortunate was the Redmond weather which took its toll and eventually ended my "sojourn."

More later.......


Anonymous said...

Was the name of the WHQL manager Jerry C. ?

Anonymous said...

The problem I always run into with UI automation is that the events (error pop-ups) that you need to watch for are too diverse and poorly documents for me to proactivley code for all of them.

I kept running into bugs that required a lot of manual research to find what was breaking the automation and then I could add yet another dialog handler because the automation was expected to be "robust" even though the code we were testing wasn't.