Watir vs Selenium vs SahiOut of these 3, which have you or has your company chosen to work with? Pros & Cons please. I'll be comparing them myself as well, but I'd like to hear what others have to say.
Also, please state which you have all tried (so that I know you have a good comparison of the 3).
ACCEPTED]
I made a presentation for a seminar I had to hold for our R&D team. Here is basically what I came up with:
Pros:
Cons:
Pros:
Cons:
Pros:
Cons:
There are a lot of other small stuff that I didn't include.
I decided to use Selenium because I like the freedom that you get with all the languages it supports and because development is usually quicker because of its IDE recorder.
Edit Jul 3 '09: I have done some of the same tasks in Watir and they seem to be more stable and straightforward in Watir (written in Ruby) than what I wrote in Selenium (in Java). Selenium has a lot of timing problems but Watir automatically waits for the page to finish loading, and can detect when it's finished loading. Whereas with Selenium, you have to use the pause command (or Thread.sleep() if you use it in Java).
Edit Feb 7 '14: The info presented here is really old and is probably outdated. Also, there exist many more solutions for automated testing in the browser now.
One problem I have found with Selenium is that it is very difficult to test complex multi-site apps. My company has an app that uses a login process via Facebook (using the Facebook API). In Selenium 1.0, which uses Javascript as its enabling engine, the single domain policy is enforced so that when I open my main site (http://beta.mysite.com) and then click on the Connect to Facebook button which opens a window in the www.facebook.com domain for logging in, my Selenium process loses the connection between the two windows and the app locks up at login. I have tried many different approaches to solve this and have not come up with an answer.
I switched to Watir because it provides an "attach" method which lets me programmatically access both the original domain window and the facebook window from the same test.
Also, as noted by several people, Selenium has timing issues. It is possible to use clickAndWait if a click is known to load a new page, but in our app, there are situations where clicking MAY cause an Ajax call and a dynamic in-page change in which case you do NOT want to clickAndWait (because your test app will hang waiting for a page to load that will never load) or it may cause a page load/reload, in which case NOT waiting is suicide because your app will simply run past the situation and start doing things before the page is fully loaded and ready to go.
If you can't tell which is going to happen (and in our case, you can't without introspecting the javascript which is MUY DIFICIL), you are up the proverbial creek without a paddle.
(Yes, I know, who in the hell would write javascript that does one or the other alternatively depending on certain contexts? I certainly wouldn't have written it that way and I will not defend the implementation of the app, I just have to figure out how to test it LOL!)
Anyway, this is also relatively easy to handle in Watir since you just let it decide whether to wait or not. To be fair, Selenium 2.0 (using WebDriver) will fix some of these problems, but unfortunately, the API will then be different. So its not an easy thing to figure out.
BTW, I was "driving" the Selenium API with Python and it took me about 2 hours to learn enough Ruby to be dangerous LOL and also to use Watir. I have a background in Smalltalk and Objective-C and Ruby comes quite second-naturedly (as did Python). In fact, I expect I am going to end up on the fence, because there are some awesomely nice, robust and symmetric things in Ruby that are much better than Python, but Python also has its pluses. I still think code is cleaner in Python and easier to read for the most part, but as for power (particularly in terms of metaprogramming), I think Ruby is the better engine.
Jon
I haven't looked into Sahi, but I can tell you why we have recently chosen Selenium RC over watir. It all comes down to programming language. We are a Java shop, and no matter how easy or great Ruby is, it is much easier to coax developers into writing tests if they don't have to change contexts or IDEs. We found functionality to be similar and had a preference for watir's object model, but being able to write Selenium RC tests in Java gave it the edge.
If you are looking at open-source web testing, you should also consider webdriver. It is a good alternative to Selenium RC, and is actually going to become Selenium 2.0 at some point in the future.
I have to agree with Dennis that the pro-Sahi comments are probably from the Sahi team or some other interested party. When we tested Sahi in our shop a few months ago, it performed very poorly in comparison to Selenium. I don't know if it has improved, but at that point in time it was definitely not a usable product.
I'm using Sahi and I think it very powerfull tool for functional testing. My opinions follows:
Pros:
Cons:
In conclusions a good choice for me ^_^ thanks Narayan
Roby
P.S: I tested also watin, selenium (integrated into Visual Studion using C# library).
Just to add another to the realm: We use iMacros because of its good Flash/Ajax testing support: http://wiki.imacros.net/Selenium
Over the last year I've tried two of these tools and the clear winner for me was Sahi.
I started with Selenium. I really liked the concept and wanted badly for it to work, as my alternative at the office was QuickTest Pro (absolutely horrible for web work (and arguably any work)). Unfortunately, selenium doesn't support IE, and though it pains me to say, that is a deal breaker for my work. But on top of that, I wasn't able to completely automate much without MANY difficulties; I found Selenium needed a lot of hand holding. I eventually had to admit defeat with Selenium and look elsewhere.
I then found Sahi... which is similar in concept to Selenium but also supported IE (and all browsers) and was able to successfully automate a few things on my first try. I also liked that Sahi's script language was mostly javascript (with "$"s on vars) and that it's built in library handles much of the heavy lifting. It's facility to access objects with _near and _under is genius! It's light weight and agile... perfect for my needs.
The biggest downside of Sahi to me is the controller can be a bit quirky... but once you realize how it works and get used to it, it's not a problem. A small price to pay for the power it gives you. Everything else has just worked.
I would like to see more folks use Sahi as I want it to stay around. I honestly feel it's a better tool than Selenium currently. I hope more people give it a try.
