Automate Your Browser Testing

This was a talk given at the 2016 Des Moines Web Geeks Social Event.

Slides

The talk’s slides are embedded below. Use the left/right arrow keys to navigate.

The slides can also be viewed at eheikes.github.io/browser-testing-talk. Its code repository is available at github.com/eheikes/browser-testing-talk.

Edited Transcript

I’m going to talk about automating your browser testing. I’ll quickly introduce myself. My name is Eric Heikes. I’ve been focusing on front-end development the last few years, so that’s stuff like Javascript, HTML, CSS. I work at Jack Henry & Associates, which is a financial services company. Specifically, I work on the Banno team, which is a platform for financial institutions to interact with and serve their customers. That’s things like setting up marketing campaigns and seeing the metrics on that; getting information on customer accounts; building forms and getting the submission from those forms, and tracking the customers there; and then there’s also lots of other projects, like native mobile apps, that I don’t work on. But we’re always looking for smart people to join our team, so if you work in the front-end or even things like iOS or Android development, infrastructure, product design — I think we’ve doubled, since I’ve worked there, our front-end team. Maybe tripled it, so we’re growing rapidly. So that’s kind of the stuff that I work on.

I think that most people would say that writing test is kind of a chore. It’s not something that most developers look forward to, especially if you’re building a new product, or building out a new feature. It’s really easy to kind of say “let’s just get this out the door, we can add tests later”. But, I think most people would also agree that having some tests is better than no testing at all. You want your website or web app to be available for customers and to be working the way that they expect. You don’t want to push out a change and then a week later have a customer call you to let you know that the change that you made has broken the website. But, by the same token, you don’t want to spend all your time testing your website manually. That’s just very tedious, and also very error prone, since there’s the possibility of human error there.

So how can we make this as painless as possible for developers? Unfortunately, you still have to write tests and do some setup to get the tests running. But — good news everyone — once you get that set up then you can just hand that off to the machines and have them run tests for you.

As an example, when I joined Banno 2 years ago, a few of the products were initially built, and there was a sort of a shared library that contained a lot of the common components and services for these apps. But there wasn’t a lot of test coverage for these. Over the next few months or so, we spent some time writing unit tests and browser tests, and nowadays we’ve got almost full test coverage on the shared library. There’s about 342 unit tests that run in 8 seconds. On the browser side — Matt, you can go ahead an play that in full screen. This is an example of running browser tests. It runs the browser tests, here in Chrome. The shared library includes a lot of the components like the login screen and selecting an institution, a lot of things that are shared across the apps, and some other pieces. I won’t play the whole thing. It takes about 5 minutes to run all of the tests here, and it runs basically through all of the different scenarios that we want the tests to cover.

There’s kind of a spectrum of testing, and the nomenclature that’s used varies depending on who you talk to. In general, there’s these low-level tests where you’re testing directly against the code. That’s called unit testing. What I’m going to focus on in this talk is end-to-end testing, which is the very high-level testing of the website or its components. Basically, like what the end-user would see in the browser, and the workflow that they would use. That’s what I’m going to talk about, mainly because unit testing, I think, is a little easier to set up and write tests for. In my experience, end-to-end testing is usually a little trickier, and involves a little more thinking in how to get those set up. So I’m going to talk about very high-level testing, over those workflows and the website features.

If we want to test how a website works in the browser, then we need some sort of way to communicate with the browser directly. There’s this really nice tool called Selenium WebDriver. It actually used to be two separate projects and they merged. It provides this interface between you and your tests and the browsers. It’s the software that runs on your machine, and then you send it the commands you want to run on the browser, and then it communicates with the browser to run those commands. What’s really nice is that it works on basically any computer. There’s multiple language bindings, so any language you write in you can use with that. It also supports pretty much any browser that you’d want to test against, including IE, and the new Edge browser.

So we’ve got Selenium working to help automate these browser tasks, but we still need some sort of framework to write the tests in and then run the tests. There’s a huge amount of frameworks out there. This is not going to be an exhaustive list at all. I work mainly in Javascript, so the things I know about are things like Nemo, Nightwatch, Protractor, and webdriver.io. If you work in Ruby, there’s things like Capybara. Java has JUnit and TestNG. .Net has NUnit. And there’s others out there. There’s also a thing called Selenium IDE, which I haven’t worked with personally, but it’s a Firefox plugin that gives you a graphical interface that lets you do the same thing. So if you’re not a programmer, you might want to check that out. That way you can write some tests with that plugin and then run those from there.

My experience has been with Nightwatch and Protractor. For both personal projects and professional projects, that’s what I’ve used. At Banno, many of our apps are written in Angular, so we use Protractor for those end-to-end tests, but for things that aren’t in Angular, and for some personal projects, I’ve used Nightwatch. I’m going to use Nightwatch as an example here. Whichever framework you choose, the ideas are very similar; my specific examples are going to be in Nightwatch. It’s in NodeJS. The tool includes everything you need to get started. It seems like it’s fairly mature and popular, and fairly easy to get started.

If you want to write some end-to-end tests, the first step that you need to do is to actually install the browsers that we want to test against. So, if you want to test your website in Chrome, then you’ll need to have Chrome installed. If you want to test in Firefox, you need to install Firefox on that machine. If you want to test IE, you’ll need to have IE installed. I’ll talk about that more when I get to the cloud testing.

Next you’ll need to get Selenium running on your machine. This is just an example using NodeJS and then gulp for task running. The details here are not important. You can download the Selenium tool from the website and get that installed and run easily no matter what your platform is. This just shows it downloading the Selenium standalone module and then having some tasks to get that installed and start the Selenium software.

Once you’ve got Selenium running, then you’ll need to set up your test framework. Here’s an example of Nightwatch, again in gulp. The only important part really is the Nightwatch command. That bit right there — Nightwatch is just a command line tool, so just run that on the command line. And then you pass the config file that you want to use. All test frameworks probably follow a similar setup. The rest of that’s just handling the output and checking for errors.

Every test framework has its own configuration syntax. This is the one for Nightwatch. The most important thing is pointing to the folder that has your test files in it. Nightwatch also has an output folder option that can save your report after the tests are run — I think it’s an XML format. Then there’s other things like the default URL that you want to test against, the pattern of the test files, and then every configuration is also going to have some place where you specify the browsers that you want to test against. Here there’s the “desiredCapabilities” option, and that’s got Chrome listed. So the default setting here is just going to test against Chrome. Each browser usually has its own specific options that can be tweaked as well.

Now that you’ve got that set up, then you have to actually start writing your tests. And all end-to-end tests follow the same pattern. The first thing you do in each test, what you do is you navigate to the webpage that you want to test against. Then you locate the elements on the page that you want to test. Usually you’re going to perform some actions on that page as well, whether it’s logging in or clicking on a button or something. This is the exact same way that a user would. And then you run your assertions to test that the page is acting the way it should.

Here’s a short example that actually tests against this presentation itself, in Nightwatch. Most test frameworks will also have a browser instance that you have access to that you use. So there’s the init() method first, that loads the default page. You can also load a different page that’s not the default, if you wish. Then there’s a method called waitForElementVisible() that waits until there’s a “div” element that has the “flowtime” CSS class on it, and it waits for 1 second for that to happen. Then there’s 2 assertions. The first assertion checks that there’s an “h1” element that’s visible, and then the second assertion checks that the h1 element contains the text “Automate Your Browser Testing”. And then Nightwatch has an end() method that you have to call when you’re done to clean up after it and shut everything down. So that’s a very simple test. This is me running it. I’m using the gulp “test” task, which runs Nightwatch. First it sets up Selenium, then it runs Nightwatch. And you can see there the 3 green checkmarks that means all three of my tests passed. The browser opens up real quick in the background because it only takes 3 seconds, so there’s not a lot to see there. But that’s it running.

There’s many other ways to interact with the browser. It’s going to depend on the test framework that you choose, what the specific syntax and methods are. This is a link to the Nightwatch documentation. As you can see on the left there, there’s actually 2 different styles of assertions that you can use. There’s the regular “assert” commands, and then there’s also the “expect” which is sort of a BDD style. So the assertions are your standard kind of testing things, like you want to check to make sure that an attribute exists or has a certain value, or that a CSS style exists, or whether an element is present or even visible on the page… things like that. Most testing frameworks will also have commands. These are not assertions, but they’re other ways to interact with the page. There’s things like clicking on something, you can manipulate cookies, get the title of the browser window/page, things like that. And again, the syntax is going to be specific to the testing framework, but these are all similar from framework to framework. Most of them also give you access to the Selenium driver itself, so if you need more low-level control, you can access that, usually, from these frameworks. Things like being able to check which mouse buttons are being pressed and stuff like that.

That’s running end-to-end tests on your machine with the browsers that are installed on your machine. Of course, you probably don’t want to install all these browsers on every developer’s machine. In many cases, that’s not even possible. Also, end-to-end tests run slower than unit tests typically, so if you’re in the middle of making some changes to your code, you don’t want to be interrupted and wait while the end-to-end tests run. Then, of course, there’s continuous integration. You want to make sure that your builds are automated and as fast as possible, so there are these third-party services that you can use to run these tests for you, on their servers.

Again, there’s multiple services available. Here are some of the major ones that I know of: Browserling, BrowserStack, Sauce Labs, TestingBot. At Banno we use Sauce Labs, so I’ll show you some examples using that, but again, whichever you one your choose, they’re all going to work similarly I think. They are paid services — some of them might have free tiers to them, but you are going to have to pay, usually based on the amount of time that you use for testing.

So here’s that same Nightwatch configuration, but using Sauce Labs. There’s still a default configuration there, but now there’s also a “saucelabs” configuration, and that has some extra options there that point to the Sauce Labs server. There’s probably going to be a username and password or access key that you need to use. In this case, it’s going to come from the environment so that the password doesn’t get committed into source control. It can also be specified if you’re running it in your CI environment.

Another problem comes up. Most of the time the websites that you want test against are not public websites, or not up-to-the-minute, having the changes that you just made. So these third-party services don’t have access to localhost obviously, and your private builds. So in the example, Sauce Labs, what they have is something called Sauce Connect. This is just a quote from their website, and what it does is create a tunnel between their servers and your machine that’s doing the testing. That’s pretty to set up, it’s just another tool that you download and install on your machine. Here, it’s a command line tool called “sc” that you just run with your username and your access key.

So here it shows an example of it running on my local machine. In the control panel here for Sauce Labs, you can see that there aren’t any tunnels set up. Then I switch over to the console and run the Sauce Connect tool with the username and password for the account. It takes a few seconds to establish that secure tunnel between their server and to you…. Okay, so it will set up there, and then once it’s ready it will say that last line there: “Sauce Connect is up, you may start your tests”. And there in the control panel you can see that there’s a tunnel set up. I didn’t give it a name, so it just says “Unnamed tunnel”. Then I can just run my tests as I do normally. You don’t see it in this screenshot, but it is using the “saucelabs” environment. It does the same thing, except it’s using the Selenium server with Sauce Labs. And there in the control panel you can see it running. In my console you can see the same output of the tests running. Then in the control panel there you can see the tests. What’s nice is that they give you some tools to look at your test runs. So there’s a video here, where you can see it running. It’s only a few seconds long, so there’s not much to see. There’s also a timeline that you can click through — the different commands to the Selenium server — so you can see screenshots as the tests are being run. There’s also logs. If there’s an error or something you want to check the logs of the Selenium server. When you’re done, you can just close that tunnel. And there you can see that it’s closed.

That’s running Sauce Connect on my local machine. Obviously if you’re running in a CI environment, you’ll want to use that. Here’s a Jenkins plugin that does pretty much the same thing, except it does it on your Jenkins server. So you just install that plugin and configure it, and run all of your end-to-end tests in Sauce Labs on your Jenkins server.

So, getting started with end-to-end testing… You don’t have to jump in and write all of the tests right at once and get things all set up. There are some ways to dip your toe into it. First of all, take a look at the testing frameworks that are out there for your platform and your language. There’s probably 1 or 2 really popular ones to look at. Pick the one that appeals to you. Start small. You can write tests for just a few major features or critical paths that are most important to your product. You can also just start testing on your local machine, like with Chrome, just to make sure that you have automated testing for at least one browser. Or maybe choose problematic browsers that you want to test against. And of course, as problems arise and get fixed, you can just add a test. That way, it guards against regressions and any mistakes that may break other parts of the website.

It’s been really helpful using end-to-end testing. Definitely at Banno, it’s been a huge burden off of us, being worried about any little change that might break other parts of the website. It guards against that, it guards against any regressions. And once you get some tests in, it’s pretty easy to add or change tests as you add or change features.

So there’s a link to the talk, and there’s my Twitter handle. I’ll post a link in there as well.