Steve Smith

About Me

Hi, I'm Steve. After working for almost ten years in software testing / quality assurance (QA) and test automation, I've made the switch to software development. This website is my attempt to showcase myself, stretch my coding muscles, and help others who are looking to get into software or web development.

Dev Projects

A selection of some of the projects I have been contributing to as a developer in my spare time.

Draw My Life

I am one of a small team of engineers who are working on this project, the aim of which is to create a library of drawings made by child refugees, so they may be used to lobby for more funding towards children’s mental health initiatives amongst refugee populations. The app is built on Ruby on Rails, and provides an API in order to provide images and data with a wider audience, including the Humanitarian Data Exchange (HDX).

Github: https://github.com/empowerhack/DrawMyLife-Service

ACB Calculator

The ACB Calculator is a web app (Ruby on Rails) I have created for a student doctor friend of mine. It is a tool used to calculate a patient’s Anticholinergic Burden (ACB) score based on the medicines they are taking. The app uses Javascript and JQuery to update a running score based on medicines entered and warn the user when the score hits a dangerous level. It also uses caching to work offline, as this is necessary in a hospital setting.

Live website: http://www.acbcalc.com

Github: https://github.com/stevesmith2609/acb-calc

UK Ultimate

UK Ultimate is the governing body of Ultimate Frisbee in the UK. I took over the technical running of their website in late 2012, and have been hosting, supporting, and providing patches and updates ever since. It is based on a Drupal framework (PHP), with some custom modules that I have expanded the functionality of. It is used by almost 4,000 members to pay annual fees and to keep up to date with UKU news and events.

Live website: http://www.ukultimate.com

CV

Download a pdf or Microsoft Word version

Steve Smith

Personal Statement

I’m a hard-working, professional developer, who has recently made the jump from testing (QA). I have over 10 years experience working on software development projects, and am now looking to move into a new position in which I can contribute to awesome projects, and progress my learning, especially in Ruby.


Development Experience (professional)

Full Stack Developer at Appear Here

One of two Full Stack Developers working on the Appear Here product, I am involved in all changes to the Ruby codebase (whether writing or reviewing), and a significant amount of the front-end (React, Javascript) changes. The code I write is clean and well tested in order to meet the high standards of the team.

As a full stack developer, I have also had a DevOps role. This involved managing servers on AWS, monitoring and fixing issues within the ETL system, investigating and resolving security concerns, and managing domains and SSL certificates.

Projects I’ve contributed towards include:

  • Providing a new search experience for users, becoming particularly familiar with Elasticsearch technologies;
  • Improving admin functionality based on feedback from admin users;
  • Importing data from external sources;
  • Bug fixes and tech support for the wider company.

Test Automation at Venntro

As a senior tester, I took responsibility for much of the development and configuration of the existing automated browser-testing suite. I was required to create new tests, update existing ones, fix bugs, refactor inefficient code, configure the CI suite, and onboard new team members so they could use it.

The suite is written in Ruby, on a Capybara framework, utilising Selenium Webdriver to control the browser. Tests were run both manually, and on a schedule from Jenkins, and Git and GitHub were used extensively to work on feature branches, and request and review changes.

Release Report Automation at Lyst

The daily release report took two hours to complete manually each day, so I took it upon myself to use some recent Rails learning to create an app to do the legwork.

The app queried the JIRA API to find information about tickets that had been completed that day, and also the GitHub API to find related pull requests (using regex to find ticket numbers). It reduced the amount of effort required to produce the report each evening down to a couple of minutes of light editing.


Development Experience (volunteer)

Draw My Life - Github Repo

I am one of a small team of engineers who are working on this project, the aim of which is to create a library of drawings made by child refugees, so they may be used to lobby for more funding for children’s mental health amongst refugee populations. The app is built on Ruby on Rails, and provides an API in order to provide images and data with a wider audience, including the Humanitarian Data Exchange (HDX).

ACB Calculator - Github Repo - Live Site

The ACB Calculator is a web app (Ruby on Rails) I have created for a student doctor friend of mine. It is used to calculate a patient’s Anticholinergic Burden (ACB) score based on their medication. The app uses Javascript to update a running score based on medicines entered and also works as an offline web app, necessary in a hospital setting. It was highly commended in the HSJ Patient Safety Awards 2017.

UK Ultimate - Live Site

UK Ultimate is the governing body of Ultimate Frisbee in the UK. I took over the technical running of their website in late 2012, and have been hosting, supporting, and providing patches and updates ever since. It is based on a Drupal framework (PHP), with custom modules that I have expanded the functionality of. It is used by almost 4,000 members to pay fees and to keep up to date with news and events.


Employment History

Apr 2017 - Present Full Stack Developer at Appear Here

Dec 2012 - Apr 2017 Director and QA Consultant of Ultimate Testing Ltd

Contract roles at Venntro (QA and Test Automation), Tribal Worldwide (Test Management), and BMT Defence Services (QA)

May 2015 - Sep 2015 QA Engineer at Lyst

Jul 2011 - Jun 2012 Senior Consultant at Deloitte

Aug 2010 - Jul 2011 Test Analyst at Serco (businesslink.gov.uk)

Sep 2007 - Aug 2010 IT Graduate / Test Analyst at AXA


Education

2004 - 2007 BSc Accounting, 2:1 classification
Cardiff University


1997 - 2004 4 ‘A’ Levels, 1 ‘AS’ Level, 10 GCSEs
DHSB, Plymouth



Interests and Hobbies

I enjoy playing Ultimate Frisbee, a non-contact team sport I picked up at university, and have been playing ever since. It is great exercise, and I really enjoy being part of a team. I play for Thundering Herd, a mixed-gender team based in Clapham. I am also a keen traveller, both for Frisbee tournaments, and for my own pleasure.

Blog Posts

Originally posted at http://dev.venntro.com/2016/11/reporting-cucumber-results-in-slack/

Reporting Cucumber Results in Slack

Examples from this blog post currently run on a front-end automation suite running Ruby 2.2.0.

Increasingly, we’ve moved the running of our automation testing from our development machines to an instance of Jenkins living on a dedicated CI server. This frees up our development machines (and their limited capacity) for us to use in manual testing, exploratory testing and test automation development. It also gives us a consistent environment from which to run tests, and the ability to schedule testing overnight while the test environments are not being used.

Although it has its advantages, this does somewhat create an undesired separation between our testers and the test results being reported for them. As we are a development team that loves using Slack, we’ve created a way of reporting these test results directly to our chosen messaging app. I’ll take you through how we’ve accomplished this for our regular Cucumber test runs and for more advanced Parallel Cucumber test runs.

Cucumber

Using the cucumber gem

We want to report our results at the scenario level, as feature results are too high-level, and individual step results are too low-level. My original thought was to parse the json output from the Cucumber gem to find the results I required, but as the json output works only at the step level, it would be time-consuming to create a tool to translate this into scenario results. Fortunately, the html output of cucumber already does this for us.

The standard html output from Cucumber has this header:

Header of html output of Cucumber gem

I’ve found it’s a lot easier to parse the html of this output to grab the scenario statistics from the top line, “27 scenarios (2 failed, 1 skipped, 24 passed)” than it is to infer it from hundreds of individual step results.

def get_run_stats_standard(report)
  stats = { passed: 0, failed: 0, skipped: 0, undefined: 0 }
  file = File.read(report)
  scenarios_line = file.lines.last.split('innerHTML = "').last.split('<br />').first
  stats.keys.each do |state|
    if scenarios_line.include? state.to_s
      stats[state] = scenarios_line[/(\d+) #{state.to_s}/, 1].to_i
    end
  end
end

We set all stats to 0 before reading the output generated by Cucumber and extracting the first line of text from the last line in the file (although it is in the header, the statistics are actually in the last line of the html file). scenarios_line in this instance equals "27 scenarios (2 failed, 1 skipped, 24 passed)". For each of the states that the scenario could be (passed, failed, skipped, and undefined*), we run a check to see if it occurs in the string; if it does, we grab the number from the string using regex, and if it doesn’t, we leave it at 0.

The output of this is a handy hash of the scenario results:

{ passed: 24, failed: 2, skipped: 1, undefined: 0 }

* In cases where a step in a feature file is not defined in any step definition file, although the scenario is skipped, it is reported as “undefined”.

Parallel Cucumber

Using the parallel_tests gem and report_builder gem

The standard output of report_builder gem is a rich html file to display results:

HTML output of report_builder gem

This is nice to look at, and we could parse the html as before, but we are also provided with an exceptionally helpful array of statistics, which is perfect for our use case. Here is an example of the result of running output = ReportBuilder.build_report on a recent run:

[
  288583677059,
  [
    {:name=>"broken", :count=>1, :color=>"#f45b5b"},
    {:name=>"incomplete", :count=>1, :color=>"#e7a35c"}
  ],
  [
    {:name=>"failed", :count=>1, :color=>"#f45b5b"},
    {:name=>"passed", :count=>6, :color=>"#90ed7d"},
    {:name=>"skipped", :count=>1, :color=>"#7cb5ec"},
    {:name=>"undefined", :count=>1, :color=>"#e4d354"}
  ],
  [
    {:name=>"passed", :count=>68, :color=>"#90ed7d"},
    {:name=>"failed", :count=>1, :color=>"#f45b5b"},
    {:name=>"skipped", :count=>5, :color=>"#7cb5ec"},
    {:name=>"undefined", :count=>1, :color=>"#e4d354"}
  ]
]

After the first value of this array (duration in nanoseconds), the second value is an array of feature results, the third is scenario results in an array, and the fourth is an array of individual step results.

def get_run_stats_parallel(report)
  stats = { passed: 0, failed: 0, skipped: 0, undefined: 0 }
  stats.keys.each do |state|
    unless report[2].select { |status| status[:name] == state.to_s }.empty?
      stats[state] = report[2].select { |status| status[:name] == state.to_s }.first[:count]
    end
  end
end

As before, we set all results to 0 by default, then for each of the possible scenario states, check the third value of the report to see if any hash within it has a :name matching the state. If not, we leave the count as 0, and if so, we grab the :count value of that hash.

The result is a now-familiar hash:

{ passed: 6, failed: 1, skipped: 1, undefined: 1 }

Sending Results to Slack

Once we have extracted the results, and manipulated them into a human readable format, we use the Slack API to send them to Slack. I won’t go into the full details of how to do this, as the API is well documented by Slack, but this method will get you most of the way there:

def post_to_slack(msg_text)
  uri = URI.parse("https://slack.com/api/chat.postMessage")
  Net::HTTP.post_form(uri, {
    "token" => SLACK_TOKEN,
    "channel" => SLACK_CHANNEL,
    "attachments" => [{
      text: msg_text,
      fallback: msg_text,
      color: "good",
      mrkdwn_in: ["text", "fallback"]
    }].to_json,
    "link_names" => 1,
    "username" => SLACK_USERNAME,
    "as_user" => false,
    "icon_url" => LOGO_URL
  })
end

Obviously, you’ll need to provide your own values for SLACK_TOKEN, SLACK_CHANNEL, SLACK_USERNAME, and LOGO_URL, depending on your own implementation of Slack. For SLACK_CHANNEL, you could even have it defined as a Project Parameter within Jenkins as we do, so that users can choose their own channel to report to when they kick off a build.

Results

Each night, we run our full automation suite of almost 400 test scenarios against our two staging environments. We run them in three batches:

  1. We run as many as possible using the parallel_tests gem, running 4 streams of test simultaneously for speed. We record all failures in a txt file.

  2. We re-run the failing scenarios from the first run using the standard cucumber gem (i.e. sequentially). Most of the time, failures identified in the first run happen because a simultaneous test has disrupted the test environment configuration required by the test.

  3. We run a final set of scenarios through the standard cucumber gem sequentially. These are scenarios which require the environment to be configured in a very specific way, and so have been tagged specifically for their own sequential test run.

The results of each are fed through their respective get_run_stats_* method, amalgamated, and then sent to Slack using the method above to come up with our nightly report:

Results of our nightly CI run displayed in Slack

In the morning following this run, a designated member of the team will investigate these failures, report on the cause (whether it be environmental, a bug in the tests, or a bug on one of the branches deployed to that staging environment), and take any further action as necessary to fix the bugs or improve reliability of the test suite.

However, it’s not just the large nightly runs that can take advantage of these methods; we send the results of all Jenkins builds through this code, so anyone can be given an instantaneous glimpse of how any run they’ve kicked off has done, however small.

Results of a custom automation run displayed in Slack

Originally posted at http://dev.venntro.com/2016/10/test-automation-taking-command-of-the-command-line/

Test Automation: Taking Command of the Command Line

Examples from this blog post currently run on a front-end automation suite running Ruby 2.2.0. Jenkins is used as a CI server to run tests from.

Background

The automation test suite here at Venntro has been growing steadily over the past few years, and is now able to test most of the user-facing features on the platform. However, there are some features which have always been difficult for us to automate tests for.

Search and member feeds (which members we recommend to the user on different parts of the site), have been notoriously difficult because once you make changes to the data, they need to be reindexed by the Sphinx search server before changes are actually effected on the front-end. This runs every few minutes on the test server, but as any test automation specialist will tell you, a few seconds is too long for a test to be waiting, let alone a few minutes!

We needed to be able to force this index to be rebuilt, which required us to be able to access the “scary world” of the command line on the test environment host server.

Evolution

First Iteration: Basic Commands

It turns out Ruby has many ways to access the command line built-in, and the most simple way to do this was to just encase the command we wanted to run in backticks:

def reindex_all_sphinx_indices do
  `ssh subdomain.hostname.com 'sudo /usr/bin/indexer --rotate --all'`
end

One of the documented disadvantages of using this method is that it blocks processing until the command is complete. Thankfully, for a test suite, this actually becomes an advantage; we don’t actually want the test to continue running until the indexing is complete.

Second Iteration: Combatting Stalls

It didn’t take long using this method before tests started to get stuck at this point. Unfortunately, there are many things that can happen when using the command line to cause your test to stall. For example, our script uses a sudo command, and if the user running the test does not have sudo access for the action specified, the command line will prompt for a password. As this task is being run in the background by Ruby, you will never see this prompt.

To combat stalling we reused a tactic from our front-end tests, wrapping the call in a simple timeout:

def reindex_all_sphinx_indices do
  Timeout::timeout(30) do
    `ssh subdomain.hostname.com 'sudo /usr/bin/indexer --rotate --all'`
  end
end

So, if the call takes more than 30 seconds, the test will abort as a failure, and the cause for it taking so long can be investigated.

Third Iteration: Output

Debugging was the next issue to address. It was great that we now had the ability to run command line operations in our test suite, and a way to deal with the most crippling issue, stalling, but there were other issues that needed to be debugged. If a test failed for a missed expectation after appearing to have executed the command without fail, we had no way to see what had actually happened during the call.

The command to rotate all the Sphinx indices is an interesting one to inspect. It runs through each Sphinx index, performs the rotation, and moves on to the next regardless of whether it successfully performed the rotation or not. Even if a number of the rotations fail, and these failures are shown in the output to the user, once the command is completed, Ruby does not understand these fails. As far as it can tell, the command was completed successfully.

It turns out getting hold of this output is as easy as assigning it to a variable, so that’s what I did, and immediately output it using the puts method.

def reindex_all_sphinx_indices do
  Timeout::timeout(30) do
    output = `ssh subdomain.hostname.com 'sudo /usr/bin/indexer --rotate --all'`
    puts output
  end
end

Not long after implementing this functionality, our test environment was rebuilt on a new server. By seeing this output in our test automation console output, I could alert our Operations team to the fact that the Sphinx user being used did not have access to a specific database, which was causing the index rotation to quietly fail unbeknownst to anyone not closely monitoring the logs.

This worked for some specific scenarios, but one of the review comments I got from my colleague when I proposed it was:

Matt points out this will not work if the call times out

I tested this by reducing the timeout down to a few seconds, and sure enough found that when using this backticks methods, the output will only be written to the variable output on completion of the call. Even if the timeout does its job, and puts output is moved to a rescue block, the variable output will be empty.

It was time to dig deeper into Ruby’s array of command line tools.

Fourth Iteration: IO.pipes

After a bit of research into the challenge, I stumbled across an awesome StackOverflow answer, which gave a fantastic basis for what would eventually be the method we use to execute command line operations:

def exec_with_timeout(cmd, timeout = 30)
  rout, wout = IO.pipe
  rerr, werr = IO.pipe
  stdout, stderr = nil

  begin
    puts "Executing: #{cmd}"

    pid = Process.spawn(cmd, pgroup: true, :out => wout, :err => werr)

    Timeout.timeout(timeout) do
      Process.waitpid(pid)

      wout.close
      werr.close

      stdout = rout.readlines.join
      stderr = rerr.readlines.join
    end
  rescue Timeout::Error
    Process.kill('TERM', pid)
    Process.detach(pid)

    wout.close
    werr.close

    timedout = true
    stdout = rout.readlines.join
    stderr = rerr.readlines.join
  ensure
    wout.close unless wout.closed?
    werr.close unless werr.closed?

    rout.close
    rerr.close

    puts "OUTPUT: #{stdout}" if stdout.present?
    puts "ERRORS: #{stderr}" if stderr.present?
    raise "Execution timed out: #{cmd}" if timedout
  end
end

There’s quite a lot going on, so I’ll explain piece by piece.

The Setup and Begin block

rout, wout = IO.pipe
rerr, werr = IO.pipe
stdout, stderr = nil

begin
  puts "Executing: #{cmd}"

  pid = Process.spawn(cmd, pgroup: true, :out => wout, :err => werr)

  Timeout.timeout(timeout) do
    Process.waitpid(pid)

    wout.close
    werr.close

    stdout = rout.readlines.join
    stderr = rerr.readlines.join
  end

We start by creating two IO pipes. One will record STDOUT (standard output stream from the command), and one will record STDERR (standard error stream). We then set variables stdout and stderr to nil.

After printing out the command being executed, we spawn a new Ruby Process to run the command given. We make a new process group to run it using pgroup: true, and assign the write ends of our two new IO.pipes to record the output and error streams. We store the process number as variable pid.

Next, we start our timeout block and use Process.waitpid(pid) to wait for the process we spawned to stop. If the time taken to do this does not exceed the number of seconds specified (30 by default), we close the two write ends of the IO.pipes (we cannot read from the read ends if not), and then read the read ends of each pipe, assigning them to the specified variables to print out later.

The Rescue block

rescue Timeout::Error
  Process.kill('TERM', pid)
  Process.detach(pid)

  wout.close
  werr.close

  timedout = true
  stdout = rout.readlines.join
  stderr = rerr.readlines.join

If the Begin block does time out, the rescue block will clean up. Firstly, it will terminate and detach the process we started so it does not either become a zombie process or affect any other tests unexpectedly. Then a similar process of closing and reading the IO pipes is performed. We also set a local variable, timedout to true for use in error reporting later.

The Ensure block

ensure
  wout.close unless wout.closed?
  werr.close unless werr.closed?

  rout.close
  rerr.close

  puts "OUTPUT: #{stdout}" if stdout.present?
  puts "ERRORS: #{stderr}" if stderr.present?
  raise "Execution timed out: #{cmd}" if timedout
end

The ensure block is run no matter the outcome of the Begin or Rescue blocks, and simply closes any ends of IO pipes that may still be open, and then prints out all the output and all the errors to the console if they exist.

Finally, a raise is performed if the rescue block was run. This will immediately fail the test being run, with a very clear message; this particular command being run took an unexpectedly long time to complete. This, along with all the output and errors that were written right up to the time of failure or timeout should give all the information needed to analyse the output.

Hints & Tips

Access

When running commands on a remote host, whether you have access to run those commands is going to be an issue. You should investigate ssh access, tty, known hosts, and use of the sudo command; you will probably need to have a discussion with your Ops team to discuss which commands need running, by which users, and how to do that in a secure way. I found a useful way to ensure if your CI user has access to run a command was to ssh to the CI server as the user which runs automation (in our case, Jenkins uses the default user ‘jenkins’) and trying to run the command.

Test candidates

Any test which requires waiting for a cron job to do something is a prime candidate for accessing the command line and kicking off the process yourself, as cron jobs do not by default run at intervals smaller than 1 minute. I’ve also found that if you need to hit a url in order to trigger an action, but don’t want your test to navigate away from the page you’re currently on, using this method to run a curl command to hit the url works perfectly.

Tagging scenarios

We use Cucumber to organise and execute our test scenarios, which supports tagging. We tag scenarios depending on which command line operations are run during the test, for example, @ssh, so they can be run as a batch if required, or excluded from test runs if we are experiencing issues with ssh.

Conditionally printed output

If you don’t want all the output printed on every test run, with a bit of tweaking you could:

  • Only print the output if the call fails or the timeout is exceeded
  • Add an additional argument to the exec_with_timeout method to control whether output is printed or not
  • Create an environment variable to control whether output is printed or not

Learning Resources

In my journey to become a developer, I have followed several online courses; the following are those I would highly recommend for others.

12 in 12 Ruby on Rails Challenge

Mackenzie Child's 12 in 12 Ruby on Rails Challenge is a course of 12 challenges to build a fully functional web app using Rails. The intention is to complete one challenge each week. The first challenge takes about 6 hours, but stick with it, and as you become more competent with Rails, it'll reduce until you can keep up pace with Mackenzie's walkthroughs.

Videos: https://www.youtube.com/playlist?list=PL23ZvcdS3XPLNdRYB_QyomQsShx59tpc-

Javascript30

Javascript30 is a course of 30 javascript-based "things to build" created by Wes Bos. Each challenge takes about 30 minutes to complete, and teaches vanilla javascript rather than relying on other frameworks or libraries. A lot of what I've learnt through this course has helped me with the javascript and styling on this website.

Website: http://www.javascript30.com