EuroSTAR Conferences Blog

This blog will focus on discovering context, a key aspect of all software testing.  Context is king, and essential to understand what you are testing, how it could be used, and how it could fail. It is also important to understand the dynamics of projects, organisations and even metrics. Even something as seemingly simple as time can have various contexts - the time of day (core hours, out of hours), time of week (business hours, weekend), time of season (peak, off-peak), time of year (weekly, fortnightly, monthly, etc.), or an unpredictable time.  Usage is another important context, a newbie, typical or veteran user could be using a system in a regular, irregular or extreme fashion. As we test more, we learn more about these patterns (at least our subconscious brains do) and our context discovering becomes more advanced. 

Lets start out by looking at a most extreme example of discovering context.  Wilson Greatbatch was a scientist investigating a device to record heart rhythms, who accidentally used a massively overpowered resistor. Instead of recording, it pulsed with a slow regular rhythm. The invention became the pacemaker, replacing a giant fixed machine.  Harry Coover was a military research scientist, who tried using a new mystery compound in gun sights without success then later in aircraft cockpit canopies without success, because the substance quickly became a gooey mess.  It took a while before he also realised it bonded things together with an incredible strength. Of course, we now all know it as superglue!

At least with software and projects our contexts are more easily discoverable (but not always!). Can anyone think of other extreme examples of discovering context?

Hmh... wrong password, so I submit it again. And my password is stolen. Why?

I'm calling myself evil tester. For many people it sounds negative term but I think it describes my testing style very well. Evil tester is not evil for people but for software.He also knows how criminals could take benefit from software problems.

For example I've found lack of validation at redirection URL. Such issue is quite often at login because it is good idea to redirect user to previous page after succesful login. But if the url is not validated, evil person can redirect the user to some other page. That doesn't sound bad - right?

What if the destination page looks just like normal login page with text about unsuccesful login like above? How many noticed that URL had wrong domain? Would you have noticed it during normal login? Evil tester knows how to take benefit from the weakness. And tells that.

In talking with other testers about their experiences on projects that apply iterative methods, I often hear a lot of friction in aligning their work with that of developers. Old habits of Us and Them do not disappear overnight, by placing all disciplines in the same team. Testers struggle to get a testable product in the first stages of an iteration, and have a hard time catching up near the end.

It doesn't have to be like that at all. Collaboration is matter of give and take. Showing an interest in each other's work is a necessary first step. Second is the realization that openness in a team is a must. Too often I hear that testers have limited access to unit tests. At the very least, one should be able to read all test code, to minimize overlap in test effort.

Code should be readable, even for a tester without a programming background, given a short walkthrough by the programmer. (If not, then it's probably not going to be easy to maintain the code at a later date.) Any tester worth his or her salt should be able to review whether the tests are adequate in terms of covering decision paths and boundary values. Quite likely, the tester can add value by suggesting improvements and additional test cases. If those were adopted into the project, to be run at every build of the system, it would strengthen the base of the so-called testing pyramid, and improve testing overall in the project.

In an ideal world, testers and developers make a habit of designing tests together, possibly pair programming from time to time. In my experience, testers often have to prove themselves to the team, before they are consulted early enough in the design of tests at the unit and integration levels.

Especially the integration (unit integration and system integration) levels are interesting for testers. Most business logic will be checkable at this level. It would be a shame if it were excluded from automated testing in the style of unit testing, just because it involves more than a single unit. Unit testing is just a label; you can use the tools for it at higher levels as well. Experience has shown that numerous defects are inserted precisely at this integration level, and that it is hard and time consuming to expose them by end-to-end testing or through a GUI. Even worse, such tests tend to be very brittle, and hard to maintain.

Now suppose you are a tester in a team that does not automate many tests at the integration level. My best advice is to discuss this in the team, and not to accept quick fixes. It's very frustrating for a tester to hear that a bug was already fixed, and no test was added to the project. That means that you would need an old version of the product to reproduce the defect, and that manual testing of the fixed application only proves that you cannot easily reproduce the aberrant behavior. It's much, much better for everyone's confidence in the solution if your can agree on something that is called 'defect TDD':
1. When a defect is found, tester and developer discuss what the most appropriate spot is for a test to expose the defect. Quite often, the defect can and should be caught at the unit / unit integration level.

2. The new test is written, and it is run to see that it fails.

3. The code is fixed and the test passes.

4. Refactor if possible, and re-run the tests.

Unlike regular TDD, this approach for defects comes with very little overhead. (Because the code is already complete, there is no need to write mocks for classes that are about to be written. As noted above, it's okay if the test involves more than one class.)

Once a team is accustomed these practices (sharing information about unit and integration tests, securing that a defect stays fixed, placing more and more tests at the integration level, etc), testers will find that they have more control over their situation. There should be no debug cycle near the end. In fact, it ought to become harder to find defects.

Surely, a situation may still arise where a large part of the functionality is declared "ready for test" in the last week of the iteration. The point is that such functionality will already have undergone a lot of testing. Also, the team will be more quality-aware, and may see testing more as a team responsibility. With the provision that no-one tests their own work, developers can certainly chip in to reduce the testing workload. Developers will be able to do some non-confirmatory testing. Especially if their quality awareness has reached a point where they take pride not just in their code but also in the unit and integration tests that come with that code.

As I said, collaboration is a matter of give and take. In the next part of this series, I will discuss ways in which we as testers can make life easier for developers and designers.

I am really curious how this system works in different kind of cases. The content management system and blog is using ASP.NET. Unfortunately that framework is so poorly done (like many other modern frameworks) that it requires JavaScript to most actions. E.g. I could not login without JavaScript enabled. Also part of links weren't working anymore without Javascript.

But most important thing is to find out if I can write blogs without JavaScript. I used Web Developer plugin for Firefox at testing. I had JavaScript enabled when I logged in, then I came to posting page, disabled JavaScript and reloaded whole page again. It gave me the text area for blog writing. I am adding manually tags to see if the text gets formatted. If this works, I can then write plain html to text editor, add links and image tags, then post the form.

Develop an EXPERIENCE, not a mindless tool

After reading Vesna Leonard's blog post titled - 'You are NOT like any other QA or tester I've ever met', one statement stood out:

'You are NOT developing software…a mindless tool….you are developing an EXPERIENCE that lives and breathes with its users.'

The statement sums up the essence of the post and how as a tester, she cares for the stakeholders more than the process.

I'm trying to hack the image to this post. I'm not sure if it works or not. I used Firebug for that. What did I do? I first pasted image to document from any source where you can copy the image. It would result img-tag to document, but the image is broken. Then I used Inspect Element to text edit component and modified the src of image to correct one.

Unfortunately this forces me to put images to outside Eurostar Conference's site. I hope server won't die to the load.

Oh and about screenshot. Facebook is trying to fool me to submit Skype password to it. That is horrible requirement from security point of view. I've taught to my kids that they don't submit passwords to any external place. (And if the image does not appear below this, you can see it at https://kiva-mesta.net/~teme/2010-08-17-mit-vit.jpg .)

So the image should be above this. And I didn't try any security violation. ;) No onmouseover-tags, no broken images with error handling. If I were doing security testing to this site, I'd try those.

We Testers are a strange bunch. I'm convinced that we never fully switch off that part of our brain that likes to find flaws in other people's work, even away from the office on a night out.

The Test Team and I recently headed out for a night on the town. As we haven't been out on a night out in a while, I thought I would do the honours of organising a well deserved night out and the make the most of the weather we were having at the time. As it happens, the place in mind that would have been perfect got called off due to bad weather. Not a total surprise if you're English or follow cricket!

Planning:

One of the main problems I had found with organising a night out is that some of the testers have more of a disposable income than others. Some people want to go to a fairly nice restaurant with pricey beers, whereas others are more reluctant and want to go to a cheap and cheerful pub with proper pub grub and reasonably prices Ales. Some people won't come if we go to the pricey place, and some people won't come if we go to the cheap place.

Equivalence Partitioning:

The difficulty in this was trying to strike a balance and ensure that enough people are pleased with the decision without annoying others too much. One method I used to try and find a happy medium was using Equivalence Partitioning. I placed the number of testers who wouldn't go to the really fancy or really cheap place and placed those in invalid partitions. I then placed everyone else into two valid partitions, about 50% towards the fancy end, and 50% in the cheaper end. I then made a scale of pubs from cheap to fancy and placed those underneath the partitions and picked a pub that was roughly in the middle. The majority of people were happy with the decision; a pub chain selling fancy burgers with both reasonably priced and expensive Ales!

On the night out/Testing in the Wild

A number of issues can crop up during the course of the evening, however.

Requirements issues:

The food or drink does not meet the specification, or does not match the user's expectations. For example, the menu states that the salad comes with olives, where in reality the salad only contains one olive. A regular diner would probably have not even noticed the amount of olives buried within the salad, and have probably already forgotten what the menu said as soon as they put it back into the menu holder.

Performance Issues:

There is usually a gripe about how the waiter/waitress isn't performing as expected. These never get logged or raised with management, so they're never aware of them, however testers readily express their discontent with other testers.
As the evening progresses, the restaurant becomes more and more busy, and the once attentive waiter/waitress is now rushed off their feet serving other customers.

Load Issues:

Someone always eats or drinks way too much *Ahem*

Poor management and leadership:

Unfortunately in this instance, as I was the organiser, I was also the leader. You can see above that I had done my best to decide on a place that suits most people, however, once you decide to leave that place it's a free for all. I don't think you can plan a night out too much and it should be up to the group to decide where to go. As it happens though, there's always someone who looks at you and wants you to make a decision.

Moans, gripes and jokes aside, it was actually a good night and everyone seemed to have a good time. We have a few more nights out planned in the coming weeks and months which should be a good laugh.

*Sigh* One bug found again from the blog software. This is re-post, because the submit form accepted my previous topic:"Get that #¤%¤ bug fixed" but failed to sanitize it properly. So nobody could have commented it and I couldn't share direct links to it.

Usually we know what bugs should be fixed. But that's not always clear to other people at project. This problem appears usually with security issues. Project and product managers don't usually see how some security issue or risk can be so destructive, that they should be high priority issues. Most proof of concepts from security issues are not real exploits, but rather "Hello world"-types because creating good proof of concept just costs too much to be justified.  Instead of writing POC, one should know how to write story about problem. And that's what I'm now days usually doing when I submit security bug.

Every story has some common components. So must have every bug report which is telling the story. They should be:

• Technical details at the beginning. The problem must be found easily by developer.
• Actors. I'm always giving some kind of description who they are. Almost always there is one villain and victim, but villain can be also some innocent person, who just tries to get her work done more quickly.
• Motivation. The villain has to have good motivation to exploit the bug.
• Result. It should hurt someone, kill the business or reveal sensitive data.

For example:

Many PHP-applications has weak session handling. If the attacker can give own value for PHPSESSID, it is accepted and used by the application. Those can happen with cross site scripting bug or local access to the computer and tempering directly the cookie storage of Firefox. The tested application - women's social site about home violence WSSAHV.COM - has weak session handling, but the tester didn't find XSS. So report with story would be:

(Technical details before the story. Also describing how the session should work.)

There is family and members are Hilma (wife) and Onni (husband). Onni is jealous. At some point he finds out that Hilma is using site WSSAHV.COM. Of course he suspects that she's cheating or telling lies about him there. He directly doesn't want to attack against her and force to give the password so he starts hacking and notices the weak session handling.

He has the administrator rights to their shared Windows desktop, he has also access to Hilma's personal files. He goes to Firefox profile files, opens cookies.sqllite with sqllite. There he inserts new cookie for WSSAHV.COM. Domain is 'WSSAHV.COM', path '/', name is 'PHPSESSID ' and value is 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', expiration is set to July 15th, 2020.

Next time when Hilma logs in to WSSAHV.COM, Onni has access to her profile from different computer. He just sets PHPSESSID at his machine to 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'. Onni can alter the profile, read the private message, see the hidden areas where Hilma might have access and so on. This violates the privacy and meaning of the site. It also places the users to danger.

Two years ago, EuroSTAR's theme was "The Future". I think that it should be an ongoing mission of all of us, to explore in what ways we can improve our profession, and ourselves. What new frontiers are just around the corner, and should become commonplace in just a couple of years?

1. Virtualization. It is already there, but I think it can be used to much greater effect, now that the number-crunching power of computers does exceed what is needed in many cases, and extra gigabytes are dirt cheap. Soon, testers will use virtual machines that emulate the target platform in such a fashion, that you can also take a step back after a defect is encountered.
Then, you just hand over the virtual machine to a colleague, who can reproduce the defect instantly.  

This is not science fiction, it is already there. Virtualization and emulation tools already exist, they just need to be perfected a bit.

   Image size should be reduced: right now, an image of a virtual machine can easily be 4+ GB.

   High-performance corporate networks and Internet connection would make a big difference. People will want to send their virtual machines over a network.

2. Soft skills / consultancy skills. Demand for solitary minds in testing is dwindling. Today's and tomorrow's projects require testers to communicate well with colleagues and business people alike. Recruitment of testers (and selection of who stays on companies that reduce their staff) will focus on soft skills that are hard to train, rather than experience and knowledge that can be taught.

3. Career paths. Employers of testers must provide career opportunities and growth options that fit the new scheme. This includes differentiation for affinity towards business or technology. There must be room for individuals with aptitude towards consultancy and leadership. Also, technical excellence (e.g. tool champions) must be rewarded appropriately.
For some really technical testing jobs, it will effectively be a requirement that the 'tester' has experience as a developer. Thus, it must be a step up - not down - to choose such a career path.

4. Solid developer testing (as in the Agile testing pyramid) will gradually become the norm in the industry, allowing testers to do what they are good at: non-confirmatory testing.
Again, this is nothing new, and it does not require any new technology. However, for the standards to be raised across the field (and for agile development and testing to become everyone's frame of reference), that does take time. In some countries, universities have been teaching students XP practices for years.

5. Test frameworks become integrated with middleware solutions, to the point that they become the deciding feature for choosing such software. For example, imagine a middleware suite for modeling a business process flow and a big information system around it. All the big names in the industry have their own suite, and they all have loads of features. It may be deceptively easy to develop a large workflow application with those packages, and very hard to test the result. Tool suites that really give good support for testing the application, will have a decisive competitive advantage, as those would reduce the total development cost.

6. In-memory databases are gradually becoming accepted as a stable solution that can yield great cost savings. Back in 2008, I encountered no real differences (other than the speed) when I tested an application built using an in-memory database system. The main difference for testing was that we could use loads of DbUnit tests in our CI build, without any need to place the Db tests in a nightly build scheme.
Here, it is just the human factor that slows down adoption of a technologically sound solution. If the main performance bottleneck in many of our information systems is the response time of a costly database system (or cluster of database servers), then it stands to reason that more companies will take the plunge and replace them with a lean and mean in-memory system.

Of course, this is just my own opinion. Please feel free to comment on the trends I outlined above, and add your own.

(This was prompted by another blog post.) It has been asked many times. If we may assume that most of our projects are staffed with well-trained professionals, then why can we find so many defects in our applications?

In many cases, you have to be aware that you are not just testing the application, but by default also the process that led to it. I have been in projects where I started testing in late stages of development, simply because I wasn't called in earlier. Even in the presence of a fair number of unit tests, I would find many defects if there was something amiss with the process. For example, if there was little or no communication with the customer that had commissioned the product, then I would find many defects that were due to assumptions made by the development team. If requirements had undergone many changes, then it is likely that those changes caused many tests that were built early on, to fail. I have been on a team where I saw that the total number of unit tests in the project had suddenly gone down instead of up. When I politely asked my colleague about it, he informed me that he had been pressed for time to incorporate the scope changes, and that he had outcommented the old tests. He wasn't slacking off; in fact I knew him to be a really hard-working individual who put in a lot of unpaid overtime. He had simply been overruled, and pressed into releasing the software with a mediocre code coverage. What was defective about the process, is that the team was not allowed time to do the rework in the next iteration. Until we managed to convince our customer of the importance, that is.

I want to encourage any testers to analyze the causes behind the defects that they encounter, and look for a pattern. If you then discuss the causes of the problems with your colleagues, then as a team you should be able to improve your process over time. In so doing, it should become harder to find defects. Agile testing is like that: more focused on preventing rather than detecting defects.