02 May

Shubhra Maji's Community Choice Submission

02 May 2012 by paul

Presentation Title: TiP it! - Testing in Production for Agile Testers Shubhra Maji

Company: Microsoft Corporation

Country: India

Testers have been doing a great job in finding bugs before customers do. That applies well in the conventional software service model. But with more and more SaaS/Cloud applications being deployed into production, one can increasingly see a shift in this paradigm. Does "Test first ship next" really matter in the context of shorter release cycles, scalability and integration?

For example, think of services such as social networking, online retail and internet search engines with millions and billions of monthly visitors. With such huge scales of release, it is next to impossible for such service providers to be looking for every single major bug before each hourly release. "Ship first, test next" is the way to go for agile testers in these scenarios. That being said, can one's staging environment truly replicate such highly scalable and complex production environment? The answer is partly yes. Testing in Production solves a lot of these problems if a tester can implement the right practices to gain optimal quality of service in "real" world scenarios.

In this presentation, I would like to pick examples from Microsoft, Google, Facebook, Amazon and explain how a TiP model works in these organizations. Then, I would walk into the shoes of an agile tester and describe her role as a Tester in Production. I would also discuss best practices for Web and Performance testing such as A/B Testing, Exposure Control, Shadowing etc. and demonstrate how testing Cloud services jells well with TiP.

Walk away with knowledge of Testing in Production Learn what others are doing in Testing in Production space from real case studies Understand prevalent TiP methodologies and identify practices that best suit one's production setup.


Shubhra Maji is a Program Manager of Visual Studio Test Tools product. An MBA graduate from Indian School of Business, Shubhra has 7 years of experience as a developer, consultant, technical architect and product manager with extensive experience in Enterprise Applications, Business Intelligence and Content Management.

Apart from owning the automation testing capabilities in Visual Studio, Shubhra's role in Microsoft primarily revolves around assisting Microsoft customers and engaging with Microsoft partners in the testing space while laying roadmap for the product. Shubhra blogs at http://blogs.msdn.com/b/visualstudioalm/ . To learn more about Shubhra, please check her LinkedIn profile at http://www.linkedin.com/in/shubhramaji



42 comment(s) for “Community Choice: Shubhra Maji”

  1. Gravatar of Marc van 't Veer
    Marc van 't Veer Says:
    This is the new world for a tester: a profesional beta tester with a big scope and the responsibility to find defects before the customer (proactief). Interesting summary and hope to hear your story.
  2. Gravatar of Aseem
    Aseem Says:
    With shortening of ALM cycles, testing in production is getting important day by day. Learning how other teams are doing it and learning from them makes this a good topic to attend.
  3. Gravatar of Rik Marselis
    Rik Marselis Says:
    I would say: test first, then ship, then continue testing. Before and after deployment different quality characteristics have to be looked after.
  4. Gravatar of Shubhra Maji
    Shubhra Maji Says:
    Hi Marc,
    Thank you for your comment. In this session, we will look into two aspects of software delivery - customer value delivery (when?) and defect resolution (how long?). We know for a fact that zero-defect is far from real in a highly agile large scale software service. Question is how soon can we deliver value to our customers and how soon can we resolve defects? Sooner we release, greater are the risks of defect. On the other hand, the more time we spend on defect resolution before customer finds it, the longer we are keeping customers from experiencing our new value-added services. Best case would be to strike a balance between the two. Depending on the case in point, one can chose the appropriate TiP methodology to establish a tradeoff such that “finding defect before the customer” does not necessarily remain as the top priority of an agile tester as does “delivery of value”.
    I will show you examples of how organizations have been able to quickly deliver customer value as well as promptly resolved defects through TiP.
    Open to more comments!
  5. Gravatar of Shubhra Maji
    Shubhra Maji Says:
    Hi Aseem,
    TiP breaks down impediments to shorter delivery cycle time and impediments to reduced mean time to resolve defects (MTTR). As the software delivery models across organizations are shifting towards agile from the conventional waterfall, these two factors are becoming increasingly significant when it comes to delivering value to the end customers and reducing waste in the cycle.
  6. Gravatar of Mathew Aniyan
    Mathew Aniyan Says:
    Timely topic as more and more enterprises move into the cloud.
  7. Gravatar of Mathew Aniyan
    Mathew Aniyan Says:
    Timely topic as more and more enterprises move into the cloud.
  8. Gravatar of Raj
    Raj Says:
    Excellent forward looking thinking. As a tester, I would love to attend this session. +100 votes from my team and me.
  9. Gravatar of Shubhra Maji
    Shubhra Maji Says:
    Thank you Raj. You have rightly identified that this topic will talk about shift in conventional techniques towards radically innovative approaches to keep pace with faster release cycles.
  10. Gravatar of Shubhra Maji
    Shubhra Maji Says:
    Thank you Mathew for pointing towards the cloud-friendly enterprises!
    Think about testing for concurrent access by 1 million users and each user testing a different part of your application hosted on the cloud! How close can your staging production environment get to real usage scenarios like this when we think of such a scale? Can your tests mimic these real-time scenarios? If so, can you catch a bug and be sure that there are no more out there?
    Lots of discussion around these and similar questions will be covered in my talk.
  11. Gravatar of Neeraja
    Neeraja Says:
    This is a great topic in the age of agile development/testing and shorter release cycles!
  12. Gravatar of Vishnu
    Vishnu Says:
    Great Topic. Looking forward to understand how efficiently can we test when we make frequent releases
  13. Gravatar of Abhishek Awasthi
    Abhishek Awasthi Says:
    Excellent choice of topic. Looking forward to hear more from you.
  14. Gravatar of mrityunjay singh
    mrityunjay singh Says:
    Intresting, would love to know about how testing Cloud services jells well with TiP.

  15. Gravatar of Lynne
    Lynne Says:
    I would like to find out more on this
  16. Gravatar of Deepak
    Deepak Says:
    Ship First, test next. Yes the world is moving to this. Seeing many projects specially service oriented adopting this. Looking for a really helpful session here!
  17. Gravatar of Shubhra Maji
    Shubhra Maji Says:
    @Neeraja: Thank you for appreciating the topic. The age of agile testing is new when compared with the age of agile development. With testers being more and more involved in the agile project cycle now, it becomes quite essential that the testing methodologies adapt to the concept of shorter cycle time and faster time to market. More about this in my talk.
  18. Gravatar of Shubhra Maji
    Shubhra Maji Says:
    @Vishnu: Thank you. The more frequently a product is released, the more a product team can adapt to changing customer needs. But, there is more to it. With frequent releases, you get an opportunity to push bug fixes to the customer, maybe after a few customers might have encountered the bug (which does not sound that great). But having been able to push a bug fix before the millions of customers get to it is a great idea. Hence, faster release cycles is a self-sustaining model. More will be covered in my session.
  19. Gravatar of Shubhra Maji
    Shubhra Maji Says:
    @Abhishek: Thank you for appreciating the topic. Lots of new thinking and discussions coming up around agile testers/cloud services/release cycles and more!
  20. Gravatar of Shubhra Maji
    Shubhra Maji Says:
    @Mrityunjay: Hosting a cloud service leads to scale, scale caters to heterogeneous and evolving customer demands, which in turn leads to shorter release cycles. This is where agile kicks in - in almost all facets of the release cycle, including production testing - and there you would find the right TiP methodologies to test in production rather than in staging. More about the agile tester's role in TiP in my talk!
  21. Gravatar of Shubhra Maji
    Shubhra Maji Says:
    @Lynn: Sounds great that this topic interests you.
  22. Gravatar of Shubhra Maji
    Shubhra Maji Says:
    @Deepak: Thank you. You have rightly mentioned how the service enterprises are embracing the concepts of TiP to be on the edge and to reduce the time gap between idea inception and customer consumption. Love to see more votes for Ship first, Test next!
  23. Gravatar of Saurabh Sharma
    Saurabh Sharma Says:
    Hi Shubhra,
    This should be a very interrested topic. Hourly releases and scale at which the application are used instantly will certainly require new quality assurance process. Providing a seemless service while testing in production seems counter intutive. It will great to understand the TiP process via your presentation and practical examples.

  24. Gravatar of Elad
    Elad Says:
    TiP is becoming more and more popular both in enterprises and for small companies when testing SAAS.
    As TiP it is relatively new in the industry, it is really interesting to see how each and every company chooses the relevant TiP mythologies and implement those.
    Looking forward for that discussion!
  25. Gravatar of Alex
    Alex Says:
    Hi Shubhra,

    This is a very interesting subject to talk about.

    I attended RoWST (Romanian Workshop on Software Testing) at the end of March, and Maaret Pyhäjärvi presented an experience report about how they decided to release their new version and test in production as they weren't able to fully replicate the production environment.

    I was wandering if you've been involved in projects that followed the TiP approach, and how you and your team decided what to test before the release and what to test afterwards.

  26. Gravatar of AK
    AK Says:
    Interesting thought. Even from a startup's perspective TFSN approach could be detrimental. Given the fact that startups in tech space have to evolve at a fast pace it makes sense for them to take an incremental approach and focus on fixing bug after developmental and get the product out first.
  27. Gravatar of Anu
    Anu Says:
    Exciting topic to pick! The future belongs to cloud based services - testing them in production is the only way to catch a whole class of issues on time and at scale. Great choice!
  28. Gravatar of Doron
    Doron Says:
    It sounds interesting , but I guess it's not applicable for all environments and context.
    In embedded RT Systems in defense industry you can't "Ship first, test next". So you should explain in your topic where this approach is aplicable , in my case you can relesa a prototype to customer evaluation and test in paralel with him , but the production have to wait untill all testing is completed.
  29. Gravatar of Tarun Arora
    Tarun Arora Says:
    A very impressive subject! I would be very keen to attend this session.
  30. Gravatar of Shubhra Maji
    Shubhra Maji Says:
    @Saurabh: A near-seamless service while testing in production could be achieved through fast responsiveness. Moving over to TiP is not expected to be an abrupt change. The more gradual your team can make this change happen, the faster you learn to respond to surprises. Gradually you could see your testing infrastructure fit itself into the TiP model while ensuring near-seamless service while cutting cost on up-front testing and ensuring faster go-to-market.
  31. Gravatar of Shubhra Maji
    Shubhra Maji Says:
    @Elad: Rightly said. TiP is not a one-stop solution for all. Depending on the kind of software service provisioning, release cadence, scale of release, integration complexity and the risks of user data disruption, a production testing team needs to choose the appropriate TiP methodology.
  32. Gravatar of Shubhra Maji
    Shubhra Maji Says:
    @Alex: I would cover experience of teams across Microsoft while moving over to TiP. Great to hear that Testing in Production is gaining momentum and you got to hear Maaret's experience with TiP on a live project.
  33. Gravatar of Shubhra Maji
    Shubhra Maji Says:
    @AK: Thank you. TiP is just what a startup needs to catch customer's attention and stay ahead of tough competition in this dynamic environment.
  34. Gravatar of Shubhra Maji
    Shubhra Maji Says:
    @Anu: Thank you and sounds great that you too believe TiP is the future as is cloud based services. Although the transition is going to be gradual, it is exciting to see that small to large enterprises are opening up to this new way of testing!
  35. Gravatar of tapas
    tapas Says:
    Quite an absorbing topic in my opinion. I specifically agree to the tradeoff point you have mentioned about striking a balance between “finding defect before the customer” and “delivery of value”. I would be eagerly awaiting to learn more about this in the session. Goodluck !
  36. Gravatar of Ankur Bhardwaj
    Ankur Bhardwaj Says:
    Seems to be an interesting one. Waiting and hoping to listen a lot on this. Thanks Shubhra for coming with such a great idea!!!
  37. Gravatar of Shubhra Maji
    Shubhra Maji Says:
    @Doron: Thank you for your comments.
    Ship first Test next has several variations which can be defined under each TiP methodology. For sensitive data testing e.g. in defense industry as you have rightly pointed out, teams would follow what we call "Controlled Test Flight" wherein releases would be made available internally for "dog-fooding" or externally as "Beta release". In both cases, you can get valuable feedback from targeted end-users which you could then be incorporated in the final product. The final product would then pass through usual testing before production as usual but with more confidence because it was Beta tested. Does it reduce time-to-market? Yes. Does it ensure protection of sensitive information? Yes. It is testing in production, but at a controlled level, but that gives you the benefits of TiP.
    In my talk, we will have more discussions around issues related to sensitive data handling such as PII misuse, leaking of new competitive features, exposure of test artifacts and the right thing to do for each of these cases.
  38. Gravatar of Shubhra Maji
    Shubhra Maji Says:
    @Tarun/Ankur: Thanks for appreciating the topic.
  39. Gravatar of Shubhra Maji
    Shubhra Maji Says:
    @tapas: Thank you for appreciating the topic. It is great to see how this topic is getting more and more interesting. Change in conventional methodologies is hard to implement all at once, but it is the gradual shift that I would be emphasizing on. And the tradeoffs that I have talked about will help define how each team needs to execute the shift in testing practices and give the agile tester more flexibility while ensuring quality of delivery.
  40. Gravatar of Samyajit Das
    Samyajit Das Says:
    @Shubhra: Excellently written piece!! Well done! :-)
  41. Gravatar of Shubhra Maji
    Shubhra Maji Says:
    @Samyajit: Thank you for appreciating the topic. You may vote for it using the voting link.
  42. Gravatar of testwithus
    testwithus Says:
    SWIFT Interview questions on


    For selenium solution visit

    For QTP interview questions



Leave comment: