Building vs buying your test automation framework is a strategic decision that will impact every phase of your app’s development and product lifecycle. There are several different frameworks and services available on the market – some you pay for, and some you don’t. There’s a certain allure to the tailored specificity and control you get from building your own solution in-house, but many businesses opt to browser search for pre-made tools, making this an interesting decision for app and game developers.
Buying a tool can provide immediate access to advanced testing technologies, data analysis, rapid deployment capabilities, and ongoing professional support, making it an appealing option for dynamic industries like mobile app and game development. But is this always the best way of effectively automating your app testing? Let’s explore this a little more and help you decide on the most appropriate strategic choice for your business.
Table of Contents
Three Options – Build from Scratch, Use Open Source or Buy from A Third Party
It’s important to consider your business’s specific needs when choosing a test automation tool – what works for you might not work for someone else. You need to have a clear understanding of the resources available to you and your long-term goals before you can make a call as to whether you’re going to build or buy.
Fundamentally, however, you have three distinct options.
- Build your own testing tool/framework in-house from scratch.
- A hybrid approach, where you take open-source software off the shelf and use it to help you build tests.
- The hands-off approach – outsourcing your testing to a third-party tool or managed service provider.
Each option has its merits. For example, building a custom automated test solution offers complete control, but also requires significant investment in time and expertise. Open-source tools, though cost-effective, may require substantial customization and are dependent on community support. Lastly, third-party testing tools offer quick deployment and professional support but may lack the same level of customization.
Building from Scratch
This is an option that many businesses initially settle on as it appears to offer them the greatest degree of flexibility. There’s a certain attractiveness to having full control over the software testing criteria and environment as well as having a bespoke testing tool specific to your app. Building a custom testing tool provides flexibility but comes with the challenge of maintaining the system. A custom solution is often best when applications require very specific testing protocols or handling of proprietary data.
Why building from scratch might be a good thing
If you’ve got very niche, very specific use cases that you’ve researched and discovered that off-the-shelf tools simply don’t account for, you may not have any other option other than to “DIY”. This does give you an element of control that you simply don’t get with the other approaches, however, and to some businesses this is very appealing.
Additionally, you get total control over the feature set, updates and security measures. This allows you to make them as detailed and as robust as you want to, and you aren’t relying on anyone else to provide any of these constituent parts; for example, if you’re particularly security-conscious, you may sleep better at night knowing that it’s very unlikely an early build of your app or game is going to get leaked due to the attention you’ve paid to your custom testing tool’s security measures.
Why building from scratch might cause you problems
Firstly, building from scratch is time intensive. Off-the-shelf testing tools generally go live in weeks – custom built tools go live in months. If you have the time to spare, then this isn’t an issue, but many development teams spend their days being hounded by marketing and product teams to meet launch deadlines, and it’s entirely possible that the custom approach might cause you ship a lot later than you first anticipated.
The other thing you should consider when taking the “DIY” approach is that often, you’re reliant on a small group of people who have the prerequisite knowledge to build these testing tools. If those people somehow are no longer involved – whether they’ve had a promotion, moved to a different team, moved to another company or even have just decided that they don’t want to be involved any more, it can be very difficult to make progress.
Even replacing that developer can be problematic, as not only do you have to find someone with the right skillset, but even if you do, this person may not like the approach the previous individual has taken and may want to re-factor or even completely re-write some of the code. Often this leads to custom-built testing tools becoming nothing more than expensive shelfware after a few years.
Using Open Source Software
This option appears to offer the best of both worlds – you need some kind of in-house expertise to be able to implement an open source solution and with that comes some degree of customisation, but at the same time a vastly reduced development time, and subsequently, a reduced cost. There are pitfalls to be aware of, however.
Why an open source solution might be a good thing
The big draw for using an open source testing solution is cost. Open source frameworks like Appium are usually free to use, which vastly reduces development time and cost. You will also save on external vendor costs to maintain and update the software. Community support and guidance is nearly always free to access, and some open-source projects benefit from shared development where contributions from a global community are made to the software which can include things like bug fixes, new features and security enhancements – all at no cost to you.
The other draw for adopting this hybrid approach is that it saves time. By taking the framework and customising it in-house, your development team can focus on customising rather than building basic functionalities from scratch, as these have already been built into the core of the framework. As we know, development time = cost, and with this approach it’s possible to conduct perfectly adequate testing with vastly reduced development resource.
Why an open source solution could cause you problems
While there are some great resources out there for some of these popular open-source test frameworks, you are almost entirely dependent on the community for bug fixes and feature updates. Community reliance can be very unpredictable, and you have to consider that people aren’t being paid to develop these frameworks – so it’s entirely up to them when and how they work on specific issues. Don’t expect any kind of formal support.
Another thing you should consider is that many open source projects get abandoned. This is unlikely for large tools like Appium or Selenium but if you’re using something more obscure, it’s crucial you choose one with a vibrant community. If you base your testing tool on a project that gets abandoned six months after you start testing, continuing to use it will become increasingly difficult.
Like any customisation, you will also need to set aside budget for security auditing – customisations critical to business operations such as test automation tools must always be regularly audited for security vulnerabilities. You should also consider costs associated with risk mitigation; investment in off-site or cloud backup solutions is a very sensible move.
Buying from A Third-Party Service Provider
This is where you hand off the logistics of performing tests to a third party managed service provider with their own testing software. A great example of this is our T-Plan Robot, where you will not need any coder-level knowledge to operate the tool. We can also work with you to design and construct tests to your requirements, leaving you to focus on evaluating the test results and implementing the required fixes.
Why a third-party vendor might be the best choice for you
Firstly, a third-party vendor will offer dedicated technical support both in and outside of business hours. Businesses that cannot afford downtime or extended troubleshooting sessions value this very highly, as any technical hitches become the service provider’s problem and obligation to resolve. Vendors also handle maintenance, updates and security, meaning your only concern should be how you improve your app or game following testing.
One thing that is very attractive to businesses large and small is that third party testing tools are generally ready to deploy out of the box, significantly reducing the time and effort needed to get started, particularly when compared to an open-source or custom-built solution. Vendors will also usually help you with integration (or in some cases, even just do it for you) which reduces your development team’s troubleshooting time on potential integration issues.
Another thing you should consider is that third party vendors have to ensure their products comply with industry standards and regulations – so you can be sure that the software you are using is likely far less vulnerable and poses less of a risk to your organisation than anything custom-built in-house, where you will have to assign budget to security auditing and testing.
Why a third-party vendor might not be suitable
This all largely depends on the managed service provider that you choose. Some third party vendors offer less customisation than building from scratch. However, when you partner with T-Plan, we work with you to ensure that our tool accurately meets the needs of your business – to the point where we’re confident that there’s nothing you can do from scratch that we can’t match with our testing tool.
How a managed service provider like T-Plan can save you time and money when testing mobile apps and games
For businesses that require rapid deployment or lack the internal capability to manage and customise complex testing tools effectively, buying in your testing capability from a third-party vendor like T-Plan can be a compelling choice – particularly when compared to the difficulties many businesses face either trying to build a tool from scratch or integrate an open-source framework.
Typically, bought tools offer reliability and convenience while also offering robust technical support. There are specific business cases, however, to buying in your testing tool from a managed service provider, which include but aren’t limited to:
Quick Deployment and Immediate ROI
It goes without saying that buying in a tool is quicker than building it from nothing. While custom-built solutions take months to become operational, purchased tools can be implemented to go live within weeks. This accelerated timeline leads to an much quicker return on investment as teams can begin testing and refining products much sooner.
Access to Industry-Leading Testing Technologies
Third party providers like T-Plan invest continuously in integrating the latest technologies and methodologies into their tools. By contracting with a managed service provider, businesses gain access to advanced features that might be too resource-intensive to develop and maintain in-house. This access ensure that mobile apps and games are tested with cutting-edge tools that can simulate complex user environments and detect issues more effectively.
Professional Support and Maintenance
Managed service providers offer comprehensive ongoing maintenance and support services, significantly alleviating the workload of internal teams. Businesses benefit from the peace of mind that comes with having expert assistance readily available, which can help reduce downtime and resolve technical glitches swiftly. This professional support is crucial for maintaining the stability and reliability of testing operations.
Scalability and Flexibility
As a business grows and its products evolve, the need for a scalable and flexible testing tool becomes critical. Purchased tools typically allow for easier scaling and adaptation to more complex testing scenarios. Additionally, many third-party solutions offer customisable modules or add-ons that can tailor the tool to specific needs, without the need for extensive redevelopment or overhauls.
Check out a free demo of T-Plan by clicking the link below.
A fourth option – the best of both worlds?
There is another model that is very popular with those who have tried the open-source option and have become disillusioned with it due to the inherent drawbacks, and this is those who have opted for an open-source framework that is wrapped within a managed testing tool. This is exactly what you get with T-Plan – an image-based testing tool that incorporates Selenium (which is an object based automation tool.) This really gives you the best of both worlds – a hybrid tool, if you like.
This can be very beneficial for users requiring a low/no-code solution. Frameworks like Gherkin/Appium/Selenium/Cucumber etc are all very developer-oriented, and very code-focused. For example, think of this like the difference between coding a website from scratch using HTML versus using some kind of page builder or WYSIWYG solution. The client gets the ease of use of using an off-the-shelf system, but without having to support it, research it, code it, etc.
Now there are products on the market claiming to do this, and many of them are free. However, what you’ll find is that these tools are being developed to the benefit of the company that developed them. For example, Facebook released a testing tool called WebDriverAgent in 2015 designed for iOS automation testing, but this was very much influenced by what Facebook thought was relevant – not necessarily what you think is relevant for testing your apps and games. A third-party tool that wraps your desired framework within it to provide a GUI-driven testing tool (where you can work with the vendor to specify exactly what you want to test) is likely to be much more suitable.
How to Get the Best Out of a Third-Party Testing Vendor (Like T-Plan)
Working with a third-party vendor, if you’ve decided that this is the route you want to go down, is not just about outsourcing tasks – it’s about building a strategic partnership. To make the most of the relationship it’s important to adopt a collaborative approach. Here are a few essential practices to ensure you’re getting the best possible value from your vendor:
Ensure robust SLAs are in place
With T-Plan, robust service legal agreements are standard. This ensures that both parties have clear expectations of performance metrics, response times and quality standard. These agreements serve as a foundation for accountability and quality assurance.
Establish clear communication channels, expectations, and metrics
Effective communication is critical to the success of any partnership. Having clear channels and protocols for communication ensures that any issues, updates or changes and managed effectively. Setting clear expectations and agree-upon metrics from the outset helps both parties measure progress and success.
Insist on comprehensive documentation
Comprehensive documentation should cover all aspects of the testing procedures – including test plans, test cases, user stories, and any anomalies or bugs founds. Good documentation not only supports current project needs but also provides a valuable resource for future testing cycles and training new team members.
