While both cloud-based and on-premise testing both have their advantages and disadvantages, and in all likelihood you will be best served using a mixture of both, there is a debate that exists in the industry about which is better. Of course, cloud-testing is much more scalable, likely cheaper and in many situations it’s just easier, there’s still a lot to be said for actually physically testing devices where possible.
In this article we’re going to give you some food for thought in terms of which testing methods you should be considering for your next mobile app or game release.
Table of Contents
An overview of Cloud-Based Testing
Imagine having the ability to test your mobile game or app on thousands of different device configurations all at once. This is what cloud-based platforms offer you – a huge array of software and hardware variants at the touch of a button, without the need to spend vast sums of money on physically possessing these devices.
There are distinct advantages to this approach over on-premise testing – mainly around cost and scalability, but let’s touch on some of the most compelling reasons why you’d opt for a cloud-based solution.
Advantages of cloud-based testing over on-premise testing:
Scalability: During peak times, such as a major release cycle, cloud resources can be scaled up infinitely (depending on budget) to meet the increased demand and then scaled down to reduce cost when fewer resources are needed. If you’re facing variable workloads and tight deadlines along with a need to keep budget lean, this can be a game-changer as opposed to actually physically buying and maintaining devices that might sit unused outside of peak times.
Cost Effectiveness: Think of cloud-based testing as a “pay-as-you-go” testing model, where you pay for resources as and when you need them. This is very attractive to smaller developers with smaller budgets where underutilised devices represent quite a large financial burden. One thing many businesses overlook is that if you have hundreds of physical devices, you have to actually store them somewhere and you have to provide them with power – and as we know, space and electricity aren’t free. Cloud-based testing solves this problem.
Device and OS Accessibility: As we’ve mentioned, a cloud-based testing tool will give you access to any device configuration you can think of. This doesn’t just mean device type and OS, but different screen sizes and resolutions, right down to variable hardware configurations. Actually keeping hold of this many real devices can be a real challenge (and a huge cost) that cloud-based testing completely solves.
Parallel Testing Capabilities: Cloud-based testing supports running multiple tests simultaneously, significantly cutting down on the time needed to execute large suites of tests, accelerating the development cycle.
Geographical Testing: Allows teams to test applications in different geographical server locations, which is crucial for verifying application performance and functionality in varied user conditions globally.
[H3] Challenges you might experience with cloud-based testing:
There are drawbacks to cloud-based testing – and while many of these can be mitigated with good management (or a good managed service provider) you should still ensure you consider these points in your evaluation:
Security Risks. With cloud-based testing your data is stored on remote servers, which inherently increases the risk of data breaches and unauthorised access compared to an on-premise solution. This means that you need to be very vigilant and possibly consider investing in additional security protocols to keep your data secure – adding another layer of cost.
Dependence on Internet Connectivity: Cloud-based testing can’t occur without a stable and fast internet connection. If your internet service suffers an outage, and you don’t have a backup line, you simply can’t test. And of course, this will inevitably happen at the worst possible time, either right in the middle of a major testing exercise or just before a major release. If your connection is intermittent, it can also lead to inconsistent and inconclusive testing results, which can lead to your team wasting time chasing false alarms.
It is also worth mentioning that this is highly dependent on performance bandwidth and latency. For example, most cloud-based services are in the USA. If you are in, for example, Bangalore, most cloud-based services will be close to unusable for you due to the latency and performance delays associated with data having to travel back and forth between the US and India.
Limited Customisation. Sometimes, cloud testing tools come with a set menu of options, limiting how much you can customise the testing environment. If you need to test on highly specific hardware setups that aren’t supported by your cloud testing provider, it’s going to be a challenge to actually successfully perform the test; whereas if you were using an on-premise solution, you could just go out and buy or build this hardware yourself.
Emulated and virtualised services. While this isn’t such a problem for Android devices as these are easier to emulate and virtualise, Apple make it very difficult to emulate their hardware. Additionally there are numerous license restrictions to running Apple services in the cloud; in fact it’s almost impossible to do from a licensing standpoint, and Apple themselves don’t really have a cloud-server presence, so it can be extremely difficult to run accurate tests on Apple devices without actually having the physical device present with you as you would with an on-premise test.
And yes – Apple devices in-cloud can be offered as a real service, where someone else does the testing for you on real devices, but this is not scalable.
An overview of On-Premise Testing
On-premise testing might seem a bit old-fashioned, but there are actually very specific security and privacy benefits to performing on-site testing as opposed to running all your tests in the cloud. On-premise testing could involve physically owning these devices yourself, or you could look to contract this out to a device farm that already owns the devices and hires them out to businesses looking to test mobile apps.
Advantages of on-premise testing over cloud-based testing:
More control over environment. On-premise testing offers total control over both hardware and software environments, enabling precise test conditions that match the production settings.
Security enhancements. With all data handled locally, on-premise testing provides enhanced security, crucial for projects handling sensitive information.
Customisation and integration. On-premise testing allows for deep integration with internal development tools and processes, offering the ability to tailor the testing environment to specific needs.
Reliability for interactive testing. On-premise setups often have lower latency, which is essential for testing games that require precise and high-frequency inputs.
Long-term cost efficiency. Although the initial setup for on-premise testing involves significant investment, it can be more cost-effective over time, especially for larger operations that can amortise the cost across many projects.
To discover how on-premise testing will completely transform the way you test, book a free demo with T-Plan below.
Challenges you might experience with on-premise testing:
Initial Costs: Setting up an on-premise testing environment requires substantial upfront investment in infrastructure, primarily in the purchasing of devices. If you are operating on a reduced budget this can always be a challenge, as often that money spent elsewhere might be more effective.
Scalability: Scaling on-premise testing facilities can be challenging and expensive, as it will requires additional physical space and hardware – neither of which are free.
Device Limitation: Unlike cloud-based solutions that offer access to a vast array of devices, cloud-based testing requires you to actually take physical possession of the devices in question, and if they’re old or legacy devices you may not actually be able to do this easily.
Scenario-based evaluation for cloud-based vs. on-premise testing
Scenario 1: Rapid prototype testing for an indie game developer
Context: An indie game developer working on a new puzzle game needs to verify that the game’s interface is intuitive on both smartphones and large tablets. They need to conduct specific UI tests such as checking menu layouts, touch responsiveness and visual appeal across different devices.
Why cloud-based testing would be preferable in this instance:
The kind of real-time feedback you get from cloud-based testing gives you the ability to make rapid tweaks and iterations, shortening the development cycle and fostering a more agile process. This would allow the indie developer in this scenario to more quickly assess how changes affect the game’s performance across different platforms, leading to a more refined product at launch.
Cloud-based testing also helps ensure gameplay consistency – a vital part of the user experience but can be enormously challenging to get right due to device and OS fragmentation. Cloud-based testing offers access to innumerable device configurations, enabling simulations on any number of platforms without you having to take physical ownership. In this scenario, where an indie developer with likely low budget isn’t going to have the capital or even the time to go out and buy physical devices, this helps ensure this uniform experience across different devices in a time and cost-sensitive manner.
Cloud-based testing in this instance will also help to optimise performance in several ways – it isn’t all about smooth gameplay but instead extending a game’s compatibility across diverse hardware. This process allows adjustments in game settings to match different device capabilities, crucial for an indie developer trying to capture a wide audience (including those with both older and newer devices.)
Finally perhaps one of the most important reasons you’d opt for cloud-based testing in this scenario (particularly if you were planning to launch in multiple markets) is that you can much more easily simulate game performance across various locations and network types – without having to physically be in those locations. This ensures that games and apps perform well globally, accommodating differences in network latency, data speeds and geographical differences.
Scenario 2: Large-scale performance testing for a AAA game title
Context: A major game studio requires in-depth performance testing for a new AAA title expected to push the limits of mobile hardware, where consistent performance and hardware integration are critical. The game is set in a vast open world, and the developer needs to test whether the game mechanics do not consume excessive resources and that the in-game physics behave consistently.
Why on-premise testing would be preferable in this instance:
Real-world testing conditions are a must in this scenario. Developers creating test environments need to ensure that the game is tested in real-world conditions that accurately mirror what will be experienced by real users. Therefore, a controlled on-premise setup is crucial to ensure developers can accurately observe game mechanics, react to specific hardware and network configurations and ultimately ensure the game works optimally on release. While cloud-based testing may be able to offer some of this to a degree, the only way you truly know how your game runs on a particular device is to actually run it on that device – and this is why on-premise testing is better in this particular scenario.
Security and intellectual property protection is a huge consideration for large game studios. It’s a fact that leaks are not only embarrassing but expensive. Cloud-based testing introduces significant risks regarding intellectual property and security, and the external storage of sensitive game data on cloud servers vastly increases the potential for a breach. It may not even be you that gets hacked – all it takes is for a datacentre managed by your cloud service provider to be breached and that’s it – your game data is out there. On-premise testing allows for much tighter control over this, mitigating a lot of these risks.
Networking and real-time multiplayer testing is also a large consideration for scenarios such as this. On-premise testing offers a much more reliable environment for testing dependence on stable internet connectivity and external server performance, ensuring network issues don’t compromise the integrity of the test results.
Strategic recommendations for businesses considering on-premise testing
Firstly, companies should carefully evaluate their long-term development and testing needs. If frequent, intensive testing cycles are anticipated, especially for graphically intense or highly interactive games, On-Premise testing might offer better value over time.
Then it’s best to start with what’s necessary but still plan for scalability. As the company grows and testing needs to expand, having a modular system that can grow with your needs will ensure longevity and adaptability of the testing setup.
It’s also important to invest in training for IT staff to manage and optimise On-Premise testing environments. As technologies evolve, having knowledgeable personnel who can adapt and maintain testing infrastructures is invaluable.
It’s also recommended to ensure that the On-Premise testing environment is fully integrated with existing development pipelines. This integration enhances efficiency, allowing for seamless transitions from development to testing phases, thereby reducing downtime and speeding up release cycles.
Frequently asked questions about on-premise vs cloud-based testing
How do Cloud-Based and On-Premise testing environments impact the speed of development cycles in mobile game production?
Cloud-based testing can speed up development cycles through its scalability and rapid provisioning of diverse testing environments. This enables quick iterations and faster feedback. Conversely, on-premise testing provides a stable environment but may slow down the process due to the time needed for setup and scalability limitations.
What are the specific security considerations that game developers should evaluate when choosing between Cloud-Based and On-Premise testing for mobile games?
You should consider the security risks of data breaches and unauthorised access associated with the cloud, versus the controlled but resource-intensive security management required on-premise. Ensuring data protection involves evaluating the cloud provider’s compliance with security standards versus the capacity to secure an on-premise setup.
Can On-Premise testing environments keep pace with the rapid advancement of mobile device capabilities and operating system updates as effectively as Cloud-Based solutions?
On-premise environments struggle to match the agility of cloud-based solutions in keeping up with rapid advancements in mobile technology due to the need for continual investment in new hardware. Cloud-based testing offers instant access to the latest devices and systems, providing a more flexible and up-to-date testing framework.
