CI/CD and Testing in Context of E-commerce Platform

Why are Continuous Integration (CI), Continuous Delivery (CD), and a standardized development lifecycle essential in developing e-commerce software?

I've been surprised when talking with different e-commerce merchants about how few of them have proper CI/CD processes in place and how many problems they have to deal with because of that. This is usually regardless of the e-commerce platform they use. Why don't companies invest in CI/CD? I've come across the following answers:

• Development progress would be slower if the developers needed to write code and tests.
• The project costs more because of additional code for automated tests.
• The team has no expertise to be able to set up proper CI/CD.
• The developers don't know how to write tests.

Software characteristics which have a direct impact on revenue are availability and reliability.

Availability–tells how much percentage of time your e-store is up and fully operational.

Reliability and Robustness–describe how the shop system is fail-safe, and how it can deal with exceptional situations, assuming that every incident causing downtime or critical malfunction is pure revenue loss.

How to minimize the risks then?

Investing in well-functioning development workflow, Continuous Integration and Delivery (or Continuous Deployment), and reliable infrastructure are some of the most important steps towards high availability and reliability.

What is CI/CD and what does it help with?

CI/CD is a set of practices that aim to enable frequent releases of software changes (new features, bug fixes) while lowering the probability of serious issues.

Continuous integration (CI) integrates all code changes developers make into a shared version control repository. Each change needs to be evaluated by automatic tests. CI is effective only when code changes are committed to short-lived branches and merged to the main branch (trunk) on a daily basis. This approach is called Trunk-based Development. It helps integrate smaller portions of code. So, any potential issues discovered by automatic tests are simpler to resolve and, most importantly, won't get into the production system. Moreover, by merging small pieces of code it is very uncommon to run into conflicts. Even if a conflict happens it is easy to resolve. We've applied this workflow to our Shopware and Magento-based project development.

_{A visualization of Continuous Integration}

Continuous Delivery (CD) follows from Continuous Integration and automates releasing code changes to pre-production environments so they can be tested in them. Tests can be both automatic and manual. Once the tests pass and after the product owner's approval, the changes can be deployed to the production environment.

_{Continuous Delivery–deployment to production is triggered after manual test results review, optional manual testing and final approval.}

Continuous Deployment goes one step further. It's a practice where all code changes are deployed to production in an automated and continuous manner without any manual action or signing off by anyone. Only a failed test may stop this process. This approach is the most efficient but requires thorough automated testing on multiple levels. Such deployments should be frequent, containing small improvements.

_{Continuous Deployment–fully automated deployment to production with no human involvement.}

CI/CD for infrastructure

Infrastructure as Code (IaC) has become a standard for provisioning and managing infrastructure. Tools like Terraform or Pulumi support the majority of cloud providers. The code that defines infrastructure can be tested. These tests should become a part of Continuous Integration.

How to manage features with short release cycles?

Trunk-based development and short release cycles may consequently lead to the situation where a newly developed feature isn't ready yet to be publicly available–at least not to 100% of the customers.

A handy technique to solve this is using Feature Flags (alt. Feature Toggles). They allow for toggling features, usually via predefined environment variables. Therefore, we can hide a feature if it's not intended to be public yet. Shopware already has Feature Flag support built into its core platform framework.

Feature flags and A/B Testing

Feature flags are also useful for setting up A/B tests, thanks to feature flags you don't need to prepare two versions of the application for an experiment. Instead, enable given behavior using a feature flag only for a selected group of customers.

Automated testing of e-commerce platform implementation

A typical construct is built of three levels of testing known as the Test Pyramid and consists of:

• Unit tests
• Integration tests
• End-to-end tests

Unit tests focus on individual components or functions of the application and test them in isolation. Unit tests should not require connection to any external services like a database, cache server, or message broker, and they must be fast. This means that the entire test suite should be able to complete in seconds rather than minutes. In order to write highly testable code, TDD (Test Driven Development) is really beneficial. In TDD, you start writing code by writing the tests for this code. It indeed helps to structure the code for higher testability.

Integration tests assess the interaction between multiple components or services. The tests concern the application code with the integrated services, such as the database, message broker, or APIs.

End-to-end tests validate the whole application through the UI or API, simulating real customers or users. Since such tests are the slowest, they should rather cover the business-critical functionalities. End-to-end tests must also be reliable and reproducible. A flaky test (sometimes passes, sometimes fails) doesn't bring much value. For end-to-end testing Shopware and Magento use browser-based test frameworks such as Cypress, Playwright, and Selenium.

All kinds of tests must be reliable and reproducible (they must neither depend on each other, nor in which order they are executed). Integration and end-to-end tests need pre-prepared data fixtures. A data fixture is fake data which needs to be loaded for each test scenario and cleaned up afterwards.

_{The Test pyramid}

Testing own extensions for Shopware or Magento

Testing extensions for an e-commerce platform such as Shopware or Magento as an extension vendor requires additionally a matrix of all configurations that the extension declares compatibility with. That is all supported by PHP, database server versions, and other modules and plugins. It is also important to declare the dependency requirements strictly in the dependency manager config (composer.json). That will prevent an extension from being installed in incompatible environments.

What if my project doesn’t have tests?

The best practice is to write tests along with the code from the start. Even if the project is a proof of concept. Often even conceptual projects are further developed and eventually get adopted. In our experience, writing tests from day one helps identify any contradictions in project requirements, catch defects early, and come up with a more robust application architecture.

What if the project is already mature or it is inherited from a different team and it doesn't have any tests? Adding tests to existing code is notoriously difficult, but it's a valuable tool to help the developers understand how complex software works and make changes with confidence. In such a scenario the most optimal solution is to start from end-to-end testing mission critical functionality, such as the complete purchase process, and integration with key external systems such as ERP. At the same time ensuring that all future changes include relevant tests. Although end-to-end testing is the slowest, it doesn't depend on the code. Therefore there's lots of gain with relatively low cost.

Code quality standards

Factors such as code consistency and readability, efficiency, performance, and security influence the overall project quality. Incorporating static code analysis tools allows for validating the code syntax according to once-set rules. Same as unit tests, static analysis should be run regularly before committing code to the repository, and be an integral part of the Continuous Integration pipeline. For PHP-based projects like Magento or Shopware, there are a bunch of open-source tools available, such as PHPStan, PHP_CodeSniffer, PHP Mess Detector, or Psalm.

Conclusion

Let me bring up the common question coming from many merchants. Does test automation, CI/CD, and TDD slow an e-commerce project down and increase its costs? The answer is no. It is actually the opposite. By running tests frequently during development, developers get immediate feedback. Moreover, TDD enforces good code practices. All of this results in significantly fewer bugs. With no automated tests, our customers will eventually test everything, and we definitely don't want that to happen.

• CI/CD
• Continuous Delivery
• Continuous Integration
• TDD