Modernization of a PHP Data Processing Platform
Overview of Our Client
Our client develops and maintains a data processing platform that collects information from multiple external sources, stores operation history, and generates analytical reports. The system includes a user portal, external integrations, several data exchange channels, background tasks, and monitoring tools.
The platform had been developed and extended over many years. As new functionality accumulated in the monolithic application and the volume of stored data continued to grow, its architecture began to limit further development and make changes to critical components more difficult and risky.
The main application was built with PHP 7 and Symfony. The client needed to modernize its architecture, improve data processing, and establish a more maintainable foundation for subsequent development.
- Platform: Web Application
- Industry: Data Processing / Analytics
- Project Type: Platform Modernization
Challenge
The main challenge was to modernize a mature monolithic platform without disrupting its business logic or data workflows. Its architecture created limitations in four key areas: message processing, historical data storage, logging, and maintainability.
Message reception, validation, and business rules were tightly coupled, so temporary failures could delay subsequent messages. Communication channels also used inconsistent retry and failure-handling rules.
The historical archive was stored in a relational database in a format designed for daily sets of values. To build reports, the application loaded full records and transformed them on its own side, which made migration, integrity checks, and report optimization harder as the archive grew.
Logs from different channels had no common format or shared message ID, so engineers matched records from several sources by hand, and failed messages had no dedicated search and reprocessing workflow.
Finally, the monolith contained interconnected business logic with limited automated test coverage, with much of its behavior not covered by automated tests, increasing the risk of regressions when changing data processing, reports, and billing.
The project therefore needed to:
- Separate message intake from business processing and standardize failure handling.
- Move historical data to time-series storage with controlled migration and integrity checks.
- Unify logging and enable end-to-end message tracing and reprocessing.
- Define clearer application domains and add regression tests for critical workflows.
Project Overview
The work covered four areas of the platform: message processing, historical data storage, logging, and the monolith itself. For data storage, the new format was introduced without switching off the existing path at once.
Message intake was moved to separate PHP 8.3 services connected to the main application through RabbitMQ. Existing business rules remained in the main application. Incoming data and outgoing commands are processed independently, so each flow can have its own delivery rules; temporary failures trigger a retry, and messages that could not be processed are stored separately.
For data storage, TimescaleDB was introduced alongside the relational database. Historical data was migrated with a repeatable process and parallel writes. Automated reconciliation compared values in both stores, and reads were switched gradually, with the option to return to the original source. During validation, the team fixed conversion and reconciliation errors that could cause skipped or duplicated records, and optimized indexes and archive storage.
The team also centralized logs in Elasticsearch and made them searchable through Kibana. Shared message IDs enabled end-to-end tracing: each message can be followed from reception through validation and processing to a result or an error. Successful events and errors are stored separately but linked by one ID, and a failed message can be resubmitted and checked in the same system.
For the monolith, the team analyzed its structure and defined domain areas for users, integrations, data, calculations, and analytics. Most of the effort went into recording the system’s actual behavior with automated tests that cover user scenarios, forms, business entities, reports, data reception, background tasks, and access checks.
Modernization Workflow
- Separating Message Processing: Moving message intake to PHP 8.3 services and RabbitMQ, with retries and separate storage for failed messages.
- Introducing Time-Series Storage: Adding TimescaleDB alongside the existing database.
- Migrating and Validating Data: Repeatable migration, parallel writes, automated reconciliation, and gradual read switching with rollback support.
- Centralizing Logging and Traceability: Consolidating events in Elasticsearch and Kibana and tracing each message by one ID.
- Defining Domain Boundaries: Identifying the monolith’s main business areas and responsibilities.
- Expanding Regression Tests: Covering critical workflows to protect existing application behavior.
Technology Stack
The modernization project used the following technologies:
Backend
- Migrated the entire application from PHP 7 to PHP 8.3 while decomposing the monolith into separate services.
Framework
- Upgraded Symfony to a version compatible with PHP 8.3, ensuring framework compatibility with the updated PHP environment.
Message Broker
- RabbitMQ for asynchronous communication, message delivery, and retry handling.
Time-Series Database
- TimescaleDB for historical data storage and archive management.
Logging and Search
- Elasticsearch for centralized event storage and indexing.
Log Visualization and Analysis
- Kibana for analyzing message flows, errors, and processing outcomes.
Automated Testing
- Regression tests covering user workflows, business entities, reports, data reception, background tasks, and access control.
Results
SCAND modernized the platform’s core architecture while preserving existing business logic and enabling a controlled transition to new infrastructure.
Message processing was decoupled, with independent incoming and outgoing flows and standardized error handling. Historical data was migrated to TimescaleDB through parallel writes, automated reconciliation, and gradual read switching. Centralized logging in Elasticsearch, supported by shared message IDs and Kibana, improved message traceability and error investigation.
The monolith gained clearer domain boundaries and stronger regression test coverage. The number of test classes increased more than tenfold, while line coverage in the main module reached 93.53%.
Core Team
- Backend Developer: Implemented message-processing services and integrated RabbitMQ.
- Database Engineer: Introduced TimescaleDB and supported historical data migration and reconciliation.
- DevOps Engineer: Configured centralized logging and monitoring with Elasticsearch and Kibana.
- QA Engineer: Expanded automated regression tests and validated critical application workflows.
Related Cases
- .NET Core
- Angular
- Flutter
- Xamarin Migration
- JavaScript
- PHP