Implementation of Solr-based Algorithm into Job Search Platform

New job search platform based on Apache Solr.

Challenge

SCAND team was tasked to develop a new platform for finding job vacancies. It was to be based on the existing customer’s system. The client requested improvements in the search speed, implementation of filtering based on regions and industries, mobile device compatibility, and a modern UI/UX. A prerequisite was to synchronize the data between the old system and the new platform during the testing phase and then to put the latter into operation.

In addition to the existing functionality, the new platform was to include:

  • search by keywords in job descriptions;
  • faceted search by attributes of job vacancies (regions, branches, cities, etc.);
  • creation of a RESTful API.

Approach

The architecture of the new platform was based on several services:

  • Data migration
  • Data loading
  • Search
  • Job vacancy management

Such approach helped to shorten the development time through parallel work on different services. It also provided quick and easy scalability to individual parts of the application, allowing to handle huge volumes of data.

Apache Solr, an open source enterprise search platform, was chosen both for full-text and facet search functionality. The main features of this open source enterprise search platform include:

  • databases integration;
  • flexible search relevance management;
  • caching of queries, filters, and documents;
  • web-based interface for administration;
  • processing of documents with complex formats such as Word or PDF;
  • extensibility through plug-ins.

The data migration to the new database was carried out in several stages.

Stage 1.

First of all, archived and outdated job vacancies were moved to the new database, since they accounted for most of the data.

Stage 2.

The next stage was the transfer of active and new job vacancies.

Stage 3.

The last stage included stopping the service for final synchronization of data and then switching it to a new platform and a new database.

Final synchronization touched only those jobs that were added or updated after the previous sync, so the downtime was minimized.

The implementation of the RESTful API allowed to unify the interaction with external client applications, including the new UI. It also opened up wide opportunities for creating third-party solutions for the partners of the platform.

To provide a fault-tolerant data loading, a new MySQL database with the master-master replication was developed.  Originally, data in the existing system was stored in databases with different structures. The new database allowed storing data for different regions in a uniform format.

Result

SCAND web development team has created a platform that meets UI/UX, architectural and speed requirements.

While preserving all the existing functionality, the new platform allows the customer to accelerate and improve search results significantly. As the platform uses a unified API, it also provides the ability to integrate third-party applications including those for mobiles.

Need a similar project?

Contact us

Thank you for your interest in our services. We have received your message and will respond to you shortly. Please check your mailbox for a confirmation letter from us.