Leading Jewellery Brand

The four Magento 1 sites contained similar bespoke functionality that had been developed by the client over many years, however, the functionality varied because there was no central structure, so the four separate code bases had diverged over time.

The client, who prefers to remain anonymous, provided over 100 custom Adobe XD designs that had been produced by their in-house team. In addition to tackling the back-end functionality, they also asked us to bring their designs to life in terms of a bespoke front-end theme.

Due to the complex technical nature of this build, we had to evolve our project management approach to allow for real-time discussion between our lead developer via a dedicated Slack channel. While this isn’t our standard approach, it was crucial because the client had directly developed the sites over many years and we needed their input on bespoke features. We also had to tackle a migration from Magento 2 to Adobe Commerce Cloud during the project. The client recognised the need to move their build to the next level with the additional functionality and support from the licensed version of Magento.

The migration could be split into two main areas:

• The merging of data from the four Magento 1 sites into the Magento 2 database.
• The porting of bespoke Magento 1 modules into a version that worked in the Magento 2 environment.

These modules create an order flow with several stages of notification detailing the status of the customer’s order. For the merging of the databases, we investigated the functionality provided by existing third-party modules and the core Magento data migration tool. We needed something that we could heavily customise to cover the merging of primary keys and conversion of the tables that had been created in the bespoke Magento 1 modules.

The porting of functionality from the bespoke Magento 1 modules was a lengthy task. It required learning what these modules did in enough detail so that we could write and run reliable tests after each stage of the porting process. In general, the idea was that functionality would be copied over from the Magento 1 codebases with minimal updates added as needed to allow it to work using the associated Magento 2 functionality.

Merging Of Multiple Data Sources

Magento's data migration tool is available for developers to migrate data from Magento 1 to Magento 2. However, this tool only covers basic data in the Magento installation, and it will only allow a migration from a single source site to a destination site.

The following issues needed to be addressed for a migration to be feasible:

• The data migration tool needed to handle migrating data from multiple sources without truncating the data from any of the previously migrated databases.
• Primary keys in the tables needed to be altered to avoid them clashing with each other. Clashing primary keys would result in the data from one source overwriting the data of another source.
• The migration tool needed to be able to re-migrate data if issues were found with the previous migration. The ability to specifically migrate certain tables or data after certain dates was important for this in order to target sections of data where issues were found instead of having to do a full migration every time.
• We needed to be able to connect the migration tool directly to the Magento 1 and Magento 2 servers concurrently to allow easier testing of the migration with the live data and to reduce the downtime on the launch days of the Magento 2 site.

We were able to resolve these issues after conducting significant research into how the data migration tool could be extended and altered.

Porting of Manufacturing, Consignment and Appraisal Order flow

The bulk of the porting process focused on bespoke modules that implemented a specific order flow. These modules were completely bespoke in the Magento 1 sites. To add to the complexity of the porting process, in some cases each site has slightly different functionality for certain steps or omits them completely.

The following was the order flow that required porting:

Manufacturing

• After a customer placed an order, it would be changed to a manufacturing status to indicate that this process had begun. The customer would be notified.

Consignment

• Once manufacturing was complete, the manufactured orders would be grouped into a consignment. This would then be shipped and the customer would be notified.
• Barcode PDFs could be generated by the admin from the consignment.
• An API request to the EAO would be triggered from the consignment in the admin to generate a shipment document.

EAO Consignment Arrived

• When a consignment arrived at the EAO, they would send a request to the Magento site to update the status of all orders in the consignment.

EAO Hallmarked

• When an order had been hallmarked, the EAO would send a request to the Magento site to update the order status.

EAO Shipped

• When an order had been shipped to the customer, the EAO would send a request to the Magento site to change the status of the order. The customer would be notified.

These are the main points of interest from the perspective of the end user. In reality, the functionality was much deeper than this,comprising cost price calculations, attribute conversions and separate supplier-only views. Several dozen points of functionality could be listed from a single Magento 1 site when breaking the modules down file by file.

Duplicate Data Keys

One of the major complications was handling the primary keys that conflicted between the source databases. The solution for this was initially relatively simple. We created a transform class to run all primary keys through as they were copied across. Each source database was assigned a range, to which the primary keys would be converted. For example, primary keys in database one would be converted to start from 1,000,000, database two from 2,000,000, and so on. Since the largest primary keys in any of the tables being migrated was less than one million, this ensured there were no conflicts.

However, an unforeseen complication of this process was that the core data migration tool relies on the primary keys in order to migrate static attributes for various entities. These are a new concept introduced in Magento 2. Rectifying this issue was much more complicated and required substantial rewriting of the migration tool to move the point of the primary key conversion after the static attributes had been migrated.

Constantly Updating Destination Data

During the process of developing and running various data migrations, the client was importing their own data–specifically, the product data. This was undergoing some significant changes and it was decided that this wouldn’t be migrated using the migration tool we were developing. A complication of this was that several of the data tables we were migrating from the Magento 1 databases relied on having the product data in the destination database so it could map the Product IDs. Due to the constant importing and replacing of the product inventory on the Magento 2 site, this meant the migration tool could not rely on the Products IDs being correct or, in many cases, even existing at the time of a migration.

Because of project time constraints, we were unable to wait until the product inventory was finalised. We needed to test what we could with the data we were migrating and check the Product IDs later. To tackle this issue, we developed the migration tool to allow more specific migrations that only migrate certain tables, with the sole purpose of updating the Product ID columns.

We also developed the tool to allow us to specify different ranges of data based on dates or column values. This meant that if a specific issue were found, we could run a more targeted migration instead of having to run a full migration each time.

Order Flow Refactoring

Some areas of the ported functionality were able to be copied over with minimal alterations. However, many required significant refactoring for them to work in the newer Magento 2 framework, or for them to function within a multisite environment, as they were originally not written with this in mind. Gauging the scale of this type of work is complicated. The order flow functionality that was being ported had several dozen points of specific functionality underneath the main points These were compounded by the fact that there were four sites with differing functionality. It’s almost impossible to uncover these types of issues until work is undertaken to port and test the functionality from the different sites.

A final unforeseen complication in this process was that the client would be implementing new modules and functionality that would inadvertently affect the functionality of their existing order flow. Resolutions for these various issues involved investigating the application code base with a debugger, to identify the specific points where the issues were rooted and where a workaround could be put in place.