Bringing Drush and Migrate Tools closer

Migrating data between systems can be complex, depending on the approach taken for the interaction. Drupal helps a lot on this task thanks to the Migrate API subsystem that lives inside Drupal core since 8.0-alpha6 and previously lived on a contributed module on Drupal 6 times.

For this reason, writing migrations into Drupal has been a great developer experience, that abstracts away several details, and integrates well inside crucial pieces like the Entity API subsystem.

Once written, migrations need to be executed. This can happen on a customized UI that on the back-end interacts with Migrate API, but the most popular option is to use Drush commands. There are several Drush command sets available to run your migrations, provided by modules like Migrate Tools, Migrate Run, and Migrate Manifest. They try to cover a subset of possible scenarios that can be found while dealing with migration projects; and having used all of them, I appreciate a lot the effort made behind each of them.

Starting on Drush 10.4.0, Drush project itself now ships with migrate commands.

The idea of adding migration-related commands to Drush is not new, for instance, it was suggested on 2017, but only happened on Jan 2021 2021 after some iterations and improvements by Claudiu Cristea, which started on top of the Migrate Run set of commands.

An important thing to mention is that both Migrate Run and Migrate Tools sets of commands were using the same command line names, i.e. both follow the migrate:* pattern. Hence, part of the initial changes added into Drush also responsibly declare incompatibility with both Migrate Run and Migrate Tools.

Additionally, a new issue about this was added for Migrate Tools, which has been waiting for some feedback.

This change on Drush core is a great opportunity to bring a bit closer the efforts that started on Migrate Run and Migrate Tools, which can impact positively the ecosystem of modules around migrations into Drupal.

In previous editions, DrupalCon events separated roughly Mondays for summits, Tuesday to Thursday for talks, and Friday as a full-day contribution event.

This year, DrupalCon North America has changed its format, splitting the day to have talks during the morning, and contribution time in the afternoon across the full week, trying to promote contributions over some of the active Drupal core initiatives.

This is highlighting over contributions is a great addition to the event. It also opens the door for a continued effort that can happen throughout the week.

On the other side, Adapt kindly sponsored my attendance at the conference, which meant I could use the contribution time to try to help directly.

Since I was thinking about this problem for a while, I decided to try to start the needed steps to figure out what could be done about this possible incompatibility between Drush core and Migrate Tools. It was one of those tasks that could be truly tedious, and they affect people using Migrate Tools. That affected person could be me in the near future when I need to upgrade Drush to 10.4.0 or later, so it sounded like a great match for the continuous effort during the week.

The first part of the effort was about producing a detailed evaluation to try to identify clearly the differences between the migrate related command-line interfaces that are shipped into Drush core, and the ones on Migrate Tools. The main outcome of the comparison was truly positive: they both share a lot more things than they differ on.

There are naturally some aspects in which they differ, but they are so close, that the natural recommendation I made was to mainly expand Drush core migration commands from Migrate Tools in the few aspects that they are different, and also add things directly to Drush core if they are generic enough to live there. In that way, things can still evolve in both, where Drush code is trying to be generic, and Migrate Tools can extend it to accommodate for extra features if needed. The first of those generic changes I suggested and sent a pull request about was recently merged into Drush core, where support for --idlist was added into migrate:messages command. This is great, because it hints that Drush core maintainers welcome extra changes about these commands where it makes sense. I also proposed a change to add a compatibility layer for one of Migrate Tools’ options, and even it was not accepted, it also hints that that kind of change should live at Migrate Tools instead, which is also helpful.

The next step is about trying to change Migrate Tools to be written on top of Drush core commands, for which I started a 6.x branch of the project locally. This is still a work in progress, but I should have an update about this in the following weeks, likely on the original issue.

The migration ecosystem around Drupal seems to be really healthy, and there are already many tools available to use to help migrations into Drupal. Efforts inside Drupal core Migrate API and contributed modules using and extending it have shaped a great development experience to do migrations into Drupal.

The added emphasis on contributions at DrupalCon, together with Adapt support to attend the event has allowed me to push forward the initial detailed assessment and plan to follow.

Building Migrate Tools on top of Drush core migration commands sounds like the best path forward, and there have been already some initial steps taken to bring them closer. I hope to continue this effort, and make available a new version of Migrate Tools migrate commands built on top of Drush core, as a new branch suggested for the project.

Finally, thanks to the people behind the Migrate API in Drupal core, and the contributed modules around it, you bring a lot of value into the Drupal community!