Using codemods to upgrade legacy code

Last week I helped a client plan an upgrade from an internal custom JavaScript library to a more modern UI framework. The existing code uses a tool called “jingo“ to manage dependencies. We wanted to support modern bundlers that use the new EcmaScript Modules (import and export) syntax. A straight forward mechanical transformation exists but the codebase consists of hundreds of files to convert. It would be tedious, boring work over the course of days. However, I could use codemods to write a script that would convert the syntax automatically.

In jingo, engineers write code in modules, and each module provides a list of dependencies. The developer specifies dependencies in a . separated format, just like a Java package namespace. At runtime, Jingo will convert those namespaces to URLs, and add the relevant <script> tags to the DOM before executing the module code. Anything loaded as a dependency should create new global variables in window, but generally placed in a namespace object.

Here’s a simple example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
jingo.declare({
  require: [
    'root:lib.common.utilities',
    'root:lib.common.textBox'
  ],
  as: function() {
    // find or create a namespace for our code
    var MY_APP = window.MY_APP || (window.MY_APP = {});

    // Add a class to the namespace
    MY_APP.HelloWorldController = function(rootNode) {
      this.rootNode = rootNode;

      // TextBox is a dependency, and is expected to put a class into the MY_APP namespace
      this.textBox = new MY_APP.TextBox(this.rootNode);
    }

    // TODO: Add other class methods here
  }
})

This module depends on utilities and textBox and requires that the browser load them first. Jingo makes sure they get initialized before executing the as function. At that point, the code of this module can depend on the methods and classes its dependencies create. The engineers chose to have dependencies create known classes and functions inside of a global namespace.

If you stand back and squint at this, you’ll notice that this looks kinda like a regular ES Module, where the require array is equivalent to import statements, and the as function is the main code in the module.

An engineer could translate it into modules like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
import '~/lib/common/utilities';
import '~/lib/common/textBox';

// find or create a namespace for our code
var MY_APP = window.MY_APP || (window.MY_APP = {});

MY_APP.HelloWorldController = function(rootNode) {
  this.rootNode = rootNode;

  // TextBox is a dependency, and is expected to put a class into the MY_APP namespace
  this.textBox = new MY_APP.TextBox(this.rootNode);
}

// Add other methods here

It would take a developer a three or four minutes to translate from the previous syntax to the new one, and it doesn’t take too much thought once they’ve completed a couple of files. But it would be tedious, boring, and prone to typos. It’s also complex enough that a well-crafted Find+Replace with regular expressions is unlikely to do the right thing.

Luckily great tooling for this kind of transformation exists in the form of codemods: Scripts that automate the transformation of JavaScript and typescript code.

I used a library called codemod-cli to help set up a project for developing the mod. It helps generate a sample script and some mechansisms for writing tests against the conversions. I also found this article to be helpful while getting started: Getting Started with Code Mods.

The trick to actually writing a codemod is to figure out which Abstract Syntax Tree (AST) elements need modification. An AST is a data structure that represents the code itself. Rather than operating at the text layer of characters and words, we can operate at the syntax layer and look for particular code constructs like a CallExpression (function call).

In this case I wanted to inspect and replace the call to jingo.declare. I’d need to pull out the require and as properties from its function argument and use them to generate the replacement code. Through a library called jscodeshift, codemod authors can search for nodes in the AST that match the requested pattern.

A great way to do that is with https://astexplorer.net. You can paste the code in and see the AST that would correspond to the code in question. By looking at that I could determine that I needed to find a CallExpression where the callee property is a MemberExpression with object named "jingo" and property named declare.

Once I found that, I could inspect the function call arguments, find the array of requirements and rewrite them to import statements.

Then I could find the as function and extract its body. Last, I could combine the new import statements and the function body into the replacement for the original method call.

It looks something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
const { getParser } = require('codemod-cli').jscodeshift;
const { getOptions } = require('codemod-cli');

/**
 * Convert "root:lib.common.utilities" to "~/lib/common/utilities"
 * @param namespace
 */
function buildImportPathFromNamespace(namespace) {
  return namespace.replace(/\./g, '/').replace(/^root:/, '~/');
}

module.exports = function transformer(file, api) {
  const j = getParser(api);
  const options = getOptions();

  const root = j(file.source);

  // we're looking for a call to jingo.declare.
  root
    .find(j.ExpressionStatement, {
      expression: {
        type: 'CallExpression',
        callee: { type: 'MemberExpression', object: { name: 'jingo' }, property: { name: 'declare' } },
      },
    })
    .replaceWith((path) => { // path is the CallExpression

      // if we found one, we need to do some munging:
      // 1. Extract the require array from the configuration parameter
      // 2. Extract the "as" function from the configuration parameter
      // 3. For all the required "namespaces" (dotted names) convert them to full file paths
      // 4. Rewrite the require as ES module imports
      // 5. Stick the body of the "as" function at the end of the module

      const configAst = path.value.expression.arguments[0];
      const requireArray = configAst.properties.find((prop) => prop.key.name === 'require');
      const asFunction = configAst.properties.find((prop) => prop.key.name === 'as');

      /** The list of es module imports as AST ImportDeclaration nodes */
      let imports = requireArray.value.elements
            .filter((e) => e.type === 'StringLiteral')
            .map((e) => buildImportPathFromNamespace(e.value))
            .map((importPath) => j.importDeclaration([], j.literal(importPath)));

      // return replacement for jingo.declare, which is the list of imports and the code of the module
      return [...imports, ...asFunction.value.body.body];
    });


  return root.toSource();
};

module.exports.type = 'js';

All in, the codemod wasn’t too scary. The AST Explorer website made it a breeze to inspect jscodeshift’s data model and understand what I was looking for. The docs for jscodeshift are a bit lacking though. For example it was hard to know what methods existed for generating new tokens (such as j.importDeclaration). But in the end I got through it.

I’ll probably be reaching for these tools again sometime, its good to know its possible to write scripts to move code to newer constructs without a lot of manual transformation.