It's Time to Separate: Lint and Test

Today, you get to learn a pet peeve of mine, a rare instance where I never follow what Ember blueprints say since 2020.

Since version 3.17.0, ember-cli generates apps and addons with a test script that—weirdly—checks files for lint and format errors.

{
  "scripts": {
    "build": "ember build --environment=production",
    "format": "prettier . --cache --write",
    "lint": "concurrently \"pnpm:lint:*(!fix)\" --names \"lint:\" --prefixColors auto",
    "lint:css": "stylelint \"**/*.css\"",
    "lint:css:fix": "concurrently \"pnpm:lint:css -- --fix\"",
    "lint:fix": "concurrently \"pnpm:lint:*:fix\" --names \"fix:\" --prefixColors auto && pnpm format",
    "lint:format": "prettier . --cache --check",
    "lint:hbs": "ember-template-lint .",
    "lint:hbs:fix": "ember-template-lint . --fix",
    "lint:js": "eslint . --cache",
    "lint:js:fix": "eslint . --fix",
    "lint:types": "tsc --noEmit",
    "test": "concurrently \"pnpm:lint\" \"pnpm:test:*\" --names \"lint,test:\" --prefixColors auto",
    "test:ember": "ember test"
  }
}

The lack of separation between lint and test resulted in 3 side effects that the initial implementation hadn't accounted for:

• In CI, the lint command runs twice.
• In README and CONTRIBUTING, the test commands show an incongruence: People can run pnpm test and pnpm test:ember --server. Wouldn't pnpm test --server make more sense?¹ They neglect to mention that the test command lints files, which has occasionally led to questions on Ember Discord.
• We need to add the query parameter nolint to the /tests URL, if we don't want lint results from addons to double the test report size. This issue was prevalent when we had depended on ember-cli-eslint and ember-cli-template-lint. We can still get a mixed report due to ember-cli-dependency-lint.

The fix is simple:

{
  "scripts": {
-     "test": "concurrently \"pnpm:lint\" \"pnpm:test:*\" --names \"lint,test:\" --prefixColors auto",
-     "test:ember": "ember test"
+     "test": "ember test"
  }
}

In v1 addons, and in apps that test v2 addons, I use these scripts:

{
  "scripts": {
    "test": "ember test",
    "test:ember-compatibility": "ember try:one"
  }
}

Note, the test script for ember-try uses try:one to run a single scenario, instead of try:each to run all. This way, we can run our scenarios in CI in parallel, and locally test a failing scenario quickly. Separation FTW.

While I could continue to apply these changes to my projects manually, it'd be better if we update the blueprints to reflect the Ember ecosystem of now and our improved knowledge of CI since 2020.

There isn't yet an RFC that highlights the issues above and proposes change. I think this is because we've gotten used to the problems and think less over time in the shoes of a newcomer. Let's make Ember projects a delight to work with.