Composal
Back to Reflog

/Product updates

Post-merge testing with scheduled runs

Agents merge faster than any queue can test. Scheduled runs keep testing what landed, in any environment, on whatever schedule you set.

Anthropic went from about 2,500 merges a day in May to over 30,000 in September. In their post on rebuilding source control, they say “a proper merge queue doesn’t make much sense at agent scale.” So they test after the merge instead: “we’ve invested in post-merge and pre-deploy automated testing.”

You don’t need 30,000 merges a day to have this problem. Composal tests each pull request before it merges, but two pull requests can each pass and still break checkout together. Something has to test what actually landed.

That’s what scheduled runs do.

Any environment, any schedule

Choose an environment, when to run and which scenarios. Testing agents run them on that schedule, the same way they test a pull request. Check staging every morning, production every hour, or both.

Pick scenarios by label. A schedule for smoke and checkout runs every scenario with either label, including ones you add later.

The Edit scheduled run dialog for Staging: a daily 09:00 run in America/Los_Angeles, scenarios selected by the smoke and checkout labels with 85 matching, a QA administrator credential and a Slack channel.

When the run finishes, you get a report by email or in Slack. It lists what passed and what needs attention, and links to the recording and evidence for each failure.

Runs don’t pile up. If the last run is still going, the next one is skipped. After an outage, Composal catches up one missed run, not a backlog.

Cron rules in the scenario file

For timing the schedule form doesn’t cover, add cron rules to a scenario:

on:
  schedule:
    - cron: '0 * * * *'
      environment: production
    - cron: '0 9 * * 1-5'
      timezone: America/Los_Angeles
      environment: staging

This runs every hour on production and at 09:00 on weekdays on staging. Results go to run history; cron rules don’t send reports.

Set one up

Open Scenarios or an environment in Verify and choose Schedule Run. To report to Slack, connect it in Integrations, then run /composal subscribe all in the channel. The scheduling reference covers cron and event rules.

Pull request testing checks each change before it merges. Scheduled runs check everything together after it has. Both are part of Continual Agent Verification.

Keep reading

All entries