There's no solid way that I can see. I'd be concerned about the following logic too:

IF !(backup in progress) {
    // Drat, a backup could start right here and the following checkpoint will still wait
    checkpoint;
}

If there is a backup in progress and it uses WAIT AFTER END then an explicit CHECKPOINT by another connection will fail if it has uncommitted operations (otherwise the backup would wait forever for the other connection to commit but that connection is waiting forever for the backup to end so that it can do a checkpoint).

From the historical client API, you could attempt to start a backup. I think the attempt will fail if a backup is already in progress. I'm not sure if you could concoct a "do nothing" backup statement that would fail if another backup is in progress but succeed otherwise. I haven't checked but maybe the BACKUP statements just queue up rather than failing if another is in progress.

If you resort to any of these approaches, you didn't hear it from me :)

Starting with V12, it seems that backups (both with BACKUP DATABASE and DBBACKUP) are done by a temporary connection (like same kind on event, methinks).

According to the V12 docs, the temporary connection is called "BackupDB", in my test cases with the name "INT: BackupDB".

However, I can't claim whether this connection is only active while the backup process is "active", i.e. while it doesn't wait on other connections to commit.