OpenBSD now enforcing no invalid NUL characters in shell scripts

Our favorite operating system is now changing the default shell (ksh) to enforce not allowing invalid NUL characters in input that will be parsed as parts of the script.

The commit message reads,

List:       openbsd-cvs
Subject:    CVS: cvs.openbsd.org: src
From:       Theo de Raadt <deraadt () cvs ! openbsd ! org>
Date:       2024-09-23 21:18:33

CVSROOT:	/cvs
Module name:	src
Changes by:	deraadt@cvs.openbsd.org	2024/09/23 15:18:33

Modified files:
	bin/ksh        : shf.c 

Log message:
If during parsing lines in the script, ksh finds a NUL byte on the
line, it should abort ("syntax error: NUL byte unexpected").  There
appears to be one piece of software which is misinterpreting guidance
of this, and trying to depend upon embedded NUL.  During research,
every shell we tested has one or more cases where a NUL byte in the
input or inside variable contents will create divergent behaviour from
other shells.  (ie. gets converted to a space, is silently skipped, or
aborts script parsing or later execution).  All the shells are written
in C, and majority of them use C strings for everything, which means
they cannot embed a NUL, so this is not surprising.  It is quite
unbelievable there are people trying to rewrite history on a lark, and
expecting the world to follow alone.

If there is ONE THING the Unix world needs, it is for bash/ksh/sh to
stop diverging further by permitting STUPID INPUT that cannot
plausibly work in all other shells.  We are in a post-Postel world.

It remains possible to put arbitrary bytes *AFTER* the parts of the
shell script that get parsed & executed (like some Solaris patch files
do).  But you can't put arbirary bytes in the middle, ahead of shell
script parsed lines, because shells can't jump to arbitrary offsets
inside the input file, they go THROUGH all the 'valid shell script
text lines' to get there.

This was in snapshots for more than 2 months, and only spotted one
other program depending on the behaviour (and that test program did
not observe that it was therefore depending in incorrect behaviour!!)

ok ingo.  Softer ok's from various others.

So here it is again, an example of OpenBSD making software behavior saner for all of us.

Please note that to enjoy this change, you need to be running OpenBSD-current, since change occurred after OpenBSD 7.6 was tagged.