5.9.1 Invalid multibyte characters

sed’s regular expressions do not match invalid multibyte sequences in a multibyte locale.

In the following examples, the ascii value 0xCE is an incomplete multibyte character (shown here as �). The regular expression ‘.’ does not match it:

$ printf 'a\xCEb\n'
a�e

$ printf 'a\xCEb\n' | sed 's/./X/g'
X�X

$ printf 'a\xCEc\n' | sed 's/./X/g' | od -tx1c -An
  58  ce  58  0a
   X      X   \n

Similarly, the ’catch-all’ regular expression ‘.*’ does not match the entire line:

$ printf 'a\xCEc\n' | sed 's/.*//' | od -tx1c -An
  ce  63  0a
       c  \n

GNU sed offers the special z command to clear the current pattern space regardless of invalid multibyte characters (i.e. it works like s/.*// but also removes invalid multibyte characters):

$ printf 'a\xCEc\n' | sed 'z' | od -tx1c -An
   0a
   \n

Alternatively, force the C locale to process each octet separately (every octet is a valid character in the C locale):

$ printf 'a\xCEc\n' | LC_ALL=C sed 's/.*//' | od -tx1c -An
  0a
  \n

sed’s inability to process invalid multibyte characters can be used to detect such invalid sequences in a file. In the following examples, the \xCE\xCE is an invalid multibyte sequence, while \xCE\A3 is a valid multibyte sequence (of the Greek Sigma character).

The following sed program removes all valid characters using s/.//g. Any content left in the pattern space (the invalid characters) are added to the hold space using the H command. On the last line ($), the hold space is retrieved (x), newlines are removed (s/\n//g), and any remaining octets are printed unambiguously (l). Thus, any invalid multibyte sequences are printed as octal values:

$ printf 'ab\nc\n\xCE\xCEde\n\xCE\xA3f\n' > invalid.txt

$ cat invalid.txt
ab
c
��de
Σf

$ sed -n 's/.//g ; H ; ${x;s/\n//g;l}' invalid.txt
\316\316$

With a few more commands, sed can print the exact line number corresponding to each invalid characters (line 3). These characters can then be removed by forcing the C locale and using octal escape sequences:

$ sed -n 's/.//g;=;l' invalid.txt | paste - -  | awk '$2!="$"'
3       \316\316$

$ LC_ALL=C sed '3s/\o316\o316//' invalid.txt > fixed.txt