View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0000276fileGeneralpublic2021-07-26 15:242024-04-06 13:55
Reporterabathur Assigned Tochristos  
PrioritynormalSeverityminorReproducibilityalways
Status resolvedResolutionreopened 
Platformintel/x86_64OSmacOSOS Version10.15
Product Version5.40 
Fixed in Version5.46 
Summary0000276: between 5.37 and 5.39, file starts identifying bin/sh script with patched shebang as awk/perl
DescriptionI noticed a patched copy of esh 0.1.1 getting identified as an "awk or perl script" by newer versions of file (confirmed I see this in file 5.39 from nixpkgs and file 5.40 from homebrew).

I've attached an unpatched copy named `esh` and a patched copy named `nix_esh`, but I assume the salient part is the fact that it has had its shebang patched from /bin/sh to /nix/store/pcjan45rssdn01cxx3sjg70avjg6c3ni-bash-4.4-p23/bin/sh

Output here is captured on macOS, but I've confirmed the same behavior with file 5.39 in Linux (NixOS).
Steps To Reproduce# system/macOS file
$ file --version
file-5.37
magic file from /usr/share/file/magic

# unpatched /bin/sh shebang
$ file esh
esh: POSIX shell script text executable, ASCII text

# shebang patched by Nix to /nix/store/pcjan45rssdn01cxx3sjg70avjg6c3ni-bash-4.4-p23/bin/sh
$ file nix_esh
nix_esh: a /nix/store/pcjan45rssdn01cxx3sjg70avjg6c3ni-bash-4.4-p23/bin/sh script text executable, ASCII text

# file from nixpkgs
$ file --version
file-5.39
magic file from /nix/store/77p3lid93i5xjgdi9vkj3zqcpf2zddlw-file-5.39/share/misc/magic

# unpatched /bin/sh shebang
$ file esh
esh: POSIX shell script, ASCII text executable

# shebang patched by Nix to /nix/store/pcjan45rssdn01cxx3sjg70avjg6c3ni-bash-4.4-p23/bin/sh
$ file nix_esh
nix_esh: awk or perl script, ASCII text

# file from homebrew
$ file --version
file-5.40
magic file from /usr/local/Cellar/libmagic/5.40/share/misc/magic

# unpatched /bin/sh shebang
$ file esh
esh: POSIX shell script, ASCII text executable

# shebang patched by Nix to /nix/store/pcjan45rssdn01cxx3sjg70avjg6c3ni-bash-4.4-p23/bin/sh
$ file nix_esh
nix_esh: awk or perl script, ASCII text
TagsNo tags attached.

Activities

abathur

2021-07-26 15:24

reporter  

esh (4,302 bytes)    
#!/bin/sh
# vim: set ts=4:
#---help---
# Usage:
#   esh [options] [--] INPUT [VARIABLE ...]
#   esh <-h | -V>
#
# Process and evaluate an ESH template.
#
# Arguments:
#   INPUT      Path of the template file or "-" to read from STDIN.
#   VARIABLE   Variable(s) specified as NAME=VALUE to pass into the template
#              (the have higher priority than environment variables).
#
# Options:
#   -d         Don't evaluate template, just dump a shell script.
#   -o FILE    Output file or "-" for STDOUT. Defaults to "-".
#   -s SHELL   Command name or path of the shell to use for template
#              evaluation. It must not contain spaces. Defaults to "/bin/sh".
#   -h         Show this help message and exit.
#   -V         Print version and exit.
#
# Environment Variables:
#   ESH_AWK    Command name of path of the awk program to use.
#              It must not contain spaces. Defaults to "awk".
#   ESH_SHELL  Same as -s.
#
# Please report bugs at <https://github.com/jirutka/esh/issues>.
#---help---
set -eu

# Set pipefail if supported.
if ( set -o pipefail 2>/dev/null ); then
    set -o pipefail
fi

readonly PROGNAME='esh'
readonly VERSION='0.1.1'

AWK_PROGRAM=$(cat <<'AWK'
function fputs(str) {
	printf("%s", str)
}
function read(len,  _str) {
	if (len == "") {
		_str = buff
		buff = ""
	} else if (len > 0) {
		_str = substr(buff, 1, len)
		buff = substr(buff, len + 1, length(buff))
	}
	return _str
}
function skip(len) {
	buff = substr(buff, len + 1, length(buff))
}
function flush(len,  _str) {
	_str = read(len)

	if (state == "TEXT") {
		gsub("'", "'\\''", _str)
	}
	if (state != "COMMENT") {
		fputs(_str)
	}
}
BEGIN {
	FS = ""
	buff = ""

	if (shell) {
		print("#!" (shell ~ /\// ? shell : "/usr/bin/env " shell))
	}
	print("set -eu")
	print("if ( set -o pipefail 2>/dev/null ); then set -o pipefail; fi")
	print("__print() { printf '%s' \"$*\"; }")
	print(vars)

	fputs("\nprintf '%s' '")
	state = "TEXT"
}
{
	buff = $0
	while (buff != "") {
		print_nl = 1

		if (state == "TEXT" && match(buff, /<%/)) {
			flush(RSTART - 1)  # print buff before "<%"
			skip(2)  # skip "<%"

			flag = substr(buff, 1, 1)
			if (flag != "%") {
				fputs("'\n")  # close text
			}
			if (flag == "%") {  # <%%
				skip(1)
				fputs("<%")
			} else if (flag == "=") {  # <%=
				skip(1)
				fputs("__print ")
				state = "TAG"
			} else if (flag == "#") {  # <%#
				state = "COMMENT"
			} else {
				state = "TAG"
			}
		} else if (state != "TEXT" && match(buff, /%>/)) {
			flag = RSTART > 1 ? substr(buff, RSTART - 1, 1) : ""

			if (flag == "%") {  # %%>
				flush(RSTART - 2)
				skip(1)
				flush(2)
			} else if (flag == "-") {  # -%>
				flush(RSTART - 2)
				skip(3)
				print_nl = 0
			} else {  # %>
				flush(RSTART - 1)
				skip(2)
			}
			if (flag != "%") {
				fputs("\nprintf '%s' '")
				state = "TEXT"
			}
		} else {
			flush()
		}
	}
	if (print_nl && state != "COMMENT") {
		fputs("\n")
	}
}
END {
	if (state == "TEXT") {
		fputs("'\n")
	}
}
AWK
)
readonly AWK_PROGRAM

help() {
	sed -En '/^#---help---/,/^#---help---/p' "$0" | sed -E 's/^# ?//; 1d;$d;'
	exit ${1:-0}
}

convert() {
	local input="$1"
	local vars="$2"
	local evaluate="${3:-yes}"

	if [ "$evaluate" = yes ]; then
		$ESH_AWK -v vars="$vars" -- "$AWK_PROGRAM" "$input" | $ESH_SHELL -s
	else
		$ESH_AWK -v vars="$vars" -v shell="$ESH_SHELL" -- "$AWK_PROGRAM" "$input"
	fi
}

: ${ESH_AWK:="awk"}
: ${ESH_SHELL:="/bin/sh"}
EVALUATE='yes'
OUTPUT=''

while getopts 'dho:s:V' OPT; do
	case "$OPT" in
		d) EVALUATE=no;;
		h) help 0;;
		o) OUTPUT="$OPTARG";;
		s) ESH_SHELL="$OPTARG";;
		V) echo "$PROGNAME $VERSION"; exit 0;;
		'?') help 1 >&2;;
	esac
done
shift $(( OPTIND - 1 ))

[ $# -ge 1 ] || help 1 >&2

INPUT="$1"; shift
[ "$INPUT" != '-' ] || INPUT=''

# Validate arguments.
for arg in "$@"; do
	case "$arg" in
		*=*) ;;
		*) echo "$PROGNAME: illegal argument: $arg" >&2; exit 1;;
	esac
done

# Format variables into shell variable assignments.
vars=''; for item in "$@"; do
	vars="$vars\n${item%%=*}='$(
		printf %s "${item#*=}" | $ESH_AWK "{ gsub(/'/, \"'\\\\\\\''\"); print }"
	)'"
done

if [ "${OUTPUT#-}" ]; then
	tmpfile="$(mktemp)"
	trap "rm -f '$tmpfile'" EXIT HUP INT TERM
	convert "$INPUT" "$vars" "$EVALUATE" > "$tmpfile"
	mv "$tmpfile" "$OUTPUT"
else
	convert "$INPUT" "$vars" "$EVALUATE"
fi
esh (4,302 bytes)   
nix_esh (4,710 bytes)    
#!/nix/store/pcjan45rssdn01cxx3sjg70avjg6c3ni-bash-4.4-p23/bin/sh
# vim: set ts=4:
#---help---
# Usage:
#   esh [options] [--] INPUT [VARIABLE ...]
#   esh <-h | -V>
#
# Process and evaluate an ESH template.
#
# Arguments:
#   INPUT      Path of the template file or "-" to read from STDIN.
#   VARIABLE   Variable(s) specified as NAME=VALUE to pass into the template
#              (the have higher priority than environment variables).
#
# Options:
#   -d         Don't evaluate template, just dump a shell script.
#   -o FILE    Output file or "-" for STDOUT. Defaults to "-".
#   -s SHELL   Command name or path of the shell to use for template
#              evaluation. It must not contain spaces. Defaults to "/nix/store/pcjan45rssdn01cxx3sjg70avjg6c3ni-bash-4.4-p23/bin/bash".
#   -h         Show this help message and exit.
#   -V         Print version and exit.
#
# Environment Variables:
#   ESH_AWK    Command name of path of the awk program to use.
#              It must not contain spaces. Defaults to "/nix/store/8kpxw5na07ggdl2bs8kiwysif7120r6g-gawk-5.1.0/bin/awk".
#   ESH_SHELL  Same as -s.
#
# Please report bugs at <https://github.com/jirutka/esh/issues>.
#---help---
set -eu

# Set pipefail if supported.
if ( set -o pipefail 2>/dev/null ); then
    set -o pipefail
fi

readonly PROGNAME='esh'
readonly VERSION='0.1.1'

AWK_PROGRAM=$(cat <<'AWK'
function fputs(str) {
	printf("%s", str)
}
function read(len,  _str) {
	if (len == "") {
		_str = buff
		buff = ""
	} else if (len > 0) {
		_str = substr(buff, 1, len)
		buff = substr(buff, len + 1, length(buff))
	}
	return _str
}
function skip(len) {
	buff = substr(buff, len + 1, length(buff))
}
function flush(len,  _str) {
	_str = read(len)

	if (state == "TEXT") {
		gsub("'", "'\\''", _str)
	}
	if (state != "COMMENT") {
		fputs(_str)
	}
}
BEGIN {
	FS = ""
	buff = ""

	if (shell) {
		print("#!" (shell ~ /\// ? shell : "/usr/bin/env " shell))
	}
	print("set -eu")
	print("if ( set -o pipefail 2>/dev/null ); then set -o pipefail; fi")
	print("__print() { printf '%s' \"$*\"; }")
	print(vars)

	fputs("\nprintf '%s' '")
	state = "TEXT"
}
{
	buff = $0
	while (buff != "") {
		print_nl = 1

		if (state == "TEXT" && match(buff, /<%/)) {
			flush(RSTART - 1)  # print buff before "<%"
			skip(2)  # skip "<%"

			flag = substr(buff, 1, 1)
			if (flag != "%") {
				fputs("'\n")  # close text
			}
			if (flag == "%") {  # <%%
				skip(1)
				fputs("<%")
			} else if (flag == "=") {  # <%=
				skip(1)
				fputs("__print ")
				state = "TAG"
			} else if (flag == "#") {  # <%#
				state = "COMMENT"
			} else {
				state = "TAG"
			}
		} else if (state != "TEXT" && match(buff, /%>/)) {
			flag = RSTART > 1 ? substr(buff, RSTART - 1, 1) : ""

			if (flag == "%") {  # %%>
				flush(RSTART - 2)
				skip(1)
				flush(2)
			} else if (flag == "-") {  # -%>
				flush(RSTART - 2)
				skip(3)
				print_nl = 0
			} else {  # %>
				flush(RSTART - 1)
				skip(2)
			}
			if (flag != "%") {
				fputs("\nprintf '%s' '")
				state = "TEXT"
			}
		} else {
			flush()
		}
	}
	if (print_nl && state != "COMMENT") {
		fputs("\n")
	}
}
END {
	if (state == "TEXT") {
		fputs("'\n")
	}
}
AWK
)
readonly AWK_PROGRAM

help() {
	/nix/store/fnzsi837b2xqqsfiq7hb61v5xka98avl-gnused-4.8/bin/sed -En '/^#---help---/,/^#---help---/p' "$0" | /nix/store/fnzsi837b2xqqsfiq7hb61v5xka98avl-gnused-4.8/bin/sed -E 's/^# ?//; 1d;$d;'
	exit ${1:-0}
}

convert() {
	local input="$1"
	local vars="$2"
	local evaluate="${3:-yes}"

	if [ "$evaluate" = yes ]; then
		$ESH_AWK -v vars="$vars" -- "$AWK_PROGRAM" "$input" | $ESH_SHELL -s
	else
		$ESH_AWK -v vars="$vars" -v shell="$ESH_SHELL" -- "$AWK_PROGRAM" "$input"
	fi
}

: ${ESH_AWK:="/nix/store/8kpxw5na07ggdl2bs8kiwysif7120r6g-gawk-5.1.0/bin/awk"}
: ${ESH_SHELL:="/nix/store/pcjan45rssdn01cxx3sjg70avjg6c3ni-bash-4.4-p23/bin/bash"}
EVALUATE='yes'
OUTPUT=''

while getopts 'dho:s:V' OPT; do
	case "$OPT" in
		d) EVALUATE=no;;
		h) help 0;;
		o) OUTPUT="$OPTARG";;
		s) ESH_SHELL="$OPTARG";;
		V) echo "$PROGNAME $VERSION"; exit 0;;
		'?') help 1 >&2;;
	esac
done
shift $(( OPTIND - 1 ))

[ $# -ge 1 ] || help 1 >&2

INPUT="$1"; shift
[ "$INPUT" != '-' ] || INPUT=''

# Validate arguments.
for arg in "$@"; do
	case "$arg" in
		*=*) ;;
		*) echo "$PROGNAME: illegal argument: $arg" >&2; exit 1;;
	esac
done

# Format variables into shell variable assignments.
vars=''; for item in "$@"; do
	vars="$vars\n${item%%=*}='$(
		printf %s "${item#*=}" | $ESH_AWK "{ gsub(/'/, \"'\\\\\\\''\"); print }"
	)'"
done

if [ "${OUTPUT#-}" ]; then
	tmpfile="$(mktemp)"
	trap "rm -f '$tmpfile'" EXIT HUP INT TERM
	convert "$INPUT" "$vars" "$EVALUATE" > "$tmpfile"
	mv "$tmpfile" "$OUTPUT"
else
	convert "$INPUT" "$vars" "$EVALUATE"
fi
nix_esh (4,710 bytes)   

christos

2021-07-30 08:44

manager   ~0003630

With the HEAD of the file code this reports:
$ ./file -m ../magic/magic.mgc ~/nix_esh
/Users/christos/nix_esh: a /nix/store/pcjan45rssdn01cxx3sjg70avjg6c3ni-bash-4.4-p23/bin/sh script, ASCII text executable

abathur

2021-07-31 22:28

reporter   ~0003633

Thanks--I'll keep an eye out for the next release.

(I do see the same after figuring out how to rebuild the Nix package from the latest commit on the GH mirror).

christos

2021-10-28 15:33

manager   ~0003654

Release has been out.

abathur

2024-04-04 14:22

reporter   ~0004025

This morning I noticed that this regressed at some point between 5.43 and 5.44:

$ file --version
file-5.44
...

$ file /nix/store/y2jm9585rj81y9ks04b7ky2631ahgv3s-esh-0.1.1/bin/esh
/nix/store/y2jm9585rj81y9ks04b7ky2631ahgv3s-esh-0.1.1/bin/esh: awk or perl script, ASCII text

$ file --version
file-5.43
...
$ file /nix/store/y2jm9585rj81y9ks04b7ky2631ahgv3s-esh-0.1.1/bin/esh
/nix/store/y2jm9585rj81y9ks04b7ky2631ahgv3s-esh-0.1.1/bin/esh: a /nix/store/7xmqfgfgipjypqprhz0xw6bd3jb58z3y-bash-5.2p26/bin/sh script, ASCII text executable

christos

2024-04-06 13:55

manager   ~0004026

Fixed again, thanks!

Issue History

Date Modified Username Field Change
2021-07-26 15:24 abathur New Issue
2021-07-26 15:24 abathur File Added: esh
2021-07-26 15:24 abathur File Added: nix_esh
2021-07-30 08:43 christos Assigned To => christos
2021-07-30 08:43 christos Status new => assigned
2021-07-30 08:44 christos Status assigned => feedback
2021-07-30 08:44 christos Note Added: 0003630
2021-07-31 22:28 abathur Note Added: 0003633
2021-07-31 22:28 abathur Status feedback => assigned
2021-10-28 15:33 christos Status assigned => resolved
2021-10-28 15:33 christos Resolution open => fixed
2021-10-28 15:33 christos Note Added: 0003654
2024-04-04 14:22 abathur Status resolved => feedback
2024-04-04 14:22 abathur Resolution fixed => reopened
2024-04-04 14:22 abathur Note Added: 0004025
2024-04-06 13:55 christos Status feedback => resolved
2024-04-06 13:55 christos Fixed in Version => 5.46
2024-04-06 13:55 christos Note Added: 0004026