means that when grep is exec-ed, it receives these arguments:
argv[0] = "grep" /* possibly full path, irrelevant here */
argv[1] = "echo <contents of data variable>"
argv[2] = NULL
If this does what you want, it means that you wanted to look for strings starting with "echo ". All of arv[1] is the pattern to look for (because it doesn't look like a command line option). Since there are no other arguments, grep is scanning its standard input.
OTOH
grep `cat filelist.txt`
gets execed like this:
argv[0] = "grep"
argv[1] = "<first 'word' from filelist.txt>"
argv[2] = "<second 'word' from filelist.txt>"
...
argv[n] = "<last 'word' from filelist.txt>"
argv[n+1] = NULL
Needless to say, grep doesn't work this way; it doesn't accept multiple patterns as arguments. We cannot use "grep foo bar baz file" to scan file for occurrences of foo, bar or baz. Rather, what this means is to scan bar, baz and file for occurrences of foo.
Also, there is the risk that material from filelist.txt looks like command line options for grep. If filelist.txt contains "-v foo bar" then grep `cat filelist.txt` means exactly the same thing as "grep -v foo bar".
grep has a -f option which can specify a file whose lines are patterns:
grep -f filelist.txt
If we use the -F option, those patterns are interpreted as fixed strings, like the single pattern given on the command line.
fgrep is the way to go here. It uses a different algorithm that makes only one pass through the file while searching for all strings simultaneously. An extension of trie search.
I don't understand how this is an answer. If I follow your argument correctly, and totally ignoring for a moment what ${data} might evaluate to, then for the first command to produce any output it is necessary for the output of ls to at least contain the string "echo". But we can see that it does not, yet the claim (which I haven't tested) is that it does produce output. Why is that?
0:terada:~$ mkdir testdir
0:terada:~$ cd testdir
0:terada:~/testdir$ data="testfile1 testfile2 testfile3"
0:terada:~/testdir$ touch $data
0:terada:~/testdir$ ls -1
testfile1
testfile2
testfile3
0:terada:~/testdir$ ls | grep "echo ${data}"
1:terada:~/testdir$
I've seen this sort of shenanigan more than once in comp.unix.shell; usually followed by "oops, I somehow confused myself and can no longer reproduce this" type of backpedaling.
The submitted web page doesn't have a properly presented repro test case; it's writing about something which allegedly happened on some machine somewhere, possibly with inaccuracies and omissions.
We don't know what shell this is. Maybe the command that was actually run was ls | `echo ${data}`, data happened to be empty at the time, and the shell being used expanded that backtick expression to an empty argument instead of expanding to no argument. An empty regex matches everything.
This was done on CentOS 7.3.x (not CentOS 6 as I had incorrectly first noted), and the following versions of bash and grep:
"GNU bash, version 4.2.46(1)-release (x86_64-redhat-linux-gnu)"
"grep (GNU grep) 2.20"
So where $data is a newline delimited list of items/patterns:
grep "${data}"
..will indeed match each item/pattern individually and grep over each, eg, just as grep -f does against a file with a list of patterns. It doesn't work without the double quotes, and putting the echo in front was obvious derp in hindsight. See my comment where I replied to
uraza and demonstrated this.
Aha! This is a POSIX-required behavior, not documented in the GNU "info grep" or man page.
According to POSIX, the syntax is:
grep [-E|-F] [-c|-l|-q] [-insvx] -e pattern_list
What is pattern_list? "The pattern_list's value shall consist of one or more patterns separated by <newline> characters"
In the GNU man page and info, the pattern argument is referred to as the PATTERN metavariable, a noun in singular form. The only hint I can find that it can contain a plurality is this, in the man page:
-F, --fixed-strings
Interpret PATTERN as a list of fixed strings (instead of regular
expressions), separated by newlines, any of which is to be
matched.
This drops the hint that there may be multiple regular expressions in PATTERN separated by newlines, which may instead be treated as fixed strings.
A similar description is in the Info manual.
Elsewhere in both documents, PATTERN is misleadingly referred to as "the pattern".
If the output of ls is not a terminal, then it is single-filename-per-line (and sorted alphabetically):
$ ls
nomatchfile1 nomatchfile2 testfile1 testfile10 testfile2 testfile3 testfile4
$ ls | cat
nomatchfile1
nomatchfile2
testfile1
testfile10
testfile2
testfile3
testfile4
The first argument to grep is the pattern, the following arguments to grep are the files to search. If there are any files specified, stdin is not read/searched.
Is ambiguous. Is `data` an array? Is it a string with lots of spaces? Newlines? There's no way to know and I get the sense that we're being nerd sniped.
heh definitely nerd sniped ... compelled to answer when this gets up-voted so people don't think it's some special insightful question, rather than like 5 different critical errors all mixing together with strange results
I don't know what nerd sniping is, but I'm pretty sure I wasn't trying to do that, nor trick anyone. I'll try to better explain how I defined the variable when I get to a computer. Essentially I just copy pasted from the filelist.txt output, into data='copy pasted into here, which also appeared to clearly generate newlines'
Look, I'm kind of at the level of the layman trying to decipher a Numberphile video. You guys are trying to distill it down for me, and I'm desperately trying to grasp at it. But hey, at some point I may grab hold.
I thought I was serviceable in bash/builtins, then I post this and realize just how much I don't know. But hey that just means there's that much more to know and learn.
"Javascript is required for ZeroBin.net to work.
Sorry for the inconvenience."
Here's the contents of the paste:
I stumbled upon/essentially derped the:
ls | grep "echo ${data}"
... line, saying oh this should work... and it actually worked.. I didn't think anymore about it...
... but later on, I thought about it more and decided echoing the variable shouldn't actually work, and you should have to do a for loop instead... and that it should fail in the same way that piping thru a cat of the file should fail (and does fail):
ls | grep `cat ../filelist.txt`
...does not work/no output (as expected)
But, piping to grep and echoing the variable actually works and returns the wanted results/matches!! How is this possible?
It can, you just incorrectly guessed what's the required input for it to work
(the author didn't do a very good job at showing the environment in line 29).
If you pass to grep a multi-line argument as the pattern (or a pattern, for
-e option), grep will treat each line of this argument as a separate pattern.
Or at least that's what GNU grep is doing.
"echo ${data}" worked (and only somewhat and accidentally) because while the
first line of ${data} got a prefix of "echo ", the other lines stayed the
same.
`cat filelist.txt` didn't work, because grep got the content of filelist.txt
as a separate arguments (one word per line, if I guess the content correctly)
thanks to shell's word splitting, so grep treated the first filename as
a pattern and the rest of the filenames as files to search for the pattern. If
the filelist.txt contained a name that didn't exist, grep would print an error
message for ENOENT. Note that in this scenario the output of `ls' was not read
by grep at all.
If it was "`cat ...`" (double-quoted form), wordsplitting wouldn't take place
and the grep call would work as the author was expecting.
Yep, this is the correct answer! I was just writing up something similar, but you beat me to it. This is the expected behavior if the `data` variable has newlines between every word:
...the above has the same/desired output (including the missing testfile1 that the echo derp had missed).
I guess though that I still don't fully understand how/why grep is able to interpret/execute each newline individually, eg kind of cycling through it.
EDIT - I do realize dozzie pretty much explained it, and what's happening here, however still sort of bewildered that grep is able to perform the operation sort of separately on each newline as an individual argument, and sort of parse through each one...
Btw, this was after-the-fact testing. We had a list of about 800 or so unique UUIDS (there was no pattern anywhere between the UUIDS), where the UUIDs were present in some portion of the string for the filenames in working dir.
Example UUID in the list:
fd306bdfb115f1284655da6b06f826eb
There were thousands of other files in the directory that did not have any part/string of the filenames that matched the list. And using the echo $data technique found the matching files.
Is this still accidentally/happenstance matching here?
Realize now of course that 'grep -f uuidlist.txt' would have likely just done it, but kind of glad I've undertaken this exercise, even if it clearly shows my bash skills are subpar.
I've got plenty to learn no doubt, and ny intent in posting this was to learn.
It probably missed the first UUID in the list, because the search term would have been "echo $UUID". grep would correctly parse the rest of the newline-delimited list.
EDIT: and about the question in the pastebin - I guess it has to do with ${var} showing it contains spaces as opposed to `cat file.txt` having newlines.
I'll have to experiment with the -f flag. Completely forgot about this, and would likely have done the trick without having to resort to the echo variable nonsense.
Are the files all empty? Looks like echo is part of the search string and the expansion might be interpreted as the file list. ls is ignored. Just a guess...
Correction, this was done on CentOS 7.3.x (not CentOS 6 as I had incorrectly first noted), and the following versions of bash and grep:
"GNU bash, version 4.2.46(1)-release (x86_64-redhat-linux-gnu)"
"grep (GNU grep) 2.20"
So where $data is a newline delimited list of items/patterns:
grep "${data}"
..will indeed match each item/pattern individually and grep over each, eg, just as grep -f does against a file with a list of patterns. It doesn't work without the double quotes, and putting the echo in front was obvious derp in hindsight. See my comment where I replied to uraza and demonstrated this.
OTOH
gets execed like this: Needless to say, grep doesn't work this way; it doesn't accept multiple patterns as arguments. We cannot use "grep foo bar baz file" to scan file for occurrences of foo, bar or baz. Rather, what this means is to scan bar, baz and file for occurrences of foo.Also, there is the risk that material from filelist.txt looks like command line options for grep. If filelist.txt contains "-v foo bar" then grep `cat filelist.txt` means exactly the same thing as "grep -v foo bar".
grep has a -f option which can specify a file whose lines are patterns:
If we use the -F option, those patterns are interpreted as fixed strings, like the single pattern given on the command line.