[FIXED] How to unescape non-usa, en, ASCII type characters using grep?

Issue

I am using grep to parse a friend list obtained via the facebook Open Graph API. I am mostly able to do what I want with the following command, issued in bash:

grep -aiPo '"name":"(.*?)","id":"[[:digit:]]*"' friends?blahblah-access-token-stuff

which yields a list which looks like:

"name":"John Day","id":"--id ommitted--"
"name":"Andria Cast\u00f1eda","id":"--id ommitted--" // let me draw your attention here
"name":"Jane Doe","id":"--id ommitted--"

Names were changed above to preserve privacy

If you notice, there is an unescaped sequence in the middle entry, that corresponds to a tilde N. Is there an easy way to to feed such characters into a java program (my primary intention) so that java understands that \u00f1eda is unicode speak for the curly n?

I would prefer not to solve this problem by parsing the string in java and manually unescaping the unicode. I would very much prefer to instruct grep to handle this situation, or another GNU or open source tool that is widely available for bash.

At that point, I would feed the entire input as a file to a java program without having to worry about OMG, is that a unicode escape sequence!!? Java would naturally detect the unicode characters and map them to it's corresponding internal representation.

Thanks in advance!

Solution

Java understands Unicode. You provide Java Unicode escapes in the following manner:

String str = "\u00F6";

So if you pass a string such as "Andria Cast\u00f1eda" which is an escaped sequence, it should be handled correctly without any additional handling required.

Here's also a very brief, but easy to understand introduction:

Unicode in Java

If you're still not convinced, try this class:

public class UnicodeExample {

    public static void main(String[] args) {
        
        String escaped = new String("\u00f1");
        String unescaped = new String("ñ");
        System.out.println(escaped);        
        System.out.println(unescaped);
        
        if(escaped.equals(unescaped)){
            System.out.println("The strings are the same!");
        }
        else {
            System.out.println("The strings are different!");
        }

    }

}

Answered By - Michael

Answer Checked By - Cary Denson (PHPFixing Admin)

Read More

[FIXED] How to use grep output in EOF?

Issue

I am a new leaner and I am running a program with following script:

./Multiwfn >HF-Dr.out << EOF
HF.fchk
3
21
2
2
grep 'Global surface minimum:' HF-ESP.out | awk '{print $7,$8,$9,$7,$8,$9}'
EXIT
EOF

The output of this grep is something like:

0.043532 -0.032964 1.960094 0.043532 -0.032964 1.960094

I want to use the output of grep instead of input in the script, i.e. I want the script like:

./Multiwfn >HF-Dr.out << EOF
HF.fchk
3
21
2
2
0.043532  -0.032964   1.960094  0.043532  -0.032964   1.960094
EXIT
EOF

Is there any way that I hide (make non-executable) the grep input from my program and use only its output? Thank you in advance.

Solution

I do not understand the 'make non-executable' part of your question. As I understand it, you want it executed before it is given to ./Multiwfn.

There are several solutions:

1) PesaThe's comment:

./Multiwfn >HF-Dr.out << EOF
HF.fchk
3
21
2
2
$(grep 'Global surface minimum:' HF-ESP.out | awk '{print $7,$8,$9,$7,$8,$9}')
EXIT
EOF

2) Create the input in a pipe:

(echo -e "HF.fchk\n3\n21\n2\n2;grep 'Global surface minimum:' HF-ESP.out | awk '{print $7,$8,$9,$7,$8,$9}';echo EXIT) | ./Multiwfn >HF-Dr.out

3) use an intermediate file:

cat > file <<EOF
HF.fchk
3
21
2
2
EOF
grep 'Global surface minimum:' HF-ESP.out | awk '{print $7,$8,$9,$7,$8,$9}' >> file
echo EXIT >> file
./Multiwfn < file >HF-Dr.out
rm file

I'm sure that there are more ways, but this should put you on the right track.

Answered By - Ljm Dullaart

Answer Checked By - Timothy Miller (PHPFixing Admin)

Read More

[FIXED] Which is best way to grep on exec from find command?

Issue

I'm just curious which of these statements would be most resource intensive.

I expect the criteria at times to be 1000s of files, and want to make sure I implement the "safest" execution. Files themselves will be relatively small, but the amount of files might be substantially large.

The two alternatives:

sudo find /home/users -name '*.sh' -type f -exec grep -n 'rm.*tmp.*7z$' {} \+
sudo find /home/users -name '*.sh' -type f -exec grep -Hn 'rm.*tmp.*7z$' {} \;

As you can see the only difference is whether I should use the + or the ;

Solution

The first one is going to run grep fewer times than the second, which will launch one instance per file. grep's startup time is pretty fast, so it might not be much of a visible improvement, but the first one will be more efficient, the second one more resource intensive.

(You'll want to add -H to the first grep's options too, just in case it gets run with a single filename argument at some point.)

Answered By - Shawn

Answer Checked By - Marilyn (PHPFixing Volunteer)

Read More

[FIXED] How do I get the dates of files in my output from a grep search?

Issue

I can search through all my files for a specific string and output in a txt document without issue. I can't seem to figure out how to also capture the date of the files in the output results.

grep -rnw '/my_path/' -e 'search_string' > list.txt

That works for finding the files, but when I try piping any additional commands like stat, date, or ls I can't seem to get the date of the files to output with file names.

grep -rnw '/my_path/' -e 'search_string' | stat -c %n':'%z > list.txt

This does not seem to work to get the dates of the files in my output.

Solution

The problem is that grep is outputting the line that match your string, not the file name, so that in your second example your trying to call stat on a string, not on a file!

You should add a -l parameter to your grep command in order to not output the matching line but the file that contains it. Try this:

grep -lrnw '/my_path/' -e 'search_string' | stat -c %n':'%z > list.txt

[EDIT] Anyway this would not work because the stat command does not accept input from a pipe. The solution is then

stat -c %n':'%z $(grep -lrnw '/my_path/' -e 'search_string') > list.txt

Answered By - luco5826

Answer Checked By - Pedro (PHPFixing Volunteer)

Read More

[FIXED] How to grep following words from string

Issue

How can I grep the nodeport which is the "31000" under the PORT(S) , it can and will change, from the following kubernetes command output:

NAME       TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
vault-ui   NodePort   10.96.62.155   <none>        8200:31000/TCP   23m

and how can I grep the ip address(in this case 192.168.99.213) but not the port(8443) or the https:// in following output, this output also changes depending on the ip:

Kubernetes master is running at https://192.168.99.213:8443
KubeDNS is running at https://192.168.99.213:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Solution

kubecmd | awk -F[:] 'NR>1 { split($2,arr,"/");print arr[1]}'

For the first scenario, you can use awk as above. Use : as a field separator and then further split the second delimited piece on / into array arr printing the first element this array the

kubecmd | sed -En 's/(^.*Kubernetes master.*https:\/\/)(.*)(:.*$)/\2/p'

For the second scenario, use sed as above. Split the text into three sections and print the second section

Answered By - Raman Sailopal

Answer Checked By - Mary Flores (PHPFixing Volunteer)

Read More

[FIXED] how to make grep ignore first line and process other line

Issue

I need to remove line beginning with '#' in some txt file. but ignoring the first line as it header. how to make grep ignore first lines and remove any line beginning with # for rest of the lines?

cat sample.txt
#"EVENT",VERSION, NAME
1,2,xyz
1,2,abc
1,2,asd
1,2,ert
#"EVENT",VERSION, NAME
1,2,xyz
1,2,abc
1,2,xyz

cat sample.txt | grep -v "^\s*[#\;]\|^\s*$" > "out.txt"

but this removes the header too!

Solution

With sed:

sed '2,${/^#/d}' sample.txt

From second row (2) to last row ($): search (/.../) for rows beginning (^) with # and delete (d) them. Default action of sed is to print current row.

Output:

#"EVENT",VERSION, NAME
1,2,xyz
1,2,abc
1,2,asd
1,2,ert
1,2,xyz
1,2,abc
1,2,xyz

Answered By - Cyrus

Answer Checked By - Cary Denson (PHPFixing Admin)

Read More