share
Unix & LinuxHow to use grep -f and perl to replace file inline substituting new lines with ";X"
[0] [2] Arthur Accioly
[2020-04-14 17:43:54]
[ grep perl xargs ]
[ https://unix.stackexchange.com/questions/580055/how-to-use-grep-f-and-perl-to-replace-file-inline-substituting-new-lines-with ]

I have the following file:

$$ head on_this_file.txt
Instance,Session,SenderCompID,Type,SrcAddr,SrcPort,DstAddr,DstPort,Protocol,Client,MIC,curr
304,PBAR29,PBAR29,V,146.127.180.96/27,,162.69.138.157,40008,pillar,DESH,ARCX,USD
336,PBAR29,PBAR29,V,146.127.180.64/28,,162.69.138.157,40008,pillar,DESH,ARCX,USD
304,PBAR36,PBAR36,V,146.127.180.96/27,,162.69.142.4,40015,pillar,DESH,ARCX,USD
336,PBAR36,PBAR36,V,146.127.180.64/28,,162.69.142.4,40015,pillar,DESH,ARCX,USD
304,PBAR28,PBAR28,V,146.127.180.96/27,,162.69.142.109,40007,pillar,DESH,ARCX,USD
336,PBAR28,PBAR28,V,146.127.180.64/28,,162.69.142.109,40007,pillar,DESH,ARCX,USD
310,PBAR88,PBAR88,V,146.127.197.128/26,,162.69.142.207,40285,pillar,SQOL,ARCX,USD
346,PBAR88,PBAR88,V,146.127.168.64/27,,162.69.142.207,40285,pillar,SQOL,ARCX,USD
304,PBAR31,PBAR31,V,146.127.180.96/27,,162.69.138.62,40010,pillar,DESH,ARCX,USD

I'm trying to change all the lines that start with the lines on this other file:

$$ cat change_these_lines_only.txt
211
304
310
328
342

So then the lines will have ";X" in the end. For example, the first line:

304,PBAR29,PBAR29,V,146.127.180.96/27,,162.69.138.157,40008,pillar,DESH,ARCX,USD

would be:

304,PBAR29,PBAR29,V,146.127.180.96/27,,162.69.138.157,40008,pillar,DESH,ARCX,USD;X

I've tried the following command:

grep -f change_these_lines_only.txt on_this_file.txt | xargs -L1 -I {} sh -c "perl -p -i -e 's/{}\n/{};X\n/' on_this_file.txt"

But it's clearly not working:

Number found where operator expected at -e line 1, near "s/304,PBAR29,PBAR29,V,146.127.180.96/27,,162.69.138.157,40008,pillar,DESH,ARCX,USD\n/304"
Backslash found where operator expected at -e line 1, near "X\"
syntax error at -e line 1, near "s/304,PBAR29,PBAR29,V,146.127.180.96/27,,162.69.138.157,40008,pillar,DESH,ARCX,USD\n/304"
Execution of -e aborted due to compilation errors.
(...)

I've tried many modifications of this same command, but no luck. Can somebody point me in the right direction? Thank you a lot!

[+1] [2020-04-14 20:12:29] Gerard H. Pille [ACCEPTED]

Not much experience with perl, sorry.

for X in $(cat change_this_lines_only.txt)
do
  sed -i "/^${X},/s/\$/;X/" on_this_file.txt
done

Still don't know what was wrong with my other command, but this works perfectly. Thank you. - Arthur Accioly
(1) I'm afraid there are a number of problems, most important: it is much too complex. Remember to KISS! For instance, suppose a line doesnt start with "304,", but contains 304 somewhere in the middle. You'd need to change the pattern file to contain "^304,", "^211," etc. Then, in the substitute pattern, you're using "/" as a separator, but the "/" appears in the lines too, creating an invalid substitution, and perl gets lost. - Gerard H. Pille
1
[+1] [2020-04-14 20:50:24] Stalin Vignesh Kumar

Using awk but have to re-direct the output to some other file:

This way we can avoid for loop and cat command

$ awk -F"," '
FNR==NR{ a[$1]=$1;next} (FNR==1){print $0;}((NR > 1) && (a[$1]==$1)){ print $0",X"}' change_these_lines_only.txt on_this_file.txt 
Instance,Session,SenderCompID,Type,SrcAddr,SrcPort,DstAddr,DstPort,Protocol,Client,MIC,curr
304,PBAR29,PBAR29,V,146.127.180.96/27,,162.69.138.157,40008,pillar,DESH,ARCX,USD,X
304,PBAR36,PBAR36,V,146.127.180.96/27,,162.69.142.4,40015,pillar,DESH,ARCX,USD,X
304,PBAR28,PBAR28,V,146.127.180.96/27,,162.69.142.109,40007,pillar,DESH,ARCX,USD,X
310,PBAR88,PBAR88,V,146.127.197.128/26,,162.69.142.207,40285,pillar,SQOL,ARCX,USD,X
304,PBAR31,PBAR31,V,146.127.180.96/27,,162.69.138.62,40010,pillar,DESH,ARCX,USD,X

Very interesting, thank you. I was trying to understand your command and arrived at this other link (in case someone else gets confused): stackoverflow.com/questions/32481877/… - Arthur Accioly
Thanks. Sorry i could have explained ... - Stalin Vignesh Kumar
2