Detecting the use of named pipes is made rather simple thanks to the pipelist.exe tool released by sysinternals. Here is some sample output of the tool in action:
PipeList v1.01
by Mark Russinovich
http://www.sysinternals.com
Pipe Name Instances Max Instances
--------- --------- -------------
TerminalServer\AutoReconnect 1 1
InitShutdown 2 -1
lsass 6 -1
protected_storage 2 -1
ntsvcs 4 -1
scerpc 2 -1
net\NtControlPipe1 1 1
SfcApi 2 -1
net\NtControlPipe2 1 1
net\NtControlPipe3 1 1
...
This displays the pipe name and number of instances. It also shows the user the number of max instances of the named pipe. A max instance value of -1 means the pipe can handle multiple instances simultaneously.
It is important to note that named pipes can be handled in two distinct ways when it comes to the security of those pipes. Firstly a LPSECURITY_ATTRIBUTES structure can be setup to define the security attributes of the named pipe. This influences the security restrictions put upon the named pipe and the amount of access allowed to clients interacting with the named pipe. Many times you will see the security descriptor given a NULL value which defines the default ACL (Access Control List). Additionally it is important to check if the named pipe is contained within the registry key HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Services/lanmanserver/parameters/NullSessionsPipes. This key is used to determine the amount of authorization required for the named pipe. If the named pipe appears in this key it will allow Null Sessions which means no authentication is required to access the named pipe.
Now we know that named pipes are in use. But we need to track them down to find out what applications created them. Luckily sysinternals comes to the rescue once again with the tool filemon. Normally people think of filemon as a way of tracking which files are read from and written to during a processes execution. This tool also provides the option to monitor processes for named pipe creation and interaction. The option is available under the Volumes menu item. With the named pipe option checked you will then see all instances of interaction or creation of named pipes. (Note: Recently filemon and regmon were combined into Process Monitor for modern Windows Operating Systems such as Windows 2000 and up. Filemon does however, still work for the processes described above.)
Now hopefully, you have discovered which applications create a specific named pipe. Here comes the fun part. Finding the location within the binary code that actually creates the named pipe and interacts with clients.
When looking through the binary code the first thing to do is inspect the imports for CreateNamedPipe(). This the function used to create the named pipe. The following MSDN link describes the function and arguments better than I ever could: http://msdn.microsoft.com/en-us/library/aa365150(VS.85).aspx
Interacting with a client is rather simple. The ReadFile API is passed an hPipe handle which is created from the CreateNamedPipe() function. This will read data from the connected client. Additionally WriteFile is used to write reply messages back to the client. As you can see this is similar to socket communications except its just using normal file system related APIs to accomplish the job. So to find the code which interacts with the client connections following the hPipe value through the code should lead you down the right path. Additionally, clients communicate with the named pipe server using the CreateFile API to establish a connection and again using ReadFile and WriteFile to communicate accordingly .
