A little background on this is that people are having trouble communicating with a samba share on Windows 7 even though they can communicate with every other version of windows. So I gave it a try and they're right I can no longer establish an SMB connection; the server responds with an "Out of Memory" error. So now came the incredibly frustrating attempt to solve the problem (which turned out successful).

My initial thoughts were that it was because I was using smb1 and the server would only communicate with smb2.1. So I did some real thorough packet inspection, determined what would need to be changed, yelled a bunch of profanity for a couple of days before beginning, then realized this was not the solution. While I was figuring out what needed to be done, I decided to analyze samba packets which is when I realized it [samba] was still able to communicate with win7 over smb1 so that wasn't my problem (well kind of, it was at least getting further than I was but still not a fully working smb transaction).

Okay, so now that I'm pretty sure smb2.1 isn't my problem, I scour google and start finding (to summarize) that windows 7 is set up by default to not be a file server and you have to enable that in the registry of the server. Well that doesn't sound practical but I can do it to test out my code and see if there's any other problems. So I change the registry values and it works! I can now connect to the smb server without getting an "Out of Memory" error.
HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\LargeSystemCache = 1
HKLM\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters\Size = 3

Things are looking great, now lets do what we came here to do and grab data from the registry. Crap, BAD_FILE errors... Well by the name of that error I'm thinking there's three things that could be wrong: 1) The WINREG protocol has been changed so I can no longer access the registry. 2) The registry key I'm looking for does not exist in windows 7. 3) I don't have permission to the registry key but since when do you need special permission to access the registry?

Well it's not really possible to debug problem #1 so I move on to 2. Well whatta ya know, the registry key doesn't exist. That's going to cause some problems for me later (out of the scope of this article) but lets try to access a key that actually exists. Hey it works! ok lets try some more keys that definitely exist? nope those don't work... wtf? Well I know the keys exist so scratch problem #2; I got one key so it's not problem #1; that leaves problem #3, permissions.

I google registry permissions and find out there is such a thing (I haven't been a windows sysadmin in nearly 10 years, back then there were no permissions). So now I realize I can view the permissions of a hive by right clicking and going to the "Permissions..." (have you ever right clicked a hive before???) then you can view a fairly confusing permissions interface. So I compare the permissions between the working key's hive and the not working one but they're nearly identical. I make changes so they are identical but it's still no go. So this is where the banging my head against the desk comes into play. Eventually I stumble across something from google called "remote registry" and I think hey, I'm a remote user trying to access the registry! So I read up on it and really only find stuff on the theory of remote registry, no actual implementation. Sooner or later (mostly later) I start finding stuff about implementation and realize that it needs 3 things.
1) it's a service you need to turn on
***It wasn't but is now.
2) HKLM\SYSTEM\CurrentControlSet\Control\SecurePipeServers\winreg\Description must be "Registry Server"
***Already set
3) The user logging in over SMB must have read permission to HKLM\SYSTEM\CurrentControlSet\Control\SecurePipeServers\winreg
***Of course it wasn't set

So now I have all that and vioala, I can now access the registry over smb. So now I'm thinking I am so damned over dealing with SMB, I no longer care about those registry changes to make the server believe it's a file server. I actually don't even know what it means that it's doesn't think it's a file server... First of all I'm not actually try to grab files just values from the registry. Second what counts as a file server? I have file sharing/smb on obviously I want to share files. At some point I read that if you want to share to 3 computers you need it to be a file server but I'm just one computer and one connection. So I leave it alone for a couple of weeks.

Then people start telling me that setting those registry keys and having to do those things for the remote registry isn't an acceptable answer. So I decide to take another look at it and continue where I left off, packet sniffing. I notice (something I ignored before) that with those registry keys shut off, I can't connect to the registry but regshell can (regshell is an awesome tool in the newer samba source that gives you a shell prompt on a remote registry). Not only can regshell connect, if I try my connection and regshell back to back on the same machine, regshell can always connect but I cant.

So back to wireshark (another awesome tool which was incredibly helpful during all of this). I decided to forge my packets to be identical to samba but that still doesn't work. While doing that though I notice that I'm sending out a few NBSS (continuation messages and server authentication messages) packets before ever trying to connect with SMB. So the first thing I do is try to not send those out which took a while since I had no idea where they were coming from. Eventually I manage though and hey I can now communicate overs SMB without the registry key modifications. So what the hell were those NBSS packets and why were they keeping me from connecting using SMB? Well it turns out they were sent while trying to identify the port and the packets themselves weren't the problem. The problem was that the forked process that was trying to identify the port wasn't closing the connection until after my SMB communication program started. This was causing me to have 2 simultaneous connections to the SMB port which I then realized is what they meant by "file server." Only a file server would need to have simultaneous SMB connections open to the same host, so it wouldn't allow me to open a second connection. As I write this I realize that "Out of Memory" is a pretty accurate error description since it didn't have the memory available to open a second connection.

So that's where I'm at now and hoping to leave it. I can now remotely access the registry/files of a windows 7 without having to make changes to registry keys. Of course you still need to do the remote modifications changes (users logged in locally can access the registry without having to assign permissions but if you want to access the registry remotely, they need permission), but I think that's a fair thing to need to do since disabling remote access to it by default is a security enhancement.

---Eric Kinolik