Hi guys Thanks for all the replies!
I did aproach the company who originally wrote and packaged the software - but (as expected) they simply said they do not support the x64 architecture and there is no plans to do.
I eventually got it to work! As suspected, the biggest issue was that the installation package registers a dll and I could not get this to work in the beginning... I installed the application on a 32bit XP "source" machine first.
Solution (Please note this explanations might not be 100% "technically correct" but is my understanding and experience):
Firstly to keep in mind - on x64 architecture the SYSTEM32 directory actually contains 64bit DLL's!!! There is also a "Syswow64" directory in Windows which keeps your 32bit DLL's. Yeah sure you'd expect it to be the other way around - but not untypical Microsoft :-)
Secondly - when registering a 32bit DLL on x64, you MUST run the regsvr32.exe from the "Syswow64" directory - using the regsvr32.exe one in system32 is for registering 64bit dll's (there is no regsvr64.exe). ALSO to add some more confusion the regsvr32.exe sees the "syswow64" directory as "system32"... THUS I copied my 32bit dll into the syswow64 directory and ran the following from command prompt:
cd c:\windows\syswow64
regsvr32.exe c:\windows\system32\myDLL.dll (YES although physically the file is located in the syswow64 directory)
You could also just do the following: regsvr32.exe mydll.dll - but for explanation purposes I'd mention both.
So after I copied all the files from source machine to x64 server in the Program Files (x86) directory, and registered my dll as above I got it working! You might have a situation where more than one DLL is required, for that you can use "Filemon.exe" (downloaded from the web) while running regsvr32.exe and the log will tell you which other DLL's is required by the one you are trying to register...
I hope the above makes sense! Feel free to ask any questions.
Cheers
Hendrik