For RDS, I use an older version of stereo tool and run it in the background with Task Scheduler. It works very well. I set it to use an unused sound device as the input and then set the output to my 192 kHZ capable sound card. This works independently of other software I use for audio processing.
The sound quality is amazing, but dependent on the quality of your audio processing.
The audio inputs are 600 or 10,000 ohms (adjustable with the dip switches on the back). This means that connecting a standard 400 ohm audio device won't be adequate (even though Camel RF says it is) because the volume level will be too low. In order to get normal volume level, I had to overdrive my sound card which caused it to disconnect from the PC. I used and Aphex amplifier to correct the issue (available cheap on ebay).
Also, I had to purchase some XLR audio cables. There are no optical or RCA audio inputs on the transmitter. The antenna output is a type N female. This created a confusing mess trying to connect an SWR meter with UHF connections and a signal attenuator with BNC connections inline with the antenna. Local HAM radio shops or ebay are good sources for cables and adapters.
I like that pre-emphasis can be turned off via dip switch settings, which helps with using professional audio processing. Also, the HMI controls let you select audio input as stereo, L-R Mix, L or R inputs. So if you wanted to try using stereo encoding at your audio processing you can do that. Personally, I didn't have good luck with this. I was able to get great stereo separation by using the L audio input, but loudness was lacking. I'm not sure why as I adjusted my audio processing controls with an oscilloscope. So I just use the transmitter's stereo processing with pre-emphasis disabled.
Lastly, the fan is loud, whinny and irritating. I replaced it with a low db one as I couldn't stand the noise.
I have a 19" rackmount transmitter too. It's a Broadcast Warehouse model manufactured in UK. And everything you experienced is familiar to me.
I use an ART RCA to XLR audio level converter to get the audio up to pro level and an N to SO239 socket converter on the output side. One difference - I use a hardware RDS encoder.
Audio comes via Macbook and Traktor controller (a very stable and reliable combo) or direct from vinyl and analogue dj mixer to the ART device. From there it goes into the XLR audio sockets on the rig. RDS is injected via BNC sockets and audio processing is also done in the rig.
I set the rig to 'natural' compression/limiting as I value sound quality above loudness. It sounds better than anything else out there imo, but maybe I'm biased.
Out if curiosity I bought 500mw and 15w Chinese rigs. They only have 3.5mm audio input sockets and I feed RDS into them via Stereo Tool. To my surprise, it works well!
Questions for you:
Do you know if your rig is driven by a chip? And if so which one?
Do you use Stereo Tool in standalone ie paid licence mode?
What software do you use for playing music? And what do you think of it?