How to make sure that my SpeedOfMe server is showing the actual speed?

SpeedOfMe reflects the actual speed that you are experiencing via the browser. Here's how you can make sure:

Try downloading one of the larger sample files (e.g. samplefiles/sample131072k.bin) directly from the browser and watch the download speed. (In Chrome use this address to view downloads and their actual speed: chrome://downloads/ ).

It should be something close to SpeedOfMe measured speed. 

Please keep in mind that browser download speed is usually based on Megabyte/s. So you need to convert it to Mbits/s (multiply it by 8).