By default, the X server listens on port 6000 for incoming connections. If you are directly connected to the internet, or don't trust others connected to your LAN, this could be a potential security risk.
Disabling this can accomplished one of two ways. The first possible way is to run startx with the -nolisten tcp flag, i.e.
startx -- -nolisten tcp
The more permanent way to do it (globally) is to edit the startx script. It will be located at /usr/X11R6/bin/startx if you installed the base X11, or /usr/pkg/xorg/bin/startx if you installed X.org from pkgsrc.
Change the serverargs line to look like this:
If you are not the administrator of the system or you like to let free the users to choose their configuration, edit (or create) the ~/.xserverrc file and put this line to it:
exec X -nolisten tcp
Restart X, and you're all set!