SocialTagManager GetUrls only returns current user urls

I am using SocialTagManager for tagging usage in Blog posts. While everything works fine, I found that the GetUrls method doesn't return anything unless you are the owner of the term. If I looked at the API, there are only two of them:

GetUrls (Term term) - Retrieves an array of all URLs that are associated with a specified taxonomy term by aSocialTag object.

GetUrls (Term term, UserProfile profile) - Retrieves an array of all URLs that are owned by the specified user and associated with a specified taxonomy term by aSocialTag object.

The first one should return everything but strangely it's not returning any urls unless I login as the term owner. I have tried to run with elevated permission, and even impersonate as the term owner token but still no luck. Any idea?

http://msdn.microsoft.com/en-us/library/ee579916(v=office.14).aspx