Perform some operation on sharepoint site from VSTO appliaction

Our project is a VSTO application build in .NET framework 4.0 client profile.

We need to perform some operations on SharePoint site from VSTO application - like create document library, upload and delete files, etc. We need our application to work with all versions of SharePoint server.

We have tried the Microsoft.sharepoint dll with sharepoint 2010 which only supports up to .NET framework version 3.5.

How can our .net 4.0 VSTO application work with SharePoint server 2010? 

Any help will be much appreciable. 

Thanks,

Saurabh