Direct references to images should be avoided in a Silverlight project as it makes the XAP file heavier. An alternative could be to have the images on server and load images by creating bitmap of the image by using the code snippets below-
ImageControl.Source =
new BitmapImage(new Uri("www.mysite.com/images/logo.jpg", UriKind.Absolute));
After I discovered this, I was able to drastically reduce XAP size from 5MB to 250KB.