intermittent site slowness

[HollyKNY]

Any help is welcome! Would you say that sporadic site slowness is a hosting company issue? I have built several sites for customers using Genesis and PixelHappy. All are small sites, many don't have blogs, very limited plug-in use. On a few sites we have experienced excruciating sporadic slowness - like 30-60 seconds to open a page on the admin side, another 60 seconds to publish a change, 30-60 to upload a graphic (small, under 200kb), and often it just hangs, 30-60 to refresh the site in a browser, etc. Hours later, the site will be fine and fast.

Each hosting company has told me I need to optimize the DB and install caching plug-ins. Optimizing the DB has not always helped and I don't want to use a plug-in if I don't have to, because my guess is that this may be their issue. If it was a caching issue, wouldn't it be consistently slow and would a caching plug-in help with dashboard side issues? My guess is not but would love some advice.

Thanks!