Difficulty is relative - depends on your native language, on other languages you might know, whether you consider grammar, pronunciation, orthography. Also languages tend to balance these 3 elements in terms of complexity in various ways and various individuals will find different elements difficult.
If we're talking about completely mastering a language, then every single language is difficult, unless you live in the country where it's spoken. Esperanto being an exception, since it's an artificial language, and made to be easy (for westerners at least). If we're talking about spoken language then I think korean, japanese and classical arabic would win. Actually any language that has xy levels of politeness. If written - chinese hands down. For me, if written and spoken combined, also chinese - cantonese (don't know much about other dialects).
I'd never call english difficult. Yes the grammar on the proficiency level is a mess and the spelling sucks (actually that's the worst part), but, at least for the western speaker it's far easier than most languages to learn in order to be able to hold a decent conversation. The topic for me means exactly that, learning a language to some advanced level to be able to read, write and hold a proper conversation. What makes english especially easy is the position of english in today's world, the entertainment industry, the internet and the amount of books available in it. I don't know about other countries, but nowadays many kids in croatia learn english words before they learn croatian ones.
I think that the more "difficult" languages to learn to the previously mentioned degree are those that radically differ from your own (or those you know) and that have a bunch of cases for nouns, pronouns, adjectives, genders for all three, a bunch of tenses, etc. Because you have to learn tons of suffixes which largely differ depending on everything mentioned, and which have various exceptions. I think I'd find finnish more challenging than most "mainstream" languages since it has 15-16 cases. I have no idea how long it would take me to learn to get them all right. I haven't looked into greek much but it doesn't seem like a walk in the park either.