I similarly liken Lisp to a lightsaber–if such a thing existed, the first thing you’d probably wind up doing with it would be cutting off your own arm (or head). Lisp is like that. Yes, it’s powerful, but unless you’re willing to take up life as a celibate space monk, your chances of successfully wielding said power are slim. Sadly, however, whereas the Jedi gain telekinesis and power over the weak-minded in the bargain, Lisp programmers gain power over EMACS and something other than hairstyle in common with Richard Stallman. If you ask me, it’s a rather poor trade–but I’m biased. I freely admit that I hate pretty much everything about Lisp. I understand it fairly well, and I certainly have an appreciation for the problems Lisp is designed to solve, but they aren’t problems in which I’m especially interested. If you care about lambda calculus and questions of computability and such, Lisp or another (probably related) domain-specific functional language may well be what you need (R, Wolfram, Opal, etc.).
Python, by comparison, has no learning curve. You could probably write the Python version of “Hello world” in fairly short order by pulling letters and symbols out of a bag and arranging them at random. Python’s simple, regular, consistent syntax and enormous library of modules make it very attractive to new programmers and experienced programmers who simply need to write correct, maintainable code quickly. That isn’t to say that scientific and mathematical programming is absent from Python. On the contrary, SciPy, NumPy and Matplotlib (especially when put together with iPython’s interactive notebook function) are rapidly changing the face of how many people do research and scientific computing.
Python is popular because it’s accessible, powerful, and free. Accessibility, especially in a programming language is one of the best characteristics it can have–it democratizes the power of computing and spreads its benefits to more people. Lisp dates from an era when real computers (by which I mean computers running some flavor of Unix) were not in the hands of individuals outside of academia, government, big business, and the military. Python, on the other hand, is a child of the computing environment that gave birth to the current revolution in general use of Unix and Linux. Lisp was designed for computer scientists, by computer scientists, to explore questions that p computer scientists. Python was designed by a programmer, for programmers to write programs. There are far, far more people who fit the second set of criteria than the first and that, in the end, is why Python is more popular than Lisp.
Python’s popularity arises from a combination of factors. Primary among these are:
Python’s standard libraries include quite a few modules which are of immediate practical use. They’re ideal for working with zip, gzip, tar (and gzip’d tar) files, comma-separated values (CSV), JSON and XML, various forms of “dbm” files, and many common protocols and services … quickly and “out-0f-the-box.”
Python is relatively easy to use, with a syntax that can read almost like English and a structure that approximates the pseudo-code used by programmers of almost all languages to convey concepts while eschewing finer syntactic points. The additions of list comprehensions, and later generator expressions, dictionary and set comprehensions, and decorators have make Python’s syntax a bit more complicated. But it’s a relatively smooth slope from the basic syntax that anyone can learn in about a half hour up to all of those and more.
Python is pre-installed on most non-Windows systems and has been for about a dozen years. The pre-installed version and its standard libraries is generally enough to get started and, by the time one might be chafing at any limitations from that, the language has proven itself worth the investment to download a more recent version and deal with the complexities of maintaining multiple Python versions, virtual environments, etc.
Lisp, by contrast, has not normally been pre-installed (except, perhaps, for the special case of “elisp” as part of Emacs). It’s generally not been known for including libraries for handling all sorts of practical, real-world, common, file formats, libraries (curses) and protocols; and while Lisp syntax is minimal, its semantics present a rather steep learning curve. (Perhaps better described as a brick wall).
For these reasons Lisp has, traditionally, only appealed to CS students and mostly those from the Eastern seaboard (MIT, especially).
If Clojure starts to be included with most Linux distributions and MacOS X installations then that will nail the availability. (I’d think that Clojure might become part of a de facto professional “Java” development suite; although that’s a long shot).
Even after all that Clojure is still Lisp and the semantics still present an initial challenge. It’s likely that Lisp and Clojure don’t require more over all mental effort to understand … but rather than the effort is heavily front-loaded; that it gets *much* easier after you’ve climbed that first grade.