Swarthmore is an intense community of learning. If that's what you're looking for, this is an excellent school to look at! You will be living with wonderful, good-hearted scholars, inundated with challenging and rewarding coursework, and surrounded by a lovely arboretum. The biggest downside is that you'll never want to leave "the Swarthmore Bubble". It's easy to forget there's a world out there, and even a major city twenty minutes away. Don't forget to leave campus occasionally to stay grounded--Philly is only a short train ride away.
Swarthmore is the right place for the academically intense, the earnest seeker of knowledge, and the guy that joined the math team for the fun of it. I have and had enormous respect for the people with whom I attended school. I have never met a group of more intellectually diverse, individually accomplished folks then or since. My college roommates consisted of an concert violin player and economist; an engineering guy that could fix anything; and an astrophysicist who computed Pi for fun on the weekends. I'm good friends with two out of three today. (I won't say which ones!)
This is not to say that Swarthmore doesn't party. The party was as an intense as the work. Swarthmore students were almost uniformly incredibly personally motivated. They weren't there because someone thought they should be. They didn't succeed (or fail) because that was their parents or peers expectation. They were there because they wanted to be.
There is an inherent weakness in this strength as well. Swarthmore's individual intensity can devolve into navel-gazing and an overt and thick earnestness, occasionally a humorlessness. It never became malicious or competitive, however.
I'd go again, but I wouldn't go back, if that makes sense. Good luck!