

Show HN: Beautiful Visualization of Agreement Groups in the US Senate - friggeri
http://friggeri.net/senate/

======
uptown
"However [political parties] may now and then answer popular ends, they are
likely in the course of time and things, to become potent engines, by which
cunning, ambitious, and unprincipled men will be enabled to subvert the power
of the people and to usurp for themselves the reins of government, destroying
afterwards the very engines which have lifted them to unjust dominion."

\- GEORGE WASHINGTON, Farewell Address, Sep. 17, 1796

Our democracy has evolved into what Washington had prophesied in his farewell
address ... a system where diversity of thought and political compromise are
rejected in favor of the strength of a unified political party and the power
that brings with it.

When political positions consolidate into just two choices, so do the proposed
solutions to the world's problems. These solutions are artificially
constrained for the sake of political loyalty. Ultimately, party loyalty
requires individual politicians to sacrifice their personal beliefs on
individual issues ... and once those values have been compromised, the only
thing worth clinging to is control.

~~~
OstiaAntica
Political parties are the weakest they've been in 200 years. The driving
factor in American politics are Internet-powered movements like MoveOn.org and
the Tea Party...these groups are capturing the the traditional party
structure, and they are highly issue-driven. Even in fundraising, the
traditional party committees are outflanked by the new Super PACs.

There is more diversity of thought and action in the U.S. today than at any
other time in our history.

------
friggeri
Op here, this is part of my PhD research. Please bare in mind that I'm French
and that my knowledge of U.S politics is limited to what I can read online.

If you find any errors in my analysis or have any tips or other interesting
pieces of information I'd be really happy to discuss with you.

~~~
Cushman
The visualization is totally cool, though I don't claim to understand the
analysis.

...but if it helps, you made a little English error. Wouldn't even have
mentioned if you hadn't said you were French :)

> [Republicans.] Most of which are either moderates or closer to the Democrats
> than to their own party.

> Olympia Snowe and Susan Collins are the two Republican Senators from Maine
> which are both regarded to be leading moderates within their party.

Should be "Most of whom..." and "...who are both...".

"Which" is only ever[1] used in relative clauses to describe inanimate objects
(as you do correctly five other times in the article). Humans and animals are
always either "who" (more personal) or "that" (more impersonal).

Note that's only in relative clauses-- "Which senator is yours?" is fine, but
"The senator which represents me" means that I think my senator is a card
table.[2]

[1] Outside of the odd dialect, of course.

[2] Not necessarily out of the question.

~~~
noahc
I'll assume you're correct in your explanation. I've never heard it explained
that way.

But as a native speaker "that" sounds slightly more correct than "which"
because it seems less formal, however, if I heard a non-native speaker say
"which" it wouldn't sound weird nor would it be something I would have
corrected them on.

~~~
georgefox
In American English, _which_ is normally used in non-restrictive clauses
(offset by commas), and _that_ is normally used in restrictive clauses (not
offset by commas).

Using _which_ in a restrictive clause like _The senator which represents
me..._ or _The road which runs from X to Y..._ is somewhat uncommon. Both of
those examples sound better in American English using _that_ in place of
_which_. If the subjects didn't need disambiguation, though, _that_ would no
longer be appropriate, e.g., _Senator Jones, who represents me, ..._ or
_Foobar Street, which runs from X to Y, ..._

As Cushman points out, though, _which_ used on a person doesn't quite work.
_That_ can work on a person, though, but _who_ / _whom_ can work for people in
both restrictive and non-restrictive clauses.

See <http://owl.english.purdue.edu/owl/resource/645/01/> if you want to read a
well-structured and thorough explanation.

------
drx
Would be interesting to see PCA and CA* applied to the same data.

On a quick google trip I found

[1] Principal Component Analysis of Senate Voting Patterns
(<http://escholarship.org/uc/item/3xm9z62w#page-1>)

[2] Shlomo S. Sawilowsky, Real data analysis; Chapter 28, Principal Component
Analysis of Senate Voting Patterns (on Google Books)

* Principal Component and Correspondence Analyses respectively -- <http://en.wikipedia.org/wiki/Principal_component_analysis> and <http://en.wikipedia.org/wiki/Correspondence_analysis>

~~~
friggeri
I'm not sure this would give the same kind of results, it was shown [1] that
Kleinberg's HITS algorithm is PCA, and HITS gives [2] hubs (nodes linked to
many others) and authorities (nodes linked from many hubs). I have the
intuition that this would qualify the floor leaders as authorities, but I'd
have to run the numbers.

Anyway, in order to obtain groups from PCA (that is, groups which are not only
"visual"), one would have to run some kind of clustering algorithm on the
reduced data.

[1]
[http://www.uclouvain.be/cps/ucl/doc/iag/documents/WP125_Saer...](http://www.uclouvain.be/cps/ucl/doc/iag/documents/WP125_Saerens.pdf)

[2] <http://en.wikipedia.org/wiki/HITS_algorithm>

------
EvilTerran
It does look nice... and it does raise interesting questions. I'm particularly
intrigued as to John Kerry, John Edwards, and Joe Lieberman's presences in
that anomalous 108th Congress. Can you tell us any more about how you define
an "agreement group"?

Visually, the result reminds me of [<http://xkcd.com/657/>] - is this sort of
diagram a recognised concept in data presentation, or a new idea?

[edit - hmm... <http://en.wikipedia.org/wiki/108th_Congress>]

~~~
friggeri
I was not aware of this xkcd comic, funny how things seem to pop out. My
inspiration was more in the line of Sankey diagrams. Among others, the famous
map of Napoleon's invasion of Russia by Minard
(<http://en.wikipedia.org/wiki/File:Minard.png>). I've called this
representation a "groupflow" but any name would do.

The notion of agreement departs a little bit from the classical fraction of
common votes/total votes and instead is based on a vector model of votes,
using cosine similarity as a value of agreement. The agreement groups are
computed by heuristically optimizing a quantity called the cohesion and which
turns out is really correlated to the way people perceive social communities
(the old preprint of the first paper: <http://arxiv.org/pdf/1107.3231.pdf>).

~~~
specialist
Minard's graph is brilliant.

On my to do list: a graph that shows the loss of voters casting ballots. I
needed a way to show how postal balloting disenfranchises voters (what it
gives, it equally takes away).

Voting age population -> registered to vote -> voter received ballot ->
returned ballot -> elections rcvd ballot -> signature verification ->
challenged ballots -> overvotes/undervotes, voter intent, adjudication ->
tabulated.

------
pacmon
Looks really great and seems like a useful tool. My only real complaint, is
that I use a high sensitivity on my mouse and I found it difficult to
highlight a line. If you could make the lines slightly thicker, it would be
easier to select them.

------
sesqu
For what it's worth, I thought the gray background bars were far too light -
it took me a while to find them. I also couldn't intuit what the branching
lines were all about.

Also, Chrome had some difficulty with typesetting, though it's possible that's
a localized issue.

*edit: I figured out why the boxes seem so light - it's because I view my screen at a slight angle. From a straight angle they show up fine.

~~~
friggeri
Sorry about that, I should have clarified that each Senator could be in
different groups, hence the branching. I'll add a little darkness to the
background bars.

What kind of typesetting issues did you have, and with which version of
Chrome?

~~~
sesqu
Here's a screenshot from my Chrome 18 on Linux: <http://i.imgur.com/vPwl8.png>

The visualization is quite nice, and it would be useful to link to or include
some kind of short description about the algorithm and visualization - to give
a different example, it looks like the groups are scored numerically but the
senators are simply ranked, so I wonder if the algorithm found more than two
groups per congress.

~~~
friggeri
I have absolutely no idea why it renders this way on your Chrome, maybe an
issue with TypeKit?

About the visualization, it was done semi-manually with a tool I've developed
to heuristically minimize the crossings, but their is no way to obtain this
kind of layout purely algorithmically (I'm pretty sure it's NP-hard).

    
    
      it looks like the groups are scored numerically but the senators are simply ranked
    

Not quite sure what you mean. The groups were positioned in order to have the
one with most Republicans on one side and the one with most Democrats on the
other one. There were consistently two groups for those sessions, this I can
assure you. The order of the senators inside the groups is arbitrary, in order
to attempt to minimize the number of crossings and provide a readable output.

~~~
sesqu
Ah, I assumed each senator was placed according to some measure of extremism,
though in hindsight that alone should have produced crossings.

To support this interpretation, I noticed that the groups are not aligned -
see for example George Allen's swerve in the 108th Congress. I suppose I was
expecting it too, from having seen some visualization that tried and failed to
find a tea party cluster.

------
caf
Simon Jackman calculates ideal point estimates based on non-unanimous roll
calls: <http://jackman.stanford.edu/ideal/currentSenate/x1.pdf>

I think his R source is available too.

------
joshu
Good stuff. Can you talk more about the algorithm?

Can you use this to detect issue clusters rather than voting clusters?

I have tried to build subgroup-in-graph detectors with only partial success.

~~~
friggeri
Thanks, the algorithm is a heuristic which optimize a quantity I've called
social cohesion (<http://arxiv.org/pdf/1107.3231.pdf>), the article presenting
the algorithm itself is currently under review, but basically: first it
computes the best possible group for selected edges in the graph, then it
merge groups which overlap too much. The whole difficulty is to identify how
to select those edges, how to expand around them and then what does "too much
overlap" means. I'll link to the preprint as soon as I upload it to arxiv.

What do you mean exactly by "detect" issues? C^3 should work if there is a
notion of transitivity in the network (that is, if A and B are similar and B
and C are similar, then with very high probability A and C are similar).

~~~
specialist
Great stuff. Thanks for sharing.

It'd be great to apply your algorithm per issue.

During Camp Wellstone training (grassroots political organizing), they taught
us a tool called a Power Map. The two axis are yes v no and influence. You
graph each person voting. Then you draw arrows to show who influences who. It
helps to identify where you should focus your lobbying.

One point that stood out is that a Power Map is needed to be done for each
issue, because many many votes don't come down on partisan lines (eg local and
state, ymmv).

------
LyleK
Best use of SVG I have seen yet!

------
rorrr
1) Many senators shift to the left side (more conservative) as time goes by

2) It's especially true about republicans

