Over the last month or so I've been tracking the election polls and testing out a few different ideas in order to improve a general model that I'll end up using for the upcoming Alberta election. Of course, I wasn't the only person doing this, and I was able to find at least six other sites tracking and projecting alongside.
But who did the best? Can we learn anything specific about which models produce more reliable results?
First of all, we can look at seat projections. As far as I could tell by mid-day June 7th, this was the seat projection distribution between the seven of us:
| CBC | Too Close to Call | QC125 | Lispop | Teddy on Politics | Calculated Politics | Extreme Enginerding | Average | Actual | |
|---|---|---|---|---|---|---|---|---|---|
| PC | 78 | 74 | 70 | 69 | 60 | 71 | 70 | 70.3 | 76 |
| NDP | 45 | 46 | 47 | 50 | 55 | 44 | 45 | 47.4 | 40 |
| LIB | 1 | 3 | 6 | 4 | 8 | 8 | 9 | 5.6 | 7 |
| GRN | 0 | 1 | 1 | 1 | 1 | 1 | 0 | 0.7 | 1 |
| OTH | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
Ranking these by the root sum of squares difference from the actual results, we get:
- Calculated Politics (diff: 6.48). Their method involved seat-by-seat projections, suggesting a regional breakdown that seemed to work pretty well for them!
- Too Close to Call (diff: 7.48). They also provided seat-by-seat projections, and had regional factors involved to project those. Also, most handily, their simulator was interactive, but putting the correct values into it actually made their predictions slightly worse (still second place at 7.87 though).
- (Tie: CBC and Me) (diff: 8.12). We ended up with the same predictions for the NDP, but CBC was way under for the Liberals and I was quite a bit under for the PCs. My model didn't involve individual seat projections and instead just approximated historical trends for seat ranges based on party vote share, so that's a win for simplicity I suppose.
- QC125 (diff: 9.27). Another site with seat-by-seat projections. The actual seats fell well within their expected ranges, but were all off by a little bit. I'm unsure how they came up with the seat vote projections.
- Average (diff: 9.48). In this case, the wisdom of the crowds didn't pan out.
- Lispop (diff: 12.57). Hypothetically they used a regional swing model similar to mine, so I'm not quite sure where the difference comes from here. It looks like they anticipated a much higher NDP voter base than actually happened.
- Teddy on Politics (diff: 21.95). It seems like Teddy paid more attention to leader favorability numbers than most of the rest of us, and that seems to have tilted the seat distribution against him. His was the only model to predict a minority government.
For most of the models, the seat projections came directly from the popular vote estimates. If we take a look at those, we get:
| CBC | Too Close to Call | QC125 | Lispop | Teddy on Politics | Calculated Politics | Extreme Enginerding | Average | Actual | |
|---|---|---|---|---|---|---|---|---|---|
| PC | 38.7 | 37.9 | 37.8 | 38 | 37.9 | 38.4 | 39.8 | 38.4 | 40.5 |
| NDP | 35.5 | 36 | 36.1 | 37 | 36.8 | 36.1 | 35.9 | 35.9 | 33.6 |
| LIB | 19.6 | 19.8 | 19.7 | 19 | 20.9 | 19.5 | 19.6 | 19.7 | 19.6 |
| GRN | 4.9 | 4.6 | 5 | ? | 4.5 | 4.6 | 5.2 | 4.8 | 4.6 |
| OTH | 1.3 | 1.7 | 1.4 | ? | 0 | 1.5 | 1.3 | 1.4 | 1.8 |
Ranking these again by the same criteria we get:
- Me! (diff: 1.15)
- CBC (diff: 2.69)
- Average (diff: 3.21) This is a better example of the group as a whole performing better than most individual members. This also probably makes sense as these numbers would have come mostly from the same pool of publicly available polls with a small amount of interpretation for trends and recency, as opposed to a large amount of interpretation as in the case with seat projections.
- Calculated Politics (diff: 3.29)
- Too Close to Call (diff: 3.56)
- QC125 (diff: 3.73)
- Lispop (diff: ~4.3) Note that Lispop didn't list their prediction for the green party vote total, despite projecting them to win a seat.
- Teddy on Politics (diff: 4.37)
Overall I'm really pleased with how I did, and I've learned a few tricks to use in upcoming elections. Next up will probably be Québec, hopefully with the same group of people, and we can see if this was a fluke for me or not!
Finally, here's my seat model with the actual results input as though they were one final gigantic poll at the end. Using these correct values would have resulted in the model being the most accurate seat projection of them all (diff: 4.24), which is an encouraging sign that the model itself was sound!
See you next election!
Finally, here's my seat model with the actual results input as though they were one final gigantic poll at the end. Using these correct values would have resulted in the model being the most accurate seat projection of them all (diff: 4.24), which is an encouraging sign that the model itself was sound!
