Support us at FLAME by visiting us on the web!

So this year we tried something a bit different at FLAME Festival. Rather than decide amongst ourselves as organizers whom we wanted to come and teach at the festival, we put it in the hands of our community. The germ of this idea came from a conversation Ky, Justin, and I had immediately after Pacific Fire last year. All three of us wanted to find a way to pay our instructors, but we were at a loss as to how to do so in a fair and equitable way.

I think it’s fair to say that the majority of festival-goers have little inkling of the thought that must go into the value add of each teacher to a festival. Sure, there are big names that you know will sell tickets, but will they provide a rich learning experience for the festival attendees. Put simply: will they make them want to come back? Sadly, our community is in the dark ages when it comes to putting together these kinds of assessments. We don’t have a good way of collecting information about classes after people take them (in most cases, we’ve no idea how many people even took a given class!), so being able to say a given teacher was worth the investment is tricky at best.

So along came the idea of voting. As we discussed ways that we could assess the value a given teacher was adding to the festival, it dawned on us that we could ask our attendees to tell us who they wanted to see and what they wanted to take by asking them to vote for those classes. We knew it couldn’t be the only consideration, but what it would give us was a more accurate insight into what our attendees wanted to learn and who was best equipped to provide them with that experience.

A lot of our instructors were very confused by this idea--and for good reason. It’s a radical departure from the way most festivals decide on their lineups. To the best of our knowledge, it’s not something that has ever been attempted before at a fire, flow, or juggling festival before.

There were a lot of questions we had to answer to get such a system working: who could vote? how would they vote? would this be a part of the FLAME site or something different?

Ultimately after a few frustrating weeks of attempting to find a solution that presented the user experience we were looking for, we decided to build our own site to handle the voting that would be separate from the rest of the FLAME system, but built to look like the regular site to make for as seamless a user experience as possible. This also meant that we were opening up the vote to anyone and everyone--not just FLAME attendees but really anybody anywhere in the world. This meant that there was a risk that people could game the system by getting their friends who many have had no interest in coming to FLAME to vote for them. We ultimately decided to take the risk A. because it would raise the profile of FLAME and inevitably introduce us to new people and B. because we knew we’d be promoting it on a local level as well--in other words it would be our challenge to insure the local voice was as loud as the wider voice in voting.

With the system in place, we asked teachers to begin promoting their classes and waited to see what would happen. We were pleasantly surprised by the level of engagement this produced within the community, with more than 200 people signing up for an account to vote on the site. The site was visited 1,500 times in the two weeks we had voting open with most traffic coming from shares on Facebook.

When voting closed, it was time to figure out how best to balance the vote against our available budget. At this point, we needed to assess what the individual votes for classes were worth, what the placement of all the classes was worth, and how we could measure that against our budget. This was the criteria I used:

Rather than individually counting the votes on each and every class, I went for a system of counting the number of classes each teacher got into the following brackets: the top 1-20, 21-40, 41-90, 90-120, and after 120 we didn’t bother to count the additional classes. This was partly to save time and partly because we reasoned that with only 90 available slots for workshops this year, shedding ¼ of our total would be enough work in and of itself.

Given that the highest requested travel stipend this year was $500, I proceeded from the assumption that any class that made the top 20 was in, because at that point it’s fairly easy to conclude that the teacher is indispensable to our lineup this year. Thus, I assigned point values to each of the brackets in accordance to ensure they would cancel out travel stipends appropriately. A teacher with a class in the top 20 was rewarded with 500 points per class that made it in, then 200 points for every class that made the top 21-40 slots (the logic here: a teacher that gets 2 or more classes in this bracket is also making a clear contribution to the festival), because they had more than twice as many slots to potentially fill between 41-90 I assigned 50 points for each class in this bracket (in other words, 4 classes in this bracket is the equivalent of getting a class in the 21-40 slots), and finally only assigned 20 points per class in the top 91-120 to fill out my exponential curve. With this, we had the total value of the classes the teacher had offered to our potential attendees.

Next, I assigned values based upon the quantity of classes a teacher was willing to contribute. Teachers willing to teach more classes were giving us more bang for our buck and I wanted to reward that. Again, I took the $500 travel stipend as a point of departure and allocated 500 points to any instructor who got 5 classes into the top 120 (yes, some may opt not to teach them all--more on that later) and 200 for any instructor who got 4 classes into the top 120. The more a teacher was willing to contribute, the more it saved us money by spreading out that investment across as many classes as possible. This also applied in the opposite case: we consider 3 classes offered per teacher to be the butter zone for making up a schedule. More than that and a teacher may come to feel overwhelmed. Less and it becomes progressively more difficult for us to fill out our schedule. Due to this, I assigned a deduction of 500 points to any teacher who got less than 3 classes in the top 120. We now had the value each individual teacher was contributing to filling out our schedule.

Finally, we had to weigh both of these factors against the amount of money each teacher was requesting for travel. This was easily the hardest step and required the most care to figure out how to integrate properly. Too much of an effect and it would hurt the lineup, too little and it would break the bank for us. Ultimately, I wound up multiplying this number by a given factor and subtracting it from the total number of points the instructor had accumulated up until this point. There was absolutely no hard and fast science to this. I kept changing the number until I got between 90 and 100 classes inside my budget.

One arrived at, this algorithm gave us a list of 44 teachers who were the best balance of vote results and bang for our buck that we could get. It meant that all teachers who got a class in the top 20 made it in and only 1 teacher in the top 40 wasn’t selected. Also, we came in just enough under budget to attempt another experiment.

The question we began this journey with was how best to approach compensating teachers for their involvement in the festival. On this front, we couldn’t give any kind of meaningful compensation to teachers in the top 20 because it would be spread across 17 individuals. Plus which, for all intents and purposes it would be a monetary reward for popularity, insuring that only well-known teachers could qualify for it. Beyond making it into the top 20, the other biggest value add a teacher could give us would be teaching more than 3 classes. Here was a much better opportunity for fair compensation for two reasons: each additional class a teacher taught helped us immensely as well as it being a lot to ask of the teacher to commit to. Thus, a value add to us and a precious commodity to the instructor.

Thus, any teacher offering 4 classes at FLAME will be paid $25 for that fourth class (admittedly, it’s nowhere near what their time is worth, but it is a start and a gesture at the very least) and any teacher offering a fifth class will be paid another $50 on top of the $25 for their fourth class. Thus, any teacher could stand to earn as much as $75 from us for teaching in addition to their travel stipend. 4 teachers who made the cut qualify for the 4-class bonus and of them 2 qualify for the 5-class bonus. No directors or organizers were eligible for these bonuses.

The equation then boils down to the following:

a = classes a teacher got in the top 1-20 slots
b = classes a teacher got in slots 21-40
c = classes a teacher got in slots 41-90
d = classes a teacher got in slots 91-120
e = 500 point deduction for fewer than 3 classes
f = 200 point bonus for offering 4 classes
g = 500 point bonus for offering 5 classes
h = requested travel stipend

Teacher rank R = ((500*a)+(200*b)+(50*c)+(20*d))+(e+f+g)-(1.5*h)

Those of you who are well acquainted with statistics may find errors in my logic or know of a better way to accomplish many of these same things. If so, I invite you to send me feedback. I am admittedly an armchair mathematician and this was the best I could come up with. I’d love to know how to do it better.

In the meantime, however, it took a process that can sometimes last weeks and gave us our teacher selections within 12 hours of the vote close. To me, that counts for a lot. Did we succeed in our mission? Only the experience of our attendees this year will tell.

You can see the final vote count here: http://dev.drexfactor.com/top-rated-points

Note: out of respect for the privacy for all the instructors, we will not be publicly sharing the full data sheet we used to calculate rank.