Filter by topic and date
IETF 120 post-meeting survey
- Jay Daley IETF Executive Director
20 Aug 2024
IETF 120 Vancouver was held 20-26 July 2024
The results of the IETF 120 Vancouver post-meeting survey are now available on a interactive dashboard. Thank you to all of you who responded to this survey as we use your views to continually adjust the meeting experience.
Analysis
We received 255 responses, 200 onsite and 55 remote. With 1537 registered participants, this gives the survey a maximum margin of error of +/- 5.61%.
The results for satisfaction questions include a mean and standard deviation using a five point scale scoring system of Very satisfied = 5, Satisfied = 4, Neither satisfied nor dissatisfied = 3, Dissatisfied = 2, Very dissatisfied = 1. While there’s no hard and fast rule, a mean of above 4.50 is sometimes considered excellent, 4.00 to 4.49 is good, 3.50 to 3.99 is acceptable and below 3.50 is either poor or very poor if below 3.00. The satisfaction score tables also include a top box, the total of satisfied and very satisfied, and a bottom box, the total of dissatisfied and very dissatisfied, both in percentages. Please note that a small number of questions are on a four point scale.
Question changes since the last survey
For this survey we removed the questions specific to the previous meeting, and the one-off question about the block on submitting I-Ds two-weeks before a meeting as the results from last time were sufficient (no strong movement to change). We dropped the detailed questions about side meetings for reasons explained below. We also dropped the question about different online participation mechanism in favour of a more detailed question about Meetecho.
We kept in the question about where people get their funding from and added back the matrix questions about each individual part of the agenda and structure of the meeting and side meetings that were dropped in the last survey to make space.
We did not add back in the matrix question about venue and accommodation as some aspects are covered elsewhere is new questions. This is likely to require further tidying up in future surveys.
Actions taken following the last survey
For this meeting, we made the following changes, prompted by survey feedback:
- We no longer made any mention of COVID but continued to provide masks and tests for those that wanted them.
Satisfaction
Overall satisfaction is 4.37, a good result. With some key exceptions, the satisfaction scores remain high, reflecting the various improvements made since we returned to onsite meetings.
The table below shows the satisfaction scores for the last six meetings, along with colour coded indicators for the five point scale above: excellent (🔵), good (🟢), acceptable (🟡), poor (🔴), very poor (⚫️)
| IETF 120 Vancouver | IETF 119 Brisbane | IETF 118 Prague | IETF 117 San Francisco | IETF 116 Yokohama | IETF 115 London | |
|---|---|---|---|---|---|---|
| Overall satisfaction | 4.37 🟢 | 4.24 🟢 | 4.42 🟢 | 4.30 🟢 | 4.30 🟢 | 4.28 🟢 |
| AGENDA | ||||||
| Overall agenda | 4.19 🟢 | 4.05 🟢 | 4.20 🟢 | 4.16 🟢 | 4.18 🟢 | 4.22 🟢 |
| Sessions for WGs | 4.29 🟢 | - | 4.32 🟢 | 4.22 🟢 | 4.22 🟢 | 4.22 🟢 |
| BOFs | 4.12 🟢 | - | 4.12 🟢 | 3.95 🟡 | 4.11 🟢 | 4.10 🟢 |
| Sessions for RGs | 4.20 🟢 | - | 4.09 🟢 | 4.12 🟢 | 4.14 🟢 | 4.10 🟢 |
| Plenary | 3.84 🟡 | - | 3.94 🟡 | 3.99 🟡 | 3.98 🟡 | 3.98 🟡 |
| Side meetings | 3.87 🟡 | - | 3.79 🟡 | 3.75 🟡 | 3.73 🟡 | 3.81 🟡 |
| Hackathon | 4.30 🟢 | - | 4.27 🟢 | 4.25 🟢 | 4.34 🟢 | 4.35 🟢 |
| HotRFC | 3.94 🟡 | - | 3.90 🟡 | 3.89 🟡 | 3.84 🟡 | 4.21 🟢 |
| Pecha Kucha | - | - | 4.00 🟢 | 4.15 🟢 | - | - |
| Office hours | 3.97 🟡 | - | 4.07 🟢 | 3.98 🟡 | 4.23 🟢 | 4.00 🟢 |
| Opportunities for social interaction | - | - | 4.06 🟢 | 4.11 🟢 | 3.72 🟡 | 3.98 🟡 |
| STRUCTURE | ||||||
| Overall meeting structure | 4.28 🟢 | 4.15 🟢 | 4.34 🟢 | 4.28 🟢 | 4.28 🟢 | 4.28 🟢 |
| Start time | 4.40 🟢 | - | 4.49 🟢 (9:30am) | 4.28 🟢 (9:30am) | 4.16 🟢 (9:30am) | 4.28 🟢 (9:30am) |
| Finish at 5pm Friday | 3.67 🟡 | - | 3.65 🟡 | - | - | - |
| Length of day | 4.18 🟢 | - | 4.20 🟢 | 4.30 🟢 | 4.30 🟢 | 4.32 🟢 |
| Number of days | 4.15 🟢 | - | 4.18 🟢 (5+2) | 4.27 🟢 (5+2) | 4.30 🟢 (5+2) | 4.32 🟢 (5+2) |
| Session lengths | 4.36 🟢 | - | 4.38 🟢 (60 / 90 / 120) | 4.41 🟢 (60 / 90 / 120) | 4.36 🟢 (60 / 90 / 120) | 4.32 🟢 (60 / 90 / 120) |
| Break lengths | 4.31 🟢 | - | 4.38 🟢 (30/90) | 4.32 🟢 (30/90) | 4.38 🟢 (30/90) | 4.36 🟢 (30/90) |
| Number of parallel tracks | 3.94 🟡 (8) | - | 3.94 🟡 (8) | 4.08 🟢 (8) | 4.01 🟢 (8) | 3.90 🟡 (8) |
| PARTICIPATION MECHANISMS | ||||||
| Meetecho | 4.38 🟢 | 4.50 🔵 | 4.47 🟢 | 4.35 🟢 | 4.45 🟢 | 4.45 🟢 |
| Gather | - | - | - | 3.52 🟡 | 3.46 🔴 | 3.37 🔴 |
| Zulip | - | 3.86 🟡 | 3.94 🟡 | 3.66 🟡 | 3.77 🟡 | 3.73 🟡 |
| Jabber | - | - | - | - | - | - |
| Audio streams | - | 4.27 🟢 | 4.27 🟢 | 4.02 🟢 | 4.21 🟢 | 4.04 🟢 |
| YouTube streams | - | 4.24 🟢 | 4.36 🟢 | 4.32 🟢 | 4.36 🟢 | 4.25 🟢 |
| CONFLICTS | ||||||
| Conflict avoidance | 3.93 🟡 | 3.88 🟡 | 4.00 🟢 | 3.90 🟡 | 3.94 🟡 | 3.91 🟡 |
| VENUE & ACCOMM | ||||||
| Overall accommodation | - | - | 4.34 🟢 | 4.07 🟢 | 4.09 🟢 | - |
| Overall venue | - | - | 3.90 🟡 | - | - | |
| Location | - | 4.52 🔵 | 3.60 🟡 | - | - | |
| Venue facilities | - | 4.43 🟢 | 4.07 🟢 | - | - | |
| Cost of rooms | - | 3.54 🟡 | 2.87 ⚫️ | - | - | |
| Availability of rooms | - | 4.32 🟢 | 4.07 🟢 | - | - | |
| Options for accommodation | 3.68 🟡 | 4.10 🟢 | ||||
| ONSITE | ||||||
| Overall | 4.44 🟢 | 4.37 🟢 | 4.50 🔵 | 4.29 🟢 | - | - |
| Badge collection | 4.73 🔵 | 4.71 🔵 | 4.70 🔵 | 4.69 🔵 | - | - |
| WiFi | 4.25 🟢 | 4.09 🟢 | 4.17 🟢 | 3.98 🟡 | 4.06 🟢 | 4.10 🟢 |
| QR Codes | 4.31 🟢 | 4.31 🟢 | 4.18 🟢 | 4.11 🟢 | - | - |
| Break F&B | 4.15 🟢 | 3.75 🟡 | 4.05 🟢 | 4.44 🟢 | - | - |
| Breakout seating | 3.54 🟡 | 3.93 🟡 | 3.80 🟡 | 4.08 🟢 | - | - |
| Signage | 4.08 🟢 | 3.96 🟡 | 4.15 🟢 | 4.22 🟢 | - | - |
| Coffee carts | 4.65 🔵 | 4.08 🟢 | 4.64 🔵 | 4.56 🔵 | - | - |
| Childcare | 4.43 🟢 | 3.76 🟡 | 4.12 🟢 | 4.06 🟢 | - | - |
| Welcome reception | 4.01 🟢 | 3.97 🟡 | 4.08 🟢 | - | - | - |
| Farewell reception | 4.31 🟢 | 4.15 🟢 | 4.34 🟢 | - | - | - |
Remote participation
We once again asked people why they participated remotely. The set of options has developed over time with some changes in wording and emphasis. The table below therefore combines options that were written differently in different surveys.
A number of the results here appear to vary strongly for reasons outside of the control of the IETF, while others indicate areas that the IETF can continue to work on (childcare, visas, accommodation costs).
We also asked if those who participated remotely would have wanted to participate onsite.
Putting the two together shows a fairly consistent proportion of remote participants would have participated onsite, most of whom were unable to do so because they did not have the funding.
| IETF 120 Vancouver | IETF 119 Brisbane | IETF 118 Prague | IETF 117 San Francisco | |
|---|---|---|---|---|
| WHY REMOTE | ||||
| The travel costs were too high | 47.17% | - | 4.44% | 15.25% |
| The accommodation costs were too high | 33.96% | - | 4.67% | 13.56% |
| It was too expensive to travel to the meeting | - | 75.00% | 60.00% | 62.71% |
| I could not get a visa or getting a visa is too difficult | 26.42% | 4.69% | 4.44% | 15.25% |
| I could not take a week away from home/work | 22.64% | 39.06% | 42.22% | 35.59% |
| It is my preferred way to participate | 11.32% | 15.63% | 28.89% | 18.64% |
| I did not want to participate in the whole meeting | 11.32% | 10.94% | 11.11% | 23.73% |
| I had childcare responsibilities | 5.66% | 9.38% | 15.56% | 6.78% |
| The COVID policy (mask wearing, etc) was too loose | 3.77% | 3.13% | 4.44% | 5.08% |
| I did not want to go to this location | 1.89% | 4.69% | 0% | 16.95% |
| ONSITE PREFERRED | ||||
| Yes | 72.73% | 73.44% | 68.89% | 62.71% |
| No | 3.64% | 10.94% | 2.22% | 10.17% |
| Unsure | 23.64% | 15.63% | 28.89% | 27.12% |
Funding
For the second time asked the question "What was the primary source of funding for your onsite participation?", the results of which were:
- "My employer" 67.86% (Was 75.14%)
- "A grant or bursary" 4.76% (Was 9.04%)
- "My own business" 4.37% (Was 5.65%)
- "My personal funds" 7.94% (Was 5.08%)
- "One or more clients" 5.56% (Was 5.08%)
- "Other (please specify)" 2.38% (Was 0%)
New participants
We regularly include a set of questions for those who identify themselves as new participants and these have been used to drive large changes to the new participant program over the last two years. We will provide a more detailed analysis of the results of these questions in a future blog post.
Other good results
We're clearly doing a lot right as overall satisfaction is good and there are clear improvements trends in a number of areas. For those onsite, we're pleased that the quality of coffee is back to excellent.
Other areas for improvement
Breakout seating
We continue to bring in extra seating and place it close to the breakout rooms, but the feedback shows that we're still underestimating just how much close by seating is needed.
Signage
Our signage score has improved but there are several comments in the free text responses recommending further improvements.
Opportunities for social interaction
From the comments, the new games night proved succesful and partially compensated for the lack of a social event. Given corporate budget restrictions, we should expect no more social events for at least a year and so we will continue to identify smaller improvements we can make.
Break food and beverage
While we continue to tweak the food and beverage provisions based on feedback, we are always highly constrained by the choices available at the venue and their food management processes.
Areas we might not be able to improve
Side meetings
The satisfaction score for side meetings remains at the acceptable level, lower than that for most other aspects of the meeting. We get the same feedback on side meetings each time, generally that they should be better supported (i.e. provide Meetecho, avoid scheduling conflicts, integrate into meeting agenda) but after multiple revisits, the IESG is not planning on any further changes.
Conflict avoidance
The satisfaction score for conflict avoidances between sessions has not moved over two years despite significant efforts to improve things. It is likely that we are simply not able to do any better given the number of sessions and the limited days in which we can meet.
Individual comments
The individual comments covered a number of key themes, many of which have already been covered above:
- Visas. We know of at least 50 participants who applied for visas within the recommended timeframe, but did not receive a response and so were not able to participate onsite. We will be following this up with the Canadian government to see if we can prevent similar issues for future meetings.
- Cost and availability of accommodation. This meeting was actually booked in 2014 and then renegotiated in 2017 for the IETF 107 meeting in March 2020. However, IETF 107 was switched fully online due to COVID and this meeting was rebooked for 2024. This timing introduced two issues with accommodation. The first is that since we originally booked this meeting, the cost of accommodation in Vancouver during July has risen steeply, and the second is that our guaranteed room rate stayed comparatively cheap and so the hotel were unwilling to release more rooms at that rate to help us meet demand. We do not expect such a confluence of issues for future meetings.
- COVID. There were a number of comments complaining about the infection risk at this meeting. We recognise that the risk of infection from airborne diseases, of which COVID is just one, is higher at IETF meetings given the concentration of a large number of people who have flown in from different countries, and the reduced level of immunity after a two-year pause in international travel. Consequently, we will amend our communications to make it clear that anyone who may be infectious should either wear a mask or participate remotely. We will not however, consider any return to formal restrictions or data collection, which is where we believe community consensus still sits.
- Meeting room screens. There were a number of comments about how the screens in meeting rooms are used that will be considered by the meetings team.
- Other comments. Other comments covered issues such as the finish time on Friday, Meetecho functionality, and more. These will be passed on to the appropriate people to consider.
And finally
Thank you everyone who responded to this survey, your feedback is much appreciated.