Summarizing Stand-up Calls: Experiments using GPT-3 and Google STT (Part 2)

From the Members of the DSPG team at Digite Inc. (Siddhant Bane, Shashank M, Mitesh Gupta and Ashwin Swarup)

This is the concluding part of our blog where we focus on extracting actionable items from stand-up meetings.

You can find the previous blog post here. Just to summarize we looked at how GPT-3 summarized standup calls without any text pre-processing. The summary was good but there was nothing actionable that we could extract because “ Who did what “ details are important to make something actionable.

So in order to improve results, lets think of a standup meeting flow in terms of a block diagram. You have a manager/lead/ <insert lead here> who co-ordinates the entire conversation and rest of the team pitching in whenever required.

We start our second solution iteration from this broad concept and take it forward.

The code for this blog post is shared in this google collab notebook.

Solution Second Iteration:

Diagram showing the conversation blocks during a standup call. The team lead coordinates the meeting while each team member conveys a cohesive thought

If we were to draw a block diagram for the conversation flow discussed earlier it looks something like this (diagram to the right). Clearly, the nuance here is that what each person says is equally important since she is talking about a separate update.

It clearly does not make sense to ask GPT-3 to summarize each dialog separately because of loss of context. So, we decided that we can group what each team member is saying separately and look at the results.

There are two types of actors here –

1. Team Lead who switches conversation topics
2. Team member who talks cohesively for some time on the same topic.

Let’s look at how GPT-3 performs specifically for these two actors.

Iteration 2 results where actor information is grouped together

What we noticed is that when concepts from multiple parts of the conversation are clubbed together performance drops for the team lead. This is expected because we are clubbing conversations from across the text. So sentences spoken in different contexts are being clubbed together. This is not so for team members since they coherently convey the idea for the topic in one go.

So clearly there has to be another method to overcome this challenge.

Solution Third Iteration:

Experiment with varying window sizes

This diagram shows how a moving window of 3 passes over the conversation blocks. The ‘red’ coloured window is passed over the conversation in an interactive manner.

We observed more promising results with this one. Apply a moving window on the conversation. Group conversation blocks in varying window sizes and then pass them to GPT-3. The conversation now starts chunking into concepts and ideas. We are able to capture idea transitions in the conversation. We experimented with varying window lengths and we identified that GPT-3 performed well for window size of 5.

In sliding window approach, 3 dialogues are not sufficient for proper text summarization. 5 dialogues however gives much better results. The summarization performs badly when the window is covering the end of a conversation topic as the dialogues containing the context for the conversation seem to leave the window. Also it is observed that towards the end of a topic in a conversation there are a lot of short sentences that do not have any context on their own, which leads to degraded performance in summarization.

We use the sentence completion mechanism for text summarization which is not exactly how it was intended to be used. Still by writing “tasks” as a suffix for your prompt (tl;dr or summarize in 2 lines) one can get the same results that one would expect from a text summarization model.

Next Steps

Our intent is to make these experiments more rigorous. We want to generate cards for a scrum board directly from the minutes of the meeting. Keep watching this space for more updates as we work on this project!

Shameless Plugs

Some of the other open-source work we are doing! And this!

Acknowledgements

1. https://stackoverflow.com/questions/64722585/gpt-3-prompts-for-sentence-level-and-paragraph-level-text-summarization-text-s
2. Meet Transcript: This is an excellent app and is able to leverage google transcribe https://chrome.google.com/webstore/detail/meettranscript/jkdogkallbmmdhpdjdpmoejkehfeefnb
3. Chatbot playground: https://kairon.digite.com/
4. The folks at Open-AI for letting us experiment with GPT -3 https://openai.com/