
Unfortunately, every company has its own unique culture so it’s not possible to have an answer of when to quit, that works for everyone. There are cultures that are very production centered and struggle to see the value of spending time on brain storming and creative problem solving and they are more likely to quit too early in the process. Other cultures may revere the creative work but find it hard to get to the application and production phase. In either case there needs to be some mile markers that are agreed upon and set in advance to track progress and timing. This is a lot like project management for innovation.
Setting goals and objectives and managing to plan is how you can ensure that you are keeping on track as long as you have a good understanding of how the creative process works when you set them. If you have an experienced innovation team leader then they can follow the plan and it will help them know if the production of viable ideas are indeed happening and if not it will help them pull the plug so you can focus on efforts that have more potential.
The goal is to get better at the creative innovation process so that you can produce breakthrough ideas more consistently and with manageable risk. Knowing when to say no is what innovation systems can help with.