Creating actionable plans has been shown to be helpful in promoting physical activity. However, little research has been done on how best to support the creation and execution of plans. In this paper, we interviewed 16 participants to study the role that context plays in the formulation and execution of plans for physical activity. Our findings highlight nuanced ways that contextual factors interact with each other and with individual differences to impact planning. We propose the notion of sweet spots to encapsulate how particular contextual factors converge to create optimal states for performing physical activities. We argue that the concept of sweet spots can help us better understand the nature of plans and guide the design of context-based tools for planning support. We present design guidelines to show how sweet spots can help support physical activity planning.