Although suppressing negative transfer between tasks has been a critical challenge for multi-task learning, previous approaches have dealt with multi-task architecture and optimization strategies separately for the purpose. Instead, we propose connection strength-based optimization with progressive multi-modal feature exchange as a combined method for reducing task interference by (i) conserving each task’s feature space in a shared network, and (ii) facilitating inter-task information flow in feature level. We reinterpret the connection strength, a well-known concept in network compression, to determine which channel of the shared convolutional layer has a dominant influence on each task. Based on connection strength, our optimization method projects each task’s gradient to prevent a specific task from exerting a dominant influence on the entire network by intruding on other tasks’ space. Then, these conserved features are progressively forwarded and mixed in a stage-by-stage manner from a shared single-tasking backbone, so that the network fully utilizes inter-task information by exchanging task-specific features. We propose an integrated method that conserves tasks feature space, enabling hard parameter sharing with minimized task interference and allowing it to be used for multi-modal feature exchange. Experiments demonstrate the validity of proposed methods on several dense-prediction tasks by achieving state-of-the-art performances.