The initial scores for this paper were : 4: An okay submission, but not good enough; a reject. 7: A good submission; an accept. 5: Marginally below the acceptance threshold. The main concerns of the negative reviewers were: - issues about the problem formulation - only weak baselines are considered; results below the state-of-the-art - limited novelty - missing citations - only relatively minor improvements obtained by the proposed approach The positive reviewer also acknowledges the issues with experimental evaluation (the proposed method is shown to help weak baselines that are overall below the state-of-the-art), but finds the idea of the paper interesting, original and standing out. The authors provide a rebuttal. In the follow-up discussion among the reviewers, R3 acknowledges that some of their concerns have been addressed but remains borderline negative (5) as they think the rebuttal does not alleviate the concerns regarding the overall low results and some ablations are still missing. R2 agrees on the issues with experimental evaluation pointed by R1+R3 but maintains that “given that the problem and the method are interesting and that there are no good dataset to study them, I would recommend accept.” R1 is only partially happy with the rebuttal but agrees with R2 that the current field of action recognition needs new ideas, current datasets are biased and far from reality. R1 is borderline about this work. 
AC has read the reviews, rebuttal and the paper. AC agrees with R2 that the problem and method are interesting. AC also agrees with R1 and R3 (and R2) on the main concern that the paper currently shows only improvements over non-state-of-the-art baselines. AC specially appreciates the additional results in the rebuttal that show new results on a additional dataset that has the same scene background (Diving48 dataset) as requested and also appreciated by R3. The proposed method demonstrates an improvement on this data and a favourable comparison to the comparable (though non-state-of-the-art) baseline. Overall, AC agrees with R2 that the paper studies and interesting and important problem and proposes an interesting solution. AC further thinks that, given the additional results in the rebuttal on the Diving48 dataset, the paper sufficiently demonstrates the potential of the proposed technique. The fact that the implementation of the method presented in this paper does not build on the state-of-the-art backbone architectures and hence does not demonstrate improvements over state-of-the-art results is a weakness. However, AC agrees with R2 that the problem and solution are interesting and this outweighs the short-coming of non-state-of-the-art results.