Smart Home is generally considered to be the final solution for human living problem, especially for health care of the elderly and disabled, power saving, etc. Human activity recognition in smart home is the key to achieve home automation, which enables smart services automatically run according to human mind. Recent researches have made several progresses in this field, however most of them can only recognize default activities which is probably not needed by smart home services. In addition, low scalability makes such researches infeasible out of laboratory. In this work, we unwrap this issue and propose a novel framework to not only recognize human activity, but also predict it. The framework contains three stages: recognition after the activity; recognition in progress and activity prediction in advance. With the help of RFID tags, the hardware cost of our framework is low enough to popularize. And the experiment result shows that our framework can realize good performance in activity recognition and prediction with high scalability.