Understanding route travel time and passengers route choice preference can help transportation authorities to build smart public transportation systems. More and more public transportation systems are now using smart cards to record. The smart card data record the information of the origin and destination of each individual passenger trip and the corresponding travel time. However, the detailed movement of passengers inside of the public transportation system are unobservable. Such unobserved passenger movement patterns are important to many applications including route planning, flow congestion control and others. This motivates us to develop the TripDecoder framework. TripDecoder estimates the parameters of unobservable time periods which can best-fit the observable travel time periods from smart card transactions. By estimating these parameters for pairs of stations ordered by ascending number of possible paths between them, TripDecoder is shown to infer the travel time and the routes of passengers both effectively and efficiently. In our extensive experiments using a real dataset of subway trips made in a city, we show that the prediction of trip traveling time could achieve higher accuracy than the popular online map direction service being used in the city.