{"id":92,"date":"2023-12-13T18:17:46","date_gmt":"2023-12-13T18:17:46","guid":{"rendered":"https:\/\/sites.tntech.edu\/lcasl\/?page_id=92"},"modified":"2023-12-13T18:25:10","modified_gmt":"2023-12-13T18:25:10","slug":"hmm-rl","status":"publish","type":"page","link":"https:\/\/sites.tntech.edu\/lcasl\/hmm-rl\/","title":{"rendered":"HMM RL"},"content":{"rendered":"\n
<\/div>\n\n\n\n
\n

Hidden Markov Model Based Q-Learning for Partially Observable Markov Decision Process with Discounted Rewards<\/h3>\n<\/div>\n\n\n\n
\n
\n
\n

Decision-making problems in real-world are often partially observable, and dynamic models of the environments are typically unknown. Therefore, there are demands for learning methods to estimate both the dynamic models and the decision policy, given streams of rewards and incomplete state observations. This work presents an online estimation algorithm that simultaneously estimates the model parameter and belief-state-action-value function that uses discretized belief state space. Furthermore, we establish asymptotic convergence analysis for the above estimation. Also, we show that the discretized action-value function converges to the actual action-value function as we increase the number of grids for the discretization. In addition, we provide a numerical example where the introduced estimation methods show improved performance compared to the application of standard Q-learning that does not consider incomplete state observations. While this work focuses on analysis on finite observation and action space, we consider this as a step towards the theoretical understanding of the existing deep reinforcement learning methods that estimate both models and policies given incomplete observations.<\/p>\n\n\n\n

BACKGROUND<\/h2>\n\n\n\n
\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

<\/p>\n\n\n\n

POMDP EXAMPLE<\/h2>\n\n\n\n
\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

<\/p>\n\n\n\n

HMM BASED RL FOR POMDP<\/h2>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n

PERFORMANCE OF THE POLICY TO MACHINE REPAIR PROBLEM<\/h2>\n\n\n\n
\"\"<\/figure>\n\n\n\n

Relevant Paper:<\/strong><\/p>\n\n\n\n

    \n
  1. Yoon, Hyung-Jin, Donghwan Lee, and Naira Hovakimyan. “Hidden Markov model estimation-based q-learning for partially observable Markov decision process.”\u00a02019 American Control Conference (ACC). IEEE, 2019.<\/li>\n<\/ol>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

    Hidden Markov Model Based Q-Learning for Partially Observable Markov Decision Process with Discounted Rewards Decision-making problems in real-world are often partially observable, and dynamic models of the environments are typically unknown. Therefore, there are demands for learning methods to estimate both the dynamic models and the decision policy, given streams of rewards and incomplete state […]<\/p>\n","protected":false},"author":184,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-92","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/sites.tntech.edu\/lcasl\/wp-json\/wp\/v2\/pages\/92","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sites.tntech.edu\/lcasl\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/sites.tntech.edu\/lcasl\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/sites.tntech.edu\/lcasl\/wp-json\/wp\/v2\/users\/184"}],"replies":[{"embeddable":true,"href":"https:\/\/sites.tntech.edu\/lcasl\/wp-json\/wp\/v2\/comments?post=92"}],"version-history":[{"count":3,"href":"https:\/\/sites.tntech.edu\/lcasl\/wp-json\/wp\/v2\/pages\/92\/revisions"}],"predecessor-version":[{"id":101,"href":"https:\/\/sites.tntech.edu\/lcasl\/wp-json\/wp\/v2\/pages\/92\/revisions\/101"}],"wp:attachment":[{"href":"https:\/\/sites.tntech.edu\/lcasl\/wp-json\/wp\/v2\/media?parent=92"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}