Article: Potential-Based Shaping and Q-Value Initialization are Equivalent

JAIR is pleased to announce the publication of the following article:

Wiewiora, E. (2003)
  "Potential-Based Shaping and Q-Value Initialization are Equivalent", 
   Volume 19, pages 205-208.

   For quick access via your WWW browser, use this URL:
     http://www.jair.org/abstracts/wiewiora03a.html

Abstract:
Shaping has proven to be a powerful but precarious means of improving
reinforcement learning performance. Ng, Harada, and Russell (1999)
proposed the potential-based shaping algorithm for adding shaping
rewards in a way that guarantees the learner will learn optimal
behavior.

In this note, we prove certain similarities between this shaping
algorithm and the initialization step required for several
reinforcement learning algorithms. More specifically, we prove that a
reinforcement learner with initial Q-values based on the shaping
algorithm's potential function make the same updates throughout
learning as a learner receiving potential-based shaping rewards. We
further prove that under a broad category of policies, the behavior of
these two learners are indistinguishable. The comparison provides
intuition on the theoretical properties of the shaping algorithm as
well as a suggestion for a simpler method for capturing the
algorithm's benefit. In addition, the equivalence raises previously
unaddressed issues concerning the efficiency of learning with
potential-based shaping. 

The article is available via:
   
 -- comp.ai.jair.papers (also see comp.ai.jair.announce)

 -- World Wide Web: The URL for our World Wide Web server is
       http://www.jair.org/
    For direct access to this article and related files try:
       http://www.jair.org/abstracts/wiewiora03a.html

 -- Anonymous FTP from Carnegie-Mellon University (USA):
        ftp://ftp.cs.cmu.edu/project/jair/volume19/wiewiora03a.ps
    The compressed PostScript file is named wiewiora03a.ps.Z 

For more information about JAIR, visit our WWW or FTP sites, or
contact [EMAIL PROTECTED]



-- 
Steven Minton
JAIR Managing Editor