INFORMATION THEORY

Stat 364b/664b  ENAS 454b/954b  Spring 2006

Instructor:  Andrew Barron  <Andrew.Barron@yale.edu>,
             phone: office 2-0634, dept 2-0666, home 248-5386
             office hours: Tues/Thurs 10:15-11:30 or by appointment.
Teaching Fellow: Zhenning Kong <zhenning.kong@yale.edu>,
             office hours: Wednesday at 3:00 or 4:00 pm.

Time: Tuesday, Thursday, 9:00-10:15.
Location:  24 Hillhouse (room 107)

Webpage http://www.stat.yale.edu/~arb4/stat364/coursehomepage.txt

Textbook: Elements of Information Theory, Cover and Thomas, Wiley 1991.

Topics in approximate order of presentation:  entropy and mutual
information; the asymptotic equipartition property; entropy rate of
a stochastic process; data compression and the minimum description
length principle; gambling and the stock market; Kolmogorov complexity
of algorithms; channel capacity; the maximum entropy principle; some
implications of information theory for probability limits (laws of
large numbers, central limit theorem, and conditional limit theorems);
applications to hypothesis testing and estimation.

The course is intended to introduce information theory to a broad
spectrum of students.  It may be of interest to students in mathematics,
applied mathematics, electrical engineering, computer science, physics,
bioinformatics, economics, and finance as well as statistics.  It is
suitable for both upper-level undergraduates and graduate students.

Prerequisite:  At least one course in introductory probability such
as Statistics 241 or 238.

Grade based on:
    35% weekly assignments
    10% class participation
    20% midterm exam
    35% final exam

HW1: Chapter 2, Problems 1,3,5,6,8,16,19       (Due Thurs. Jan 19)
HW2: Chapter 2, Problems 9,10,20,21,26,34,35   (Due Thurs. Jan 26)
HW3: Ch.2, Pr.24,25;  Ch.3, Pr.2,4,5,6,7       (Due Thurs. Feb 2)
HW4: Chapter 4, Problems 1,2,4,10,11,12,14     (Due Thurs. Feb 9)
HW5: Chapter 5, Problems 1,2,4,5,6,8,25        (Due Tues. Feb 21)
HW6: Chapter 5, Problems 12,13,26              (Due Tues. Feb 28)
     Problem A: 
      Let a source string be i.i.d. with P(X=a)=3/4, P(X=b)=1/8, P(X=c)=1/8.
      Use recursive arithmetic coding, as explained in class, to give the 
      binary codeword for the string
                    a a c a b a a a
HW7: Chap. 6, Pr. 1,2,3,4,6;  Chap. 15, Pr.1,3 (Due Tues. March 28)
Midterm Thursday March 4.
HW8: Ch.8, Prob 1,3,5,6,9; Ch.9, Prob 1,6      (Due Tues. April 4)
HW9: Ch.2, Pr.33; Ch.8, Pr.4; Ch.10, Pr.1,2,3  (Due Tues. April 11)
HW10: Ch.11, Prob 1,2,5; Ch.12, Prob 1,5       (Due Tues. April 18)
Extra Credit Assignment: 
      Chapter 7, Problems 1,2,3,4,5,7,9        (Due Mon. April 24)