INFORMATION THEORY

Stat 364b/664b  ENAS 454b/954b  Spring 2008 

Instructor:  Andrew Barron  <Andrew.Barron@yale.edu>, 
             phone: office 2-0634, dept 2-0666, home 248-5386
             office hours: Tues/Thurs 10:15-11:30 or by appointment.

Teaching Assistant: Peisi Yan <Peisi.Yan@yale.edu>
             
Time: Tuesday, Thursday, 9:00-10:15.
Location:  24 Hillhouse (room 107)

Website accessible via the classes*v2 server syllabus page or 
http://www.stat.yale.edu/courses.html
Solutions to homeworks from earlier weeks will be posted on the 
classes*v2 server resources page.

Textbook: Elements of Information Theory, Cover and Thomas. 
          Wiley, 2nd Edition 2006. 

Topics in approximate order of presentation:  entropy and mutual 
information; the asymptotic equipartition property; entropy rate of 
a stochastic process; data compression and the minimum description 
length principle; gambling and the stock market; channel capacity; 
the maximum entropy principle; some implications of information 
theory for probability limits (laws of large numbers, central 
limit theorem, and conditional limit theorems); applications to 
hypothesis testing and estimation; Kolmorogov complexity of algorithms.

The course is intended to introduce information theory to a broad 
spectrum of students.  It may be of interest to students in mathematics, 
applied mathematics, electrical engineering, computer science, physics, 
bioinformatics, economics, and finance as well as statistics.  It is 
suitable for both upper-level undergraduates and graduate students.

Prerequisite:  At least one course in introductory probability such 
as Statistics 241 or 238.

CLASS PARTICIPATION 10%, HOMEWORK 30%, MIDTERM 20%, FINAL 40%.

The first two Homework assignments: 

HW1: Chapter 2, Problems 1,3,4,5,8,12,37       (Due Tues. Jan 22)
HW2: Chapter 2, Problems 6,9,11,18,19,38,42    (Due Tues. Jan 29)
HW3: Chapter 3, Problems 2,3,4,5,6,10,13       (Due Tues. Feb 5)
     [Extra Credit:  Produce a corrected table for page 69]
HW4: Ch.2 Pr.32,34; Ch.3 Pr.12; Ch.11 Pr.1,16  (Due Tues. Feb 12)
HW5: Chapter 4, Problems 2,6,7,18,19,20,30     (Due Tues. Feb 19)
HW6: Chapter 5, Problems 2,4,6,9,12,13,15      (Due Tues. Feb 26)
HW7: Chap. 5, Problems  8,30,32,33,35          (Due Tues. March 4)
     Pr. A: A string is i.i.d. with P(X=a)=3/4, P(X=b)=1/8, P(X=c)=1/8.
     Use arithmetic coding to give the codeword for a a c a b a a a
MIDTERM Thursday March 6.
HW8: Chap. 6, Prob 1,5,7,10; Chap 13, Prob 4   (Due Tues. March 26)
HW9: Chap. 7, Problems 1,5,8,12,13,19          (Due Tues. April 1)
HW10: Ch.7 Prob 15,20,28,18; Ch.8 Prob 1,5     (Due Tues. April 8)
HW11: Ch.8,Pr.3; Ch.9,Pr.2,4,6; Ch.12,Pr.1,2   (Due Thurs. April 17)
HW12: Ch.11, Pr.13,18,20,22,25; Ch.12, Pr.4,8  (Due Thurs. April 24)
FINAL EXAM Monday May 12, 9am.