B019012 - INFORMATION THEORY

Lessons are held in Italian, and edicational matherial is also in italian

The course aims at providing a basic knowledge about the representation in a compact form of the information and the reliable transmission of the information over a noisy channel. In particular:
- Knowledge about basic concepts of the information representation
- Knowledge about essential elements of source coding and source coding techniques
- Knowledge about the challengs and the characterization of reliable transmission over noisy channels.
- Knowledge about channel coding principles and channel coding techniques

At the end of the course, the student is expected to be able to:
- understand the principles that are the basis of the current standards of compression of data, video, audio
- apply compression techniques
- understand the most popular techniques of channel coding and reliable transmission of data over noisy channels
- apply channel coding techniques

The course deals with the problem of an efficient and reliable message transmission. It introduces the information as something defined and measurable and answers to two fundamental questions for any information system: what is the highest compression level and the maximum data transmission rate. The course also provides source and channel encoding elements, used to approach the previously defined theoretical limits.

The student is expected to have a basic knowledge about: signals and systems; probability, random variables and processes, and their characterization in time and frequency domain; vector and matrix representation.

The course will consist of:
Frontal lectures conducted using slides provided by the teacher (available on Moodle Platform)
Final seminars on new areas of application of information theory

The final exam consists of an oral test with theoretical questions and exercises.
The theoretical questions are intended to verify
- Understanding the information concept
- Understanding the theory behind compression and channel encoding
- Understanding the mechanisms and principles that allow both source and channel code to be constructed
- Understanding the characteristics of source and channel codes and performance evaluation metrics
The exercises are intended to verify
- ability to apply theoretical concepts
- ability to apply the analyzed coding techniques

PART I- INTRODUCTION – Information measure and Entropy. Discrete memoriless sources entropy. Sources of information with memory. Continuous sources.

PART II – SOURCE CODING - Introduction to source coding: codes classification. Kraft's and McMillan’s inequalities. Average length of a code. First Shannon's theorem on reversible coding. Huffman coding. Arithmetic coding. Lempel-Ziv coding. RUn-Length coding.

PART III – RATE DISTORTION THEORY - Quantization. Distortion and its measure. Rate-Distortion theory. Rate-distortion curve of a source. Rate-distortion function for Gaussian variables.

PART IV- CHANNEL CAPACITY – Channel models for the transmission of information. Channel equivocation. Channel capacity. Decision rules and error probability. Repetition codes. Hamming distance. Fano inequality. Second Shannon's theorem on the reliable transmission over noisy channels. Gaussian channel capacity. Shannon limit and region of reliable communication.

PART V- CHANNEL CODING - Error control codes. Detection and correction of errors. Block codes. Linear codes. Hard decoding of linear codes. Cyclic codes. BCH codes. Reed-Solomon codes. Concatenated codes. Interleaving techniques.Convolutional codes. Decoding of convolutional codes: the Viterbi's algorithm. Soft decoding. Gain of a channel code.
Seminars (i) on Turbo Coding and LDPC and (ii) physical layer security.

The whole teaching material is available on Moodle Platform.

As additional texts for further insights:
T. M. Cover, J. A. Thomas: Elements of Information Theory. John Wiley & Sons, New York, 2nd ed 2006
N. Abramson: Information Theory and Coding. McGraw-Hill, New York, 1963.
S. Lin, D. J. Costello Jr.: Error Control Coding: Fundamentals and Applications. Prentice-Hall, 1983.
K. Sayood: Introduction to data compression, Elsevier 4a ed., 2012
S. Benedetto, E. Biglieri, V. Castellani: Digital Transmission Theory, Prentice Hall, 1988.
J. G. Proakis: Digital Communications. McGraw-Hill, 4a Ed., 2001.
A. Papoulis, S.U. Pillai, Probability, Random Variables, and Stochastic Processes, 4th ed., McGraw-Hill, 2002.

The exam calendar is available on line.

Slides and other material will be provided during the course by the teacher on Moodle platform.