Problem 2.

Alice and Bob are working together to estimate a function mapping the chemical com- position of a solar array to the power output. To collect training data Alice and Bob work in turns in the lab, making new compositions and measuring power output. Suppose Bob is not as carefull as Alice in making his measurements. This leads to some questions:

(Q1) Should Alice and Bob ignore Bob’s measurements in estimating their function? (Q2) How can they incorporate both sets of measurements in a reasonable way?
We can capture the situation with a simple mathematical model. Let fw : X → R be a

function defined by a feature map φ : X → RM and a vector of parameters w ∈ RM , fw(x) = wT φ(x)

Let TA and TB be two sets of training examples. We assume the errors in the training examples are independent but are larger in TB compared to TA.

For (x, y) in TA we assume y = fw(x) + e with error e distributed according to a Normal distribution N(0,σA2 ). For (x,y) in TB we assume y = fw(x) + e with error e distributed according to a Normal distribution N (0, σB2 ). The errors are independent and σB2 > σA2 .

Suppose we know σA2 and σB2 . What is the maximum likelihood estimate of w? wML = maxp(TA,TB|w)

w

(a) Show that wML minimizes a sum of weighted squared differences. The sum should have one term per example in TA and one term per example in TB. Justify your answer.

(b) Show how to compute wML by solving a linear system. Justify your answer.
(c) What does this mathematical model say about questions (Q1) and (Q2) above? (d) Suppose we don’t know σA and σB. How can we estimate w?

 

 

ENGN 2520 / CSCI 1420
Spring 2017
Homework 2
Due Wednesday February 22 at 4pm INCLUDE THIS COVER PAGE WITH YOUR HOMEWORK
NAME:
BANNER ID:
BROWN EMAIL:
COLLABORATED WITH:
(LEAVE THIS BLANK) problem grade memo
1
2
3
total 1 IMPORTANT: Students may discuss and work on homework problems in groups. However, each student must write down their solutions independently. All of the work submitted
should be your own. Each student should write on the problem set the set of people with
whom they collaborated. Problem 1
Let x be a real valued random variable with a uniform distribution p(x) on some unknown
interval [a, b]. Suppose we have a training set T with k independent samples from p(x).
What is the maximum likelihood estimator for [a, b]? Justify your answer. Problem 2
Note that (Q1) and (Q2) below are hypothetical questions. You should answer questions
(a), (b), (c) and (d).
Alice and Bob are working together to estimate a function mapping the chemical composition of a solar array to the power output. To collect training data Alice and Bob work
in turns in the lab, making new compositions and measuring power output. Suppose Bob is
not as carefull as Alice in making his measurements. This leads to some questions:
(Q1) Should Alice and Bob ignore Bob’s measurements in estimating their function?
(Q2) How can they incorporate both sets of measurements in a reasonable way?
We can capture the situation with a simple mathematical model. Let fw : X → R be a
function defined by a feature map φ : X → RM and a vector of parameters w ∈ RM ,
fw (x) = wT φ(x)
Let TA and TB be two sets of training examples. We assume the errors in the training
examples are independent but are larger in TB compared to TA .
For (x, y) in TA we assume y = fw (x) + e with error e distributed according to a Normal
distribution N (0, σA2 ). For (x, y) in TB we assume y = fw (x) + e with error e distributed
according to a Normal distribution N (0, σB2 ). The errors are independent and σB2 > σA2 .
Suppose we know σA2 and σB2 . What is the maximum likelihood estimate of w?
wML = max p(TA , TB |w)
w (a) Show that wML minimizes a sum of weighted squared differences. The sum should
have one term per example in TA and one term per example in TB . Justify your answer.
(b) Show how to compute wML by solving a linear system. Justify your answer.
(c) What does this mathematical model say about questions (Q1) and (Q2) above?
(d) Suppose we don’t know σA and σB . How can we estimate w? 2 Problem 3
In this problem you will experiment the least absolute deviation method for regression. The
data for this problem is available on the course website. The data is similar to what you
used for Homework 1, but there are a few outliers in the training set. You should review the
notes on robust regression from class.
You will use polynomial basis functions to estimate a polynomial fw (x) using (1) sum
of squared differences and (2) sum of absolute deviations. For (1) you should solve a linear
system. For (2) you should use ’linprog’ in Matlab to solve the resulting linear program.
Type ’help linprog’ in the Matlab prompt to learn how to use that package.
(a) Use the training data to estimate two degree 2 polynomials, one with each regression
method. Make a plot showing the training set and the two polynomials you estimate. You
should clearly label the polynomials in the plot according to which regression method was
used for each one.
(b) Repeat part (a) using degree 4 polynomials.
(c) What can you say about the differences between the two approaches for regression
based on these experiments?
Submit your Matlab source code along with your homework. You should include the
plots for parts (a) and (b) in your writeup. 3

Attachments:

• 1493702226-hw2.pdf