Issues with the current implementation

Lets say we have d1 to d100 documents in the corpus. For query q1, if documents d1, d2 and d3 are relevant, then {q1, {d1, d2, d3}} forms a training example. Our task is to have a function which will eventually learn to rank these documents in the correct order w.r.t to q1.

We compute 25 features for each pair {q1, d1}, {q2, d2} and {q3, d3}. So now our training example looks like this {3, 25}. We assign score to each document by multiplying the each of the individual features with a weight and adding all the values. If we have features x1, ... x25 and we have weights w1, ... w25, our score for the document is x1.w1 + x2.w2 + ... x25.w25 (dot product of the weights and the features). We start off with some weights (not the learned weights) and have the score for each training example. Lets assume that the relevance for d1 is r1, d2 is r2 and d3 is r3 w.r.t query q1 { r3 > r2 > r1 }. We try to train the network to learn the weights in such a way that the output score for the features generated for {q1, d3} > {q1, d2} > {q1, d1}.

Now simply using cross product as the output will not be good enough for training the network for multiple reasons. We apply softmax on the outputs from the network to determine the probability with which each document is relevant to the query (softmax helps train the network better w.r.t to the loss). For the previous example lets assume that the network output the scores {s1, s2, s3} for {{q1, d1}, {q2, d2}, {q3, d3}}. Softmax output for the 3 documents will be {exp(s1) / S, exp(s2) / S, exp(s3) / S} where S = exp(s1) + exp(s2) + exp(s3). We now to train the network to match the softmax outputs to {exp(r1) / R, exp(r2) / R, exp(r3) / R}, where R = exp(r1) + exp(r2) + exp(r3). Note that since softmax is a monotonically increasing function we are still training the network to learn the weights in such a way that the output score for {q1, d3} > {q1, d2} > {q1, d1}.

The issue with the two functions mentioned is that they while calculating S and R ( the sum of the exponents ), they are currently taking documents which are not relevant to the query into account. So the softmax output for {q1, d3} right now is exp(s1) / S, where S is calculated as um of exponents of the neural network outputs over all possible {qi, di} and it is mapped to exp(r) / R where R is calculated as the sum of exponents over all the relevance judgements for all the documents ( which doesnt make any sense ).