Next select just the front and back edges, and skip the edges that make up the bottom of the arm hole. Hold shift and pull just those edges up. Do it again another one or two times as needed, checking both front and side views to line up with the reference, and then the perspective view to more accurately shape it as needed.

Doing the shoulders and connecting to the collar can be tricky. Instead of bring up an entire row of edges, I selcted JUST the edge circled below. Same as always, just hold down SHIFT and drag the edge to create new faces. Then in vertex mode, I clicked the Target Weld button and dragged the verts along the top of those edges to the base of the collar.

For the two edges shown selected in the image, I used Bridge. When you have two edges selected and you click Bridge it will create a new poly between the two edges.

Use Bridge on the remaining edges on the front of the shirt and in the back where you can to fill the gaps. Weld any verts together that you need to, and cut in any new edges where needed to make it work.

For this part, I find it's best to just work in the perspective viewport so you have a better feeling for what you need to fix and adjust.

For the Back, the number of edges didn't match perfectly so I had to do some cutting to make it all fit correctly.

Now is a good time to look your model over and see if there are any areas that you can optomize. Any place where you've got several edges close together where you could probably do just as good with fewer. Just remember that any edges you remove, you also have to remove any isolated verties by selecting them afterwards and clicking the remove button.

If you need to, select all the polys and set them all to the same smoothing group again. Using Bridge tends to add extra smoothing group info that you can get rid of by clicking Clear All and then assigning a smoothing group to everything again.