Basically, both cases depict a lovelorn artist extensively staring at a mirror. The Lady of Shalott is locked in a tower, and only looks at the world by the reflection of the window through her mirror. Everyday she weaves an image of what she sees. One day she glimpses Lancelot and falls in love. She realizes what she sees in the mirror and weaves is only shadows of the world, not the real thing. This realization combined with her utter lack of romantic prospects with Lancelot compels her to commit suicide. Obviously the outcome of the myth is where the connections run thin. But, in a way, Emma is Stephen's Lancelot. In the scene in question he has just finished writing a poem about Emma which is similar to the lady's "reflection of reality' ; the poem does not match the real life event. Both characters have never spoken to their intended love and, while the lady weaves a tapestry of Lancelot instead of writing a poem, both works of art are inspired by their beloved. Perhaps, then, when Stephen is staring in the mirror, he is contemplating how the art he has just created in based not on reality itself, but on reality filtered through his emotions as represented by the mirror.
Of course, the fact that the most famous telling of the Lady of Shalott is a poem by Tennyson complicates things. Stephen hates Tennyson. All I have to say in that regard is that the Tennyson v. Byron scene indicates Joyce is familiar with Tennyson's poetry and therefore may have read the poem I'm referencing.
As an ending note, I would be remiss if I didn't mention the more obvious mythical allusion, more familiar to most members of this class, Narcissus. Most are familiar with the myth of the Greek boy who was so entranced with his own reflection that he wasted away staring at it. If Joyce is drawing this parallel, perhaps Stephen's thoughts as he sat in front of his mother's mirror were more along the lines of, "that is an incredible artist sitting in front of me! I'm so talented! Great poem writing, Stephen!"
The reason I think its worth discussing both of these possible allusions is because of the weight they lend to this moment in Stephen's life. Because it is an image embedded in myth, sitting in front of a mirror for an extended period of time takes on both a romantic and a larger-than-life quality. Thus, in this moment we see a boy becoming (or believing he is becoming) an artist and, accordingly, becoming a figure worthy of mythic status.