We can do no better than to quote (much abridged) from Henry Alford's 19th-century New Testament commentary (still the best for such historical questions, and easily downloadable from archive.org) on this passage:
["Jesus' brothers" are mentioned in] Matt xii. 46 : Mark iii. 31 : Luke viii. 19 : the two next are the present passage and Mark vi. 3, where they are mentioned in connexion with His mother and sisters ; the four others are in John ii. 12; vii. 3, 5, 10. The last is in Acts i. 14. … In another place, 1 Cor. ix. 5, Paul mentions οἱ ἀδελφοί … Their names, as stated here and in Mark vi. 3, were Jacob, Joseph (or Joses), Simon, and Judas, all of them among the commonest of Jewish names...
These persons are found … with ... Mary, the mother of the Lord… Νot a word is anywhere dropped to prevent us from inferring that they … were His relations in the same literal sense as we know … Nothing is said from which it can be inferred whether Joseph had been married before he appears in the Gospel history ;—or again, whether these were, according to the flesh, older or younger than our Lord… The silence of the Scripture narrative leaves it free for Christians to believe these to have been real (younger) brethren and sisters of our Lord, without incurring any imputation of unsoundness of belief as to His miraculous conception.
[There is ancient and modern opinion] that they were all sons of Alpheus (or Clopas) and Mary the sister of the mother of our Lord ; and so cousins of Jesus, and called, agreeably to Jewish usage, His "brothers." This is the view taken in the remarkable fragment of Papias,… adopted by Jerome … and very generally received in ancient and modern times. But it seems to me that a comparison of the Scripture testimonies cited above will prove it untenable…
[There is also opinion] that they were children of Joseph by a former marriage (or even by a later one with Mary wife of Clopas, to raise up seed to his dead brother,—as Clopas is said to have been): but this needs no refutation. This view was taken by several early Fathers, e. g. Hilary, Epiphanius, and mentioned by Origen… This however, while by no means impossible, and in some respects agreeing with the apparent position of these brothers as older (according to the flesh) than the Lord (John vii. 3), has no countenance whatever in Scripture, either in their being called sons of any other woman, or in any distinct mention of Joseph as their father, which surely in this case would be required.